Track Points (AllTracker)

How it fits

Typical backends

Caveats

• I/O contractThe output is a oneof of three arms resolved once at deploy time from the downstream connection — the [TrackedPoint] sparse path, the dense OpticalFlow field, or the per-pixel visibility Mask. A single deployment serves exactly one arm.
• I/O contractOn the sparse [TrackedPoint] arm the worker follows the points named by the query_points config, given in the first (query) frame's coordinates (pixels, or normalised [0, 1] when query_points_normalized is set). The arm requires a non-empty query_points; it is ignored by the dense arms.
• I/O contractEach TrackedPoint carries a stable id (the query-point index), the point's current image-pixel position, a trajectory of past positions (oldest first; the last element equals the current position) bounded to max_trajectory, and a visible flag set from the confidence sampled at the point and thresholded by confidence_threshold.
• I/O contractThe sparse path anchors every prediction to the fixed first (query) frame, so each tracked point keeps the same identity and a continuous trajectory across the whole stream; it does not re-anchor as frames advance.
• I/O contractThe dense OpticalFlow is anchored to the query frame of the current sliding window, not the first frame ever seen; once the window fills, the oldest frame is dropped and the dense query frame advances forward in time. This advancing-window behaviour applies to the dense arms only.
• I/O contractOn the dense arms the first frame has nothing to track against, so it emits a zero OpticalFlow and an all-false Mask sized to the input frame. On the sparse arm the query frame emits each point at its query position with a singleton trajectory and visible true.
• I/O contractFrames are downscaled for inference (longest edge clamped, each side snapped to a multiple of 8), but emitted positions, OpticalFlow, and Mask are mapped back to the original frame resolution before they leave.
• AccuracyVisibility is a thresholded confidence value; a point or pixel marked not-visible (occluded or out of frame) still carries a predicted position so the trajectory and flow stay continuous. Apply the visibility flag or Mask downstream when occluded regions must be excluded.
• CostA larger temporal window and buffer raise GPU memory and per-frame latency; the model is built for GPU and CPU inference is impractical at video frame rates.
• State lifetimeThe model, geometry, query points, and all numeric settings are fixed at startup; changing any of them requires a restart.