Track Objects (SAM 3)

How it fits

Typical backends

Caveats

• I/O contractTwo inputs that must come from the same frame: an Image and a seed list, resolved once as either [BoundingBox] (box detector) or [Segmentation] (mask detector). The seed is a trigger, not a per-frame requirement — frames with an empty seed list run pure propagation and every already-registered object keeps being tracked.
• State lifetimeTracking is stateful per stream: SAM 3's memory accumulates across frames and a track id persists for an object even through occlusion, so a detector need only re-seed occasionally. A restart drops every id; objects re-seed with new ids.
• Parameter interactionA seed whose box overlaps a live track at or above seed_iou_gate is absorbed (the existing id and memory are kept); a seed below the gate against every live track registers a NEW id and pins that seed's class id. Lower the gate to re-seed more eagerly, raise it to suppress duplicate ids on noisy detections.
• Accuracyconfidence_threshold is the per-frame presence-score floor; an object scoring below it is treated as absent on that frame (occluded / out of frame) and not emitted, but its memory is retained so it re-appears later with the SAME id.
• CompatibilityThe SAM 3 checkpoint is supplied through the model file slot and loaded locally at startup; no weights are fetched at runtime, so the slot must be populated before deploy. The tracker and the SAM 3 concept detector share the facebook/sam3 backbone.
• CostA full promptable segmentation model runs and propagates memory on every frame, so a GPU is expected for usable frame rates; the model runs in fp32 (fp16 is numerically unstable on the mask-decoder path) and CPU execution is possible but slow.
• I/O contractThe output shape is fixed once from the connected downstream type: [TrackedSegmentation] for id + per-pixel segmentation (the default, mask-native arm), [Track] for id + mask-bbox + center trajectory, [TrackedObject] for id + confidence, [BoundingBox] for id-less mask boxes, or [UInt64] for the bare id set. Every arm derives from the same per-frame propagated-instance list, so counts and ids stay consistent across arms.
• I/O contractThe [Track] arm fills each track's `trajectory` with a bounded history (last 64) of mask-bbox center points, oldest first, accumulated per track id across frames; an id that disappears drops its history. The box and segmentation carry the class id pinned from the seed that started the track, so the class stays meaningful downstream.