Detect Objects (SAM 3)

How it fits

Typical backends

Caveats

• Hard constraintThe concept prompt must be a non-empty string at startup, and any comma-separated entry that trims to empty aborts boot; a malformed runtime update to the prompt is rejected and the previous value is kept.
• LatencyMultiple comma-separated phrases run sequentially, so per-frame latency scales linearly with the number of phrases.
• I/O contractThe output is a oneof resolved at deploy time by the connected sink, and every shape is a projection of one per-detection record so the shapes agree detection-for-detection: [BoundingBox] is box geometry plus class, [Rectangle<Double>] is box geometry only, [DetectedClass] is class id plus confidence only, UInt64 is the detection count, and Image is the source frame with boxes drawn. The output list is unbounded — a common concept on a crowded scene can produce many detections per call. Each phrase maps to a class id via the configured phrase-to-id mapping, and any phrase absent from the map defaults to class id 0.
• State lifetimeEvery configuration key except the prompt is read once at startup; only the prompt is re-read on each call, so changing the device or thresholds requires a redeploy. The checkpoint is loaded offline from the bound model artifact; nothing is fetched from the HuggingFace hub at runtime.
• FallbackAn empty/zero-size frame yields the empty-but-valid value for the resolved arm (an empty list, a zero count, or a blank frame on the overlay arm). A per-frame inference fault (such as a transient CUDA out-of-memory) is caught and logged, and the same empty-but-valid value is emitted for that frame, so every input still produces exactly one output and one bad frame never tears the worker down. The compute device is resolved once at startup; a `cuda` request falls back to CPU when CUDA is unavailable.