Detect Objects Zero Shot (HuggingFace)

How it fits

Typical backends

Caveats

• I/O contractThe output is a oneof resolved at deploy time by the connected sink, and every shape is a projection of one per-detection record so the shapes agree detection-for-detection: [BoundingBox] is box geometry plus class, [Rectangle<Double>] is box geometry only, [DetectedClass] is class id plus confidence only, UInt64 is the detection count, and Image is the source frame with boxes drawn. These detectors are box-only, so no instance-mask shape is offered.
• Parameter interactionlabels is the open-vocabulary class list: each text query becomes a detectable class whose id is its index in the list, so id 0 is the first query and id 1 the second. An empty labels list detects nothing and emits the safe empty value for the resolved arm.
• Latencymodel is hot-swappable at runtime and triggers a model reload (and a GPU memory reclaim on CUDA) on the next call, producing a one-call latency spike. labels and threshold change in place with no reload.
• Parameter interactionthreshold is the minimum box confidence to keep a detection. text_threshold is the minimum text-to-box matching confidence and applies only to the Grounding DINO family; OWL-family checkpoints ignore it.
• AccuracyOpen-vocabulary detection quality depends on the wording of labels; short noun phrases such as "a person" or "a red car" detect more reliably than single bare nouns, and the same query can match differently across the Grounding DINO and OWLv2 families.
• Hard constraintmodel must be an open-vocabulary detector checkpoint loadable by the HuggingFace AutoModelForZeroShotObjectDetection auto-class (Grounding DINO, OWLv2, OWL-ViT, OmDet-Turbo); a closed-set detection checkpoint will fail to load.
• FallbackA frame that decodes empty, or an empty labels, yields the safe empty value for the resolved arm (an empty list, a zero count, or a blank overlay frame); every frame emits exactly one output. device is read once at startup and a value starting with `cuda` silently falls back to CPU when CUDA is unavailable.