Detect Objects (HuggingFace)

How it fits

Typical backends

Caveats

• I/O contractThe output is a oneof resolved at deploy time by the connected sink, and every shape is a projection of one per-detection record so the shapes agree detection-for-detection: [BoundingBox] is box geometry plus class, [Rectangle<Double>] is box geometry only, [DetectedClass] is class id plus confidence only, UInt64 is the detection count, and Image is the source frame with boxes drawn. These backends are object-detection only, so no instance-mask shape is offered.
• Parameter interactionThe backend is chosen at load time from the resolved model repo contents: a `.pt` file selects the Ultralytics path, then a `.onnx` file selects the ONNX runtime path, otherwise use_pipeline toggles between the high-level detection pipeline and the eager processor+model path.
• Latencymodel and use_pipeline are hot-swappable at runtime and trigger a model reload (and a GPU memory reclaim on CUDA) on the next call, producing a one-call latency spike.
• Parameter interactionthreshold is the base confidence floor; class_thresholds overrides it per class id, and object_classes acts as an allow-list where an empty list keeps every class. All key off the checkpoint's own class id space.
• AccuracyThe class id mapping is checkpoint-dependent, so downstream logic that branches on a class id will misfire if model is swapped for a checkpoint with a different class order.
• Hard constraintAn Ultralytics path requires detection weights; classification or segmentation weights load but produce wrong outputs rather than erroring.
• FallbackA frame that decodes empty yields the safe empty value for the resolved arm (an empty list, a zero count, or a blank overlay frame); every frame emits exactly one output. device is read once at startup and is not mutable; a value starting with `cuda` silently falls back to CPU when CUDA is unavailable.