Detect Objects by Visual Prompt (Ultralytics YOLOE)

How it fits

Typical backends

Caveats

• I/O contractExactly one arm is emitted, fixed at deploy by the consumer: boxes with class, geometry-only rectangles, classes only, instance masks with class, bare masks, a detection count, or the frame with boxes drawn. Every arm is a projection of the same prompted detection set, so they agree detection-for-detection per frame; the [Segmentation] and [Mask] arms also require the bound checkpoint to produce instance masks.
• Hard constraintThe model binding must resolve to exactly one YOLOE segmentation weights file; zero or more than one aborts startup. Prompt-free weights lack the visual-prompt branch and will not work.
• Hard constraintThe reference binding must resolve to exactly one image file; the visual prompt is embedded from it once at startup.
• Parameter interactionrefer_boxes is a list of native BoundingBox regions in the reference image; each box's rectangle marks one example region and its class id is the visual-prompt class. An empty list aborts startup.
• Parameter interactionThe emitted class id is exactly the class id carried by the matching refer_boxes box, so downstream rules index the same id space the prompt defined.
• Accuracythreshold defaults low because cross-image visual prompting scores below text prompting; raise it to suppress weak matches. iou bounds NMS on every arm.
• State lifetimeAll config keys, the bound weights, and the embedded visual prompt are captured once at startup; runtime changes have no effect and require a redeploy.
• FallbackA frame whose shorter side is under 16 pixels is silently treated as a no-match frame: the detector cannot resolve an object at that size, so it emits the same empty projection a no-detection frame does on every arm (an empty list, a zero count, or the untouched frame on the overlay) rather than raising. Such tiny probe frames therefore never drop on their arm.