Detect Objects Open Vocabulary (Florence 2)

How it fits

Typical backends

Caveats

• I/O contractThe query arrives as a per-message String input, so the phrase can change at any time; an empty phrase or an empty frame yields an empty result rather than an error.
• I/O contractThe class id indexes the query phrase: the phrase is split on commas and periods into an ordered catalogue, and a detection's label is mapped back to its phrase index. A label matching no phrase is dropped. Florence carries no per-detection score, so each detection is reported at full confidence.
• AccuracyOpen-vocabulary detection localizes only what the phrase names; a phrase with no visible referent returns an empty list.
• LatencyRegions are decoded autoregressively with beam search, so per-frame latency is higher and more variable than a single-shot detector, which is impractical for live video — especially if the host falls back to CPU.
• CompatibilityThe checkpoint loads offline from the bound model artifact (a snapshot directory holding config.json, the safetensors weights, and the processor/tokenizer files) with `local_files_only`; `trust_remote_code` runs the snapshot's own modeling code, so no network fetch occurs. Bind a Florence-2 checkpoint to the model slot at deploy.
• State lifetimeThe model and device are resolved once at startup; changing either requires a redeploy.