Analyze Image (HuggingFace)

How it fits

Typical backends

Caveats

• I/O contractEach Image produces exactly one String answer; there is no 1-to-0 drop. A frame that decodes empty yields the empty string, and a generation fault is logged and degraded to the empty string rather than dropping the frame.
• LatencyThe model runs the transformers generation loop in the worker process, so throughput is governed by the GPU and max_tokens. This is a single in-process model, not a batched server; for high-throughput serving prefer a dedicated vLLM/SGLang backend.
• Parameter interactionmodel and max_pixels are hot-swappable at runtime and trigger a model + processor reload (and a GPU memory reclaim on CUDA) on the next frame, producing a one-call latency spike. prompt, max_tokens and temperature are re-read every frame with no reload.
• Parameter interactiontemperature 0.0 is greedy and deterministic; any positive value enables sampling and diversifies the answer. max_pixels caps the visual-token budget — larger frames are downscaled by the processor to bound memory and latency.
• Fallbackdevice is read once at startup and is not mutable; a value starting with `cuda` silently falls back to CPU when CUDA is unavailable, which is impractical for live generation.
• Hard constraintmodel must be a repo loadable by transformers `AutoModelForImageTextToText` + `AutoProcessor`. A gated or private repo requires the hf_token secret; a model whose architecture is not an image-text-to-text class fails to load at startup.