Generate Video (HuggingFace)

How it fits

Typical backends

Caveats

• Hard constraintmodel MUST be a diffusers text-to-video checkpoint whose `model_index.json` resolves to a text-to-video pipeline (DiffusionPipeline auto-selects the class). The default `ali-vilab/text-to-video-ms-1.7b` (ModelScopeT2V, Apache-2.0, ungated, ~6GB VRAM at fp16) resolves to `TextToVideoSDPipeline` at 256x256; `cerspense/zeroscope_v2_576w` also loads with this pipeline.
• Hard constraintwidth and height MUST be multiples of 8. text-to-video-ms-1.7b is trained at 256x256; very large resolutions exhaust GPU memory because decoding the full clip is the memory peak.
• Parameter interactionnum_frames sets the clip length; text-to-video-ms-1.7b trains at 8 fps, so 16 frames is ~2s of playback. Higher counts raise VAE decode memory and latency.
• Parameter interactionnum_inference_steps and guidance_scale shape quality: text-to-video-ms-1.7b's DDIM scheduler wants ~25-50 steps at guidance_scale ~9.0. Fewer steps trade quality for latency.
• Parameter interactionnegative_prompt unset disables negative guidance; set it to steer generation away from artifacts.
• Parameter interactionseed unset (`Nothing`) draws a fresh random latent per prompt; a `Just` value makes the clip deterministic for a fixed prompt and config.
• State lifetimemodel is hot-swapped when its value changes at runtime — the pipeline is rebuilt and GPU memory reclaimed; the previous pipeline stays live if the new id fails to load. device and hf_token are captured once at startup.
• Fallbackdevice `cuda` falls back to CPU when no CUDA backend is visible — inference continues at CPU speeds, which is impractical for diffusion video.
• I/O contractEvery input prompt emits at least one Image. A non-empty prompt runs inference and emits num_frames frames; an empty prompt does not run inference but still emits exactly one black width x height frame, and any generation fault degrades to one black frame — the output stream never drops an input.