Animate Portrait Audio (Sonic)

@tencent1 VersionGenerate

Audio-driven talking head with Sonic: consumes a driving AudioFrame stream and animates a bound source portrait through a Stable-Video-Diffusion backbone so lip motion and head movement follow the audio, emitting the result as a sequence of Image frames.

How it fits

Typical backends

Animate a portrait from a recorded speech or song file and publish the generated frames over HTTP for a downstream consumer.

Synthesize speech from text and drive the bound portrait with it, streaming the talking head to a browser viewer.

Caveats

I/O contractThis is an offline diffusion clip generator: a window of driving audio is buffered, then a temporally consistent clip is produced and emitted frame by frame onto a single Image stream; an empty audio window yields no frames.
CompatibilityThe diffusion backbone has no CPU path and needs a large-memory CUDA GPU; incoming audio is resampled to 16 kHz mono before driving the model.
Hard constraintThe source portrait is bound once at startup from its configured image input; pointing it at a directory uses the first image found, and changing the animated subject requires a redeploy.
CostDiffusion sampling is the dominant cost: more sampler steps and a larger generation resolution raise fidelity and per-clip latency, and optional frame interpolation doubles the output frame rate at extra compute.
LimitationThe model and weights are research/non-commercial licensed and the backbone carries its own community license, so this component is unsuitable for commercial deployments.

Versions

43d6a542defaultlatest
Automated release
6/29/2026