Real-time talking heads on a single GPU

01 · Overview

Lifelike talking heads, without the data centre

Large video models make the most convincing digital humans, but they need clusters of server GPUs to run live. Small motion models run anywhere, but they look like puppets. Maya-Lite is built to have both: a 1.3B diffusion model that streams a talking head at up to 96 FPS on one consumer graphics card.

Two ideas carry it. Streaming-aware pre-training keeps lip-sync precise even though live audio arrives in short 1.32-second pieces. Oracle-guided bidirectional distillation cuts generation to a few steps while teaching the model to correct its own drift, so a stream can run indefinitely without the face slowly changing.

Maya-Lite comes in two variants built on the same backbone. Fast is tuned for speed and Pro for detail, so you can pick the trade-off your use case needs.

02 · Positioning

Why 1.3B

Existing talking-head models force a choice. Lightweight models such as SadTalker and Ditto drive an abstract motion rig, which is fast but loses the full texture of the face. Diffusion models such as Hallo3 and AniPortrait model every pixel, but run far below real time or can't stream at all.

We wanted every box ticked: streaming, real time, infinite length and a full pixel-level face, at a size that fits on one GPU.

Full-face pixel model: the face is generated in a pixel-level video latent rather than driven through abstract motion coefficients.

03 · Variants

Fast and Pro

Both variants share the same 1.3B diffusion transformer. They differ in the video autoencoder that turns pixels into the tokens the model works on, and that one choice sets the balance between speed and detail.

Same voice, same face, both variants

04 · Streaming

Built for streaming audio

In a live conversation the model never hears a full sentence in advance. Audio arrives in pieces of about 1.32 seconds, and speech encoders like wav2vec produce unstable, shifting features from clips that short. That shows up as mouths that lag or mumble.

The other streaming problem is the cold start. The first chunk of a stream has no previous video to continue from, only the reference image. During training, one sample in ten starts from a single frame instead of a run of real motion, so the opening second of a stream looks as natural as the rest.

05 · Method

How Maya-Lite is trained

Learning from an oracle

06 · Data

Training data

A 1.3B model has less room to absorb noise than a 14B one, so data quality matters more. We filtered more than 10,000 hours of raw talking-head footage down to 782 hours of clean, tightly aligned video.

• 330Kclips
• 782 hof aligned video
• 60Kspeakers
• 15languages
• 512²face-centred crops

The filtering pipeline

1. DeduplicateSource IDs and MD5 hashes remove repeats across sources.
2. SliceScene detection cuts footage into coherent 5–50 s clips.
3. StandardizeEvery clip is re-timed to 25 FPS.
4. CropFace detection centres a 512×512 crop on the speaker.
5. Remove jump cutsOptical flow flags abrupt scene transitions.
6. Drop faceless footageClips with too many frames missing a face are cut.
7. Filter occlusionPose keypoints catch hands covering the face.
8. Check lip-syncSyncNet scores discard poorly aligned audio.
9. AnnotateLanguage, age, gender and ethnicity labels for every clip.

How it compares

07 · Results

Benchmarks

We evaluate on HDTF and VFHQ against six talking-head models, in both streaming and offline modes. In streaming mode Maya-Lite Pro leads on lip-sync and video coherence on both benchmarks, and Maya-Lite Fast keeps most of that quality at 96 FPS.

Streaming costs very little quality. On HDTF, Maya-Lite Pro's lip-sync confidence moves only from 6.04 offline to 5.73 when streaming, still ahead of every other model in either mode. The audio cache and cold-start training are doing their job.

08 · Roadmap

What's next

Maya-Lite is optimized for the face and head. At 1.3B parameters it has less capacity for complex physical motion than our larger models, so large body movements and detailed hand gestures are less precise than the face itself. For full-body presenters, use Maya-2. Next, we're working on:

• Scale the architecture so Maya-Lite models the whole upper body, not just the face and head, without giving up real-time speed.
• Better large-amplitude movement and hand gestures, where a 1.3B model has less capacity than our larger models.
• Push the Pro variant to real time on a single consumer GPU.