Lip sync video and make talking avatars

Lip sync covers two jobs. Re-sync takes a clip of someone speaking and rewrites the mouth to new audio: a corrected line, another language, another voice. Talking avatar takes one still portrait and an audio track and returns a clip of that face speaking, with expressions, blinks and head movement. This guide prepares the speech and the face, runs each job with its main model, and shows how to check the mouth frame by frame.

Before you start

• Rule of thumb from the Knowledge Center: if you already have a video of the character, re-sync it, so the original motion and camera stay. If you only have a still, animate the portrait; the clip runs as long as the audio.
• The audio decides most of the result. The face decides the rest. Fix the inputs before you try another model.
• Lip sync is for speech, not singing. Sync-3 gives generic mouth shapes on music.
• Runs are billed by length. Trim the source to the spoken part first, and check the price on the button.
• Use invented faces, or a real person's only with their consent.

1. 1Speechnarration model, clean, natural pace
2. 2Faceapproved portrait or source clip
3. 3Synclip sync model
4. 4Checkclosing sounds, identity

Step 1: Prepare the speech

• Clean speech. No music or effects under the voice. If the track has a bed, separate the voice first, sync, then remix.
• Natural pace. Clipped or rushed text-to-speech gives choppy mouth movement. Generate speech with a narration model and listen to it first. See Narration and speech.
• More expression. With text-to-speech, exclamation marks and capital letters in the script give stronger delivery, and the mouth follows it.
• No silence at the start. A pause before the first word can push the sync off. Trim it.
• Know the length. Compare the audio with the video or the model's limit, and decide what happens when they differ (Step 4).

Step 2: Prepare the face

• Mouth fully visible: front-facing or three-quarter, no hand or microphone over it. Sync-3 detects hands and microphones near the mouth, but a clear mouth still gives the sharpest result.
• Even light on the face. Deep shadows around the mouth make teeth and lips smear.
• Room around the head for a portrait, above the head and below the chin: the model adds head movement. Head and shoulders at eye level, on a plain background, gives the model the most to work with.
• Little camera movement in a source clip. Fast moves and cuts make the sync drift.
• Generate the portrait first with an image model, at the framing you want, and approve it like any first frame. The output keeps the image's framing and aspect ratio.

Step 3: Open the video panel and pick the model

From the top menu, choose Create, then Videos. Click the Model card, then View all models to open the library, and search for the model by name, or for "lip sync" to see them all. Click one to read its inputs in the panel on the right, then click Use this model. For a walkthrough of the library, see Generate video.

Step 4: Re-sync an existing clip with Sync-3 Lipsync

Sync-3 Lipsync re-syncs the mouth in any existing video to a new audio track, for dubbing, voice-over and fixing a line. It processes the whole frame (no face crop needed), outputs up to 4K, and can open the lips of a character who wasn't speaking in the source.

The sync mode decides what happens when the video and the audio aren't the same length:

Other models that re-sync a clip: Sync Lipsync v2 (keeps the speaker's style, handles several speakers), Sync Lipsync React-1 (fast), Veed Lipsync v2 and Kling LipSync (short, quick clips; also takes text-to-speech). The Knowledge Center notes that the Sync models work best on realistic faces.

Step 5: Animate a portrait

Minimax H3 Max Lip Sync takes one portrait and 5 to 15 seconds of audio, and no text prompt: the image and the audio drive everything. It outputs at 480P, 768P, 1080P or 2K, the aspect ratio always follows the image, and it works on photos, illustrated characters and stylized 3D or mascot art, in any spoken language.

Other portrait models:

Split long scripts at sentence breaks when they're longer than the model's limit. Make one clip per part with the same portrait, then join them. See Presenter video with avatar.

Step 6: Check the mouth

Watch the clip at full speed with the sound on, then step through the frames on the hard sounds. Lips must close on "p", "b" and "m". Watch it twice more: once for the face, once for the teeth.

Make it repeatable

• One portrait and one voice per presenter: keep the approved portrait (or source clip) and the voice setting in a collection.
• A workflow per series: it takes a script, makes the speech and passes both to the lip sync model, so each new video is one run with the same face and voice.

Common problems

What's next

To say the same thing in other languages, go to Dub and translate video. To make the speech, see Narration and speech.