ai-video-to-talking
Add or replace dialogue in AI-generated video with sync. labs (sync.so). Takes a clip from Veo, Kling, Sora, Runway, Hailuo, Luma, Seedance, Wan or any other video generator, with the user's audio or a line that sync. voices, and lip-syncs the character so they actually speak. Also plans the source clip so it lip-syncs well, and handles scenes with several shots or two characters talking. Use when the user wants an AI character, avatar, UGC-style ad actor or generated presenter to talk, add a voice to an AI video, or make a talking AI video from a prompt. Not for real footage that already has a voiceover to match (lipsync), still images (talking-photo), or translating a video (translate-and-dub).
- License 许可
- MIT
- 最近更新
- 2026-11-14
- Stars
- ★ 1.6k
- Forks
- 491
01 前言 Front Matter
- name
- ai-video-to-talking
- license
- MIT
- description
- Add or replace dialogue in AI-generated video with sync. labs (sync.so). Takes a clip from Veo, Kling, Sora, Runway, Hailuo, Luma, Seedance, Wan or any other video generator, with the user's audio or a line that sync. voices, and lip-syncs the character so they actually speak. Also plans the source clip so it lip-syncs well, and handles scenes with several shots or two characters talking. Use when the user wants an AI character, avatar, UGC-style ad actor or generated presenter to talk, add a voice to an AI video, or make a talking AI video from a prompt. Not for real footage that already has a voiceover to match (lipsync), still images (talking-photo), or translating a video (translate-and-dub).
- compatibility
- Needs the sync. MCP server or a SYNC_API_KEY. See references/connect.md. sync. doesn't generate video; the clip comes from the user or another tool.
02 资源文件 5
- 参考文档agents/openai.yaml233 B
- 参考文档references/api.md4.1 KB
- 参考文档references/connect.md3.3 KB
- 参考文档references/media-input.md3.8 KB
- 参考文档references/source-clip.md2.1 KB
03 SKILL.md 正文
Make an AI video talk
Delivers an MP4 of the generated character speaking the agreed line, in a voice that fits them, with matching lips. sync. doesn't make the clip itself. It comes from the user, or from a video-generation tool you can use in this session.
If no sync. tools are available, follow references/connect.md first. Without the MCP tools, use the REST requests in references/api.md.
Steps
- Get the clip.
- The user has one: save it as in references/media-input.md.
- The user only has an idea: if you can generate video here, write the prompt with references/source-clip.md so the face lip-syncs well. If you can't, give the user that prompt to paste into their video tool, and continue once they have the clip.
- Check the clip before spending anything: one clearly visible face per shot, the mouth unobstructed, and how many shots or cuts it has. If it has cuts or two characters, plan which part each line covers now. The timeline is built in step 7, after the lines, voices and cost are agreed.
- The line. Use the user's exact words. If they want you to write it, draft a line that fits the character and the clip's length, about 2.5 words per second (an 8-second clip holds about 20 words), and get approval.
- The voice. Either the user's own audio (save it like the clip), or a script and a voice: call
voices_listand suggest two or three that fit the character's age, energy and setting. Clone a voice withvoices_cloneonly if the user explicitly asks and owns, or has permission to use, the sample. Cloning saves a voice to the account, can count against plan limits, and isn't idempotent: confirm the sample and the name, call it once, and after a lost or failed response list voices withvoices_listto see whether it was saved before trying again. Never retry it on your own or change the sample or name to get around an error. - Fit the timing. Generated clips are short, so the speech often runs longer:
- Speech a little longer:
bounce(the default) orloopkeeps the character on screen. - Speech much longer: shorten the line, generate a longer clip, or use
remapto slow the video down to fit. - Speech shorter:
silencekeeps the rest of the clip without speech,cut_offends the video with the speech.
- Speech a little longer:
- Quote, confirm, create. Call
generations_estimate-costwithmodel: "sync-3"and thescriptoraudioDuration, plusvideoDurationandsyncModewhen the mode iscut_offorsilence. This prices the lipsync only; for a script, say the length is approximate and that speech synthesis is billed separately. Show the clip, the line, the voice and the estimate, and wait for approval. Then calllipsync_createonce withvideoAssetId, the speech (scriptwithvoiceId, or the audio),model: "sync-3", anyoptions.sync_mode, a newidempotencyKey, a new UUID asprojectCreationKeyand a shortprojectName. sync-3 handles stylized faces, profiles and partial occlusion best. - Several shots or speakers. For a scene where the speaker changes, or the line is spread over cuts, use
lipsync_create-timelinewhen it's available: one clip per spoken part, each with its audio, itsstartTimeandendTimeon the video, and aspeaker({"frameNumber", "x", "y"}on the face, or{"detect": true}). Clips can't overlap. Voice each character's lines first, so every clip has its own audio: either the user's recordings, or speech previews. Speech previews are paid, so get approval for them first. Then callspeech-previews_createwith aprojectIdfromprojects_createand its own new UUIDidempotencyKey(saved before the call), pollspeech-previews_getuntilSUCCESS, and use the returnedurlas that clip'saudioUrl. Quote the timeline render and get approval before callinglipsync_create-timelinewith a newidempotencyKeyand thatprojectId(withprojectIntent: "explicit"). Without that tool, lipsync each shot separately and offer to join them with ffmpeg. - Wait and deliver. Poll
generations_getwithwait: trueuntil it's final, then give the exactoutputMediaUrl. In a coding agent, offer to download it.
Rules
- Never invent the words without approval. Never start a second paid generation to check progress.
- Assume the user has the rights to their own media, and don't ask about consent on routine requests. Decline only when the request is to make a real, identifiable person, such as a politician or celebrity, appear to say something they didn't say, in order to deceive.
- If the generator already gave the clip dialogue and the user only wants different words, this is the same flow with their new line or audio.
04 同仓库更多 Skills
worktable-review-with-annotations
Create, inspect, reply to, update, or resolve situated Worktable annotations. Use when feedback belongs to a Doc section, block, text range, or HTML Doc, or when the user asks to address existing review feedback and coordinate a requested artifact edit.
worktable-find-and-synthesize-context
Find, assess, and synthesize context from Worktable Docs and Records. Use when the user asks what a workspace contains, wants workspace-backed research, needs current facts reconciled across artifacts, or requests a cited synthesis without changing workspace state.
worktable-create-or-update-docs
Create, edit, rename, or maintain narrative Worktable Docs. Use for plans, decisions, briefs, notes, research syntheses, and other durable prose when the user requests a new artifact or a safe update to an existing one.
worktable-create-or-manage-records
Organize repeated structured information as durable Worktable Records. Use for research datasets, inventories, directories, pipelines, trackers, and other collections whose items need identity, fields, filtering, links, validation, or individual updates.
worktable-create-or-manage-interactive-html
Create or improve durable visual and interactive artifacts in Worktable. Use for dashboards, trackers, calculators, mini apps, slide presentations, and other requests that benefit from a polished HTML Doc.
worktable-collaborate-threads
Read, answer, and hand off messages through durable Worktable threads with users or other authenticated participants. Use when the user asks to check addressed messages, reply to a participant, continue a thread, request another agent's help, wait briefly for a response, or watch for comments and messages sent to the agent.