LatentSync is ByteDance’s open-source lip-sync system: an end-to-end method based on audio-conditioned latent diffusion models that makes a person in a video appear to speak any given audio, without intermediate motion representations. Given a video and a speech track, it regenerates the mouth region so lip movements match the audio precisely while keeping the rest of the frame intact. The project has about 6,200 stars on GitHub, and version 1.6, trained at 512×512 resolution to reduce blurriness, was released in June 2025.
Earlier diffusion-based lip-sync pipelines often produced blurry mouths, temporal flicker or poor performance on non-English content. LatentSync attacks those problems directly in the latent space of a Stable Diffusion model, conditioning generation on the audio so synchronization is learned rather than stitched together from landmarks. Version 1.5 added a temporal layer for consistency across frames, improved results on Chinese videos and cut stage-two training VRAM to about 20 GB; version 1.6 then raised training resolution to 512×512 for sharper detail. Model checkpoints are published on Hugging Face, a hosted demo space is available for quick tests, and the accompanying paper (arXiv 2412.09262) explains the method in full.
LatentSync is free and open source under the Apache-2.0 license. You can download the code and model weights and run them on your own GPU at no cost beyond compute.
LatentSync is for video creators, dubbing and localization teams, virtual-avatar builders and researchers who need high-quality lip synchronization and want a modern diffusion-based method they can run and adapt themselves.