UniDiffuser is the official open-source code and model release for the paper “One Transformer Fits All Distributions in Multi-Modal Diffusion,” from Tsinghua University’s machine learning group. With about 1,500 stars on GitHub, it shows how a single transformer can handle text and image generation tasks that normally need separate models.
Most diffusion systems are specialists: one model turns text into images, another captions images, and joint generation needs yet another architecture. UniDiffuser instead treats text and images as distributions to be modeled together, learning the joint, marginal, and conditional distributions within one transformer framework. In practice, that means a single model can perform text-to-image generation, image-to-text captioning, image variation, text variation, and joint generation of paired text and images. The repository provides the training and inference code plus released model weights, and the model is also integrated into the Hugging Face Diffusers library. Interactive demos are available through Hugging Face Spaces, Colab, and Replicate, so you can try the capabilities before running anything locally. The project was last updated in 2023 and stands as a reference implementation for unified multimodal diffusion research.
UniDiffuser is free and open source under the AGPL-3.0 license. You can run the models locally, experiment through the free online demos, or use the Diffusers integration at no cost beyond your own compute.
Machine learning researchers and engineers studying multimodal generation who want a well-known, citable implementation of unified diffusion โ and developers curious about how far a single model can stretch across text and image tasks.