NExT-GPT is an any-to-any multimodal large language model from the NExT++ research group: it can take in text, images, audio and video in any combination and respond in any of those modalities too. Published as an ICML 2024 paper with full code and model checkpoints, it was one of the first open systems to demonstrate a single LLM genuinely chatting across modalities in both directions. The repository has around 3,600 stars on GitHub and is BSD-3-Clause licensed.
Rather than training a giant model from scratch, NExT-GPT connects a frozen LLM core (Vicuna) to existing expert encoders and generators: ImageBind supplies a shared embedding space for understanding images, audio and video, while Stable Diffusion, AudioLDM and Zeroscope handle generation on the output side. Lightweight projection layers and modality-switching instruction tuning align everything, so only a small fraction of parameters is trained. The result is a system that can, for example, watch a video, discuss it, and answer with a generated image or spoken reply. Training code, data recipes and checkpoints are all released for researchers to reproduce and extend.
NExT-GPT is completely free and open source under the BSD-3-Clause license, including code and checkpoints for research use. The only cost is your own GPU compute.
Multimodal AI researchers who want a reproducible reference for any-to-any generation, and engineers prototyping assistants that need to move fluidly between text, images, audio and video.