Ultravox is a fast, open-source multimodal LLM built for real-time voice interaction. Unlike classic voice pipelines that chain a separate speech-recognition step into a text model, Ultravox understands speech directly, which makes spoken conversations with an AI noticeably quicker and more natural.
The architecture extends an open-weight LLM with a multimodal projector that maps audio straight into the LLM’s representation space, skipping the ASR stage entirely and cutting the latency it adds. Versions have been trained on Llama 3, Mistral and Gemma backbones; the default model builds on Llama 3.3 70B, with a smaller 8B variant also published on Hugging Face. Ultravox currently takes audio in and streams text out, and because the audio path is direct, it is designed to eventually exploit paralinguistic cues such as timing and emotion. The repository includes the full training code โ typically only the projector is trained, with the LLM and audio encoder frozen โ plus evaluation tooling and an inference path through partners such as Baseten. Around the model sits Ultravox Realtime, the hosted platform for building production voice agents. The project is MIT-licensed, has more than 4,500 stars on GitHub, and reached version 0.7 in late 2025.
Freemium. The Ultravox model and code are open source under the MIT license and free to self-host. The hosted Ultravox Realtime platform is paid: pay-as-you-go starts after a free allowance of minutes, with Pro and Enterprise tiers for higher concurrency and support.
Developers building voice agents, call automation and spoken interfaces who want an open, speech-native model they can self-host or fine-tune, with a managed platform available when they would rather not run inference themselves.