WhisperFusion is an open-source project from Collabora that enables seamless, ultra-low-latency voice conversations with an AI. With about 1,650 stars on GitHub, it combines real-time speech recognition, a large language model and speech synthesis into one streaming pipeline that you can talk to naturally, without push-to-talk delays.
WhisperFusion builds on Collabora’s WhisperLive and WhisperSpeech projects. Your speech is transcribed in real time by OpenAI’s Whisper, a Mistral large language model generates the reply, and WhisperSpeech turns the answer back into spoken audio. Both Whisper and the LLM are optimized to run as TensorRT engines, while WhisperSpeech is accelerated with torch.compile, keeping latency low enough for genuine back-and-forth conversation rather than walkie-talkie exchanges. The project demonstrates what a fully self-hosted voice assistant stack can look like running on your own GPU hardware, without sending audio to third-party voice APIs — an appealing blueprint for privacy-conscious voice interfaces.
A demonstration video linked from the repository shows the system holding a spoken conversation end to end, and the codebase is meant to be experimented with — each stage of the pipeline, from transcription to synthesis, can be studied or swapped independently of the others.
WhisperFusion is free and open source; you can download and run it from GitHub at no cost. There is no hosted plan — the trade-off is that you need suitable GPU hardware of your own to hit its real-time latency targets.
Developers prototyping voice assistants and voice interfaces, and researchers exploring low-latency spoken dialogue systems who want every component of the pipeline under their own control. It is an engineering showcase as much as a product, aimed at technical users.