mistral.rs

mistral.rs is a fast, flexible LLM inference engine written in Rust. Despite the name, it is not limited to Mistral models: it runs a wide range of open large language models — Llama, Qwen, Gemma, Phi and many more — locally or as an OpenAI-compatible server, with a strong focus on speed and quantization. The project has earned about 7,800 stars on GitHub, appears on trending lists constantly, and is actively developed in 2026.

Overview

The engine is built for people who want serious local inference without Python overhead. It loads models directly from Hugging Face, can quantize them in place (including its compact UQFF format) to fit consumer hardware, and accelerates across CUDA, Metal and CPU backends. Serving is a first-class use case: mistral.rs exposes an OpenAI-compatible API, supports paged attention and continuous batching for throughput, and can also be embedded as a Rust library when you want inference inside your own application. Beyond plain text models it supports vision and other multimodal architectures, embeddings and speech models in recent releases, which makes it one of the broadest single-binary runtimes in the local-LLM ecosystem. Configuration is deliberately simple — point it at a model and it works out the rest.

Key Features

• High-performance LLM inference engine written in Rust
• Runs Llama, Mistral, Qwen, Gemma, Phi and many other open models
• In-situ quantization, including the compact UQFF format, for consumer hardware
• OpenAI-compatible API server with paged attention for throughput
• CUDA, Metal and CPU acceleration; usable as a server or embedded Rust library
• Multimodal support for vision models, plus embeddings and speech models

💳 Pricing

mistral.rs is free and open source under the MIT license. The models you run with it are open weights from repositories such as Hugging Face.

Who Is It For

mistral.rs is for developers and local-LLM enthusiasts who want fast, quantized inference and an OpenAI-compatible endpoint without a Python stack.