OpenLLM, from the BentoML team, lets you run any open-source large language model as an OpenAI-compatible API endpoint, locally or in the cloud, with a single command. The project has collected more than 12,500 stars on GitHub, is released under the Apache-2.0 license, and was actively maintained in 2026.
Serving LLMs well means juggling runtimes, quantization, batching and API compatibility. OpenLLM hides that complexity: pick a model — DeepSeek, Llama, Qwen, Mistral and many others — and it starts a production-grade server speaking the OpenAI API format, so existing clients and tools connect without changes. It integrates with the BentoML ecosystem for containerization, scaling and deployment, supports fine-tuning workflows, and can run models with optimized inference backends. For teams that prefer not to operate GPUs themselves, the same stack deploys to BentoCloud, BentoML’s managed platform.
The project’s design goal is operational sanity: models are loaded with sensible defaults for their architecture, streaming responses work out of the box, and the server exposes the metrics and concurrency controls needed for real traffic. Because the endpoint mimics OpenAI’s API, migration paths are painless — applications written against hosted APIs can be pointed at a self-hosted OpenLLM instance with little more than a URL and key change, which makes it a popular on-ramp for teams bringing inference in-house for cost, privacy or compliance reasons.
OpenLLM is free and open source for self-hosting. BentoML also offers BentoCloud, a paid managed platform, for teams that want hosted deployment and operations handled for them.
OpenLLM is for ML engineers and platform teams who want to standardize LLM serving on open models — prototyping locally with the same stack they will later scale in production.