Xinference

Xinference is an open-source model serving platform from Xorbits that lets you run large language models, speech models and multimodal models behind one unified API. Its GitHub repository has earned more than 9,600 stars, and the project is updated almost daily, with releases continuing through October 2026.

Overview

The platform’s promise is simple: swap GPT for an open model by changing a single line of code. Xinference exposes an OpenAI-compatible REST API, so existing applications keep working while the backend model changes. Under the hood it can dispatch to high-performance engines such as vLLM, SGLang and llama.cpp, automatically handling model loading, GPU placement and request routing. It runs just as well on a laptop for experimentation as it does across a cluster in production, and it supports text, embedding, reranking, image, audio and video models through the same control plane.

Key Features

• OpenAI-compatible API for chat, completions, embeddings and more, minimizing migration work
• Multiple inference backends, including vLLM, SGLang, llama.cpp and Transformers
• Broad model support: LLMs, vision-language models, speech recognition and synthesis, image generation
• Distributed deployment across multiple machines and GPUs with a built-in cluster manager
• Web UI and CLI for launching, monitoring and managing models
• Quantized model support (GGUF, AWQ, GPTQ) to fit large models on modest hardware

💳 Pricing

Xinference is free and open source under the Apache License 2.0. There is no paid tier for the software itself; you supply the hardware or cloud instances it runs on.

Who Is It For

Xinference is for ML engineers and platform teams who self-host open models and want one stable API across many model types and engines. It is equally useful to individual developers who want a tidy local server for experimenting with the latest open-weight releases.