LitServe is an open-source Python framework from Lightning AI for building custom AI inference servers, available on GitHub with about 4,000 stars. Where high-level serving tools force you into their abstractions, LitServe lets you define exactly how inference works for your models, agents or pipelines in plain Python.
LitServe is built on FastAPI but optimized for AI workloads, with specialized multi-worker handling that delivers at least twice the throughput of a plain FastAPI setup. You implement setup, decode, predict and encode steps, and the framework handles batching, streaming, GPU autoscaling and concurrency. It works with any engine or model type: LLMs via vLLM, vision and audio models, multi-model pipelines, RAG systems and agents. Servers can be self-hosted anywhere or deployed in one command to Lightning Cloud, which adds autoscaling GPUs, monitoring and serverless scale-to-zero. The project is actively maintained with frequent releases.
LitServe itself is free and open source under the Apache 2.0 license, and self-hosting costs nothing. Lightning AI’s managed cloud is freemium: a generous free tier is included, with pay-as-you-go pricing for GPU capacity and enterprise features beyond it.
LitServe is for ML engineers and Python developers who need to serve custom models or compound AI systems and want production performance and scaling without giving up control of the inference logic.