vLLM

vLLM is an open-source LLM inference and serving engine with 94,000+ stars on GitHub, built around the PagedAttention algorithm to deliver state-of-the-art serving throughput for large language models.

Overview

Originally developed in the Sky Computing Lab at UC Berkeley, vLLM has grown into one of the most active open-source AI projects, maintained by a community of more than 2,000 contributors. Its core idea, PagedAttention, manages attention key and value memory the way an operating system manages virtual memory, dramatically reducing KV-cache waste and letting a single GPU serve far more concurrent requests. Around that core it layers continuous batching, chunked prefill, prefix caching and fast CUDA/HIP graph execution. vLLM seamlessly supports more than 200 model architectures from Hugging Face — decoder-only LLMs, mixture-of-experts models, multimodal models, embedding models and more — and runs on NVIDIA, AMD and Intel GPUs, CPUs and accelerators such as Google TPUs.

Key Features

• PagedAttention – efficient KV-cache memory management for much higher serving throughput per GPU.
• Continuous batching – incoming requests are batched dynamically, with chunked prefill and prefix caching.
• Broad quantization support – FP8, INT8, INT4, GPTQ, AWQ, GGUF and more to shrink memory and cost.
• Speculative decoding – n-gram, suffix and EAGLE methods to cut generation latency.
• 200+ model architectures – Llama, Qwen, DeepSeek, Gemma, Mixtral, multimodal and embedding models from Hugging Face.
• OpenAI-compatible server – drop-in API server, plus Anthropic Messages API and gRPC support.
• Distributed inference – tensor, pipeline, data, expert and context parallelism, with efficient multi-LoRA serving.

💳 Pricing

vLLM is completely free and open source under the Apache License 2.0. You self-host it on your own GPUs or cloud instances, so your only cost is the hardware you run it on — a big reason it underpins so many production LLM deployments.

Who Is It For

ML engineers, platform teams and anyone serving open-weight language models at scale who needs maximum throughput, broad model support and an OpenAI-compatible API without a commercial inference bill.