FlashInfer is an open-source kernel library and JIT compilation engine for LLM serving, providing the high-performance attention and sampling kernels that power fast inference engines. Developed by researchers from the University of Washington, CMU and OctoAI, it has earned about 6,700 stars on GitHub, is Apache 2.0 licensed, and is under intense active development in 2026.
LLM serving performance lives or dies in its GPU kernels: attention over long, paged KV caches, batched decoding, speculative sampling and MoE routing all need specialized implementations. FlashInfer supplies them as a library with a just-in-time compiler that generates optimized kernels for the shapes and hardware at hand โ including prefill and decode attention, paged and ragged KV-cache layouts, FP8 and low-bit paths, and fused operations for modern architectures. In practice, most users meet FlashInfer through the engines that integrate it: vLLM, SGLang and MLC-LLM all build on it for state-of-the-art throughput and latency on NVIDIA GPUs. The project is documented in an MLSys 2025 paper and ships Python APIs that let engine builders compose kernels without writing CUDA by hand. Its block-sparse and customizable attention variants also make it a research platform for experimenting with new inference algorithms.
Free. FlashInfer is open source under the Apache 2.0 license with no commercial tier; it is a library you build into your own serving stack.
Inference-engine developers and performance engineers building or tuning LLM serving systems who need proven, composable GPU kernels instead of maintaining their own CUDA attention implementations.