Lookahead Decoding is an open-source research implementation from Hao AI Lab that speeds up large language model inference by breaking the sequential, token-by-token bottleneck. Hosted on GitHub with about 1,300 stars under the Apache-2.0 license, it accompanies the ICML 2024 paper of the same name, with code updates through 2025.
Standard autoregressive decoding generates one token per step, which wastes the parallelism modern GPUs offer. Lookahead decoding instead treats decoding as solving a system of equations โ building on Jacobi iteration โ and predicts multiple future tokens in parallel within each step, then verifies and keeps the longest correct prefix. Crucially, it needs no draft model and no datastore, unlike speculative decoding approaches, and it reduces the number of decoding steps roughly linearly as more compute per step is applied. The repository includes the sampling implementation, FlashAttention support and demos, such as accelerating LLaMA-2-Chat 7B generation, so researchers can reproduce the paper’s results and drop the technique into their own inference code. A detailed LMSYS blog post and a public roadmap in the repository walk through the design, the measured speedups and the advanced token-prediction features added since the first release.
Lookahead Decoding is free and open source under the Apache-2.0 license. There is no hosted service or paid version; you run the code on your own hardware as part of your own LLM inference stack.
LLM researchers and inference engineers who want to cut generation latency without training or maintaining a separate draft model, and anyone studying parallel decoding techniques from a reference implementation.