Medusa is an open-source framework that accelerates large language model generation by adding multiple decoding heads to a model, letting it predict several future tokens at each step and verify them in parallel. Developed by researchers from Princeton and collaborators, the GitHub repository has around 2,770 stars and was last actively updated in 2024. It reports typical speedups of 2.2 to 3.6 times on inference with no loss in output quality.
Standard autoregressive decoding produces one token per forward pass, which wastes GPU capacity. Medusa attaches lightweight extra heads on top of the model’s final hidden states; each head drafts a candidate token for a future position, and a tree-based attention scheme verifies many candidate continuations in a single pass, accepting the longest correct prefix. The framework supports fine-tuning the heads on your own data (Medusa-1) or training them jointly with the base model (Medusa-2), and it integrates with popular open models.
The project page and paper report results across model sizes, showing the approach scales from 7B chat models to much larger deployments, and the repository provides training recipes plus evaluation scripts for measuring acceptance rates and wall-clock speedups honestly. Medusa has been widely adopted and re-implemented inside mainstream inference engines, and its head-based design influenced later work on self-speculative decoding. Example notebooks make it straightforward to test on your own model and workload.
Medusa is free and open source under the Apache-2.0 license. There are no paid tiers; your only cost is inference compute, which the framework is designed to reduce.
ML engineers serving LLMs in production and researchers working on inference optimization who want speculative-decoding speedups without maintaining a separate draft model.