GPT-Fast is a compact reference implementation of fast transformer text generation written in pure PyTorch by the PyTorch team at Meta. It has about 6,300 stars on GitHub and was updated in 2025.
GPT-Fast does something unusual: it refuses to be a framework. The whole implementation is under 1,000 lines of Python with no dependencies beyond PyTorch and sentencepiece, and the authors explicitly invite you to copy-paste and fork it. What it demonstrates is how much inference speed native PyTorch can deliver when you combine torch.compile with the right techniques: int8 and int4 weight-only quantization (plus a pure-PyTorch GPTQ path), speculative decoding with a draft model, and tensor parallelism across up to eight GPUs. It supports the LLaMA 2, 3, and 3.1 families, CodeLlama, Mistral, and Mixtral 8x7B, on both NVIDIA and AMD hardware, with published benchmarks reaching roughly 197 tokens per second for Llama-2-7B with int4 quantization on A100-class GPUs. An in-depth walkthrough published by the PyTorch team explains the design choices line by line, which is part of why the repository is so often recommended as reading material. Community extensions have since applied the same playbook to additional model families, but the original stays intentionally minimal.
GPT-Fast is completely free and open source under the BSD 3-Clause license.
Inference engineers and researchers who want to understand โ and reuse โ the concrete techniques behind fast LLM generation, rather than treat the inference stack as a black box.