TorchAO (the pytorch/ao repository) is PyTorch’s native architecture optimization library, providing quantization and sparsity techniques for both training and inference. Maintained by the PyTorch team, it lets you compress large models โ especially LLMs such as Llama โ with int8, int4 and float8 formats so they run faster and use far less memory, without leaving the PyTorch ecosystem. The project has around 3,000 stars on GitHub and is under very active development, with updates continuing through 2026.
Quantization libraries often sit awkwardly outside the framework, forcing custom kernels and fragile export steps. TorchAO takes the opposite approach: its techniques are built on native PyTorch primitives and compose with torch.compile, so a quantized model still behaves like a normal PyTorch model. It covers post-training quantization for inference, quantization-aware training for accuracy-critical fine-tuning, and cutting-edge float8 training recipes used to speed up large-scale LLM training. The same APIs also support sparsity and newer low-precision data types such as MX formats, making TorchAO the reference toolkit for efficient PyTorch deployment on server, desktop and edge hardware. It is developed in the open alongside PyTorch itself, so new hardware and model architectures are typically supported quickly.
TorchAO is free and open source as part of the PyTorch project. There are no licenses to buy and no hosted version โ the library runs entirely within your own PyTorch environment.
TorchAO is aimed at ML engineers and researchers who train or serve PyTorch models and need production-grade quantization to cut inference cost, memory footprint or training time.