AWQ

AWQ (Activation-aware Weight Quantization) is an award-winning method for compressing large language models to low-bit weights so they run faster and use far less memory. Developed at MIT’s HAN Lab, it won the MLSys 2024 Best Paper Award and is one of the most widely used quantization approaches in the LLM ecosystem.

Overview

The llm-awq repository is open source on GitHub with about 3,650 stars and was updated in 2025. AWQ protects the small fraction of weights that matter most, identified from activation statistics, which lets it quantize models to INT3 or INT4 with minimal accuracy loss and no retraining. The repo includes the AWQ search procedure, a pre-computed model zoo covering Llama, OPT, CodeLlama, StarCoder, Vicuna and multimodal models such as LLaVA and VILA, a memory-efficient 4-bit linear layer in PyTorch, and optimized CUDA kernels. Its TinyChat runtime demonstrates 4-bit inference running several times faster than FP16 on hardware from an RTX 4090 down to a Jetson Orin edge device. The techniques from AWQ have been absorbed into mainstream inference stacks, so models quantized with this repository’s search procedure can be served by popular runtimes while keeping the original accuracy characteristics the paper demonstrated.

Key Features

• Accurate INT3/INT4 weight quantization without retraining the model
• Pre-computed quantized model zoo for popular LLMs and vision-language models
• Efficient CUDA kernels for both context and decoding stages
• TinyChat runtime delivering up to roughly 3x faster inference than FP16
• Chunked prefilling for faster multi-round question answering
• Supports instruction-tuned and multi-modal models alike

💳 Pricing

AWQ is free and open source under the MIT license. Researchers and engineers can use the code, kernels and model zoo at no cost.

Who Is It For

ML engineers deploying LLMs on limited hardware, from edge devices to cost-sensitive servers, who need smaller, faster models without giving up output quality.