GPTQ-for-LLaMa was one of the most influential open-source quantization tools of the early LLM era: an efficient implementation of the GPTQ post-training quantization algorithm for LLaMA models. Published by qwopqwop200 under the Apache-2.0 license, it collected about 3,070 stars on GitHub and was last updated in 2024. By compressing 16-bit weights down to 4 bits (and other widths), it made large models runnable on a single consumer GPU for the first time, kick-starting the local-LLM ecosystem.
GPTQ performs one-shot weight quantization using approximate second-order information, dramatically shrinking model size and memory bandwidth while preserving most of the original quality. This repository packaged the method for the LLaMA family with fast CUDA kernels for both the quantization process and low-bit inference, support for multiple bit widths and group sizes, and workflows for evaluating perplexity after compression. Much of its code and many of its ideas were later absorbed into mainstream tooling such as AutoGPTQ and llama.cpp’s quantizers.
At its peak, GPTQ-for-LLaMa was the standard way the community shared compressed models: thousands of GPTQ checkpoints for LLaMA, LLaMA-2 and derivative models were published on model hubs, and popular front-ends of the era integrated this code to load them. The repository supports quantizing with different calibration datasets and evaluating the perplexity cost of compression, which made it useful both practically and as a reference implementation for the GPTQ paper’s algorithm. Modern successors exist, but the project remains an important piece of local-LLM history.
GPTQ-for-LLaMa is free and open source under the Apache-2.0 license. There are no paid tiers; you run it on your own hardware.
LLM engineers and local-model enthusiasts interested in model compression, and researchers studying post-training quantization methods and their trade-offs.