ExLlama is an open-source inference library for running Llama-family language models with GPTQ-quantized weights, written as a leaner, more memory-efficient alternative to the Hugging Face Transformers implementation. When local LLM inference was taking off, ExLlama became a favorite in the community for squeezing large models onto consumer GPUs with good generation speeds. The repository has about 3,000 stars on GitHub.
ExLlama focuses on doing one thing well: fast, low-memory text generation with 4-bit GPTQ models. Its custom CUDA kernels and careful memory management let models that would not fit โ or would crawl โ under a general-purpose stack run comfortably on a single consumer graphics card. The library exposes both a simple generator API for scripts and the building blocks used by popular front-ends of its era. Development later moved to its successor, ExLlamaV2, which broadened hardware and format support; the original ExLlama repository was last updated in 2023 and remains available as a compact, well-understood reference implementation of GPTQ inference. Its design influenced much of the local inference tooling that followed, and its kernels are still embedded in some downstream projects that value its simplicity and predictable memory behavior.
ExLlama is completely free and open source under the MIT license. There is no commercial version โ you run it locally with your own models and hardware.
ExLlama is for local-LLM enthusiasts and developers who want to understand or reuse a compact GPTQ inference engine, and for anyone maintaining projects built during the first wave of consumer LLM tooling.