ExLlamaV2 is a fast inference library for running large language models locally on modern consumer-class GPUs. It is built around the EXL2 quantization format, which squeezes capable models into the limited VRAM of desktop graphics cards while keeping quality loss small.
Where many inference engines target datacenter hardware, ExLlamaV2 is unapologetically aimed at the local-first crowd running models on a single PC GPU. EXL2 supports flexible bitrates from roughly 2 to 8 bits per weight, and different layers of a model can use different bitrates, so you can trade size against quality with fine granularity instead of a one-size-fits-all quant. The library includes a dynamic batching generator for serving multiple sequences efficiently and supports speculative decoding to squeeze out extra tokens per second. It is also the engine behind TabbyAPI, the server its documentation recommends for exposing models over an API. One status note from the repository itself: the project is currently archived, with development continuing in its successor, ExLlamaV3 โ but ExLlamaV2 remains widely used and installable from PyPI or prebuilt wheels. Its quantization tooling converts models from common formats, and the library exposes both a low-level API for embedding and generator classes for interactive use. It is MIT-licensed and has more than 4,600 stars on GitHub.
Free. ExLlamaV2 is open source under the MIT license and runs entirely on your own hardware.
Local LLM enthusiasts and developers who want to run the largest possible models on a consumer GPU, and builders of local inference servers looking for an efficient EXL2 backend.