MARLIN is an open-source, highly optimized FP16xINT4 matrix multiplication kernel for large language model inference, developed by IST-DASLab. It delivers close to the ideal 4x speedup of 4-bit weight quantization at batch sizes where earlier kernels gave up. The project is licensed under Apache-2.0, has about 1,200 stars on GitHub, and was last updated in 2024.
Weight-only quantization shrinks a model to 4-bit integers, but actually realizing the theoretical speedup is hard: the GPU must keep global memory, L2 cache, shared memory, tensor cores and vector cores busy simultaneously while dequantizing weights on the fly. MARLIN (Mixed Auto-Regressive Linear) was engineered to do exactly that, sustaining near-ideal speedups up to batch sizes of 16 to 32 tokens, compared with the 1 to 2 tokens typical of prior kernels. That makes it valuable for real serving scenarios such as speculative decoding, sampling with multiple candidates, and medium-batch inference. Coming from IST-DASLab, the group behind widely used compression research like GPTQ-related tooling, MARLIN has been integrated into mainstream inference stacks and is a reference implementation for fast quantized LLM serving.
MARLIN’s design pays attention to the unglamorous details that decide real-world speed: asynchronous weight loading, carefully staged dequantization, and avoiding global memory round-trips for intermediate results. The repository includes benchmarks against earlier kernels, and its techniques influenced later generations of quantized inference kernels in popular serving engines, where its layouts and scheduling ideas were widely adopted.
MARLIN is free and open source under the Apache-2.0 license. There are no paid tiers; you compile and run the kernel on your own NVIDIA GPU infrastructure.
MARLIN is for LLM inference engineers and framework maintainers who serve 4-bit quantized models and need high throughput at realistic batch sizes, as well as GPU kernel researchers studying quantization performance.