LLMLingua is Microsoft’s open-source prompt compression framework: it shrinks long prompts and KV caches by up to 20x with minimal performance loss, making LLM inference faster and cheaper. The research behind it appeared at EMNLP 2023 and ACL 2024. The project has earned about 6,800 stars on GitHub, is MIT licensed, and remains active in 2026.
Long prompts are expensive: every token costs latency and money, and retrieval-augmented or few-shot prompts balloon quickly. LLMLingua uses a small language model to estimate which tokens in a prompt carry the information and which can be dropped, then compresses the prompt at a coarse-to-fine granularity before it reaches the big model. The follow-up LongLLMLingua targets long-context scenarios such as document QA, improving the model’s perception of key information in the middle of long inputs, while LLMLingua-2 trains a dedicated compressor for task-agnostic, faster compression. Reported results show up to 20x shorter prompts with minimal degradation on benchmarks like GSM8K and Big-Bench Hard, translating directly into lower API bills and quicker responses for RAG pipelines, agents and long-document workflows. The library integrates with Hugging Face models and is straightforward to insert before any downstream LLM call.
Free. LLMLingua is open source under the MIT license; any savings come from the API tokens and latency it removes from your own workloads.
LLM application developers โ especially on RAG, agent and long-document workloads โ who want to cut inference cost and latency by compressing prompts before they hit an expensive model.