GPTCache is an open-source semantic cache for large language model applications, created by Zilliz. Ordinary caches only help when a query repeats word-for-word; GPTCache embeds each query and serves a cached answer whenever a new question is semantically similar to one already answered — cutting API bills and response times for LLM apps. The project has about 8,200 stars on GitHub and is actively maintained in 2026.
GPTCache sits between your application and the model API. Incoming prompts are embedded and searched against stored question-answer pairs in a vector store; above a similarity threshold, the cached answer returns immediately instead of paying for a new generation. The library is modular — you can choose the embedding model, vector database (including Milvus, FAISS and others) and similarity evaluation strategy — and it integrates directly with LangChain and LlamaIndex as well as with OpenAI-style APIs. It is a practical cost-optimization layer for chatbots and RAG systems whose users ask many near-duplicate questions. The project includes benchmarking tools to measure hit rates and quality trade-offs before you commit to it, eviction policies to keep the cache fresh, and a Docker-based deployment for running it as a shared service. Because a wrong cache hit is worse than a miss, GPTCache’s similarity evaluation is configurable per application, and teams typically tune it on their own traffic — support questions and FAQ-style queries tend to cache extremely well, while highly personal or time-sensitive prompts can be excluded.
GPTCache is completely free and open source under the MIT license.
GPTCache is for developers running LLM features in production who want to stop paying full price — and full latency — for the same questions asked in slightly different words.