LMDeploy is an open-source toolkit from the InternLM team for compressing, deploying, and serving large language models, published on GitHub with around 8,100 stars. It is best known for TurboMind, its high-performance inference engine that squeezes maximum throughput out of production serving workloads.
LMDeploy covers the three jobs of putting an LLM into production. For compression, it supports weight quantization methods such as AWQ and GPTQ as well as KV-cache quantization, shrinking memory footprints so big models fit smaller GPUs. For deployment, its TurboMind engine โ descended from FasterTransformer technology โ delivers high-throughput offline inference, while a PyTorch backend offers broader model compatibility. For serving, LMDeploy exposes an OpenAI-compatible API server, so applications can switch to it with minimal code changes. It supports a long list of popular open models, including LLaMA, Qwen, InternLM, Mistral, and many vision-language variants, and provides both a Python pipeline API for offline batch jobs and CLI tools for quick experiments. The project is developed by Shanghai AI Laboratory’s ecosystem team and is updated very actively, with commits continuing through late 2026.
LMDeploy is free and open source under the Apache-2.0 license. There is no paid tier or managed cloud offering; you deploy it on your own GPU infrastructure.
LMDeploy is for engineers deploying open LLMs in production who need higher throughput and lower memory usage than a naive serving setup provides โ especially teams already working with the InternLM, Qwen, or LLaMA model families.