TensorRT-LLM

TensorRT-LLM is NVIDIA’s open-source library for large language model inference, with more than 14,900 stars on GitHub, providing an easy-to-use Python API to define LLMs and run them with state-of-the-art optimizations on NVIDIA GPUs.

Overview

Serving large language models in production is a performance problem: naive inference wastes GPU memory and leaves throughput on the table. TensorRT-LLM attacks that problem with the TensorRT compiler stack, wrapping optimized kernels, quantization, in-flight batching, and distributed execution behind a Python API that feels familiar to anyone who has defined a model in PyTorch. It contains components to build both Python and C++ runtimes that orchestrate inference execution in a performant way, and it supports today’s demanding workloads — mixture-of-experts models, long contexts, and the latest Blackwell-generation GPUs alongside earlier architectures. Model definitions cover the major open model families, and serving features such as speculative decoding and KV-cache management are built in rather than bolted on. Maintained directly by NVIDIA with contributions from across the ecosystem, the repository is updated continuously and is one of the reference stacks for high-performance LLM serving on NVIDIA hardware.

Key Features

• Python API for LLM definition – define models in Python, with PyTorch and TensorRT backends.
• State-of-the-art optimizations – fused kernels, quantization, and in-flight batching squeeze maximum throughput from each GPU.
• Mixture-of-experts support – efficient MoE execution for modern sparse model architectures.
• Blackwell and beyond – support for NVIDIA’s newest GPU generations as well as earlier ones.
• Python and C++ runtimes – components to build runtimes that orchestrate inference execution efficiently.
• Speculative decoding and KV-cache features – latency and memory optimizations for production serving.
• Distributed inference – tensor and pipeline parallelism for models too large for a single GPU.

💳 Pricing

TensorRT-LLM is free and open source from NVIDIA. You download it, run it on your own NVIDIA GPUs or cloud instances, and pay only for the hardware — there is no license fee or per-token charge for the library itself.

Who Is It For

ML platform engineers and infrastructure teams serving open-weight language models on NVIDIA GPUs who need maximum throughput and minimum latency, and researchers benchmarking optimized inference.