NVIDIA Dynamo is an open-source, datacenter-scale inference serving framework for deploying generative AI models across fleets of GPUs. It is the successor line to NVIDIA’s Triton Inference Server efforts for the LLM era, designed to squeeze maximum throughput and lowest latency out of multi-GPU, multi-node deployments. The project has about 8,300 stars on GitHub and is under very active development in 2026.
Dynamo’s headline technique is disaggregated serving: it splits LLM inference into prefill and decode phases that run on different GPU pools, each scaled and optimized independently. A KV-cache-aware smart router sends each request to the worker most likely to answer it efficiently, and the framework plans resource allocation across the cluster. Dynamo works with the inference engines teams already use โ vLLM, SGLang and TensorRT-LLM among them โ and is built for Kubernetes-based production environments serving large reasoning and multimodal models at scale. Around that core, Dynamo adds components such as a planner that continuously right-sizes prefill and decode capacity against live traffic, a KV block manager that tiers the key-value cache across GPU memory, CPU memory and SSD to raise effective capacity, and a low-latency interconnect layer for moving cache data between workers. These pieces target the economics of large deployments, where a few points of GPU utilization translate into very large sums of money.
NVIDIA Dynamo is free and open source. Running it requires your own GPU infrastructure; it is aimed at datacenter-scale deployments rather than single machines.
Dynamo is for infrastructure and platform teams serving LLMs at serious scale who need higher GPU utilization and lower latency than a single-engine setup can deliver.