llm-d Router is the intelligent entry point for LLM inference requests in the llm-d project, a Kubernetes-native stack for distributed large-model serving backed by a broad industry community. Formerly known as the Inference Scheduler, the router decides which model replica should serve each request. The GitHub repository has around 430 stars and is under intense active development in 2026.
Serving LLMs at scale is not like load-balancing web requests: replicas differ in KV-cache state, queue depth and prefix reuse, and naive round-robin routing wastes expensive GPU capacity. llm-d Router sits in front of a fleet of model servers โ typically vLLM โ using the Kubernetes Gateway API and the Endpoint Picker component consolidated into this repository. It scores candidate endpoints with signals such as prefix-cache affinity, load and latency, then routes each request to the replica most likely to answer it quickly, improving throughput and time-to-first-token. The InferenceObjective and InferenceModelRewrite APIs let operators express routing policy declaratively. It is infrastructure for serious deployments: multi-node, multi-model inference where routing intelligence translates directly into GPU savings.
The router is one piece of the wider llm-d project, which also covers scheduling and disaggregated serving patterns for inference clusters. Development involves contributors from across the cloud-native ecosystem, and the APIs it implements are being standardised through the Kubernetes Gateway API community, so the concepts and tooling transfer to other inference platforms as the standard matures.
llm-d Router is free and open source under the Apache-2.0 license. There is no commercial tier; you run it on your own Kubernetes infrastructure.
Platform and infrastructure engineers operating large-scale LLM serving on Kubernetes who need smarter-than-round-robin routing to squeeze more out of their GPU fleets.