The Gateway API Inference Extension is an official Kubernetes SIG project that optimizes self-hosting generative models on Kubernetes. Published on GitHub with about 790 stars under the Apache-2.0 license and updated in October 2026, it turns a standard gateway into an inference-aware gateway for LLM serving.
Plain load balancing is a poor fit for LLM traffic, where request cost varies and model servers have their own queues and caches. The extension leverages Envoy’s external processing (ext-proc) to upgrade any gateway that supports both ext-proc and the Kubernetes Gateway API into an inference gateway, adding model-aware routing concepts on top. The project works closely with the vLLM ecosystem and the llm-d initiative to accelerate LLM serving optimizations; some components, such as the endpoint picker, have since moved to the llm-d repositories as the architecture evolves.
The core idea is that an inference gateway should understand the model traffic it routes: requests target named models, backends differ in load and capability, and a smart gateway can make better placement decisions than round-robin DNS or generic HTTP load balancing. The extension defines the Kubernetes-side APIs and the ext-proc integration that lets existing gateways act on that information. As the ecosystem matures, pieces such as the endpoint picker have been factored into the llm-d project, with this repository remaining the home of the Gateway API inference specification work.
The extension is free and open source under the Apache-2.0 license. It runs in your own clusters; your costs are only the Kubernetes and GPU infrastructure you already operate.
It is for platform engineers and infrastructure teams self-hosting LLMs on Kubernetes who need smarter traffic management than a generic ingress or load balancer can provide.