SMG, the Shepherd Model Gateway, is an engine-agnostic LLM gateway written in Rust by the Lightseek team. It puts one consistent API in front of many different inference engines and hosted providers, so applications stop caring which backend actually serves a model. The project is released under the Apache-2.0 license, has about 659 stars on GitHub, and was actively updated in October 2026.
SMG offers full OpenAI and Anthropic API compatibility while routing traffic to engines such as vLLM, TensorRT-LLM, TokenSpeed and SGLang, or to hosted services like OpenAI and Gemini. Its design centers on a gRPC pipeline that the project describes as an industry first for LLM gateways, combined with KV cache-aware routing that sends requests to the backend most likely to reuse cached context. Around that core it adds chat history handling, tokenization caching, support for the Responses API and embeddings, WASM plugins for custom logic, MCP integration and multi-tenant authentication. For platform teams running several engines side by side, SMG acts as the single front door for inference traffic.
Operators get a single place to apply authentication, routing policy and plugin logic instead of re-implementing those concerns inside every application. The project publishes releases on GitHub and distributes container images, which makes it straightforward to trial in front of an existing inference engine before adopting it more widely.
SMG is free and open source under the Apache-2.0 license. You self-host the gateway and pay only for your own infrastructure and any hosted model APIs you route to.
Shepherd Model Gateway is for platform and infrastructure teams that serve LLMs from multiple engines and want one compatible, cache-aware API layer in front of all of them.