Shimmy is a local LLM inference engine written entirely in Rust. It serves models through an OpenAI-compatible API using WebGPU, so it can run on virtually any GPU — NVIDIA, AMD or Apple Silicon — from a single binary, with no Python and no llama.cpp in the stack. The project has about 6,000 stars on GitHub and was actively developed in 2026.
Most local inference stacks assume CUDA or a specific runtime. Shimmy bets on WebGPU as the portable GPU layer: the same binary talks to whatever graphics hardware the machine has, while loading GGUF model files natively — the format the local-model community already uses. Its server speaks the OpenAI API, so existing clients and tools can point at a local Shimmy instance unchanged, and it is published as a crate on crates.io for embedding into Rust applications. The project emphasizes audited, dependency-light engineering: one Rust binary you can drop onto a machine and run.
Shimmy’s WebGPU choice is aimed squarely at the machines other engines leave behind: laptops with integrated or AMD graphics, Macs, and mixed hardware fleets where installing CUDA is not an option. The trade-off is raw peak throughput — a hand-tuned CUDA stack will still win on high-end NVIDIA servers — but for local development, demos and personal model serving, a single portable binary that just runs is often the more useful tool.
Free. Shimmy is open source and runs entirely on your own hardware; you only supply the models.
Developers who want portable local LLM serving — especially on non-NVIDIA hardware — and prefer a small Rust binary over a Python-based inference stack.