Mesh LLM pools GPUs and memory across multiple machines and exposes the combined capacity as a single OpenAI-compatible API. Start with one node, add more later, and the mesh decides whether a model runs locally, routes to a peer, or is split across machines when it is too large for any single box.
Mesh LLM is open source on GitHub with about 3,530 stars and active development in 2026. It is built around Skippy, a native inference engine based on a patched llama.cpp runtime that handles model management, split execution and OpenAI-style serving, while the Mesh layer adds peer discovery, transport, routing, plugins and an admin console. The result is distributed AI for the people: spare capacity on your own machines, or on machines shared privately or publicly, can power chat and agent workloads through one familiar endpoint at localhost. SDKs and a website round out the project, and the whole system builds from a single Rust workspace. Models too large for one machine can be executed with pipeline-style stage splits across nodes, so a small group of consumer GPUs can collectively serve models that none of them could hold alone.
Mesh LLM is free and open source under the Apache-2.0 license. You contribute or rent your own hardware; there is no per-token bill from the project itself.
Developers, labs and communities that want to combine scattered GPU capacity into one dependable LLM endpoint for agents and chat, without depending on a single cloud provider.