Paddler is an open-source load balancer and serving platform for self-hosting large language models at scale. With about 1,700 stars on GitHub and active development in 2026, it is built around the llama.cpp ecosystem and gives teams privacy, reliability and cost control instead of dependence on closed model providers.
Paddler runs inference through a built-in llama.cpp engine and balances requests across a fleet of hosts using agents that can be added dynamically, which makes it work well with autoscaling setups. It supports request buffering so capacity can scale from zero hosts, dynamic model swapping without redeploying, and an integrated web admin panel for management, monitoring and testing. Observability metrics come built in. Compared with heavier serving stacks, Paddler aims for fewer moving parts and simple deployment on CPU or GPU hardware, serving both LLMs and vision-language models, which makes it a pragmatic alternative to projects like llm-d or Docker Model Runner.
Because Paddler speaks the language of the llama.cpp ecosystem, models already converted to GGUF for local use can typically be served through it without another conversion step, which shortens the path from experiment to production.
Paddler is free and open source under the Apache-2.0 license. There is no per-token billing from the project itself; you pay only for the infrastructure you serve models on, which is exactly the predictable cost control the platform is designed to give product teams.
Product teams embedding LLM features who want predictable inference costs, and organizations that need to keep model traffic on their own infrastructure for privacy or compliance reasons. It fits platform engineers who are comfortable running llama.cpp in production.