API for Open LLM is an open-source backend that exposes open-source large language models through an OpenAI-style API. Applications built against the familiar OpenAI interface can talk to self-hosted models such as LLaMA, Qwen, ChatGLM, and Baichuan simply by changing the endpoint. The project has earned more than 2,400 stars on GitHub and was last actively updated in September 2024.
Every open model family historically shipped its own demo scripts and request formats, which made swapping models painful for application developers. This project normalizes that chaos: it loads a wide range of open models โ including the LLaMA family, BLOOM, Falcon, Baichuan, Qwen, Xverse, ChatGLM generations, CodeLLaMA, and SQLCoder โ and serves them behind endpoints that mirror the OpenAI chat completions style. That means existing tools, SDKs, and frameworks that expect an OpenAI-compatible server can be pointed at your own hardware instead. Docker support makes it straightforward to package a model and its serving layer together for deployment on a GPU server. It also supports streaming-style responses, so interactive chat applications built for the OpenAI API keep their token-by-token behavior when pointed at a local model instead.
API for Open LLM is completely free and open source under the Apache-2.0 license. Your only costs are the GPU hardware or cloud compute you use to run the models themselves.
This project is for developers and teams self-hosting open models who want a standard OpenAI-shaped interface in front of them. It is especially handy when migrating an application from a commercial API to open weights without rewriting the client code.