OpenRLHF is an open-source framework for reinforcement learning from human feedback and agentic reinforcement learning, built to make post-training large language models practical at scale. Hosted on GitHub under the Apache-2.0 license, it has collected more than 10,100 stars and is actively maintained, with updates continuing into 2026. It is one of the reference open implementations for PPO-style alignment training.
OpenRLHF describes itself as an easy-to-use, scalable, and high-performance agentic RL framework based on Ray. It combines vLLM for fast generation, Ray for distributed orchestration, and asynchronous training loops so that sampling and learning can overlap instead of blocking each other. Beyond classic RLHF with PPO, it supports newer algorithms such as DAPO and REINFORCE++, rejection sampling fine-tuning, and training for vision-language models. Integration with the Hugging Face Transformers ecosystem means most open-weight models can be dropped in with minimal glue code. Documentation is hosted on Read the Docs, with examples covering single-node experiments through to multi-node cluster runs.
OpenRLHF is free and open source under the Apache-2.0 license. There is no paid tier, hosted version, or usage fee: you run it on your own GPU infrastructure and only pay for your compute. Documentation and community support channels are also free.
OpenRLHF is aimed at ML engineers and research teams who fine-tune open-weight language models with human or automated feedback: alignment researchers, teams building agentic workflows that learn from rewards, and labs that need PPO-class training without writing a distributed training stack from scratch.