Prometheus Eval is an open-source family of evaluator language models and benchmarks for scoring LLM outputs: its Prometheus judge models grade responses against custom criteria, as a free alternative to using GPT-4 as a judge. Released under the Apache-2.0 license, the project has about 1,100 stars on GitHub and was last updated in 2025.
Evaluating open-ended generation is the bottleneck of LLM development: human review is slow, and calling a frontier model as judge is expensive and opaque. Prometheus addresses this with open models fine-tuned specifically for evaluation, able to assign scores and written feedback following user-supplied rubrics for any task. The family has evolved through Prometheus 2, which unified absolute and relative grading, to M-Prometheus, multilingual judge models in 3B, 7B and 14B sizes that report state-of-the-art results on multilingual meta-evaluation benchmarks. The repository also hosts BiGGen-Bench, a benchmark of 765 instances across 9 capabilities and 77 tasks with instance-specific criteria, used to compare more than a hundred frontier models. Everything is built for the LLM-as-a-judge workflow, with vLLM serving support for fast batch evaluation.
Using Prometheus as a judge also improves evaluation hygiene: the same open judge can be re-run by anyone, making scores reproducible in a way that calls to a changing proprietary model are not. The project provides the prompts and instructions used to elicit consistent judgments, and its rubrics format has been reused by other evaluation toolkits building on the Prometheus line of work.
Prometheus Eval is free and open source under the Apache-2.0 license. The judge models and benchmarks can be downloaded and run on your own infrastructure with no per-judgment API fees.
Prometheus Eval is for LLM developers, research teams and LLM-ops engineers who need scalable, rubric-based evaluation of model outputs without depending on a closed frontier model as their only judge.