OpenAI Evals is an open-source framework for evaluating LLMs and LLM systems, together with a community registry of benchmarks. It has more than 19,600 stars on GitHub and remains the reference evaluation toolkit published by OpenAI, last updated in April 2026.
Shipping LLM features without evaluations is guesswork, and Evals exists to make measurement routine. It provides a standard way to define test cases, run models against them, and grade the results, including model-graded evaluations where a strong model judges outputs against a rubric. The accompanying registry collects ready-made evals for common capabilities and safety checks, so teams can start from existing benchmarks and then add private evals that reflect their own products and failure modes.
A typical workflow starts small: a team writes an eval for a known weakness, watches current models fail it, improves the prompt or system, and reruns the eval to confirm the fix without breaking anything else. Basic evals check exact matches and structured outputs, while model-graded evals judge tone, correctness, and reasoning quality on open-ended tasks. Results are logged in a consistent format, which makes runs comparable over time and across models. Many organizations wire their eval suites into continuous integration, so a model upgrade or prompt change automatically faces the same battery of tests before it reaches users.
Evals is free and open source. Running evaluations consumes model API calls, which are billed by your provider according to your own account.
OpenAI Evals is for AI product and research teams who need repeatable, data-driven comparisons of models and prompts, regression testing before releases, and a shared language for discussing model quality.