Simple Evals is OpenAI’s lightweight, open-source library for evaluating large language models. It packages a set of well-known academic and OpenAI benchmarks behind one minimal codebase, so anyone can run the same evaluations OpenAI uses and compare models on equal footing.
Rather than a heavyweight framework, Simple Evals is deliberately small: a shared sampling interface, per-benchmark eval definitions, and reporting that makes results easy to reproduce and audit. The bundled benchmarks cover broad knowledge and reasoning (MMLU, GPQA, MATH), coding (HumanEval), multilingual grade-school math (MGSM), reading comprehension (DROP), and OpenAI’s own factuality and research-oriented evals such as SimpleQA, BrowseComp and HealthBench. Samplers are included for both the OpenAI API and Anthropic’s Claude API, and adding a new model usually means implementing one small interface. One important caveat, stated in the repository itself: as of July 2025 OpenAI no longer updates the library for new models or benchmarks, and keeps it public mainly as the reference implementation for HealthBench, BrowseComp and SimpleQA. Because the code is small and dependency-light, teams also use it as a template for adding their own benchmarks alongside the bundled ones. The repository also documents the exact prompts and settings behind each reported number. The project is MIT-licensed and has more than 4,600 stars on GitHub.
Free. Simple Evals is open source under the MIT license; you only pay for the model API calls your evaluations make.
Researchers, model developers and evaluation teams who want transparent, reproducible benchmark numbers for frontier models without adopting a large evaluation framework, and anyone who needs the canonical SimpleQA, BrowseComp or HealthBench implementations.