HumanEval is OpenAI’s original evaluation harness for measuring how well language models write code, published on GitHub where it has about 3,413 stars under the MIT license. It accompanies the paper “Evaluating Large Language Models Trained on Code”, the study that introduced the Codex model and made HumanEval one of the most cited benchmarks in AI.
The benchmark consists of hand-written programming problems, each with a function signature, a docstring, and unit tests. A model generates completions, the harness executes them in a controlled environment, and quality is reported with the now-standard pass@k metric, which estimates the chance that at least one of k samples passes all tests. The repository includes the dataset files, example problems and solutions, and the scoring code used in the original research. OpenAI now marks the repository as deprecated and unmaintained, provided for historical reference, and it carries strong warnings: the harness exists to run untrusted model-generated code, so it should only be used inside a robust security sandbox. Even so, it remains the canonical reference implementation that countless later code benchmarks build on. Results on HumanEval are still quoted in model cards and launch announcements today, which is why having the original dataset and scoring code publicly available matters for verifying claims. The problems were also designed to test reading comprehension: each prompt specifies behavior in natural language, so the model must understand the specification, not just produce plausible code.
HumanEval is free and open source under the MIT license.
HumanEval is for researchers and engineers evaluating code models, students studying how LLM benchmarks work, and anyone who needs the reference implementation behind a widely reported benchmark score.