EvalPlus is an open-source framework for rigorously evaluating the code that large language models generate. It has 1,832 stars on GitHub, was last updated in 2025, and its benchmarks were presented at NeurIPS 2023 and COLM 2024. EvalPlus takes popular coding benchmarks and makes them far harder to game with expanded, carefully verified test suites.
Standard code benchmarks often accept solutions that are subtly wrong because their test coverage is thin. EvalPlus fixes this by multiplying the number of test cases: HumanEval+ extends the original HumanEval problems with roughly 80x more tests, and MBPP+ does the same for MBPP, with corrected ground-truth solutions. The project also includes EvalPerf for measuring the efficiency of generated code, not just its correctness. The resulting datasets have been used in the official evaluations of Meta Llama, Qwen-Coder, DeepSeek-Coder, StarCoder2, and other major model releases. Because the expanded test suites catch code that merely looks correct, pass rates on EvalPlus benchmarks are consistently lower, and more realistic, than on the originals. That is why modern model cards increasingly report HumanEval+ and MBPP+ numbers alongside the classic scores.
EvalPlus is free and open source under the Apache-2.0 license. The datasets and evaluation code can be used in academic and commercial model evaluation at no cost.
EvalPlus is for LLM researchers, model developers, and ML teams who need trustworthy pass@k numbers for code models, and for anyone comparing coding assistants on something stricter than the original HumanEval.