MLE-bench is an open-source benchmark from OpenAI for measuring how well AI agents perform at machine learning engineering. It has 1,781 stars on GitHub and is still maintained in 2026. Note that MLE-bench is its own distinct project: it is a different product from the similarly named tool also listed on Tooloso, and this page covers OpenAI’s benchmark only.
Rather than testing models on static questions, MLE-bench drops an agent into real Kaggle competitions and asks it to do the full engineering job: explore the data, build features, train models, and produce a valid submission before a deadline. Performance is graded against the actual human leaderboard of each competition, including medal thresholds. OpenAI released the dataset construction code, the evaluation logic, and the agent scaffolds it evaluated, along with a public leaderboard (temporarily paused for new submissions while a fairer process is developed). Each run gives the agent a fixed time budget and a sandboxed environment, mirroring how a human data scientist would enter the same competition. Because the tasks are real competitions, memorized solutions do not transfer, which makes the benchmark a cleaner signal of genuine engineering ability rather than recall.
MLE-bench is free and open source. Anyone can download the benchmark and evaluate their own agents; the only costs are the compute and model API usage of the runs themselves.
MLE-bench is for agent developers and research labs who want a demanding, realistic measure of autonomous ML engineering ability, well beyond toy coding tasks.