Terminal-Bench 1.0

Terminal-Bench is an open-source benchmark that measures how well AI agents perform complicated, real-world tasks in a terminal, from building software to configuring servers. This repository is the original 1.0 release of the benchmark, with around 2,600 GitHub stars.

Overview

Each Terminal-Bench task drops an agent into a containerized command-line environment with a goal — for example compiling a project, cleaning a dataset or setting up a service — and a set of automated tests that verify the end state. Because success is checked by tests rather than by a judge model, scores are hard to game and easy to reproduce. Tasks span software engineering, system administration, data science and security-flavoured chores, and many take a competent human well over an hour. Terminal-Bench has become a standard yardstick for coding and computer-use agents, and its public leaderboard tracks frontier models and agent frameworks. The 1.0 task set in this repository is the frozen first release; the project site at tbench.ai covers the benchmark’s current versions.

Key Features

• Hand-built tasks in realistic terminal environments
• Containerized setups that run the same way on any machine
• Automated test suites verify results objectively
• Public leaderboard comparing models and agent harnesses
• Harness for running your own agent against the full suite
• Difficulty calibrated so frontier agents still fail many tasks

💳 Pricing

Terminal-Bench is free and open source under the Apache-2.0 license. The task definitions, harness and tests are all in the repository; you only pay for the model API calls your agent makes while attempting the tasks.

Who Is It For

Agent developers, model evaluators and researchers who want a tough, reproducible measure of command-line competence.