OSWorld is a free, open-source benchmark for multimodal AI agents that operate real computers. Instead of toy simulations, agents are given open-ended tasks inside actual Ubuntu and Windows virtual machines โ using real applications like Chrome, LibreOffice and VLC โ and are scored by whether the task genuinely got done.
OSWorld is an open-source project on GitHub with about 3,200 stars, presented at NeurIPS 2024 and actively developed by the XLang Lab into 2026. It provides a scalable, reproducible execution environment with hundreds of tasks spanning web browsing, file management, office productivity and multi-app workflows, each with programmatic setup and verification. The benchmark has become a standard yardstick for computer-use agents: results showed a large gap between human performance (around 72%) and the best models, and OSWorld’s leaderboard now tracks progress across GPT, Claude and open agent systems, including work on standardised variants used by major labs for training and evaluation.
OSWorld is completely free and open source under the Apache-2.0 license. Researchers can run the full benchmark on their own infrastructure at no cost.
Tasks are defined with declarative configuration files covering initial state setup, required files and getter functions for scoring, so new tasks can be contributed by the community and reproduced exactly by other labs โ one reason OSWorld results are widely cited when new computer-use models are announced.
OSWorld is for AI researchers and agent developers who need a rigorous, real-world testbed for computer-use agents โ to evaluate models, compare agent frameworks or train agents on realistic desktop tasks.