OLMo Eval

OLMo Eval is the Allen Institute for AI’s open-source workbench for evaluating language models throughout the model development loop. Built by the team behind the OLMo model family and updated continuously into 2026, it has more than 70 stars on GitHub and is licensed under Apache 2.0.

Overview

Where many evaluation scripts run one benchmark one way, OLMo Eval is organized as a registry of benchmark tasks that compose into suites, with named variants capturing few-shot counts, prompt formatting and scoring methods — so a three-shot, bits-per-byte-scored coding benchmark is a name, not a pile of flags. A harness abstraction separates how a run executes from what a task is, which means the same task can run plainly or tool-augmented without modification. Inference is supported through vLLM for local models, LiteLLM for commercial APIs, and a mock provider for dry runs and debugging. The workbench also covers multi-turn agentic evaluation with tool calling inside sandboxed environments via Docker, Podman or Modal, LLM-as-judge scoring with auxiliary providers including locally served judges, and both aggregate and instance-level prediction storage, with inspection tooling for viewing instances, formatted prompts, token arrays and raw model responses. Note that this is the evaluation harness from the OLMo team — the OLMo models themselves are a separate project.

Key Features

• Task registry and suites – benchmarks compose into reusable suites with named few-shot, format and scoring variants.
• Multiple inference backends – vLLM for open models, LiteLLM for commercial APIs, and a mock provider for debugging.
• Agentic evaluation – multi-turn, tool-calling evaluations in Docker, Podman or Modal sandboxes.
• LLM-as-judge scoring – auxiliary judge models, including locally served ones, for open-ended tasks.
• Deep inspection – browse instances, formatted prompts, token arrays and model outputs after a run.
• Reproducible by default – a uv-managed environment with a checked-in lockfile for identical setups.

💳 Pricing

OLMo Eval is free and open source under the Apache 2.0 license. You run it on your own hardware; any API-based evaluation is billed by the model provider you choose.

Who Is It For

It is for model developers, research labs and ML platform teams who need rigorous, repeatable evaluation — including agentic and judged evals — while training, fine-tuning or comparing language models.