OpenCompass

OpenCompass is a comprehensive open-source platform for evaluating large language models. It benchmarks models from OpenAI, Anthropic, Google, Qwen, GLM, DeepSeek and many others across more than 100 datasets, covering knowledge, reasoning, coding, science, language understanding, long context and safety. The project has about 7,500 stars on GitHub, is released under the Apache 2.0 license, and was actively developed in 2026.

Overview

Serious model evaluation is an engineering discipline of its own: datasets need careful preparation, prompts and inference settings change results, and scores are only useful if runs are reproducible. OpenCompass packages that discipline. It separates datasets, models, evaluators and summarizers into swappable components, distributes large evaluation jobs across machines, and produces standardized result tables that feed public leaderboards such as CompassRank. The platform supports API models and locally hosted open models alike, includes both objective benchmarks and subjective evaluation workflows, and is maintained by the community around Shanghai AI Laboratory’s InternLM ecosystem. For teams choosing or fine-tuning a model, it is the difference between vibes and evidence.

Key Features

• 100+ benchmark datasets across knowledge, reasoning, code, science and safety
• Works with API models (OpenAI, Claude, Gemini) and local open models
• Distributed evaluation for running large benchmark suites efficiently
• Standardized, reproducible result reports and leaderboard tooling
• Both objective scoring and LLM-based subjective evaluation support
• Extensible design for adding custom datasets, models and metrics
• Published evaluation methodology used by community leaderboards to compare frontier models
• Supports both subjective and objective evaluation in a single platform

💳 Pricing

Free. OpenCompass is open source under the Apache 2.0 license; costs are limited to your own compute and any model API usage during evaluation.

Who Is It For

Research labs, model developers and platform teams who need trustworthy, comparable measurements of LLM quality before shipping a model or switching providers.