EvalScope is a one-stop evaluation framework for large models, built by the ModelScope community. With a single command it can benchmark a model’s capabilities, stress-test inference performance and visualize the results, covering LLMs, vision-language models and more.
EvalScope is open source on GitHub with about 3,570 stars and very active development in 2026. It ships with industry-recognized benchmarks such as MMLU, C-Eval and GSM8K, and supports evaluating many model types including LLMs, VLMs, embedding and reranker models and AIGC systems. It can evaluate models served through OpenAI-compatible APIs, local inference backends and custom adapters, and it adds inference performance stress testing so teams can measure throughput and latency alongside quality. Results are presented in visual reports that make it easy to compare models and track regressions across versions. Beyond capability benchmarks, its stress-testing mode reports tokens per second, time to first token and latency percentiles under configurable concurrency, which is what serving teams actually need when comparing inference backends or tuning GPU deployments. The framework is a flagship project of the ModelScope community and integrates with the models and datasets hosted there, so an evaluation can go from a model page to a full benchmark report without glue code.
EvalScope is free and open source under the Apache-2.0 license. The only costs are the compute or API calls used by the models you evaluate.
ML teams and researchers who need a standardized, repeatable way to measure model quality and serving performance before choosing or shipping a model.