MTEB, the Massive Text Embedding Benchmark, is an open-source evaluation toolbox on GitHub with about 3,470 stars under the Apache-2.0 license, actively maintained in 2026. It provides a standardized way to measure how good embedding and retrieval models really are, across many languages, domains, and modalities, instead of relying on vendor claims or a single cherry-picked dataset.
The toolbox covers the full range of embedding evaluation tasks: text classification, clustering, pair classification, reranking, retrieval, semantic textual similarity, bitext mining, and summarization, including low-resource languages and multimodal settings. Models are loaded through a simple Python API or the command line, tasks are selected by name, and results are written out in a comparable format. MTEB also powers a public interactive leaderboard where embedding models are ranked on the same footing, which has made it a reference point for the whole retrieval community. Installation is a single pip or uv command, and models from Sentence Transformers and similar libraries plug in directly. Because the same tasks and metrics are used for every model, results are directly comparable across papers and products, and the benchmark has become the standard citation when new embedding models are announced. The toolbox is maintained by a broad open-source community of researchers and practitioners.
MTEB is free and open source under the Apache-2.0 license. You only need your own compute to run the evaluations.
MTEB is for NLP engineers choosing an embedding model for search or RAG, researchers publishing new models, and teams that need reproducible quality numbers before shipping a retrieval system.