The Picovoice Speech-to-Text Benchmark is an open-source framework for benchmarking speech recognition engines against each other on common datasets, using word error rate as the scoreboard. Anyone can rerun the measurements and check the numbers, which is precisely what makes the framework useful as a shared reference. It is a minimalist, extensible harness that has become a reference point for comparing cloud APIs and on-device engines on equal footing. The project is hosted on GitHub, where it has earned roughly 700 stars, and it is actively maintained, with updates in 2026.
Vendors quote accuracy numbers measured on their own chosen data, which makes cross-engine comparison nearly impossible. This benchmark fixes that by running many engines โ Picovoice’s own Cheetah among them, plus Mozilla DeepSpeech, Google Speech-to-Text, AWS Transcribe, PocketSphinx and others โ over shared public corpora including LibriSpeech, TED-LIUM, Mozilla Common Voice, Multilingual LibriSpeech, VoxPopuli and FLEURS. It reports word error rate consistently across engines and datasets, and the framework is deliberately small so researchers can add a new engine or dataset without fighting the harness. Published results in the repository give an at-a-glance view of how offline, privacy-friendly engines stack up against cloud services.
The benchmark framework is free and open source under the Apache License 2.0. Note that some of the engines it measures are commercial products with their own licensing โ the harness itself costs nothing.
This benchmark is for speech engineers, researchers and product teams evaluating speech-to-text engines who want reproducible, apples-to-apples accuracy numbers instead of vendor marketing claims.