--- title: Measuring benchmark optimization in speech recognition description: New research from Hugging Face and collaborators introduces three tests—consensus disagreement, masked entity retrieval, and orthographic switching—to measure... canonical: https://daily.dev/posts/measuring-benchmark-optimization-in-speech-recognition-dztn5d4uy twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Measuring benchmark optimization in speech recognition | daily.dev og:description: New research from Hugging Face and collaborators introduces three tests—consensus disagreement, masked entity retrieval, and orthographic switching—to measure... og:url: https://daily.dev/posts/measuring-benchmark-optimization-in-speech-recognition-dztn5d4uy og:image: https://api.daily.dev/og/posts/dZTn5d4Uy.png og:image:alt: Measuring benchmark optimization in speech recognition og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Measuring benchmark optimization in speech recognition **[Hugging Face](https://daily.dev/sources/huggingface)** · 10 min read · 0 upvotes · 0 comments ## Summary New research from Hugging Face and collaborators introduces three tests—consensus disagreement, masked entity retrieval, and orthographic switching—to measure 'benchmaxxing' in automatic speech recognition. Testing 11 widely used open-source ASR models on VoxPopuli and LibriSpeech, researchers found that top-scoring models often reproduced known benchmark transcript errors even when audio contradicted them, recovered silenced numbers at elevated rates, and switched spelling conventions to match the expected benchmark's style, all suggesting models detect acoustic cues tied to specific datasets rather than purely transcribing what they hear. These behaviors weakened significantly on freshly collected audio from the same domains, indicating inflated benchmark scores don't fully generalize. A new 'Benchmark fitting' tab has been added to the Open ASR Leaderboard to surface these effects, and researchers recommend benchmark developers use temporal or speaker-based test splits instead of simple random splits. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How can I tell if an ASR model is overfitting to benchmark transcripts instead of genuinely transcribing audio Three probes can reveal this: consensus disagreement (checking if a model reproduces known erroneous reference transcripts), masked entity retrieval (silencing numbers in audio to see if the model still outputs the reference's number), and orthographic switching (checking if spelling choices match a benchmark's convention even when audio is ambiguous). Testing 11 open-source models on VoxPopuli and LibriSpeech showed several top performers reproducing benchmark errors 18-30% of the time. _daily.dev surfaces research like this for engineers weighing which ASR benchmark scores to trust when picking a model._ ### Why do some speech recognition models transcribe numbers that were silenced in the audio Some ASR models autocomplete silenced numbers because they detect acoustic cues tied to a specific benchmark dataset and default to reproducing that benchmark's known reference transcript rather than relying purely on the audio. On LibriSpeech, some of the strongest benchmark-performing models reproduced masked numbers in roughly 30-40% of examples even though the number was removed from the audio, with the effect weakening on freshly collected data. _anyone comparing ASR models for production use can track findings like this on daily.dev before trusting a leaderboard score._ ### What is the Open ASR Leaderboard benchmark fitting tab It is a feature added to the Open ASR Leaderboard that quantifies reference error rates from VoxPopuli and orthographic switching across public datasets, letting users see how much a model's reported score may be inflated by benchmark-specific memorization rather than genuine transcription ability. The underlying scripts and un-normalized model outputs are open-sourced on GitHub. _daily.dev helps developers stay on top of leaderboard changes when choosing an ASR model to rely on._ ## Similar posts on daily.dev - [Open TTS Leaderboard: Scalable Evaluation for Multilingual Text-to-Speech and Voice Cloning](https://daily.dev/posts/open-tts-leaderboard-scalable-evaluation-for-multilingual-text-to-speech-and-voice-cloning-utmqtcomu) · Hugging Face · 0 upvotes · 0 comments - [Adding Benchmaxxer Repellant to the Open ASR Leaderboard](https://daily.dev/posts/adding-benchmaxxer-repellant-to-the-open-asr-leaderboard-nmod3ukfj) · Hugging Face · 0 upvotes · 0 comments --- Tags: [#data-science](https://daily.dev/tags/data-science), [#speech-recognition](https://daily.dev/tags/speech-recognition), [#whisper](https://daily.dev/tags/whisper) [View this post on daily.dev](https://daily.dev/posts/measuring-benchmark-optimization-in-speech-recognition-dztn5d4uy) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Measuring benchmark optimization in speech recognition","url":"https://daily.dev/posts/measuring-benchmark-optimization-in-speech-recognition-dztn5d4uy","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/measuring-benchmark-optimization-in-speech-recognition-dztn5d4uy"},"datePublished":"2026-08-21T13:17:33.099Z","dateModified":"2026-09-14T07:06:28.450Z","description":"New research from Hugging Face and collaborators introduces three tests—consensus disagreement, masked entity retrieval, and orthographic switching—to measure...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/e90145d764d9677be3cbf0a8d438dbf3?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/e90145d764d9677be3cbf0a8d438dbf3?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Hugging Face","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Hugging Face","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/f1f55c67d81a4330acf5b90b26b0c8e1","url":"https://daily.dev/sources/huggingface"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/measuring-benchmark-optimization-in-speech-recognition-dztn5d4uy","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"data-science,speech-recognition,whisper","timeRequired":"PT10M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Hugging Face","item":"https://daily.dev/sources/huggingface"},{"@type":"ListItem","position":3,"name":"Measuring benchmark optimization in speech recognition"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/measuring-benchmark-optimization-in-speech-recognition-dztn5d4uy#faq","mainEntity":[{"@type":"Question","name":"How can I tell if an ASR model is overfitting to benchmark transcripts instead of genuinely transcribing audio","acceptedAnswer":{"@type":"Answer","text":"Three probes can reveal this: consensus disagreement (checking if a model reproduces known erroneous reference transcripts), masked entity retrieval (silencing numbers in audio to see if the model still outputs the reference's number), and orthographic switching (checking if spelling choices match a benchmark's convention even when audio is ambiguous). Testing 11 open-source models on VoxPopuli and LibriSpeech showed several top performers reproducing benchmark errors 18-30% of the time. daily.dev surfaces research like this for engineers weighing which ASR benchmark scores to trust when picking a model."}},{"@type":"Question","name":"Why do some speech recognition models transcribe numbers that were silenced in the audio","acceptedAnswer":{"@type":"Answer","text":"Some ASR models autocomplete silenced numbers because they detect acoustic cues tied to a specific benchmark dataset and default to reproducing that benchmark's known reference transcript rather than relying purely on the audio. On LibriSpeech, some of the strongest benchmark-performing models reproduced masked numbers in roughly 30-40% of examples even though the number was removed from the audio, with the effect weakening on freshly collected data. anyone comparing ASR models for production use can track findings like this on daily.dev before trusting a leaderboard score."}},{"@type":"Question","name":"What is the Open ASR Leaderboard benchmark fitting tab","acceptedAnswer":{"@type":"Answer","text":"It is a feature added to the Open ASR Leaderboard that quantifies reference error rates from VoxPopuli and orthographic switching across public datasets, letting users see how much a model's reported score may be inflated by benchmark-specific memorization rather than genuine transcription ability. The underlying scripts and un-normalized model outputs are open-sourced on GitHub. daily.dev helps developers stay on top of leaderboard changes when choosing an ASR model to rely on."}}]} ```