--- title: Open ASR Leaderboard adds Hindi and Indian English with... description: Hugging Face's Open ASR Leaderboard has added its first Global South languages, Hindi and Indian English, via two new evaluation sets built with Voice Arena... canonical: https://daily.dev/posts/open-asr-leaderboard-adds-hindi-and-indian-english-with-disaggregated-evaluation-eb09oangu twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Open ASR Leaderboard adds Hindi and Indian English with disaggregated evaluation | daily.dev og:description: Hugging Face's Open ASR Leaderboard has added its first Global South languages, Hindi and Indian English, via two new evaluation sets built with Voice Arena... og:url: https://daily.dev/posts/open-asr-leaderboard-adds-hindi-and-indian-english-with-disaggregated-evaluation-eb09oangu og:image: https://api.daily.dev/og/posts/Eb09OaNgU.png og:image:alt: Open ASR Leaderboard adds Hindi and Indian English with disaggregated evaluation og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Open ASR Leaderboard adds Hindi and Indian English with disaggregated evaluation **[Collections](https://daily.dev/sources/collections)** · 2 min read · 0 upvotes · 0 comments ## Summary Hugging Face's Open ASR Leaderboard has added its first Global South languages, Hindi and Indian English, via two new evaluation sets built with Voice Arena called Monsoon en-IN and hi-IN. The datasets include 4,888 speakers across hundreds of Indian districts recorded on personal phones in spontaneous conversation, with 16 metadata columns per clip (age, gender, geography, device, occupation) and speakers split into disjoint groups to prevent test-set tuning. Because Hindi has many valid written forms for the same spoken word, the leaderboard uses a new metric, Orthographically-Informed Word Error Rate (OIWER), scored against a lattice of accepted spellings instead of a single reference transcript, alongside a released open-source tool called voi-oiwer. Disaggregated scoring by metadata reveals that models with nearly identical overall WER can diverge sharply in performance across regions, exposing biases that aggregate scores hide. ## Content The Open ASR Leaderboard, run by Hugging Face, has added its first Global South languages: Hindi and Indian English. The two new evaluation sets, called Monsoon en-IN and hi-IN, were built with Voice Arena and mark the first Indic languages on the leaderboard's multilingual tab, which previously covered only European languages. ## What the datasets look like The sets cover 4,888 speakers across hundreds of districts in India, recorded on their own phones in spontaneous conversation rather than studio conditions. Each clip carries 16 metadata columns — age, gender, geography, device type, occupation, and more — and the speakers are split into four disjoint groups so no speaker appears in more than one split. Private held-out splits mean models can't be tuned against the test set. The practical effect is that a model has fewer shortcuts. It can't lean on familiar speakers, clean audio, or a single rigid transcript. The evaluation design is doing more work here than raw data volume. ## Why standard WER doesn't work for Hindi Hindi orthography has many valid written forms for the same spoken word, so scoring against a single reference transcript penalizes correct transcriptions unfairly. The leaderboard uses Orthographically-Informed Word Error Rate (OIWER) instead, scoring against a lattice of accepted spellings. An open-source tool, `voi-oiwer`, is released alongside the datasets. ## What disaggregated scores reveal The 16 metadata columns enable error analysis broken down by region, speaker age, device, and other attributes — rather than a single aggregate WER number. A worked example in the release shows two models with nearly identical overall WER diverging sharply when broken down by region, exposing weaknesses that the aggregate score hides entirely. This is the more interesting part of the benchmark design. A leaderboard number that looks competitive overall can mask a model that works well for urban speakers in one state and poorly for everyone else. ## Questions this post answers ### What is Orthographically-Informed Word Error Rate (OIWER) and why is it used for Hindi speech recognition evaluation? OIWER is a word error rate metric that scores speech recognition output against a lattice of accepted spellings rather than a single reference transcript. It was introduced because Hindi orthography allows many valid written forms for the same spoken word, so standard WER unfairly penalizes correct transcriptions that don't match one fixed reference. An open-source tool called voi-oiwer implements this scoring and was released alongside the new Hindi and Indian English evaluation datasets. _daily.dev surfaces benchmarking approaches like this for teams evaluating ASR models on non-English languages._ ### What languages did the Open ASR Leaderboard add and how large are the new evaluation datasets? The Open ASR Leaderboard, maintained by Hugging Face, added Hindi and Indian English as its first Global South languages, via two sets called Monsoon en-IN and hi-IN built with Voice Arena. The datasets cover 4,888 speakers across hundreds of Indian districts, recorded on personal phones in spontaneous conversation, with speakers split into four disjoint groups and 16 metadata columns per clip covering age, gender, geography, device type, and occupation. _Follow ASR benchmark expansions like this on daily.dev when choosing models for multilingual speech products._ ### Why can two ASR models have nearly the same overall word error rate but perform very differently in practice? Aggregate WER can hide large performance gaps because it averages results across all speakers and conditions. A worked example on the Open ASR Leaderboard showed two models with nearly identical overall WER diverging sharply once scores were disaggregated by region, meaning a model can work well for urban speakers in one state while performing poorly everywhere else, a gap the single aggregate number completely masks. _daily.dev helps engineers track evaluation methods that expose these hidden gaps before choosing a speech model._ ## Similar posts on daily.dev - [Open ASR Leaderboard: Trends and Insights with New Multilingual & Long-Form Tracks](https://daily.dev/posts/open-asr-leaderboard-trends-and-insights-with-new-multilingual-long-form-tracks-gx0k7fwmm) · Hugging Face · 3 upvotes · 0 comments --- Tags: [#data-science](https://daily.dev/tags/data-science), [#nlp](https://daily.dev/tags/nlp), [#speech-recognition](https://daily.dev/tags/speech-recognition), [#huggingface](https://daily.dev/tags/huggingface) [View this post on daily.dev](https://daily.dev/posts/open-asr-leaderboard-adds-hindi-and-indian-english-with-disaggregated-evaluation-eb09oangu) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Open ASR Leaderboard adds Hindi and Indian English with disaggregated evaluation","url":"https://daily.dev/posts/open-asr-leaderboard-adds-hindi-and-indian-english-with-disaggregated-evaluation-eb09oangu","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/open-asr-leaderboard-adds-hindi-and-indian-english-with-disaggregated-evaluation-eb09oangu"},"datePublished":"2026-08-28T21:49:15.392Z","dateModified":"2026-08-28T21:49:58.204Z","description":"Hugging Face's Open ASR Leaderboard has added its first Global South languages, Hindi and Indian English, via two new evaluation sets built with Voice Arena...","image":"https://pbs.twimg.com/media/HQ1xxQyaAAAqRS3.jpg","thumbnailUrl":"https://pbs.twimg.com/media/HQ1xxQyaAAAqRS3.jpg","isAccessibleForFree":true,"articleSection":"Collections","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Collections","logo":"https://media.daily.dev/image/upload/s--fk_6ycEi--/f_auto,q_auto/v1780996001/logos/collections?_a=BAMAMiWQ0","url":"https://daily.dev/sources/collections"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/open-asr-leaderboard-adds-hindi-and-indian-english-with-disaggregated-evaluation-eb09oangu","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"data-science,nlp,speech-recognition,huggingface","timeRequired":"PT2M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Collections","item":"https://daily.dev/sources/collections"},{"@type":"ListItem","position":3,"name":"Open ASR Leaderboard adds Hindi and Indian English with disaggregated evaluation"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/open-asr-leaderboard-adds-hindi-and-indian-english-with-disaggregated-evaluation-eb09oangu#faq","mainEntity":[{"@type":"Question","name":"What is Orthographically-Informed Word Error Rate (OIWER) and why is it used for Hindi speech recognition evaluation?","acceptedAnswer":{"@type":"Answer","text":"OIWER is a word error rate metric that scores speech recognition output against a lattice of accepted spellings rather than a single reference transcript. It was introduced because Hindi orthography allows many valid written forms for the same spoken word, so standard WER unfairly penalizes correct transcriptions that don't match one fixed reference. An open-source tool called voi-oiwer implements this scoring and was released alongside the new Hindi and Indian English evaluation datasets. daily.dev surfaces benchmarking approaches like this for teams evaluating ASR models on non-English languages."}},{"@type":"Question","name":"What languages did the Open ASR Leaderboard add and how large are the new evaluation datasets?","acceptedAnswer":{"@type":"Answer","text":"The Open ASR Leaderboard, maintained by Hugging Face, added Hindi and Indian English as its first Global South languages, via two sets called Monsoon en-IN and hi-IN built with Voice Arena. The datasets cover 4,888 speakers across hundreds of Indian districts, recorded on personal phones in spontaneous conversation, with speakers split into four disjoint groups and 16 metadata columns per clip covering age, gender, geography, device type, and occupation. Follow ASR benchmark expansions like this on daily.dev when choosing models for multilingual speech products."}},{"@type":"Question","name":"Why can two ASR models have nearly the same overall word error rate but perform very differently in practice?","acceptedAnswer":{"@type":"Answer","text":"Aggregate WER can hide large performance gaps because it averages results across all speakers and conditions. A worked example on the Open ASR Leaderboard showed two models with nearly identical overall WER diverging sharply once scores were disaggregated by region, meaning a model can work well for urban speakers in one state while performing poorly everywhere else, a gap the single aggregate number completely masks. daily.dev helps engineers track evaluation methods that expose these hidden gaps before choosing a speech model."}}]} ```