--- title: Scaling Indic Pretraining: 2.8 Trillion Tokens and an... description: NVIDIA, IIT Madras, AI4Bharat, and Bodhan AI jointly released Nemotron-pretraining-Indic-v1, a roughly 2.8 trillion token pretraining corpus covering 22 Indian... canonical: https://daily.dev/posts/scaling-indic-pretraining-2-8-trillion-tokens-and-an-open-data-generation-pipeline-gngeiqcjb twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Scaling Indic Pretraining: 2.8 Trillion Tokens and an Open Data Generation Pipeline | daily.dev og:description: NVIDIA, IIT Madras, AI4Bharat, and Bodhan AI jointly released Nemotron-pretraining-Indic-v1, a roughly 2.8 trillion token pretraining corpus covering 22 Indian... og:url: https://daily.dev/posts/scaling-indic-pretraining-2-8-trillion-tokens-and-an-open-data-generation-pipeline-gngeiqcjb og:image: https://api.daily.dev/og/posts/GNGeIQcjB.png og:image:alt: Scaling Indic Pretraining: 2.8 Trillion Tokens and an Open Data Generation Pipeline og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Scaling Indic Pretraining: 2.8 Trillion Tokens and an Open Data Generation Pipeline **[Hugging Face](https://daily.dev/sources/huggingface)** · 12 min read · 0 upvotes · 0 comments ## Summary NVIDIA, IIT Madras, AI4Bharat, and Bodhan AI jointly released Nemotron-pretraining-Indic-v1, a roughly 2.8 trillion token pretraining corpus covering 22 Indian languages (native script for all, romanized for twelve), along with an open GPU-accelerated data generation pipeline. The pipeline translates English documents from Nemotron-CC-v2 using Gemma-4-31B-it and Bodhan AI's Indic Translate, then runs quality assessment (an MQM-inspired rubric, 75/100 threshold) and targeted repair, recovering 438.4 billion tokens. A ten-billion-token subset was used for continued pretraining of Nemotron 3.5 Super VL, producing gains of 4.98 to 9.82 percentage points across five Indic benchmarks (MILU, Indic MMLU-Pro, Indic ARC-Challenge/Easy, Indic GSM8K) and improvements on four IndiVibe language-quality dimensions versus Nemotron 3 Super. The corpus, generation plugin for NeMo Data Designer, and usage guidance (loading via Hugging Face Datasets, building training mixtures, adapting prompts) are provided for developers to build or extend Indic language models. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How many tokens are in the Nemotron-pretraining-Indic-v1 corpus and how many languages does it cover? The corpus contains approximately 2.8 trillion tokens spanning 22 Indian languages, including 2,601.0 billion native-script tokens and 224.6 billion romanized tokens (counted with the Gemma-4 tokenizer). Native-script and romanized data are both provided for twelve languages, while ten additional languages have native-script data only, covering 34 language-script configurations in total. _Teams sizing a multilingual training run can track large dataset releases like this via daily.dev._ ### What quality threshold is used to accept translated text in the Nemotron Indic data pipeline? Translations must score at least 75 out of 100 on a rubric inspired by Multidimensional Quality Metrics (MQM), which evaluates meaning, fluency, terminology, style, and formatting using the same Gemma-4-31B-it model that performed the translation. Lower-scoring outputs get feedback-based repair and reassessment against the same threshold, which recovered 438.4 billion additional tokens for the twelve-language workflow. _Anyone building a translation QA pipeline can compare rubric-based approaches like this through daily.dev._ ### Did adding Indic continued pretraining data actually improve Nemotron 3.5 Super VL's benchmark scores? Yes, scores rose on all five reported Indic knowledge and reasoning benchmarks after a continued pretraining stage that allocated 10 billion of a 100 billion token budget to synthetic and natural Indic text. Gains ranged from 4.98 to 9.82 percentage points on MILU, Indic MMLU-Pro, Indic ARC-Challenge, Indic ARC-Easy, and Indic GSM8K (GSM8K excluded Odia due to context limits). _Developers weighing continued pretraining mixtures can follow benchmark results like these on daily.dev._ ## Similar posts on daily.dev - [Towards Speed-of-Light Text Generation with Nemotron-Labs Diffusion Language Models](https://daily.dev/posts/towards-speed-of-light-text-generation-with-nemotron-labs-diffusion-language-models-0ltn2qf0e) · Hugging Face · 0 upvotes · 0 comments - [Building a Fast Multilingual OCR Model with Synthetic Data](https://daily.dev/posts/building-a-fast-multilingual-ocr-model-with-synthetic-data-tuii4mnwu) · Hugging Face · 0 upvotes · 0 comments --- Tags: [#nvidia](https://daily.dev/tags/nvidia) [View this post on daily.dev](https://daily.dev/posts/scaling-indic-pretraining-2-8-trillion-tokens-and-an-open-data-generation-pipeline-gngeiqcjb) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Scaling Indic Pretraining: 2.8 Trillion Tokens and an Open Data Generation Pipeline","url":"https://daily.dev/posts/scaling-indic-pretraining-2-8-trillion-tokens-and-an-open-data-generation-pipeline-gngeiqcjb","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/scaling-indic-pretraining-2-8-trillion-tokens-and-an-open-data-generation-pipeline-gngeiqcjb"},"datePublished":"2026-10-16T01:49:47.042Z","dateModified":"2026-10-16T01:50:10.478Z","description":"NVIDIA, IIT Madras, AI4Bharat, and Bodhan AI jointly released Nemotron-pretraining-Indic-v1, a roughly 2.8 trillion token pretraining corpus covering 22 Indian...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/0a113a62aa913e10c5b941ed964edf2d?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/0a113a62aa913e10c5b941ed964edf2d?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Hugging Face","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Hugging Face","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/f1f55c67d81a4330acf5b90b26b0c8e1","url":"https://daily.dev/sources/huggingface"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/scaling-indic-pretraining-2-8-trillion-tokens-and-an-open-data-generation-pipeline-gngeiqcjb","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"nvidia","timeRequired":"PT12M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Hugging Face","item":"https://daily.dev/sources/huggingface"},{"@type":"ListItem","position":3,"name":"Scaling Indic Pretraining: 2.8 Trillion Tokens and an Open Data Generation Pipeline"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/scaling-indic-pretraining-2-8-trillion-tokens-and-an-open-data-generation-pipeline-gngeiqcjb#faq","mainEntity":[{"@type":"Question","name":"How many tokens are in the Nemotron-pretraining-Indic-v1 corpus and how many languages does it cover?","acceptedAnswer":{"@type":"Answer","text":"The corpus contains approximately 2.8 trillion tokens spanning 22 Indian languages, including 2,601.0 billion native-script tokens and 224.6 billion romanized tokens (counted with the Gemma-4 tokenizer). Native-script and romanized data are both provided for twelve languages, while ten additional languages have native-script data only, covering 34 language-script configurations in total. Teams sizing a multilingual training run can track large dataset releases like this via daily.dev."}},{"@type":"Question","name":"What quality threshold is used to accept translated text in the Nemotron Indic data pipeline?","acceptedAnswer":{"@type":"Answer","text":"Translations must score at least 75 out of 100 on a rubric inspired by Multidimensional Quality Metrics (MQM), which evaluates meaning, fluency, terminology, style, and formatting using the same Gemma-4-31B-it model that performed the translation. Lower-scoring outputs get feedback-based repair and reassessment against the same threshold, which recovered 438.4 billion additional tokens for the twelve-language workflow. Anyone building a translation QA pipeline can compare rubric-based approaches like this through daily.dev."}},{"@type":"Question","name":"Did adding Indic continued pretraining data actually improve Nemotron 3.5 Super VL's benchmark scores?","acceptedAnswer":{"@type":"Answer","text":"Yes, scores rose on all five reported Indic knowledge and reasoning benchmarks after a continued pretraining stage that allocated 10 billion of a 100 billion token budget to synthetic and natural Indic text. Gains ranged from 4.98 to 9.82 percentage points on MILU, Indic MMLU-Pro, Indic ARC-Challenge, Indic ARC-Easy, and Indic GSM8K (GSM8K excluded Odia due to context limits). Developers weighing continued pretraining mixtures can follow benchmark results like these on daily.dev."}}]} ```