--- title: Accelerating researchers and developers building... description: GitHub has released the GitHub Multilingual Repositories Dataset, a CC0-1.0 open metadata dataset covering over 80 million classification rows across more than... canonical: https://daily.dev/posts/accelerating-researchers-and-developers-building-multilingual-ai-with-a-new-open-dataset-uftb7vhgm twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Accelerating researchers and developers building multilingual AI with a new open dataset | daily.dev og:description: GitHub has released the GitHub Multilingual Repositories Dataset, a CC0-1.0 open metadata dataset covering over 80 million classification rows across more than... og:url: https://daily.dev/posts/accelerating-researchers-and-developers-building-multilingual-ai-with-a-new-open-dataset-uftb7vhgm og:image: https://api.daily.dev/og/posts/uFtB7vHGm.png og:image:alt: Accelerating researchers and developers building multilingual AI with a new open dataset og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Accelerating researchers and developers building multilingual AI with a new open dataset **[GitHub Blog](https://daily.dev/sources/ghblog)** · 5 min read · 1 upvotes · 0 comments ## Summary GitHub has released the GitHub Multilingual Repositories Dataset, a CC0-1.0 open metadata dataset covering over 80 million classification rows across more than 40 million public repositories. It identifies non-English natural-language content in READMEs, issues, and pull requests using three classifiers (fastText, gcld3, lingua-py) with confidence scores, without collapsing them into a single label. Key findings include Portuguese topping non-English READMEs with 3M+ repositories, while Korean leads in issue text. The dataset is intended for researchers building multilingual AI tools, studying language representation in open source, and creating evaluation sets for AI coding assistants. It explicitly avoids being a ground-truth benchmark and warns against inferring person-level attributes from repository metadata. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Similar posts on daily.dev - [Hugging Face Releases FineTranslations, a Trillion-Token Multilingual Parallel Text Dataset](https://daily.dev/posts/hugging-face-releases-finetranslations-a-trillion-token-multilingual-parallel-text-dataset-ilbhxxjto) · InfoQ · 0 upvotes · 0 comments - [Datasets for Training a Language Model](https://daily.dev/posts/datasets-for-training-a-language-model-prbkjavpx) · Machine Learning Mastery · 4 upvotes · 0 comments --- Tags: [#github](https://daily.dev/tags/github), [#nlp](https://daily.dev/tags/nlp) [View this post on daily.dev](https://daily.dev/posts/accelerating-researchers-and-developers-building-multilingual-ai-with-a-new-open-dataset-uftb7vhgm) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Accelerating researchers and developers building multilingual AI with a new open dataset","url":"https://daily.dev/posts/accelerating-researchers-and-developers-building-multilingual-ai-with-a-new-open-dataset-uftb7vhgm","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/accelerating-researchers-and-developers-building-multilingual-ai-with-a-new-open-dataset-uftb7vhgm"},"datePublished":"2026-06-15T19:21:25.751Z","dateModified":"2026-06-17T19:54:03.649Z","description":"GitHub has released the GitHub Multilingual Repositories Dataset, a CC0-1.0 open metadata dataset covering over 80 million classification rows across more than...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/44efa9dc0cccacc3ab8cf252baa2fad1?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/44efa9dc0cccacc3ab8cf252baa2fad1?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"GitHub Blog","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"GitHub Blog","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/106cf162b88840808484d4b5429b59b1","url":"https://daily.dev/sources/ghblog"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/accelerating-researchers-and-developers-building-multilingual-ai-with-a-new-open-dataset-uftb7vhgm","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"github,nlp","timeRequired":"PT5M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"GitHub Blog","item":"https://daily.dev/sources/ghblog"},{"@type":"ListItem","position":3,"name":"Accelerating researchers and developers building multilingual AI with a new open dataset"}]} ```