--- title: Hugging Face x Polars | daily.dev description: Polars now natively reads datasets directly from Hugging Face Hub using hf:// URIs, applying predicate and projection pushdown to speed up downloads and... canonical: https://daily.dev/posts/hugging-face-x-polars-mmi5wl80l twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Hugging Face x Polars | daily.dev og:description: Polars now natively reads datasets directly from Hugging Face Hub using hf:// URIs, applying predicate and projection pushdown to speed up downloads and... og:url: https://daily.dev/posts/hugging-face-x-polars-mmi5wl80l og:image: https://api.daily.dev/og/posts/mmI5WL80l.png og:image:alt: Hugging Face x Polars og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Hugging Face x Polars **[Polars Blog](https://daily.dev/sources/polars)** · 4 min read · 0 upvotes · 0 comments ## Summary Polars now natively reads datasets directly from Hugging Face Hub using hf:// URIs, applying predicate and projection pushdown to speed up downloads and processing. It supports Parquet, CSV, and JSON/NDJSON formats via both lazy (scan_xxx) and eager (read_xxx) APIs, glob patterns for querying multiple files, auto-converted Parquet access via the @~parquet branch, and authentication via environment variable, parameter, or the Hugging Face CLI for private or gated datasets. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How do I read a Hugging Face dataset directly into Polars? Use a Hugging Face URI in the form hf://datasets/repository/path with Polars' read_csv, read_parquet, or scan_ functions. For example, pl.read_csv("hf://datasets/commoncrawl/statistics/tlds.csv") loads the file directly. Polars natively supports Parquet, CSV, and JSON/NDJSON formats and applies predicate and projection pushdown to speed up downloads. _daily.dev surfaces practical integration guides like this for teams wiring up ML data pipelines._ ### How do I authenticate to a private or gated Hugging Face dataset when using Polars? Provide a Hugging Face access token in one of three ways: set the HF_TOKEN environment variable, pass storage_options={"token": ACCESS_TOKEN} directly to functions like pl.read_parquet, or run huggingface-cli login so Polars picks up the cached token from the HF_HOME directory (default ~/.cache/huggingface). _Developers securing dataset access can track integration details like this via daily.dev._ ### How can I query only part of a large Hugging Face Parquet dataset without downloading it all? Use Polars' lazy API, such as pl.scan_csv or scan_parquet, with the hf:// URI, then chain group_by, agg, sort, and collect. Because Polars applies predicate and projection pushdown, it can skip irrelevant data regions and avoid transferring or loading the entire dataset into memory, saving significant time on large Parquet files. _Track efficient data-loading techniques like this on daily.dev when optimizing ML data pipelines._ ## Similar posts on daily.dev - [DuckDB and Hugging Face: Querying Datasets Directly](https://daily.dev/posts/duckdb-and-hugging-face-querying-datasets-directly-sge1f136a) · DuckDB · 0 upvotes · 0 comments - [open-index/hacker-news · Datasets at Hugging Face](https://daily.dev/posts/open-index-hacker-news-datasets-at-hugging-face-dudbdksae) · Hacker News · 0 upvotes · 0 comments - [Introducing swift-huggingface: The Complete Swift Client for Hugging Face](https://daily.dev/posts/introducing-swift-huggingface-the-complete-swift-client-for-hugging-face-73sbminvc) · Hugging Face · 0 upvotes · 0 comments - [Hugging Face for Snowflake Data Engineers: A Comprehensive Guide](https://daily.dev/posts/hugging-face-for-snowflake-data-engineers-a-comprehensive-guide-ptrfez54x) · Infostrux · 1 upvotes · 0 comments --- Tags: [#machine-learning](https://daily.dev/tags/machine-learning), [#python](https://daily.dev/tags/python), [#data-engineering](https://daily.dev/tags/data-engineering), [#polars](https://daily.dev/tags/polars) [View this post on daily.dev](https://daily.dev/posts/hugging-face-x-polars-mmi5wl80l) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Hugging Face x Polars","url":"https://daily.dev/posts/hugging-face-x-polars-mmi5wl80l","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/hugging-face-x-polars-mmi5wl80l"},"datePublished":"2026-08-16T08:53:19.019Z","dateModified":"2026-08-16T08:55:31.851Z","description":"Polars now natively reads datasets directly from Hugging Face Hub using hf:// URIs, applying predicate and projection pushdown to speed up downloads and...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/5434e7bec80a8efdf8c46b2f6f76f432?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/5434e7bec80a8efdf8c46b2f6f76f432?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Polars Blog","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Polars Blog","logo":"https://media.daily.dev/image/upload/s--xo60W3UW--/c_limit,w_256/f_auto,q_auto/v1786870390/logos/polars?_a=BAMAMicg0","url":"https://daily.dev/sources/polars"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/hugging-face-x-polars-mmi5wl80l","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"machine-learning,python,data-engineering,polars","timeRequired":"PT4M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Polars Blog","item":"https://daily.dev/sources/polars"},{"@type":"ListItem","position":3,"name":"Hugging Face x Polars"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/hugging-face-x-polars-mmi5wl80l#faq","mainEntity":[{"@type":"Question","name":"How do I read a Hugging Face dataset directly into Polars?","acceptedAnswer":{"@type":"Answer","text":"Use a Hugging Face URI in the form hf://datasets/repository/path with Polars' read_csv, read_parquet, or scan_ functions. For example, pl.read_csv(\"hf://datasets/commoncrawl/statistics/tlds.csv\") loads the file directly. Polars natively supports Parquet, CSV, and JSON/NDJSON formats and applies predicate and projection pushdown to speed up downloads. daily.dev surfaces practical integration guides like this for teams wiring up ML data pipelines."}},{"@type":"Question","name":"How do I authenticate to a private or gated Hugging Face dataset when using Polars?","acceptedAnswer":{"@type":"Answer","text":"Provide a Hugging Face access token in one of three ways: set the HF_TOKEN environment variable, pass storage_options={\"token\": ACCESS_TOKEN} directly to functions like pl.read_parquet, or run huggingface-cli login so Polars picks up the cached token from the HF_HOME directory (default ~/.cache/huggingface). Developers securing dataset access can track integration details like this via daily.dev."}},{"@type":"Question","name":"How can I query only part of a large Hugging Face Parquet dataset without downloading it all?","acceptedAnswer":{"@type":"Answer","text":"Use Polars' lazy API, such as pl.scan_csv or scan_parquet, with the hf:// URI, then chain group_by, agg, sort, and collect. Because Polars applies predicate and projection pushdown, it can skip irrelevant data regions and avoid transferring or loading the entire dataset into memory, saving significant time on large Parquet files. Track efficient data-loading techniques like this on daily.dev when optimizing ML data pipelines."}}]} ```