--- title: Polars in Aggregate: Improved I/O, Hugging Face support,... description: Polars 1.0 through 1.2.1 introduce a cloud file cache manager for CSV/IPC/NDJSON scans that persists downloaded data locally, improved Hive partitioning... canonical: https://daily.dev/posts/polars-in-aggregate-improved-i-o-hugging-face-support-and-more-ysbtvpkhz twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Polars in Aggregate: Improved I/O, Hugging Face support, and more | daily.dev og:description: Polars 1.0 through 1.2.1 introduce a cloud file cache manager for CSV/IPC/NDJSON scans that persists downloaded data locally, improved Hive partitioning... og:url: https://daily.dev/posts/polars-in-aggregate-improved-i-o-hugging-face-support-and-more-ysbtvpkhz og:image: https://api.daily.dev/og/posts/YsBtVpKhz.png og:image:alt: Polars in Aggregate: Improved I/O, Hugging Face support, and more og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Polars in Aggregate: Improved I/O, Hugging Face support, and more **[Polars Blog](https://daily.dev/sources/polars)** · 7 min read · 0 upvotes · 0 comments ## Summary Polars 1.0 through 1.2.1 introduce a cloud file cache manager for CSV/IPC/NDJSON scans that persists downloaded data locally, improved Hive partitioning support including datetime partition keys and writing Hive-partitioned Parquet datasets, a reimplemented Struct data type with outer nullability support, native Hugging Face dataset filesystem support for querying datasets directly via hf:// URIs, long-awaited right join support alongside existing join types, and binary view memory mapping for IPC/Arrow files enabled by pyarrow 17.0.0 compatibility. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How do I query a Hugging Face dataset directly with Polars without downloading the whole file? Polars now provides native support for the Hugging Face filesystem, letting you use scan_xxx or read_xxx functions with an hf:// URI, such as pl.read_parquet("hf://datasets/roneneldan/TinyStories/data/train-*.parquet"). This enables predicate pushdown and Polars' native pre-fetcher, avoiding full dataset downloads that were required with the older fsspec-based approach. _daily.dev helps data engineers track library updates like Hugging Face integration in Polars before they hit production._ ### Does Polars support right joins? Yes, Polars added right join support starting with version 1.0, completing the set of standard join types alongside inner, left, full, cross, semi, and anti. The architecture existed earlier but wasn't prioritized until this release; right joins work the same way as other join types via the how parameter in the join method. _Developers choosing a DataFrame library for joins can follow Polars feature updates on daily.dev._ ### Can Polars write Hive partitioned Parquet datasets? Yes, as of Polars 1.0-1.2.1, DataFrames can be written in Hive partitioned format by passing the partition_by argument to write_parquet, for example df.write_parquet("out/", partition_by=["date1", "date2"]). This creates a directory structure with date1=value/date2=value subfolders containing Parquet files, matching the standard Hive layout. _Data engineers building partitioned storage pipelines can track Polars I/O changes on daily.dev._ ## Similar posts on daily.dev - [Release of Polars 2.0](https://daily.dev/posts/release-of-polars-2-0-84cwf427y) · Polars Blog · 3 upvotes · 0 comments - [Announcing Polars 1.44](https://daily.dev/posts/announcing-polars-1-44-nyx9iebfn) · Polars Blog · 1 upvotes · 0 comments - [Polars Built a GPU Hook for NVIDIA. It Works on the Apple Silicon GPU Too](https://daily.dev/posts/polars-built-a-gpu-hook-for-nvidia-it-works-on-the-apple-silicon-gpu-too-n91scp2q0) · Medium · 0 upvotes · 0 comments - [DuckDB and Hugging Face: Querying Datasets Directly](https://daily.dev/posts/duckdb-and-hugging-face-querying-datasets-directly-sge1f136a) · DuckDB · 0 upvotes · 0 comments --- Tags: [#python](https://daily.dev/tags/python), [#big-data](https://daily.dev/tags/big-data), [#data-engineering](https://daily.dev/tags/data-engineering), [#polars](https://daily.dev/tags/polars) [View this post on daily.dev](https://daily.dev/posts/polars-in-aggregate-improved-i-o-hugging-face-support-and-more-ysbtvpkhz) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Polars in Aggregate: Improved I/O, Hugging Face support, and more","url":"https://daily.dev/posts/polars-in-aggregate-improved-i-o-hugging-face-support-and-more-ysbtvpkhz","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/polars-in-aggregate-improved-i-o-hugging-face-support-and-more-ysbtvpkhz"},"datePublished":"2026-08-16T08:53:13.156Z","dateModified":"2026-08-16T08:53:36.816Z","description":"Polars 1.0 through 1.2.1 introduce a cloud file cache manager for CSV/IPC/NDJSON scans that persists downloaded data locally, improved Hive partitioning...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/5434e7bec80a8efdf8c46b2f6f76f432?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/5434e7bec80a8efdf8c46b2f6f76f432?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Polars Blog","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Polars Blog","logo":"https://media.daily.dev/image/upload/s--xo60W3UW--/c_limit,w_256/f_auto,q_auto/v1786870390/logos/polars?_a=BAMAMicg0","url":"https://daily.dev/sources/polars"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/polars-in-aggregate-improved-i-o-hugging-face-support-and-more-ysbtvpkhz","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"python,big-data,data-engineering,polars","timeRequired":"PT7M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Polars Blog","item":"https://daily.dev/sources/polars"},{"@type":"ListItem","position":3,"name":"Polars in Aggregate: Improved I/O, Hugging Face support, and more"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/polars-in-aggregate-improved-i-o-hugging-face-support-and-more-ysbtvpkhz#faq","mainEntity":[{"@type":"Question","name":"How do I query a Hugging Face dataset directly with Polars without downloading the whole file?","acceptedAnswer":{"@type":"Answer","text":"Polars now provides native support for the Hugging Face filesystem, letting you use scan_xxx or read_xxx functions with an hf:// URI, such as pl.read_parquet(\"hf://datasets/roneneldan/TinyStories/data/train-*.parquet\"). This enables predicate pushdown and Polars' native pre-fetcher, avoiding full dataset downloads that were required with the older fsspec-based approach. daily.dev helps data engineers track library updates like Hugging Face integration in Polars before they hit production."}},{"@type":"Question","name":"Does Polars support right joins?","acceptedAnswer":{"@type":"Answer","text":"Yes, Polars added right join support starting with version 1.0, completing the set of standard join types alongside inner, left, full, cross, semi, and anti. The architecture existed earlier but wasn't prioritized until this release; right joins work the same way as other join types via the how parameter in the join method. Developers choosing a DataFrame library for joins can follow Polars feature updates on daily.dev."}},{"@type":"Question","name":"Can Polars write Hive partitioned Parquet datasets?","acceptedAnswer":{"@type":"Answer","text":"Yes, as of Polars 1.0-1.2.1, DataFrames can be written in Hive partitioned format by passing the partition_by argument to write_parquet, for example df.write_parquet(\"out/\", partition_by=[\"date1\", \"date2\"]). This creates a directory structure with date1=value/date2=value subfolders containing Parquet files, matching the standard Hive layout. Data engineers building partitioned storage pipelines can track Polars I/O changes on daily.dev."}}]} ```