--- title: Prototype on a laptop, scale to 16 billion rows: one... description: A walkthrough of building a data pipeline with Polars that scales from a local three-hour subset of Polymarket orderbook data to the full month of 16 billion... canonical: https://daily.dev/posts/prototype-on-a-laptop-scale-to-16-billion-rows-one-polars-query-i2c7c0vkv twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Prototype on a laptop, scale to 16 billion rows: one Polars query | daily.dev og:description: A walkthrough of building a data pipeline with Polars that scales from a local three-hour subset of Polymarket orderbook data to the full month of 16 billion... og:url: https://daily.dev/posts/prototype-on-a-laptop-scale-to-16-billion-rows-one-polars-query-i2c7c0vkv og:image: https://api.daily.dev/og/posts/I2C7c0vKv.png og:image:alt: Prototype on a laptop, scale to 16 billion rows: one Polars query og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Prototype on a laptop, scale to 16 billion rows: one Polars query **[Polars Blog](https://daily.dev/sources/polars)** · 13 min read · 0 upvotes · 0 comments ## Summary A walkthrough of building a data pipeline with Polars that scales from a local three-hour subset of Polymarket orderbook data to the full month of 16 billion rows on Polars Cloud, using the exact same LazyFrame query. The pipeline decodes JSON payloads, aggregates into small Parquet artifacts (market profiles, activity pulses, price charts), and feeds a Plotly Dash dashboard. Key techniques include streaming engine affinity, dt.truncate with group_by instead of group_by_dynamic for partitionable aggregations, sorting Parquet files for row-group skipping, and swapping local .collect()/.sink_parquet() calls for .remote(ctx) to run distributed on a cluster without rewriting logic. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How can I write a data pipeline that works the same on my laptop and on a distributed cluster with Polars? Build one LazyFrame and change only the execution call: use .collect() or .sink_parquet() locally, and .remote(ctx).sink_parquet() to run the identical query distributed across a cluster via Polars Cloud. Because the query plan is unchanged, a subset prototyped on a laptop with a few hours of data scales unmodified to a full dataset of 16 billion rows, eliminating a second implementation that would otherwise drift from the original. _See how daily.dev surfaces workflows for teams avoiding duplicate pipeline logic across scales._ ### Why should I use dt.truncate with group_by instead of group_by_dynamic for time-bucketed aggregations in Polars? group_by_dynamic requires the index column to be sorted, which forces an expensive global sort on unsorted data and is hard to distribute. Truncating a timestamp to a bucket with dt.truncate and then using a plain group_by produces the same result for non-overlapping tumbling windows and is fully partitionable, since each worker can aggregate its own chunk independently, though empty buckets are omitted rather than shown as zero rows. _Developers optimizing Polars aggregations can track these patterns on daily.dev._ ### How does sorting a Parquet file by a column improve query performance when filtering on that column? Parquet files store row-group statistics such as min and max values per column, and a reader checks those statistics to skip row groups whose range cannot contain the filter value before reading any data. Sorting a file by the filter column, such as market_id, before writing concentrates each value's rows into fewer row groups, which maximizes how many row groups get skipped entirely. _Anyone tuning Parquet-based query performance can keep such tips handy on daily.dev._ ## Similar posts on daily.dev - [Crunching data? Put that laptop hardware to use](https://daily.dev/posts/crunching-data-put-that-laptop-hardware-to-use-1fcnht8lv) · Simple Thread · 0 upvotes · 0 comments - [Polars vs DuckDB vs Pandas: Benchmark on 33 Million Rows](https://daily.dev/posts/polars-vs-duckdb-vs-pandas-benchmark-on-33-million-rows-p8chasjqs) · Medium · 0 upvotes · 0 comments - [650GB of Data \(Delta Lake on S3\). Polars vs DuckDB vs Daft vs Spark.](https://daily.dev/posts/650gb-of-data-delta-lake-on-s3-polars-vs-duckdb-vs-daft-vs-spark--bbkmmear9) · Hacker News · 1 upvotes · 0 comments - [I Rewrote a Real Data Workflow in Polars. Pandas Didn’t Stand a Chance.](https://daily.dev/posts/i-rewrote-a-real-data-workflow-in-polars-pandas-didn-t-stand-a-chance--dpixpj7fx) · Towards Data Science · 0 upvotes · 0 comments - [Decathlon Switches to Polars to Optimize Data Pipelines and Infrastructure Costs](https://daily.dev/posts/decathlon-switches-to-polars-to-optimize-data-pipelines-and-infrastructure-costs-5gcd7b1tn) · InfoQ · 70 upvotes · 0 comments --- Tags: [#data-engineering](https://daily.dev/tags/data-engineering), [#distributed-systems](https://daily.dev/tags/distributed-systems), [#polars](https://daily.dev/tags/polars) [View this post on daily.dev](https://daily.dev/posts/prototype-on-a-laptop-scale-to-16-billion-rows-one-polars-query-i2c7c0vkv) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Prototype on a laptop, scale to 16 billion rows: one Polars query","url":"https://daily.dev/posts/prototype-on-a-laptop-scale-to-16-billion-rows-one-polars-query-i2c7c0vkv","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/prototype-on-a-laptop-scale-to-16-billion-rows-one-polars-query-i2c7c0vkv"},"datePublished":"2026-08-16T08:53:15.668Z","dateModified":"2026-08-16T08:55:22.413Z","description":"A walkthrough of building a data pipeline with Polars that scales from a local three-hour subset of Polymarket orderbook data to the full month of 16 billion...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/5ab1d61be8a87751afb98bff4d2e461e?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/5ab1d61be8a87751afb98bff4d2e461e?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Polars Blog","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Polars Blog","logo":"https://media.daily.dev/image/upload/s--xo60W3UW--/c_limit,w_256/f_auto,q_auto/v1786870390/logos/polars?_a=BAMAMicg0","url":"https://daily.dev/sources/polars"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/prototype-on-a-laptop-scale-to-16-billion-rows-one-polars-query-i2c7c0vkv","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"data-engineering,distributed-systems,polars","timeRequired":"PT13M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Polars Blog","item":"https://daily.dev/sources/polars"},{"@type":"ListItem","position":3,"name":"Prototype on a laptop, scale to 16 billion rows: one Polars query"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/prototype-on-a-laptop-scale-to-16-billion-rows-one-polars-query-i2c7c0vkv#faq","mainEntity":[{"@type":"Question","name":"How can I write a data pipeline that works the same on my laptop and on a distributed cluster with Polars?","acceptedAnswer":{"@type":"Answer","text":"Build one LazyFrame and change only the execution call: use .collect() or .sink_parquet() locally, and .remote(ctx).sink_parquet() to run the identical query distributed across a cluster via Polars Cloud. Because the query plan is unchanged, a subset prototyped on a laptop with a few hours of data scales unmodified to a full dataset of 16 billion rows, eliminating a second implementation that would otherwise drift from the original. See how daily.dev surfaces workflows for teams avoiding duplicate pipeline logic across scales."}},{"@type":"Question","name":"Why should I use dt.truncate with group_by instead of group_by_dynamic for time-bucketed aggregations in Polars?","acceptedAnswer":{"@type":"Answer","text":"group_by_dynamic requires the index column to be sorted, which forces an expensive global sort on unsorted data and is hard to distribute. Truncating a timestamp to a bucket with dt.truncate and then using a plain group_by produces the same result for non-overlapping tumbling windows and is fully partitionable, since each worker can aggregate its own chunk independently, though empty buckets are omitted rather than shown as zero rows. Developers optimizing Polars aggregations can track these patterns on daily.dev."}},{"@type":"Question","name":"How does sorting a Parquet file by a column improve query performance when filtering on that column?","acceptedAnswer":{"@type":"Answer","text":"Parquet files store row-group statistics such as min and max values per column, and a reader checks those statistics to skip row groups whose range cannot contain the filter value before reading any data. Sorting a file by the filter column, such as market_id, before writing concentrates each value's rows into fewer row groups, which maximizes how many row groups get skipped entirely. Anyone tuning Parquet-based query performance can keep such tips handy on daily.dev."}}]} ```