--- title: Benchmarking single node vs distributed | daily.dev description: A benchmark comparison pits distributed Polars against single-node Polars using the PDS-H (TPC-H derived) workload at scale factor 1000 on similarly priced AWS... canonical: https://daily.dev/posts/benchmarking-single-node-vs-distributed-jhyke2wzr twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Benchmarking single node vs distributed | daily.dev og:description: A benchmark comparison pits distributed Polars against single-node Polars using the PDS-H (TPC-H derived) workload at scale factor 1000 on similarly priced AWS... og:url: https://daily.dev/posts/benchmarking-single-node-vs-distributed-jhyke2wzr og:image: https://api.daily.dev/og/posts/JHykE2wZr.png og:image:alt: Benchmarking single node vs distributed og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Benchmarking single node vs distributed **[Polars Blog](https://daily.dev/sources/polars)** · 7 min read · 0 upvotes · 0 comments ## Summary A benchmark comparison pits distributed Polars against single-node Polars using the PDS-H (TPC-H derived) workload at scale factor 1000 on similarly priced AWS instances. Overall, distributed Polars edges out single-node (0.85x total time), but the picture is nuanced: single-node wins heavily on join-heavy queries (Q8, Q9) due to shuffle costs, while distributed wins on network I/O-bound queries (Q6, Q14, Q15) because AWS's network credit system favors many small instances over one large one. NUMA effects on the large single-node machine also hurt its performance, since the streaming engine's work-stealing crosses NUMA boundaries and causes memory contention. A secondary test on a smaller single-NUMA-node instance shows single-node winning most queries except I/O-heavy ones. The team plans bloom filter pushdown to reduce shuffle costs and is working on NUMA-aware scheduling. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Is single-node processing always faster and cheaper than distributed processing for large datasets? Not always. Benchmarking distributed Polars against single-node Polars on a terabyte-scale PDS-H workload showed distributed compute running 0.85x the total time of single node overall, though results vary by query. Single node wins on join-heavy queries due to shuffle costs, while distributed wins on network I/O-bound queries because AWS's network credit system favors many small instances over one large one. _Weighing single-node against distributed compute costs? daily.dev surfaces benchmarking data like this for that decision._ ### Why would a distributed cluster of small instances outperform a single large instance on network-bound queries in AWS? AWS network bandwidth does not scale linearly with instance size and instead uses a credit system that favors many small machines. In one comparison, an m8i.32xlarge single node ran at a sustained 50 Gbps with no burst, while 32 m8i.xlarge instances had an aggregate baseline of about 60 Gbps and could burst to 400 Gbps, roughly 8x the single node, though burst credits deplete after about an hour of full use. _Engineers sizing AWS clusters for I/O-bound workloads can track findings like this on daily.dev._ ### Why does NUMA architecture hurt performance on large single-node machines running Polars? On large machines with multiple NUMA nodes, cross-node memory access is expensive, and a work-stealing scheduler that steals tasks across NUMA regions causes memory contention that degrades performance. Testing on an m8i.32xlarge instance, which spans two NUMA nodes, showed the streaming engine's cross-NUMA work stealing led to slower results, while a single-NUMA-node instance avoided this penalty on most queries. _Debugging NUMA-related slowdowns in data engines is easier when you follow deep dives like this on daily.dev._ ## Similar posts on daily.dev - [650GB of Data \(Delta Lake on S3\). Polars vs DuckDB vs Daft vs Spark.](https://daily.dev/posts/650gb-of-data-delta-lake-on-s3-polars-vs-duckdb-vs-daft-vs-spark--bbkmmear9) · Hacker News · 1 upvotes · 0 comments - [Polars vs DuckDB vs Pandas: Benchmark on 33 Million Rows](https://daily.dev/posts/polars-vs-duckdb-vs-pandas-benchmark-on-33-million-rows-p8chasjqs) · Medium · 0 upvotes · 0 comments - [Release of Polars 2.0](https://daily.dev/posts/release-of-polars-2-0-84cwf427y) · Polars Blog · 1 upvotes · 0 comments - [Decathlon Switches to Polars to Optimize Data Pipelines and Infrastructure Costs](https://daily.dev/posts/decathlon-switches-to-polars-to-optimize-data-pipelines-and-infrastructure-costs-5gcd7b1tn) · InfoQ · 70 upvotes · 0 comments --- Tags: [#aws](https://daily.dev/tags/aws), [#data-engineering](https://daily.dev/tags/data-engineering), [#distributed-systems](https://daily.dev/tags/distributed-systems), [#polars](https://daily.dev/tags/polars) [View this post on daily.dev](https://daily.dev/posts/benchmarking-single-node-vs-distributed-jhyke2wzr) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Benchmarking single node vs distributed","url":"https://daily.dev/posts/benchmarking-single-node-vs-distributed-jhyke2wzr","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/benchmarking-single-node-vs-distributed-jhyke2wzr"},"datePublished":"2026-08-16T08:53:16.253Z","dateModified":"2026-08-16T08:54:53.273Z","description":"A benchmark comparison pits distributed Polars against single-node Polars using the PDS-H (TPC-H derived) workload at scale factor 1000 on similarly priced AWS...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/5434e7bec80a8efdf8c46b2f6f76f432?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/5434e7bec80a8efdf8c46b2f6f76f432?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Polars Blog","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Polars Blog","logo":"https://media.daily.dev/image/upload/s--xo60W3UW--/c_limit,w_256/f_auto,q_auto/v1786870390/logos/polars?_a=BAMAMicg0","url":"https://daily.dev/sources/polars"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/benchmarking-single-node-vs-distributed-jhyke2wzr","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"aws,data-engineering,distributed-systems,polars","timeRequired":"PT7M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Polars Blog","item":"https://daily.dev/sources/polars"},{"@type":"ListItem","position":3,"name":"Benchmarking single node vs distributed"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/benchmarking-single-node-vs-distributed-jhyke2wzr#faq","mainEntity":[{"@type":"Question","name":"Is single-node processing always faster and cheaper than distributed processing for large datasets?","acceptedAnswer":{"@type":"Answer","text":"Not always. Benchmarking distributed Polars against single-node Polars on a terabyte-scale PDS-H workload showed distributed compute running 0.85x the total time of single node overall, though results vary by query. Single node wins on join-heavy queries due to shuffle costs, while distributed wins on network I/O-bound queries because AWS's network credit system favors many small instances over one large one. Weighing single-node against distributed compute costs? daily.dev surfaces benchmarking data like this for that decision."}},{"@type":"Question","name":"Why would a distributed cluster of small instances outperform a single large instance on network-bound queries in AWS?","acceptedAnswer":{"@type":"Answer","text":"AWS network bandwidth does not scale linearly with instance size and instead uses a credit system that favors many small machines. In one comparison, an m8i.32xlarge single node ran at a sustained 50 Gbps with no burst, while 32 m8i.xlarge instances had an aggregate baseline of about 60 Gbps and could burst to 400 Gbps, roughly 8x the single node, though burst credits deplete after about an hour of full use. Engineers sizing AWS clusters for I/O-bound workloads can track findings like this on daily.dev."}},{"@type":"Question","name":"Why does NUMA architecture hurt performance on large single-node machines running Polars?","acceptedAnswer":{"@type":"Answer","text":"On large machines with multiple NUMA nodes, cross-node memory access is expensive, and a work-stealing scheduler that steals tasks across NUMA regions causes memory contention that degrades performance. Testing on an m8i.32xlarge instance, which spans two NUMA nodes, showed the streaming engine's cross-NUMA work stealing led to slower results, while a single-NUMA-node instance avoided this penalty on most queries. Debugging NUMA-related slowdowns in data engines is easier when you follow deep dives like this on daily.dev."}}]} ```