--- title: 650GB of Data (Delta Lake on S3). Polars vs DuckDB vs... description: A performance comparison testing DuckDB, Polars, Daft, and Spark on a 650GB Delta Lake dataset stored in S3, using a single 32GB EC2 instance. DuckDB completed... canonical: https://daily.dev/posts/dn6tb3icm twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: 650GB of Data (Delta Lake on S3). Polars vs DuckDB vs Daft vs Spark. | daily.dev og:description: A performance comparison testing DuckDB, Polars, Daft, and Spark on a 650GB Delta Lake dataset stored in S3, using a single 32GB EC2 instance. DuckDB completed... og:url: https://daily.dev/posts/dn6tb3icm og:image: https://api.daily.dev/og/posts/DN6tB3icm.png og:image:alt: Post cover image og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # 650GB of Data (Delta Lake on S3). Polars vs DuckDB vs Daft vs Spark. **[DuckDB dev](https://daily.dev/sources/duckdbdev)** · [@mehdio](https://daily.dev/mehdio) · 2 upvotes · 0 comments ## Summary A performance comparison testing DuckDB, Polars, Daft, and Spark on a 650GB Delta Lake dataset stored in S3, using a single 32GB EC2 instance. DuckDB completed the aggregation query in 16 minutes, Polars in 12 minutes, Daft in 50 minutes, and PySpark in over an hour. The experiment demonstrates that single-node data processing frameworks can effectively handle large lakehouse datasets without requiring expensive distributed clusters, challenging the assumption that distributed computing is necessary for most data workloads. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Similar posts on daily.dev - [Polars vs DuckDB vs Pandas: Benchmark on 33 Million Rows](https://daily.dev/posts/polars-vs-duckdb-vs-pandas-benchmark-on-33-million-rows-p8chasjqs) · Medium · 0 upvotes · 0 comments - [Pandas Should Go Extinct](https://daily.dev/posts/pandas-should-go-extinct-1jqzeauiq) · Lobsters · 0 upvotes · 0 comments - [Crunching data? Put that laptop hardware to use](https://daily.dev/posts/crunching-data-put-that-laptop-hardware-to-use-1fcnht8lv) · Simple Thread · 0 upvotes · 0 comments - [From Spreadsheets to DuckLake: A Data Engineer's Road to DuckDB](https://daily.dev/posts/from-spreadsheets-to-ducklake-a-data-engineer-s-road-to-duckdb-ucbstyyau) · DuckDB · 0 upvotes · 0 comments - [Modern DataFrames in Python: A Hands-On Tutorial with Polars and DuckDB](https://daily.dev/posts/modern-dataframes-in-python-a-hands-on-tutorial-with-polars-and-duckdb-ysa2sfbov) · Towards Data Science · 0 upvotes · 0 comments --- Tags: [#data-engineering](https://daily.dev/tags/data-engineering), [#duckdb](https://daily.dev/tags/duckdb), [#polars](https://daily.dev/tags/polars), [#spark](https://daily.dev/tags/spark) [View this post on daily.dev](https://daily.dev/posts/dn6tb3icm) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"DiscussionForumPosting","mainEntityOfPage":"https://daily.dev/posts/dn6tb3icm","headline":"650GB of Data (Delta Lake on S3). Polars vs DuckDB vs Daft vs Spark.","text":"Shared: 650GB of Data (Delta Lake on S3). Polars vs DuckDB vs Daft vs Spark.","url":"https://daily.dev/posts/dn6tb3icm","datePublished":"2025-11-30T10:31:01.465Z","dateModified":"2026-10-14T02:23:33.747Z","author":{"@type":"Person","name":"mehdio","url":"https://daily.dev/mehdio","image":"https://avatars.githubusercontent.com/u/19834862?v=4","interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"EndorseAction"},"userInteractionCount":560}},"image":"https://media.daily.dev/image/upload/s--VDukGCjf--/f_auto/v1722860399/public/Placeholder%2002","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":2},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"sharedContent":{"@type":"WebPage","url":"https://api.daily.dev/r/BbKMmEAR9"},"isPartOf":{"@type":"WebPage","url":"https://daily.dev/squads/duckdbdev","name":"DuckDB dev"}} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"DuckDB dev","item":"https://daily.dev/squads/duckdbdev"},{"@type":"ListItem","position":3,"name":"650GB of Data (Delta Lake on S3). Polars vs DuckDB vs Daft vs Spark."}]} ```