--- title: From Spreadsheets to DuckLake: A Data Engineer's Road to... description: A data engineer with nearly two decades of experience describes moving from spreadsheets and Spark clusters to DuckDB and DuckLake for simpler analytics... canonical: https://daily.dev/posts/from-spreadsheets-to-ducklake-a-data-engineer-s-road-to-duckdb-ucbstyyau twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: From Spreadsheets to DuckLake: A Data Engineer's Road to DuckDB | daily.dev og:description: A data engineer with nearly two decades of experience describes moving from spreadsheets and Spark clusters to DuckDB and DuckLake for simpler analytics... og:url: https://daily.dev/posts/from-spreadsheets-to-ducklake-a-data-engineer-s-road-to-duckdb-ucbstyyau og:image: https://api.daily.dev/og/posts/uCbStyYau.png og:image:alt: From Spreadsheets to DuckLake: A Data Engineer's Road to DuckDB og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # From Spreadsheets to DuckLake: A Data Engineer's Road to DuckDB **[DuckDB](https://daily.dev/sources/duckdb)** · 8 min read · 0 upvotes · 0 comments ## Summary A data engineer with nearly two decades of experience describes moving from spreadsheets and Spark clusters to DuckDB and DuckLake for simpler analytics infrastructure. Matt Martin says DuckLake stores metadata in a SQL database, so catalog operations that take seconds with Iceberg resolve in under a millisecond. He highlights three DuckDB v2.0 alpha features: dynamic ATTACH statements that accept environment variables, asynchronous I/O that cut S3 file-read times by 67% to 70% in his benchmarks, and the new Quack remote protocol for connecting DuckDB instances over HTTP. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Why are Iceberg catalog queries slower than DuckLake catalog queries? Iceberg writes several metadata files per transaction, including metadata JSON, manifest lists and manifests, so as data grows a query must resolve thousands of these files before it can even plan. DuckLake instead stores all metadata in a SQL database, so catalog operations become indexed table queries. One data engineer reports catalog operations that take seconds with Iceberg resolving in under a millisecond with DuckLake. _daily.dev readers weighing Iceberg against DuckLake for a lakehouse can track real-world performance comparisons like this one._ ### How much faster is reading files from S3 in DuckDB 2.0 compared to earlier versions? Reading 1,000 and 10,000 files from Amazon S3 took 67% to 70% less time with the DuckDB v2.0 alpha compared to earlier versions, according to benchmarks from a data engineer testing the release. The speedup comes from asynchronous I/O: DuckDB now issues file list and fetch requests in the background instead of sequentially, with no code changes required. _Teams planning a DuckDB upgrade can follow benchmark results like this on daily.dev before migrating S3-heavy workloads._ ### How do I attach an AWS Glue Iceberg catalog in DuckDB using an environment variable instead of hard-coding the account ID? In DuckDB v2.0, the ATTACH statement accepts expressions, so you can call getenv() to pull the account ID, username, password, or connection string from an environment variable at runtime, for example: ATTACH getenv('AWS_ACCOUNT_ID') AS glue_catalog (TYPE iceberg, ENDPOINT_TYPE glue). Prior to v1.5, the ATTACH path had to be a literal and calling getenv raised an error. _Developers scripting Iceberg or Glue connections can keep up with DuckDB syntax changes like this via daily.dev._ ## Similar posts on daily.dev - [DuckLake 1.0: Data Lake Format with SQL Catalog Metadata](https://daily.dev/posts/ducklake-1-0-data-lake-format-with-sql-catalog-metadata-i1k4mw1ne) · InfoQ · 0 upvotes · 0 comments - [DuckLake: The Lakehouse That's Just SQL and Parquet](https://daily.dev/posts/ducklake-the-lakehouse-that-s-just-sql-and-parquet-vyaa3jmht) · Planet Python · 1 upvotes · 0 comments - [DuckLake Architecture Deep Dive](https://daily.dev/posts/ducklake-architecture-deep-dive-yuky5mgag) · MotherDuck · 0 upvotes · 0 comments - [Building a Data Lakehouse with DuckDB and DuckLake](https://daily.dev/posts/building-a-data-lakehouse-with-duckdb-and-ducklake-x05dlsmkc) · Towards Data Science · 1 upvotes · 0 comments --- Tags: [#data-engineering](https://daily.dev/tags/data-engineering), [#apache-spark](https://daily.dev/tags/apache-spark), [#duckdb](https://daily.dev/tags/duckdb), [#apache-iceberg](https://daily.dev/tags/apache-iceberg) [View this post on daily.dev](https://daily.dev/posts/from-spreadsheets-to-ducklake-a-data-engineer-s-road-to-duckdb-ucbstyyau) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"From Spreadsheets to DuckLake: A Data Engineer's Road to DuckDB","url":"https://daily.dev/posts/from-spreadsheets-to-ducklake-a-data-engineer-s-road-to-duckdb-ucbstyyau","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/from-spreadsheets-to-ducklake-a-data-engineer-s-road-to-duckdb-ucbstyyau"},"datePublished":"2026-10-29T11:56:56.925Z","dateModified":"2026-10-29T11:57:28.581Z","description":"A data engineer with nearly two decades of experience describes moving from spreadsheets and Spark clusters to DuckDB and DuckLake for simpler analytics...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/4f76bda54715fd96aa2b9e27997d2032?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/4f76bda54715fd96aa2b9e27997d2032?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"DuckDB","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"DuckDB","logo":"https://media.daily.dev/image/upload/s--3luD3Wei--/f_auto/v1714800008/logos/duckdb","url":"https://daily.dev/sources/duckdb"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/from-spreadsheets-to-ducklake-a-data-engineer-s-road-to-duckdb-ucbstyyau","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"data-engineering,apache-spark,duckdb,apache-iceberg","timeRequired":"PT8M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"DuckDB","item":"https://daily.dev/sources/duckdb"},{"@type":"ListItem","position":3,"name":"From Spreadsheets to DuckLake: A Data Engineer's Road to DuckDB"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/from-spreadsheets-to-ducklake-a-data-engineer-s-road-to-duckdb-ucbstyyau#faq","mainEntity":[{"@type":"Question","name":"Why are Iceberg catalog queries slower than DuckLake catalog queries?","acceptedAnswer":{"@type":"Answer","text":"Iceberg writes several metadata files per transaction, including metadata JSON, manifest lists and manifests, so as data grows a query must resolve thousands of these files before it can even plan. DuckLake instead stores all metadata in a SQL database, so catalog operations become indexed table queries. One data engineer reports catalog operations that take seconds with Iceberg resolving in under a millisecond with DuckLake. daily.dev readers weighing Iceberg against DuckLake for a lakehouse can track real-world performance comparisons like this one."}},{"@type":"Question","name":"How much faster is reading files from S3 in DuckDB 2.0 compared to earlier versions?","acceptedAnswer":{"@type":"Answer","text":"Reading 1,000 and 10,000 files from Amazon S3 took 67% to 70% less time with the DuckDB v2.0 alpha compared to earlier versions, according to benchmarks from a data engineer testing the release. The speedup comes from asynchronous I/O: DuckDB now issues file list and fetch requests in the background instead of sequentially, with no code changes required. Teams planning a DuckDB upgrade can follow benchmark results like this on daily.dev before migrating S3-heavy workloads."}},{"@type":"Question","name":"How do I attach an AWS Glue Iceberg catalog in DuckDB using an environment variable instead of hard-coding the account ID?","acceptedAnswer":{"@type":"Answer","text":"In DuckDB v2.0, the ATTACH statement accepts expressions, so you can call getenv() to pull the account ID, username, password, or connection string from an environment variable at runtime, for example: ATTACH getenv('AWS_ACCOUNT_ID') AS glue_catalog (TYPE iceberg, ENDPOINT_TYPE glue). Prior to v1.5, the ATTACH path had to be a literal and calling getenv raised an error. Developers scripting Iceberg or Glue connections can keep up with DuckDB syntax changes like this via daily.dev."}}]} ```