<!-- mobian-agent-page publisher="dailydev" canonical="https://daily.dev/posts/delta-lake-on-s3-qwu90riim" -->

---
title: Delta Lake on S3 | daily.dev
description: A practical guide to using Delta Lake with AWS S3 object storage, covering why Delta Lake avoids costly S3 file-listing operations via its transaction log, and...
canonical: https://daily.dev/posts/delta-lake-on-s3-qwu90riim
twitter:card: summary_large_image
twitter:site: @dailydotdev
og:type: website
og:site_name: daily.dev
og:title: Delta Lake on S3 | daily.dev
og:description: A practical guide to using Delta Lake with AWS S3 object storage, covering why Delta Lake avoids costly S3 file-listing operations via its transaction log, and...
og:url: https://daily.dev/posts/delta-lake-on-s3-qwu90riim
og:image: https://api.daily.dev/og/posts/qWU90rIim.png
og:image:alt: Delta Lake on S3
og:image:width: 1200
og:image:height: 630
og:locale: en
---

> ## Documentation Index
> Fetch the complete documentation index at: https://daily.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Delta Lake on S3

**[Delta Lake Blog](https://daily.dev/sources/delta-lake)** · 9 min read · 0 upvotes · 0 comments

## Summary

A practical guide to using Delta Lake with AWS S3 object storage, covering why Delta Lake avoids costly S3 file-listing operations via its transaction log, and how to configure Spark (delta-spark) and Python engines like Polars (delta-rs) for S3 access. Includes required JAR dependencies for Spark, credential configuration for delta-rs, safe concurrent write setups using DynamoDB as a locking provider, and required S3/DynamoDB permissions.

## Full article

daily.dev links to this article rather than hosting it. Read it at the original source: <https://delta.io/blog/delta-lake-s3>

## Questions this post answers

### What JAR dependencies do I need to add to my Spark session to read or write Delta Lake tables on S3?

You need three JARs: hadoop-aws, hadoop-common, and aws-java-sdk-bundle. Match the hadoop-aws and hadoop-common versions to your Spark version (e.g., Spark 3.5.0 pairs with hadoop-aws/common 3.3.4), and check the compatible aws-java-sdk-bundle version on Maven repository. Without these, Spark errors out because it cannot resolve s3a references.

_daily.dev surfaces practical setup guides for developers wiring Spark to cloud object storage like S3._

### Does delta-rs support reading AWS credentials from my local .aws/config file like boto3 does?

No, the delta-rs writer does not use boto3 and therefore cannot read credentials from a local .aws/config or .aws/creds file automatically. Instead, credentials must be passed explicitly through the storage_options kwarg, environment variables, EC2 instance metadata, or AWS profiles when writing Delta tables with engines like Polars, pandas, or Dask.

_developers switching Python data engines track credential quirks like this one on daily.dev._

### Why does S3 need a locking provider like DynamoDB for concurrent writes to Delta Lake tables?

S3 does not guarantee mutual exclusion, meaning there is no built-in way to ensure only one writer creates a given file, which risks data corruption during concurrent writes. Both delta-spark and delta-rs use a DynamoDB table as a locking provider to guarantee safe concurrent writes; this can be disabled via AWS_S3_ALLOW_UNSAFE_RENAME if unsafe writes are acceptable.

_teams designing safe multi-writer data lakes follow storage architecture deep dives like this on daily.dev._

## Similar posts on daily.dev

- [Expensive Delta Lake S3 Storage Mistakes \(And How to Fix Them\)](https://daily.dev/posts/expensive-delta-lake-s3-storage-mistakes-and-how-to-fix-them--iyqaxbwgz) · databricks · 3 upvotes · 0 comments
- [650GB of Data \(Delta Lake on S3\). Polars vs DuckDB vs Daft vs Spark.](https://daily.dev/posts/650gb-of-data-delta-lake-on-s3-polars-vs-duckdb-vs-daft-vs-spark--bbkmmear9) · Hacker News · 1 upvotes · 0 comments
- [Delta Lake Explained: Boost Data Reliability in Cloud Storage](https://daily.dev/posts/delta-lake-explained-boost-data-reliability-in-cloud-storage-oechbmvm4) · databricks · 0 upvotes · 0 comments
- [Building a Database on S3](https://daily.dev/posts/building-a-database-on-s3-z6q1mg0wr) · Metadata · 47 upvotes · 0 comments

---

Tags: [#data-engineering](https://daily.dev/tags/data-engineering), [#apache-spark](https://daily.dev/tags/apache-spark), [#aws-s3](https://daily.dev/tags/aws-s3), [#aws-dynamodb](https://daily.dev/tags/aws-dynamodb)

[View this post on daily.dev](https://daily.dev/posts/delta-lake-on-s3-qwu90riim)

```json
{"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]}
{"@context":"https://schema.org","@type":"TechArticle","headline":"Delta Lake on S3","url":"https://daily.dev/posts/delta-lake-on-s3-qwu90riim","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/delta-lake-on-s3-qwu90riim"},"datePublished":"2026-08-16T08:53:26.224Z","dateModified":"2026-08-16T08:55:25.435Z","description":"A practical guide to using Delta Lake with AWS S3 object storage, covering why Delta Lake avoids costly S3 file-listing operations via its transaction log, and...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/2d4c8d39ebcb0c650029e1fd93031e47?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/2d4c8d39ebcb0c650029e1fd93031e47?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Delta Lake Blog","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Delta Lake Blog","logo":"https://media.daily.dev/image/upload/s--7f2q9Asf--/f_auto,q_auto/v1786870393/logos/delta-lake?_a=BAMAMicg0","url":"https://daily.dev/sources/delta-lake"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/delta-lake-on-s3-qwu90riim","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"data-engineering,apache-spark,aws-s3,aws-dynamodb","timeRequired":"PT9M"}
{"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Delta Lake Blog","item":"https://daily.dev/sources/delta-lake"},{"@type":"ListItem","position":3,"name":"Delta Lake on S3"}]}
{"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/delta-lake-on-s3-qwu90riim#faq","mainEntity":[{"@type":"Question","name":"What JAR dependencies do I need to add to my Spark session to read or write Delta Lake tables on S3?","acceptedAnswer":{"@type":"Answer","text":"You need three JARs: hadoop-aws, hadoop-common, and aws-java-sdk-bundle. Match the hadoop-aws and hadoop-common versions to your Spark version (e.g., Spark 3.5.0 pairs with hadoop-aws/common 3.3.4), and check the compatible aws-java-sdk-bundle version on Maven repository. Without these, Spark errors out because it cannot resolve s3a references. daily.dev surfaces practical setup guides for developers wiring Spark to cloud object storage like S3."}},{"@type":"Question","name":"Does delta-rs support reading AWS credentials from my local .aws/config file like boto3 does?","acceptedAnswer":{"@type":"Answer","text":"No, the delta-rs writer does not use boto3 and therefore cannot read credentials from a local .aws/config or .aws/creds file automatically. Instead, credentials must be passed explicitly through the storage_options kwarg, environment variables, EC2 instance metadata, or AWS profiles when writing Delta tables with engines like Polars, pandas, or Dask. developers switching Python data engines track credential quirks like this one on daily.dev."}},{"@type":"Question","name":"Why does S3 need a locking provider like DynamoDB for concurrent writes to Delta Lake tables?","acceptedAnswer":{"@type":"Answer","text":"S3 does not guarantee mutual exclusion, meaning there is no built-in way to ensure only one writer creates a given file, which risks data corruption during concurrent writes. Both delta-spark and delta-rs use a DynamoDB table as a locking provider to guarantee safe concurrent writes; this can be disabled via AWS_S3_ALLOW_UNSAFE_RENAME if unsafe writes are acceptable. teams designing safe multi-writer data lakes follow storage architecture deep dives like this on daily.dev."}}]}
```