<!-- mobian-agent-page publisher="dailydev" canonical="https://daily.dev/posts/multi-cluster-writes-to-delta-lake-storage-in-s3-88pdumzij" -->

---
title: Multi-cluster writes to Delta Lake Storage in S3 | daily.dev
description: Delta Lake 1.2 introduces support for concurrent writes to Amazon S3 from multiple clusters or Spark drivers while preserving ACID transaction guarantees....
canonical: https://daily.dev/posts/multi-cluster-writes-to-delta-lake-storage-in-s3-88pdumzij
twitter:card: summary_large_image
twitter:site: @dailydotdev
og:type: website
og:site_name: daily.dev
og:title: Multi-cluster writes to Delta Lake Storage in S3 | daily.dev
og:description: Delta Lake 1.2 introduces support for concurrent writes to Amazon S3 from multiple clusters or Spark drivers while preserving ACID transaction guarantees....
og:url: https://daily.dev/posts/multi-cluster-writes-to-delta-lake-storage-in-s3-88pdumzij
og:image: https://api.daily.dev/og/posts/88PDUMZIj.png
og:image:alt: Multi-cluster writes to Delta Lake Storage in S3
og:image:width: 1200
og:image:height: 630
og:locale: en
---

> ## Documentation Index
> Fetch the complete documentation index at: https://daily.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Multi-cluster writes to Delta Lake Storage in S3

**[Delta Lake Blog](https://daily.dev/sources/delta-lake)** · 8 min read · 0 upvotes · 0 comments

## Summary

Delta Lake 1.2 introduces support for concurrent writes to Amazon S3 from multiple clusters or Spark drivers while preserving ACID transaction guarantees. Previously, S3's lack of a 'put-if-absent' consistency guarantee limited transactional writes to a single Spark driver, unlike Azure ADLSgen2 or Google GCS. The new S3DynamoDBLogStore uses a DynamoDB table with atomic PutItem operations to achieve mutual exclusion during commits to the Delta transaction log, preventing data loss from concurrent writers. Samba TV, which helped drive this feature through community contributions, now runs dozens of TBs of high-value datasets on this setup with nearly 400K DynamoDB-tracked transactions. The post also covers setup requirements, configuration keys, and recommended practices such as manually provisioning the DynamoDB table.

## Full article

daily.dev links to this article rather than hosting it. Read it at the original source: <https://delta.io/blog/2022-05-18-multi-cluster-writes-to-delta-lake-storage-in-s3>

## Questions this post answers

### Does Delta Lake support multi-cluster writes to S3?

Yes, as of Delta Lake 1.2, multi-cluster and multi-driver writes to S3 are supported with full ACID guarantees. Previously this was not possible because S3 lacks a 'put-if-absent' consistency guarantee, so writes had to originate from a single Spark driver. This limitation did not apply to Azure ADLSgen2 or Google GCS.

_Teams planning a Delta Lake upgrade can track changes like this one on daily.dev._

### How does Delta Lake achieve mutual exclusion for concurrent writes to S3?

Delta Lake's S3DynamoDBLogStore uses a DynamoDB table with atomic PutItem operations to ensure only one writer succeeds per transaction log commit. Writers first prepare a commit by writing roughly 200 bytes of metadata to DynamoDB before writing to the filesystem; since PutItem only succeeds if the entry doesn't already exist, concurrent writers competing for the same version resolve to a single winner.

_Engineers designing multi-cluster storage architectures follow implementation details like this on daily.dev._

### What are the requirements for enabling multi-cluster writes to Delta Lake on S3?

You need S3 credentials (IAM roles recommended), DynamoDB operating permissions, a compatible Apache Spark version, Hadoop's hadoop-aws connector, and the delta-storage-s3-dynamodb JAR. You must also configure the LogStore implementation and use the same DynamoDB table name and region across every Spark session and cluster node to avoid data loss.

_daily.dev helps engineers stay on top of setup requirements before rolling out infrastructure changes._

## Similar posts on daily.dev

- [Building a Database on S3](https://daily.dev/posts/building-a-database-on-s3-z6q1mg0wr) · Metadata · 47 upvotes · 0 comments
- [How S3 Enables Stateless Write-Ahead Logs Despite Latency Trade-offs](https://daily.dev/posts/how-s3-enables-stateless-write-ahead-logs-despite-latency-trade-offs-rywhyjxzx) · Arpit Bhayani · 2 upvotes · 0 comments
- [Delta Lake Explained: Boost Data Reliability in Cloud Storage](https://daily.dev/posts/delta-lake-explained-boost-data-reliability-in-cloud-storage-oechbmvm4) · databricks · 0 upvotes · 0 comments
- [Similarweb’s migration from HBase to Amazon DynamoDB](https://daily.dev/posts/similarweb-s-migration-from-hbase-to-amazon-dynamodb-fwr1gw4k9) · AWS Database Blog · 0 upvotes · 0 comments

---

Tags: [#distributed-systems](https://daily.dev/tags/distributed-systems), [#apache-spark](https://daily.dev/tags/apache-spark), [#aws-s3](https://daily.dev/tags/aws-s3), [#aws-dynamodb](https://daily.dev/tags/aws-dynamodb)

[View this post on daily.dev](https://daily.dev/posts/multi-cluster-writes-to-delta-lake-storage-in-s3-88pdumzij)

```json
{"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]}
{"@context":"https://schema.org","@type":"TechArticle","headline":"Multi-cluster writes to Delta Lake Storage in S3","url":"https://daily.dev/posts/multi-cluster-writes-to-delta-lake-storage-in-s3-88pdumzij","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/multi-cluster-writes-to-delta-lake-storage-in-s3-88pdumzij"},"datePublished":"2026-08-16T08:53:41.898Z","dateModified":"2026-08-16T08:55:55.311Z","description":"Delta Lake 1.2 introduces support for concurrent writes to Amazon S3 from multiple clusters or Spark drivers while preserving ACID transaction guarantees....","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/c71a3413aace2db59edba4f75e9bd6b9?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/c71a3413aace2db59edba4f75e9bd6b9?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Delta Lake Blog","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Delta Lake Blog","logo":"https://media.daily.dev/image/upload/s--7f2q9Asf--/f_auto,q_auto/v1786870393/logos/delta-lake?_a=BAMAMicg0","url":"https://daily.dev/sources/delta-lake"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/multi-cluster-writes-to-delta-lake-storage-in-s3-88pdumzij","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"distributed-systems,apache-spark,aws-s3,aws-dynamodb","timeRequired":"PT8M"}
{"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Delta Lake Blog","item":"https://daily.dev/sources/delta-lake"},{"@type":"ListItem","position":3,"name":"Multi-cluster writes to Delta Lake Storage in S3"}]}
{"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/multi-cluster-writes-to-delta-lake-storage-in-s3-88pdumzij#faq","mainEntity":[{"@type":"Question","name":"Does Delta Lake support multi-cluster writes to S3?","acceptedAnswer":{"@type":"Answer","text":"Yes, as of Delta Lake 1.2, multi-cluster and multi-driver writes to S3 are supported with full ACID guarantees. Previously this was not possible because S3 lacks a 'put-if-absent' consistency guarantee, so writes had to originate from a single Spark driver. This limitation did not apply to Azure ADLSgen2 or Google GCS. Teams planning a Delta Lake upgrade can track changes like this one on daily.dev."}},{"@type":"Question","name":"How does Delta Lake achieve mutual exclusion for concurrent writes to S3?","acceptedAnswer":{"@type":"Answer","text":"Delta Lake's S3DynamoDBLogStore uses a DynamoDB table with atomic PutItem operations to ensure only one writer succeeds per transaction log commit. Writers first prepare a commit by writing roughly 200 bytes of metadata to DynamoDB before writing to the filesystem; since PutItem only succeeds if the entry doesn't already exist, concurrent writers competing for the same version resolve to a single winner. Engineers designing multi-cluster storage architectures follow implementation details like this on daily.dev."}},{"@type":"Question","name":"What are the requirements for enabling multi-cluster writes to Delta Lake on S3?","acceptedAnswer":{"@type":"Answer","text":"You need S3 credentials (IAM roles recommended), DynamoDB operating permissions, a compatible Apache Spark version, Hadoop's hadoop-aws connector, and the delta-storage-s3-dynamodb JAR. You must also configure the LogStore implementation and use the same DynamoDB table name and region across every Spark session and cluster node to avoid data loss. daily.dev helps engineers stay on top of setup requirements before rolling out infrastructure changes."}}]}
```