--- title: Every fast write moves work somewhere else | daily.dev description: A deep technical exploration of how database write durability works across different storage layers. Covers the tradeoffs between writing to memory, local NVMe... canonical: https://daily.dev/posts/every-fast-write-moves-work-somewhere-else-k4xugsl0j twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Every fast write moves work somewhere else | daily.dev og:description: A deep technical exploration of how database write durability works across different storage layers. Covers the tradeoffs between writing to memory, local NVMe... og:url: https://daily.dev/posts/every-fast-write-moves-work-somewhere-else-k4xugsl0j og:image: https://api.daily.dev/og/posts/k4XUgSl0j.png og:image:alt: Every fast write moves work somewhere else og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Every fast write moves work somewhere else **[Hacker News](https://daily.dev/sources/hn)** · 18 min read · 0 upvotes · 0 comments ## Summary A deep technical exploration of how database write durability works across different storage layers. Covers the tradeoffs between writing to memory, local NVMe SSDs, object storage (like S3), and replicated WALs using Raft. Explains how Big O notation hides the real latency differences between these paths, how WAL batching improves throughput, how conditional writes on S3 handle concurrent updates, and how cleanup work (compaction, GC, replication repair) is deferred but must be bounded. Includes concrete latency numbers from AWS S3 Express One Zone and Turso benchmarks. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### What failures does fdatasync() to a local NVMe SSD protect against vs. what it doesn't? fdatasync() to a local NVMe SSD protects against process crashes and kernel crashes, but not against loss of the SSD itself or the host machine. If the host disappears before WAL records are uploaded to remote storage, a replacement can only recover from the last uploaded byte. The syscall name is the same whether the device is local or a remote network volume, but the failures survived are different. _Engineers building storage systems track durability boundaries like these on daily.dev._ ### How does WAL batch flushing work and what are the tradeoffs? A WAL writer collects appended records into a batch, closes it after a fixed number of bytes, writes, or elapsed time, then calls fdatasync() once and returns success to every writer covered by that batch position. This lets multiple client PUTs share one expensive device flush, improving throughput. The tradeoff: the earliest request in a batch waits longest, a flush error fails the entire batch, and a timed-out request may have bytes already committed to disk. _Developers designing write-heavy storage paths find discussions of batching strategies like this on daily.dev._ ### What is the data loss risk of using a local WAL in front of object storage like S3? When a local WAL is used to buffer writes before uploading to object storage, every acknowledged write that hasn't yet been uploaded is at risk if the local SSD or host is lost. The age of the oldest unuploaded WAL record determines the maximum data loss window. At 1,000 writes/second, writing one object per write produces roughly 2.6 billion S3 PUT requests per month; batching 100 writes per object reduces that to about 26 million, but increases bytes resent on failure. _Teams choosing between local WAL buffering and synchronous object PUT for their storage layer follow these tradeoffs on daily.dev._ ## Community take How the wider developer community reacted, aggregated from 1 discussion and 24 comments across hackernews (as of 2026-08-09). **TL;DR:** Commenters broadly appreciate the post's concrete framing of a well-known engineering principle, with many sharing their own names for the same idea (waterbed theory, lump under the carpet, conservation of effort). A few add technical nuances or gently note the concept isn't new. **Sentiment:** 65% positive · 25% mixed · 10% skeptical **The case for** - The post gives a concrete, tactile illustration of a principle that is easy to feel but hard to articulate. - Commenters found the specific database/storage context a useful grounding for a broadly applicable idea. - The discussion surfaced the formal name 'waterbed theory,' which several people said they'll adopt going forward. **The pushback** - Some feel the core idea is just standard engineering tradeoffs dressed up in new language. - The post's framing may give the impression that moving work is always neutral, when sometimes relocating a problem to a better-suited layer is a genuine win. - The technical angles around client-side durability and ack loss are noted as gaps the post doesn't address. **By community** - hackernews (positive): Mostly enthusiastic, with commenters trading names for the same phenomenon and adding their own examples, though a few push back that it's just basic tradeoff thinking. **Hottest debate:** Whether 'moving the problem elsewhere' is always a zero-sum shuffle or can represent a genuine architectural improvement when the destination is better suited to the task. **Open questions** - How should clients handle durability when an acknowledgment is lost in transit over HTTP? - Where exactly should backpressure be placed when write acknowledgment speed outpaces actual persistence speed? **Highlights** > Today I learned there’s a name for the general version of this idea. https://en.wikipedia.org/wiki/Waterbed_theory At a certain point in a solution everything you do to optimize (“push”) in one area will cause a negative effect in a different area (“bulge”). But this is a nice concrete example. > — [aleksiy123 on hackernews · 7 comments](https://news.ycombinator.com/item?id=49233631) > 2 angles I think DB designers don’t often think about: 1. Durability extends to the client. Replicated db might ack a write to client, but what if that ack gets lost on the way back over network? If client talks to the DB over simple HTTP, the write might first look like a failure. Can the client retry? 2. Human perception times are biological and don’t change much. But everything in the tech stack has gotten so so much faster since the 80s. Throughput matters, sure, but latency (relatively speaking), is much less of a constraint now than it was. > — [xixixao on hackernews · 1 comments](https://news.ycombinator.com/item?id=49233451) > Another fun failure mode in the same vein: if you are emulating an NVMe device and declare that a write operation is successful once bytes are in memory, you will quickly find yourself buffering arbitrarily large amounts of data as "persist to disk" (SSD speed) falls behind "acknowledge writes" (RAM speed). If you do not add backpressure to your system intentionally, it will be added for you – and you may not like where it's placed! (This also happens at the SSD level: burst writes can be very fast as data is buffered in the SSD's own RAM, then performance steady-states at the true write speed once that's saturated) > — [mkeeter on hackernews · 1 comments](https://news.ycombinator.com/item?id=49233748) > You're getting downvoted but I had the same thought. This sounds like engineering. I do believe there are some cases where pushing the problem to a different place is the right thing to do, sometimes that different place is contextually better suited to that task, so you do actually gain performance or capability from moving the problem around. > — [ambicapter on hackernews](https://news.ycombinator.com/item?id=49234800) > My friend calls it the lump under the carpet. You can move it around, but you can never get rid of it. > — [jsrcout on hackernews](https://news.ycombinator.com/item?id=49234251) **Source threads** - [hackernews](https://news.ycombinator.com/item?id=49232410) · 51 points · 24 comments ## Similar posts on daily.dev - [Why Postgres wants NVMe on the hot path, and S3 everywhere else](https://daily.dev/posts/why-postgres-wants-nvme-on-the-hot-path-and-s3-everywhere-else-lgzx97lkb) · The New Stack · 30 upvotes · 1 comments - [How S3 Enables Stateless Write-Ahead Logs Despite Latency Trade-offs](https://daily.dev/posts/how-s3-enables-stateless-write-ahead-logs-despite-latency-trade-offs-rywhyjxzx) · Arpit Bhayani · 2 upvotes · 0 comments - [WAL \+ S3: Lakebase storage for the era of agents](https://daily.dev/posts/wal-s3-lakebase-storage-for-the-era-of-agents-4trqspanr) · Neon · 25 upvotes · 2 comments --- Tags: [#distributed-systems](https://daily.dev/tags/distributed-systems) [View this post on daily.dev](https://daily.dev/posts/every-fast-write-moves-work-somewhere-else-k4xugsl0j) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Every fast write moves work somewhere else","url":"https://daily.dev/posts/every-fast-write-moves-work-somewhere-else-k4xugsl0j","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/every-fast-write-moves-work-somewhere-else-k4xugsl0j"},"datePublished":"2026-08-09T18:29:01.788Z","dateModified":"2026-08-09T22:21:10.565Z","description":"A deep technical exploration of how database write durability works across different storage layers. Covers the tradeoffs between writing to memory, local NVMe...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/2b2020145bf16134106ce1eb76744a11?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/2b2020145bf16134106ce1eb76744a11?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Hacker News","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Hacker News","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/hn","url":"https://daily.dev/sources/hn"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/every-fast-write-moves-work-somewhere-else-k4xugsl0j","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"distributed-systems","timeRequired":"PT18M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Hacker News","item":"https://daily.dev/sources/hn"},{"@type":"ListItem","position":3,"name":"Every fast write moves work somewhere else"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/every-fast-write-moves-work-somewhere-else-k4xugsl0j#faq","mainEntity":[{"@type":"Question","name":"What failures does fdatasync() to a local NVMe SSD protect against vs. what it doesn't?","acceptedAnswer":{"@type":"Answer","text":"fdatasync() to a local NVMe SSD protects against process crashes and kernel crashes, but not against loss of the SSD itself or the host machine. If the host disappears before WAL records are uploaded to remote storage, a replacement can only recover from the last uploaded byte. The syscall name is the same whether the device is local or a remote network volume, but the failures survived are different. Engineers building storage systems track durability boundaries like these on daily.dev."}},{"@type":"Question","name":"How does WAL batch flushing work and what are the tradeoffs?","acceptedAnswer":{"@type":"Answer","text":"A WAL writer collects appended records into a batch, closes it after a fixed number of bytes, writes, or elapsed time, then calls fdatasync() once and returns success to every writer covered by that batch position. This lets multiple client PUTs share one expensive device flush, improving throughput. The tradeoff: the earliest request in a batch waits longest, a flush error fails the entire batch, and a timed-out request may have bytes already committed to disk. Developers designing write-heavy storage paths find discussions of batching strategies like this on daily.dev."}},{"@type":"Question","name":"What is the data loss risk of using a local WAL in front of object storage like S3?","acceptedAnswer":{"@type":"Answer","text":"When a local WAL is used to buffer writes before uploading to object storage, every acknowledged write that hasn't yet been uploaded is at risk if the local SSD or host is lost. The age of the oldest unuploaded WAL record determines the maximum data loss window. At 1,000 writes/second, writing one object per write produces roughly 2.6 billion S3 PUT requests per month; batching 100 writes per object reduces that to about 26 million, but increases bytes resent on failure. Teams choosing between local WAL buffering and synchronous object PUT for their storage layer follow these tradeoffs on daily.dev."}}]} ```