--- title: Day 14/30 AWS System Design Patterns | daily.dev description: A flash sale exposes a classic read-modify-write race condition in a Lambda-based inventory service backed by DynamoDB. With 40,000 concurrent buyers, multiple... canonical: https://daily.dev/posts/day-14-30-aws-system-design-patterns-zrbmkwnfh twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Day 14/30 AWS System Design Patterns | daily.dev og:description: A flash sale exposes a classic read-modify-write race condition in a Lambda-based inventory service backed by DynamoDB. With 40,000 concurrent buyers, multiple... og:url: https://daily.dev/posts/day-14-30-aws-system-design-patterns-zrbmkwnfh og:image: https://api.daily.dev/og/posts/ZRbmKwNfh.png og:image:alt: Day 14/30 AWS System Design Patterns og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Day 14/30 AWS System Design Patterns **[Joud Awad](https://daily.dev/sources/iac4jsbu0lv8wbsc85fsh)** · [@joudawad](https://daily.dev/joudawad) · 2 min read · 26 upvotes · 8 comments ## Summary A flash sale exposes a classic read-modify-write race condition in a Lambda-based inventory service backed by DynamoDB. With 40,000 concurrent buyers, multiple Lambda invocations read the same stock value simultaneously and each writes back a decremented value, resulting in 183 oversold units with no errors in logs. Four potential fixes are analyzed: serializing concurrency via reserved concurrency (global bottleneck), using strongly consistent reads (doesn't solve the race), using DynamoDB's atomic conditional UpdateItem (correct solution), and distributed locking via Redis (works but adds unnecessary complexity). The correct answer is atomic conditional writes in DynamoDB using ConditionExpression, which pushes the check-and-decrement into the database as a single atomic operation. ## Content An inventory service stores product stock in DynamoDB _(managed NoSQL database)_: partition key `product_id`, attribute `stock`. The purchase flow, running on Lambda _(serverless compute)_, is three steps: read the item, check `stock >= quantity`, write the item back with the decremented value. Read, decide, write. It has worked for two years. The platform runs its first flash sale. One product — 1,200 units of a limited-edition sneaker — gets 40,000 buyers in 10 minutes. Concurrent Lambda invocations climb into the hundreds. Post-sale reconciliation: 1,383 purchases were confirmed against 1,200 units of stock. 183 customers paid for shoes that do not exist. The `stock` attribute reads 0 — it never went negative, no write ever failed, and there is not a single error in the logs. The replay of any two of those invocations tells the story. Invocation A reads `stock = 5`. Invocation B reads `stock = 5` a millisecond later. A writes 4. B writes 4. Two units left the building; the database recorded one. What is the correct fix? A) Set the function's reserved concurrency to 1 so purchases execute one at a time — this does serialize the race, by serializing everything: one purchase at a time across all products for the entire platform, a global bottleneck to protect one hot item, and it still breaks the moment a second service writes to the table B) Use strongly consistent reads so each invocation sees the latest committed value — but both invocations can perform a perfectly consistent read of `stock = 5` a millisecond apart; the race is not a stale read, it is two read-decide-write sequences interleaving between the read and the write C) Stop deciding in the client — push the check and the change into the database as one atomic operation: `UpdateItem` with `SET stock = stock - :qty` and `ConditionExpression: stock >= :qty`; DynamoDB evaluates the condition and applies the decrement atomically, and a losing writer gets a `ConditionalCheckFailedException` instead of silently overwriting D) Add a distributed lock — each invocation acquires a Redis lock on `product_id` before the read-modify-write; this can work, but it imports a lock service plus lease expiry, crash recovery, and fencing correctness, all to reproduce a guarantee the database offers in one line Answer in the comments. #DynamoDB #DistributedSystems #SystemDesign #AWS #BackendEngineering ## Community discussion Top comments from developers on daily.dev. **@joudawad** · 5 upvotes > The answer is C. > > This is the lost update problem. Read-decide-write is three operations, and the database only ever saw the third one. DynamoDB _(managed NoSQL database)_ did exactly what it was asked: two clients each said "set stock to 4," and it set stock to 4, twice. Last write wins, and the write it overwrote — along with the sneaker that write represented — is simply gone. No conflict is detected because, to the database, there was no conflict: just two valid writes. That is why the logs are clean. Lost updates don't fail; they succeed over each other. > > The fix is to collapse the three... **@joudawad** · 1 upvotes > A — Reserved concurrency of 1 turns a per-item race into a platform-wide bottleneck: every purchase of every product, single file. And it only pretends to solve the problem — the invariant now lives in a deployment setting that a second writer (an admin tool, a returns service, a batch job) silently violates. Correctness that depends on there being exactly one client is not correctness. **@trevorsuna** · 1 upvotes > Atomic conditional updates are the clean answer here because the invariant lives beside the data. Strong reads can show the latest stock value, but they still leave a gap before the write where another buyer can win the race. **@joudawad** · 1 upvotes > Also, it would mean a lot to me if you could support my content and stay in touch 🙏 > > - YouTube: [https://www.youtube.com/@system-design-lab](https://www.youtube.com/@system-design-lab) > - LinkedIn: [https://www.linkedin.com/in/joud-awad/](https://www.linkedin.com/in/joud-awad/) > - Medium Blog: [https://joudwawad.medium.com/](https://joudwawad.medium.com/) > - Substack: [https://joudawad.substack.com/](https://joudawad.substack.com/) **@joudawad** · 0 upvotes > B — Consistency answers "how fresh was my read?" The lost update doesn't care. Both invocations can read the truest possible `stock = 5` — the interleaving happens after the read, in the gap before the write. Strongly consistent reads shrink no gap; they just make both racers better informed while they collide. (They also double read cost for the privilege.) ## Similar posts on daily.dev - [DynamoDB crash course: part 3 – design patterns](https://daily.dev/posts/dynamodb-crash-course-part-3-design-patterns-gx7z1mhbm) · Planet Python · 1 upvotes · 0 comments - [We replaced Redis with MySQL for inventory reservations—and it scaled \(2026\)](https://daily.dev/posts/we-replaced-redis-with-mysql-for-inventory-reservations-and-it-scaled-2026--7uecbcooa) · Hacker News · 1 upvotes · 0 comments - [TLA\+ Modeling of AWS outage DNS race condition](https://daily.dev/posts/tla-modeling-of-aws-outage-dns-race-condition-jljhfneea) · Metadata · 1 upvotes · 0 comments --- Tags: [#career](https://daily.dev/tags/career), [#distributed-systems](https://daily.dev/tags/distributed-systems), [#aws-lambda](https://daily.dev/tags/aws-lambda), [#aws-dynamodb](https://daily.dev/tags/aws-dynamodb) [View this post on daily.dev](https://daily.dev/posts/day-14-30-aws-system-design-patterns-zrbmkwnfh) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"DiscussionForumPosting","mainEntityOfPage":"https://daily.dev/posts/day-14-30-aws-system-design-patterns-zrbmkwnfh","headline":"Day 14/30 AWS System Design Patterns","text":"A flash sale exposes a classic read-modify-write race condition in a Lambda-based inventory service backed by DynamoDB. With 40,000 concurrent buyers, multiple Lambda invocations read the same stock value simultaneously and each writes back a decremented value, resulting in 183 oversold units with no errors in logs. Four potential fixes are analyzed: serializing concurrency via reserved concurrency (global bottleneck), using strongly consistent reads (doesn't solve the race), using DynamoDB's atomic conditional UpdateItem (correct solution), and distributed locking via Redis (works but adds unnecessary complexity). The correct answer is atomic conditional writes in DynamoDB using ConditionExpression, which pushes the check-and-decrement into the database as a single atomic operation.","url":"https://daily.dev/posts/day-14-30-aws-system-design-patterns-zrbmkwnfh","datePublished":"2026-08-03T20:55:51.107Z","dateModified":"2026-09-13T21:34:14.079Z","author":{"@type":"Person","name":"Joud Awad","url":"https://daily.dev/joudawad","image":"https://media.daily.dev/image/upload/s--dOB9RaXY--/f_auto/v1773320801/avatars/avatar_iaC4JsBU0lV8wBsc85fSh?_a=BAMAMiiu0","description":"Principal Solution Architecture ","worksFor":{"@type":"Organization","name":"Metalab","logo":"https://www.google.com/s2/favicons?domain=metalab.com&sz=128"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"EndorseAction"},"userInteractionCount":85140}},"image":"https://media.daily.dev/image/upload/s--M5kLrZH4--/f_auto/v1785790552/posts/ZRbmKwNfh?_a=BAMAMicg0","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":26},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":8}],"comment":[{"@type":"Comment","text":"The answer is C.\nThis is the lost update problem. Read-decide-write is three operations, and the database only ever saw the third one. DynamoDB (managed NoSQL database) did exactly what it was asked: two clients each said “set stock to 4,” and it set stock to 4, twice. Last write wins, and the write it overwrote — along with the sneaker that write represented — is simply gone. No conflict is detected because, to the database, there was no conflict: just two valid writes. That is why the logs are clean. Lost updates don’t fail; they succeed over each other.\nThe fix is to collapse the three operations into one, executed where atomicity lives — inside the database:\nUpdateItem with UpdateExpression: SET stock = stock - :qty and ConditionExpression: stock >= :qty.\nDynamoDB evaluates the condition against the item’s current committed state and applies the decrement in the same atomic operation. Two hundred concurrent buyers of the last pair produce exactly one success and 199 ConditionalCheckFailedExceptions — and each exception is a feature: it is the “sold out” answer, delivered at the only moment it can be known truthfully.\nFor updates too complex for an atomic expression — multi-attribute changes computed in code — the same idea generalizes to optimistic locking: keep a version attribute, read it, and write with ConditionExpression: version = :theversionyou_read, incrementing it. If anyone wrote in between, your condition fails and you re-read and retry. You are not preventing the race; you are detecting it at write time and losing loudly instead of silently. SQL developers know this as UPDATE ... WHERE version = x; CPU designers know it as compare-and-swap. Same law everywhere: the check and the change must be one operation, or they are a race.\nTwo trade-offs to respect. First, failed conditions are your application’s job now — retries for version conflicts, a clean “sold out” path for stock exhaustion; a condition failure consumes write capacity, so a retry storm against a sold-out item still costs money. Second — day 1 readers will see this coming — a single flash-sale item is a hot key with a hard per-item write ceiling. Conditional writes make the contention correct; they do not make it infinite. At true ticket-drop scale, the next pattern is sharding the stock across sub-items or fronting it with a queue.","datePublished":"2026-08-03T20:56:29.613Z","url":"https://daily.dev/posts/ZRbmKwNfh#c-fX1yKe2w2","author":{"@type":"Person","name":"Joud Awad","url":"https://daily.dev/joudawad","image":"https://media.daily.dev/image/upload/s--dOB9RaXY--/f_auto/v1773320801/avatars/avatar_iaC4JsBU0lV8wBsc85fSh?_a=BAMAMiiu0"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":5}},{"@type":"Comment","text":"A — Reserved concurrency of 1 turns a per-item race into a platform-wide bottleneck: every purchase of every product, single file. And it only pretends to solve the problem — the invariant now lives in a deployment setting that a second writer (an admin tool, a returns service, a batch job) silently violates. Correctness that depends on there being exactly one client is not correctness.","datePublished":"2026-08-03T20:56:36.281Z","url":"https://daily.dev/posts/ZRbmKwNfh#c-IV5DE6MH2","author":{"@type":"Person","name":"Joud Awad","url":"https://daily.dev/joudawad","image":"https://media.daily.dev/image/upload/s--dOB9RaXY--/f_auto/v1773320801/avatars/avatar_iaC4JsBU0lV8wBsc85fSh?_a=BAMAMiiu0"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1}},{"@type":"Comment","text":"Atomic conditional updates are the clean answer here because the invariant lives beside the data. Strong reads can show the latest stock value, but they still leave a gap before the write where another buyer can win the race.","datePublished":"2026-08-04T05:06:22.700Z","url":"https://daily.dev/posts/ZRbmKwNfh#c-1bh9hk9JU","author":{"@type":"Person","name":"Trevor Suna","url":"https://daily.dev/trevorsuna","image":"https://media.daily.dev/image/upload/s--dZ7gXxpp--/f_auto/v1784081551/avatars/avatar_EMoP47rpuw8DNjhp6R1b6?_a=BAMAMicg0"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1}},{"@type":"Comment","text":"Also, it would mean a lot to me if you could support my content and stay in touch 🙏\n\nYouTube: https://www.youtube.com/@system-design-lab\nLinkedIn: https://www.linkedin.com/in/joud-awad/\nMedium Blog: https://joudwawad.medium.com/\nSubstack: https://joudawad.substack.com/","datePublished":"2026-08-03T20:56:10.058Z","url":"https://daily.dev/posts/ZRbmKwNfh#c-NcUvFncK2","author":{"@type":"Person","name":"Joud Awad","url":"https://daily.dev/joudawad","image":"https://media.daily.dev/image/upload/s--dOB9RaXY--/f_auto/v1773320801/avatars/avatar_iaC4JsBU0lV8wBsc85fSh?_a=BAMAMiiu0"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1}},{"@type":"Comment","text":"B — Consistency answers “how fresh was my read?” The lost update doesn’t care. Both invocations can read the truest possible stock = 5 — the interleaving happens after the read, in the gap before the write. Strongly consistent reads shrink no gap; they just make both racers better informed while they collide. (They also double read cost for the privilege.)","datePublished":"2026-08-03T20:56:41.971Z","url":"https://daily.dev/posts/ZRbmKwNfh#c-UTy7jHYOX","author":{"@type":"Person","name":"Joud Awad","url":"https://daily.dev/joudawad","image":"https://media.daily.dev/image/upload/s--dOB9RaXY--/f_auto/v1773320801/avatars/avatar_iaC4JsBU0lV8wBsc85fSh?_a=BAMAMiiu0"}}],"isPartOf":{"@type":"WebPage","url":"https://daily.dev/sources/iac4jsbu0lv8wbsc85fsh","name":"Joud Awad"}} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Joud Awad","item":"https://daily.dev/sources/iac4jsbu0lv8wbsc85fsh"},{"@type":"ListItem","position":3,"name":"Day 14/30 AWS System Design Patterns"}]} ```