--- title: "16/30 Days System Design Questions!" url: https://daily.dev/posts/16-30-days-system-design-questions--hegbbiuih source_url: https://daily.dev/posts/16-30-days-system-design-questions--hegbbiuih type: freeform source: "Joud Awad" author: "Joud Awad" published: 2026-05-21T16:39:56.909Z updated: 2026-05-21T16:40:33.120Z tags: ["backend", "aws-dynamodb"] reading_time: 2 upvotes: 178 comments: 15 language: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # 16/30 Days System Design Questions! **[Joud Awad](https://daily.dev/sources/iac4jsbu0lv8wbsc85fsh)** · [@joudawad](https://daily.dev/joudawad) · 2 min read · 178 upvotes · 15 comments ## Summary A multi-tenant analytics pipeline on DynamoDB hits a hot partition problem when one tenant's write volume spikes 100x overnight, causing ProvisionedThroughputExceeded errors and P99 latency to jump from 8ms to 400ms. The post presents four candidate solutions — write sharding with random PK suffix, write jitter, partition splitting via WCU bump, and time-bucketed keys — and challenges readers to identify which one is the correct fix and which is the 'senior engineer trap' that looks good on paper but fails under real incident conditions. ## Content You're running a multi-tenant analytics pipeline on DynamoDB. 200 tenants, 12K writes/sec total. Everything is fine — until it isn't. One tenant onboards a massive customer overnight. Their event volume 100x's. Now that one tenant is hitting 9K writes/sec on a single partition key. The other 199 tenants sit idle. ProvisionedThroughputExceeded errors start firing. P99 write latency spikes from 8ms to 400ms. Your on-call gets paged. Everyone is getting throttled because of one key. **Here's the setup:** • Table: events (DynamoDB, on-demand capacity) • PK: tenant_id · SK: event_timestamp • Hot tenant: ~9K WPS · All other tenants: ~15 WPS each • Single partition key absorbing all 9K writes — DynamoDB's per-partition limit is the wall Classic hot partition. What do you do? A) Write sharding — append a random suffix to the PK (tenant_id#0 … tenant_id#9). B) Jitter the writes — add randomized delay (0–500ms) on the producer client. C) Partition splitting — bump table WCU and let DynamoDB auto-split the hot partition. D) Time-bucket the key — change PK to tenant_id#YYYY-MM-DD-HH. All four show up in real production postmortems. Three of them fail at 3am. One of them is the senior engineer trap — it looks correct in the design doc and falls apart in the war room. Pick one — A, B, C, or D — and tell me why. Full breakdown in the comments (including which one trips up senior engineers every time). If your team has ever argued about hot keys at 2am, share this with them. The debate is worth more than the post. Drop your answer 👇 #30DaysOfSystemDesign #SystemDesign #SoftwareEngineering #AWS #DynamoDB ## Community discussion Top comments from developers on daily.dev. **@joudawad** · 10 upvotes > **A — Write Sharding (THE RIGHT ANSWER)** > > > DynamoDB hashes the partition key to decide which physical shard handles the write. One PK value = one shard = one wall. When that tenant hits 9K WPS, the shard caps out. > > > The fix: append a random suffix at write time — tenant_id#0 through tenant_id#9. Now the hash function sees 10 different keys and distributes writes across 10 logical partitions (~900 WPS each, well under the limit). Throttling stops, P99 drops back to single digits. > > > The read cost: scatter-gather across all 10 suffixes to query that tenant's data. For a write-heavy analytics... **@joudawad** · 9 upvotes > **C — Partition Splitting (SENIOR ENGINEER TRAP)** > > > DynamoDB _does_ auto-split partitions — so this sounds right. But the split is based on **throughput**, not **key cardinality**. If one PK value generates 9K WPS, all those writes still hash to the same destination after the split. You can't split a single key across two physical shards. > > > You'll see the partition split in CloudWatch, assume the problem is fixed, and then get paged again 10 minutes later. It's the answer that survives the design doc and dies in the war room. **@joudawad** · 5 upvotes > **D — Time-bucket the key** > > > tenant_id#2026-05-21-14 is a real pattern for time-series data — it keeps per-bucket size manageable and enables efficient TTL expiry. > > > But at any given hour, you _still_ have one partition key absorbing all 9K WPS for that tenant. You renamed the hot partition to a hot time bucket. Same problem, new name. Plus now your reads have to query 24 different PKs for a day's worth of data. **@joudawad** · 5 upvotes > **B — Jitter the writes** > > > Jitter is a real pattern — it solves the **thundering herd** problem, where N clients fire requests at the same millisecond and create a spike. Spreading those over 500ms smooths the burst. > > > But a hot partition is a **sustained** 9K WPS, not a burst. Adding delay doesn't reduce the rate — DynamoDB still sees 9K WPS on the same partition. You just made every write slower for zero benefit. **@joudawad** · 4 upvotes > Also, it would mean a lot to me if you could support my content and stay in touch 🙏 > > - YouTube: [https://www.youtube.com/@system-design-lab](https://www.youtube.com/@system-design-lab) > - LinkedIn: [https://www.linkedin.com/in/joud-awad/](https://www.linkedin.com/in/joud-awad/) > - Medium Blog: [https://joudwawad.medium.com/](https://joudwawad.medium.com/) > - Substack: [https://joudawad.substack.com/](https://joudawad.substack.com/) --- Tags: [#backend](https://daily.dev/tags/backend), [#aws-dynamodb](https://daily.dev/tags/aws-dynamodb) [View this post on daily.dev](https://daily.dev/posts/16-30-days-system-design-questions--hegbbiuih)