--- title: The Migrating Bottleneck | daily.dev description: A detailed postmortem of four successive performance bottlenecks discovered while load-testing a DynamoDB backend for a sync datastore layer. The first... canonical: https://daily.dev/posts/the-migrating-bottleneck-ep4fzxzi1 twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: The Migrating Bottleneck | daily.dev og:description: A detailed postmortem of four successive performance bottlenecks discovered while load-testing a DynamoDB backend for a sync datastore layer. The first... og:url: https://daily.dev/posts/the-migrating-bottleneck-ep4fzxzi1 og:image: https://api.daily.dev/og/posts/Ep4FzXZI1.png og:image:alt: The Migrating Bottleneck og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # The Migrating Bottleneck **[Webframp](https://daily.dev/sources/webframp)** · 13 min read · 0 upvotes · 0 comments ## Summary A detailed postmortem of four successive performance bottlenecks discovered while load-testing a DynamoDB backend for a sync datastore layer. The first bottleneck was a hot GSI partition using a single partition key value ('FILE') that caused full scans on every sync. The fix redesigned the GSI to use one partition per model with timestamps in the sort key. This revealed a second bottleneck: a single global lock serializing all workers. Namespacing locks per worker fixed that, but exposed a third bottleneck: index commits rewriting all file entries on every sync. Switching to disposable per-iteration model instances capped committed files at six, but catalog metadata from deleted models kept accumulating, causing a fourth bottleneck. Adding periodic pruning finally stabilized throughput. A cross-check against an S3 backend confirmed the catalog lifecycle issue lived in the sync coordinator above both backends, not in DynamoDB itself. Key takeaways: hot GSI partitions fail silently under low load, filter conditions should be key conditions, and metadata outliving its resources is a slow-burning performance bug. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Why does a DynamoDB Global Secondary Index with a single partition key value become a bottleneck under load? DynamoDB allocates throughput per physical partition, not per table, and hashes items across partitions by partition key value. A GSI where every item shares one partition key value (like gsi1pk = "FILE") puts all items on a single partition, which has a hard ceiling around 3,000 read capacity units regardless of table-level provisioned or on-demand capacity. _Anyone designing DynamoDB GSIs to avoid hot partitions can find real-world key redesign patterns discussed on daily.dev._ ### How do you fix a hot GSI partition in DynamoDB when syncing changed data? Restructure the partition key to have one value per logical entity (such as gsi1pk = "FILE#/") instead of one shared value, and put the timestamp first in the sort key so a query like gsi1sk > lastSyncTimestamp becomes a key condition rather than a post-read filter. This spreads writes across partitions and lets DynamoDB skip unchanged items instead of charging to read and discard them. _Developers redesigning DynamoDB access patterns for sync workloads track fixes like this on daily.dev._ ### Why does a distributed lock keep timing out even after fixing a database query bottleneck? A single datastore-level lock acquired around every sync, regardless of which model it touches, serializes all workers even when they never touch each other's data; ten workers queued behind one lock key with a 30-second TTL and 60-second wait will blow past that wait on every cycle. Scoping the lock key to a namespace so each worker gets its own lock key eliminates this cross-worker contention, dropping acquisition time to under a millisecond when uncontended. _Engineers debugging lock contention in multi-worker sync systems compare fixes like namespaced locking on daily.dev._ ## Similar posts on daily.dev - [100 ways to fail with DynamoDB indexing](https://daily.dev/posts/100-ways-to-fail-with-dynamodb-indexing-ofgdcpgin) · monday Engineering · 0 upvotes · 0 comments --- Tags: [#aws](https://daily.dev/tags/aws), [#performance](https://daily.dev/tags/performance), [#database](https://daily.dev/tags/database), [#distributed-systems](https://daily.dev/tags/distributed-systems), [#aws-dynamodb](https://daily.dev/tags/aws-dynamodb) [View this post on daily.dev](https://daily.dev/posts/the-migrating-bottleneck-ep4fzxzi1) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"The Migrating Bottleneck","url":"https://daily.dev/posts/the-migrating-bottleneck-ep4fzxzi1","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/the-migrating-bottleneck-ep4fzxzi1"},"datePublished":"2026-07-28T15:42:07.422Z","dateModified":"2026-09-13T21:44:08.471Z","description":"A detailed postmortem of four successive performance bottlenecks discovered while load-testing a DynamoDB backend for a sync datastore layer. The first...","image":"https://media.daily.dev/image/upload/s--0_ODbtD2--/f_auto/v1722860399/public/Placeholder%2008","thumbnailUrl":"https://media.daily.dev/image/upload/s--0_ODbtD2--/f_auto/v1722860399/public/Placeholder%2008","isAccessibleForFree":true,"articleSection":"Webframp","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Webframp","logo":"https://media.daily.dev/image/upload/s--NCWOwzJ8--/f_auto,q_auto/v1779608748/logos/webframp","url":"https://daily.dev/sources/webframp"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/the-migrating-bottleneck-ep4fzxzi1","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"aws,performance,database,distributed-systems,aws-dynamodb","timeRequired":"PT13M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Webframp","item":"https://daily.dev/sources/webframp"},{"@type":"ListItem","position":3,"name":"The Migrating Bottleneck"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/the-migrating-bottleneck-ep4fzxzi1#faq","mainEntity":[{"@type":"Question","name":"Why does a DynamoDB Global Secondary Index with a single partition key value become a bottleneck under load?","acceptedAnswer":{"@type":"Answer","text":"DynamoDB allocates throughput per physical partition, not per table, and hashes items across partitions by partition key value. A GSI where every item shares one partition key value (like gsi1pk = \"FILE\") puts all items on a single partition, which has a hard ceiling around 3,000 read capacity units regardless of table-level provisioned or on-demand capacity. Anyone designing DynamoDB GSIs to avoid hot partitions can find real-world key redesign patterns discussed on daily.dev."}},{"@type":"Question","name":"How do you fix a hot GSI partition in DynamoDB when syncing changed data?","acceptedAnswer":{"@type":"Answer","text":"Restructure the partition key to have one value per logical entity (such as gsi1pk = \"FILE#/\") instead of one shared value, and put the timestamp first in the sort key so a query like gsi1sk > lastSyncTimestamp becomes a key condition rather than a post-read filter. This spreads writes across partitions and lets DynamoDB skip unchanged items instead of charging to read and discard them. Developers redesigning DynamoDB access patterns for sync workloads track fixes like this on daily.dev."}},{"@type":"Question","name":"Why does a distributed lock keep timing out even after fixing a database query bottleneck?","acceptedAnswer":{"@type":"Answer","text":"A single datastore-level lock acquired around every sync, regardless of which model it touches, serializes all workers even when they never touch each other's data; ten workers queued behind one lock key with a 30-second TTL and 60-second wait will blow past that wait on every cycle. Scoping the lock key to a namespace so each worker gets its own lock key eliminates this cross-worker contention, dropping acquisition time to under a millisecond when uncontended. Engineers debugging lock contention in multi-worker sync systems compare fixes like namespaced locking on daily.dev."}}]} ```