A detailed postmortem of four successive performance bottlenecks discovered while load-testing a DynamoDB backend for a sync datastore layer. The first bottleneck was a hot GSI partition using a single partition key value ('FILE') that caused full scans on every sync. The fix redesigned the GSI to use one partition per model with timestamps in the sort key. This revealed a second bottleneck: a single global lock serializing all workers. Namespacing locks per worker fixed that, but exposed a third bottleneck: index commits rewriting all file entries on every sync. Switching to disposable per-iteration model instances capped committed files at six, but catalog metadata from deleted models kept accumulating, causing a fourth bottleneck. Adding periodic pruning finally stabilized throughput. A cross-check against an S3 backend confirmed the catalog lifecycle issue lived in the sync coordinator above both backends, not in DynamoDB itself. Key takeaways: hot GSI partitions fail silently under low load, filter conditions should be key conditions, and metadata outliving its resources is a slow-burning performance bug.
