<!-- mobian-agent-page publisher="dailydev" canonical="https://daily.dev/posts/october-2024-queue-retrospective-m4aywdpa4" -->

---
title: October 2024 queue retrospective | daily.dev
description: A detailed retrospective covers six infrastructure incidents affecting Inngest's queueing system in October 2024, including a botched Key Queues rollout that...
canonical: https://daily.dev/posts/october-2024-queue-retrospective-m4aywdpa4
twitter:card: summary_large_image
twitter:site: @dailydotdev
og:type: website
og:site_name: daily.dev
og:title: October 2024 queue retrospective | daily.dev
og:description: A detailed retrospective covers six infrastructure incidents affecting Inngest's queueing system in October 2024, including a botched Key Queues rollout that...
og:url: https://daily.dev/posts/october-2024-queue-retrospective-m4aywdpa4
og:image: https://api.daily.dev/og/posts/M4ayWdPA4.png
og:image:alt: October 2024 queue retrospective
og:image:width: 1200
og:image:height: 630
og:locale: en
---

> ## Documentation Index
> Fetch the complete documentation index at: https://daily.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# October 2024 queue retrospective

**[Inngest Blog](https://daily.dev/sources/inngest)** · 6 min read · 0 upvotes · 0 comments

## Summary

A detailed retrospective covers six infrastructure incidents affecting Inngest's queueing system in October 2024, including a botched Key Queues rollout that hurt throughput under high cardinality, cron delays from a backing store change, silent NATS publishing failures, CPU-driven queue delays fixed via sharding migration, quadratic event-matching slowdowns for waitForEvent/cancelOn, and a Postgres RDS disk issue causing UI degradation. Each incident includes root cause analysis and follow-up fixes, and the company reports P95 execution latency improvements to ~50ms after the fixes, alongside plans to eventually move away from NATS toward Kafka.

## Full article

daily.dev links to this article rather than hosting it. Read it at the original source: <https://www.inngest.com/blog/2024-10-queue-retros>

## Questions this post answers

### Why did NATS Jetstream cause data publishing failures in a distributed queue system?

NATS Jetstream silently failed to publish messages when round-robin distribution across message streams was used to reduce load on a single stream. Reverting to a permanent stream assignment restored reliable publishing, and metrics confirmed normal operation afterward. Fundamental reliability issues with NATS Jetstream publishing prompted plans to migrate away from NATS toward Kafka as a longer-term fix.

_Teams weighing NATS versus Kafka for message streaming can track real-world reliability tradeoffs on daily.dev._

### Why did adding more granular queues (Key Queues) reduce throughput under high load?

Increasing queue granularity for high-cardinality multi-tenant flow control caused an exponential proliferation of individual queues, and processing each queue required 4x the operations of processing a single queue item. Under extreme key cardinality this created many queues with very few items each, dramatically lowering per-item processing capacity and throughput, forcing a rollback.

_Engineers designing multi-tenant queue sharding can follow lessons like this on daily.dev before hitting the same wall._

### What caused delayed processing for waitForEvent and cancelOn operations in a queue-based workflow engine?

An unoptimized expression being evaluated at high volume caused the event-matching service to backlog, since matching every incoming event against every saved waitForEvent or cancelOn condition is naively a quadratic operation. A dedicated optimization service normally reduces this to near-millisecond speed but failed on one specific expression, prompting fixes to expression handling and new complexity alerting.

_Developers debugging event-matching bottlenecks in workflow engines can find similar deep dives via daily.dev._

## Similar posts on daily.dev

- [Incident report for October 20, 2026 - Event ingestion and processing disruption](https://daily.dev/posts/incident-report-for-october-20-2026---event-ingestion-and-processing-disruption-wkqorhx1v) · Inngest Blog · 0 upvotes · 0 comments
- [Durable Digest: October 2026](https://daily.dev/posts/durable-digest-october-2026-osml8w5wa) · Temporal · 0 upvotes · 0 comments
- [GitHub availability report: October 2026](https://daily.dev/posts/github-availability-report-october-2026-ukvh8pg6r) · GitHub Blog · 0 upvotes · 0 comments

---

Tags: [#postgresql](https://daily.dev/tags/postgresql), [#distributed-systems](https://daily.dev/tags/distributed-systems)

[View this post on daily.dev](https://daily.dev/posts/october-2024-queue-retrospective-m4aywdpa4)

```json
{"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]}
{"@context":"https://schema.org","@type":"TechArticle","headline":"October 2024 queue retrospective","url":"https://daily.dev/posts/october-2024-queue-retrospective-m4aywdpa4","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/october-2024-queue-retrospective-m4aywdpa4"},"datePublished":"2026-09-13T19:57:07.942Z","dateModified":"2026-09-13T19:58:10.028Z","description":"A detailed retrospective covers six infrastructure incidents affecting Inngest's queueing system in October 2024, including a botched Key Queues rollout that...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/1cbfa3e3887f23e95741029b95e2fcc0?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/1cbfa3e3887f23e95741029b95e2fcc0?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Inngest Blog","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Inngest Blog","logo":"https://media.daily.dev/image/upload/s--ieZKUa5y--/c_limit,w_256/f_auto,q_auto/v1789329423/logos/inngest?_a=BAMAMicg0","url":"https://daily.dev/sources/inngest"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/october-2024-queue-retrospective-m4aywdpa4","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"postgresql,distributed-systems","timeRequired":"PT6M"}
{"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Inngest Blog","item":"https://daily.dev/sources/inngest"},{"@type":"ListItem","position":3,"name":"October 2024 queue retrospective"}]}
{"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/october-2024-queue-retrospective-m4aywdpa4#faq","mainEntity":[{"@type":"Question","name":"Why did NATS Jetstream cause data publishing failures in a distributed queue system?","acceptedAnswer":{"@type":"Answer","text":"NATS Jetstream silently failed to publish messages when round-robin distribution across message streams was used to reduce load on a single stream. Reverting to a permanent stream assignment restored reliable publishing, and metrics confirmed normal operation afterward. Fundamental reliability issues with NATS Jetstream publishing prompted plans to migrate away from NATS toward Kafka as a longer-term fix. Teams weighing NATS versus Kafka for message streaming can track real-world reliability tradeoffs on daily.dev."}},{"@type":"Question","name":"Why did adding more granular queues (Key Queues) reduce throughput under high load?","acceptedAnswer":{"@type":"Answer","text":"Increasing queue granularity for high-cardinality multi-tenant flow control caused an exponential proliferation of individual queues, and processing each queue required 4x the operations of processing a single queue item. Under extreme key cardinality this created many queues with very few items each, dramatically lowering per-item processing capacity and throughput, forcing a rollback. Engineers designing multi-tenant queue sharding can follow lessons like this on daily.dev before hitting the same wall."}},{"@type":"Question","name":"What caused delayed processing for waitForEvent and cancelOn operations in a queue-based workflow engine?","acceptedAnswer":{"@type":"Answer","text":"An unoptimized expression being evaluated at high volume caused the event-matching service to backlog, since matching every incoming event against every saved waitForEvent or cancelOn condition is naively a quadratic operation. A dedicated optimization service normally reduces this to near-millisecond speed but failed on one specific expression, prompting fixes to expression handling and new complexity alerting. Developers debugging event-matching bottlenecks in workflow engines can find similar deep dives via daily.dev."}}]}
```

