groundcover rebuilt their alert Dispatch Center using self-hosted Temporal to replace a fragile Python/FastAPI-based system. The new architecture uses two Workflow levels: a single long-running orchestrator that scans ClickHouse every 15 seconds and fans out via SignalWithStartWorkflow, and per-alert Workflows identified by deterministic IDs (fingerprint + route + connected app). Key design decisions include Signal batching with ReceiveAsync() for throughput, cursor rewinding for at-least-once delivery, Visibility API-based backpressure capped at 5,000 concurrent Workflows, in-memory state with PostgreSQL persistence, and rate-limit handling inside Activities rather than via Temporal retries. Testing covers unit tests, Workflow integration tests, end-to-end lifecycle tests, and replay tests against saved Event Histories. The system runs self-hosted inside each customer's Kubernetes cluster as part of a BYOC deployment model.
Table of contents
What we were replacing #The architecture: Two Workflow levels #Signal batching in the per-alert Workflow #Fan-out, ordering, and backpressure #State management #Handling retries and rate limits #Testing #Debugging with groundcover #What we learned #6.8K Impressions1 Comment