Rapidflare shares how they built a production document ingestion pipeline capable of handling over a million documents using Temporal for durable workflow orchestration. The pipeline serves AI agents for technical sales in the electronics/semiconductor industry, where incomplete or stale data directly degrades agent answer quality. Key architectural decisions include: a sliding window pattern to control fan-out and respect rate limits on LLM/embedding APIs, Activity Heartbeats for long-running crawlers and clean cancellation, Continue-as-New to avoid Temporal history size limits at million-document scale, cloud storage as a data bus to bypass payload size limits, per-concern Worker isolation across separate Task Queues, and durable approval gates using Workflow Signals. The post also covers zero-downtime promotion, recurring ingestion via Temporal Schedules, and observability through a dedicated status Worker.
