Scaling Security Insights: how we achieved a 10x increase in global scanning capacity

Cloudflare's Security Insights team scaled their scanning throughput from 10 to over 120 scans per second without adding hardware. Key optimizations included: switching Kafka consumers to batch processing with goroutines, splitting consumers into 'fast lane' and 'slow lane' to prevent slow messages from blocking fast ones, replacing per-row Postgres inserts with a hybrid UNNEST/COPY bulk insert strategy, switching the internal API from active-active to active-passive to eliminate cross-continental latency, and redesigning the scheduler with independent zone scheduling, randomized scan times, and adaptive rate limiting. These changes enabled automatic scanning for all free accounts and doubled scanning frequency across all tiers.