A practical account of moving a model-serving workload from CPU/memory-based Horizontal Pod Autoscaling to KEDA's event-driven autoscaling, using queue depth (Pub/Sub, Redis, RabbitMQ, SQS) as the scaling signal instead. Covers a sample ScaledObject config, trade-offs between event source options, threshold-tuning pitfalls, cold-start mitigation via a warm minimum replica, and monitoring queue depth versus pod count. Extends the pattern to agentic AI workloads, arguing that bursty agent task queues suit event-driven scaling even better than inference serving because agents can be busy (waiting on tool calls) while using little CPU.
Questions this post answers
Why does Kubernetes horizontal pod autoscaling (HPA) fail for bursty AI inference workloads?
HPA reacts to CPU and memory usage inside pods, which is the wrong signal because a serving pod can sit at low CPU while a request queue behind it fills up. By the time resource usage rises enough to trigger scaling, the backlog has already built up, causing latency, and by the time pods scale down, teams pay for idle compute. Teams designing inference autoscaling can track patterns like queue-depth-based scaling on daily.dev.
How do I configure a KEDA ScaledObject to scale Kubernetes pods based on queue depth?
Define a ScaledObject with scaleTargetRef pointing to the deployment, minReplicaCount and maxReplicaCount, a cooldownPeriod, and a trigger such as gcp-pubsub with a subscriptionName and a SubscriptionSize mode value, for example a value of 5 meaning roughly one worker per five unprocessed messages. Setting minReplicaCount to 1 rather than 0 keeps a warm worker to absorb cold-start latency. Engineers tuning KEDA configs for inference or agent queues can follow this kind of setup guidance on daily.dev.
Which event source should I use for KEDA autoscaling: Pub/Sub, Redis, or RabbitMQ?
Pub/Sub or SQS is easiest to reason about since subscription size is a native metric and the KEDA scaler works with little tuning, but ties you to cloud provider latency and quotas. Redis is fast and gives more control but makes your scaling signal dependent on Redis availability. RabbitMQ offers more features like dead-letter queues and routing at the cost of higher operational overhead, worthwhile mainly if another team already runs it. Comparing event source trade-offs for autoscaling is easier to revisit later when saved on daily.dev.