How We Cut Kafka Consumer Deployment Costs by 83% · triva...
This title could be clearer and more informative.Try out Clickbait Shieldfor free (5 uses left this month).
A backend engineer at Trivago shares how they cut Kafka consumer deployment costs by 83% and resolved 19 P1 incidents by diagnosing a multi-layered performance problem in their PSE-kafka microservice. The root cause turned out to be three compounding issues: an in-house KafkaReceiverFlux that stopped calling poll() under load, a Mono.delayElement() ceiling tied to flatMap concurrency, and an undersized gRPC thread pool. Each fix in isolation had no measurable effect; only addressing all three together — migrating to reactor-kafka, removing the in-memory delay via an IsFinished flag, and tuning the gRPC thread pool — reduced pod count from 60 to 6, eliminated consumer lag, and cut pod startup time from ~60s to ~10s.
