A deep dive into how the `linger.ms` producer configuration affects Apache Kafka performance across different load levels. Benchmarks comparing Kafka 3.7.2 and 4.3.0 reveal that much of the observed performance difference between versions stems from the default `linger.ms` changing from 0 to 5 ms in Kafka 4.0. At low throughput (5K records/s), the 5 ms linger adds latency without improving batching. At high throughput (100K records/s), `linger.ms=20` dramatically reduces p99.9 tail latency from 700+ ms to ~23 ms. The key insight is that the per-producer-per-partition send rate — not total cluster throughput — determines whether a given linger value will improve batch sizes. Keyed workloads are especially prone to small batches because keys spread records across many partitions.
6.7K Impressions1 Comment