All tests were executed against Kafka 4.2.0 using    Dimster   .    In the last post we measured the overhead that the mechanics of share groups adds, and saw that it is pretty small. Likewise we saw that raw throughput was also comparable to consumer groups and even saw it exceed consumer group throughput on one test.   In this post we’re going to simulate processing time in the consumers  to make these benchmarks more realistic and show the utility of share groups (namely the ability to parallelize processing beyond the partition count).  We’ll see how the following two configurations play an important role in parallelizing consumption with share groups:     max.poll.records  (consumer config)     group.share.partition.max.record.locks  (broker-side config)

Jack Vanlightly

Kafka 4.2 Share Groups introduce a new parallelism model where consumers can scale beyond partition count. However, the default max.poll.records=500 causes a 'greedy-capture regime' where a few consumers monopolize the inflight record budget (group.share.partition.max.record.locks, default 2000), leaving others idle. With 300 consumers and 5ms processing time, default config yields only 4800 msg/s instead of the theoretical 60K msg/s. The fix is to set max.poll.records = group.share.partition.max.record.locks / consumers_per_partition, then reduce further to account for timing variance. With max.poll.records=30 and 312 consumers across 6 partitions, the benchmark achieves the target 60K msg/s. The post also warns about an 'accidental fair-sharing regime' that can mask the misconfiguration under low load, only to collapse under higher throughput.

Kafka Share Groups and Parallelizing Consumption — Part 1: Tuning max.poll.records — Jack Vanlightly

Two regimes: Accidental fair-sharing vs greedy-capture

Testing the stability of accidental fair-sharing