This post is a little side-quest from my “Kafka Share Groups and Parallelizing Consumption” series.   My “Kafka Share Groups and Parallelizing Consumption” series ( part 1 ,  part 2 ) has been laser focused on how different configurations and behaviors affect parallel consumption in share groups.  So far I’ve shown that you most definitely can hold share groups wrong . You could quite easily and inadvertently create a work queue and with the right combination of things going against you, see a small number of consumers dominate, leaving most consumers starved of messages. All the while lag builds and builds. You need to know the settings and what they do.  But it’s worth asking the question:  is parallelizing consumption what share groups are for?

Jack Vanlightly

A conceptual exploration of where parallelism should live in messaging systems — broker-visible (partitions, consumers, TCP connections) vs client-local (virtual threads, async tasks). Using Kafka Share Groups as context, the post introduces a simple formula for calculating required aggregate parallelism (rate × avg processing time) and argues that for highly parallel workloads, pushing parallelism to the client side is far more resource-efficient than scaling broker-managed units. It also notes the deprecation of the ParallelConsumer library and hints at the need for a new library combining client-side parallelism with Share Groups.

Broker-Visible vs Client-Local Parallelism — Jack Vanlightly

Where should your unit of parallelism live?