Unravel › Java concepts › Messaging and Kafka
Consumer groups
A consumer group is a set of consumers that share one job.
Learn these first
At a glance
Think of a checkout area with several cashiers (consumers) and several lanes (partitions).
- 1Every consumer sets a group.idAll consumers with the same id form one group. Kafka splits the topic's partitions among them: with 6 partitions and 3 consumers, each gets 2.
- 2Different groups are independentA billing group and an analytics group both read every event of orders, each with its own position. This is how Kafka does publish/subscribe and queueing in one model.
- 3If a consumer crashes or a new one joins, the group rebalances: partitions are redistributedMembers that start with more consumers than partitions leave the extras idle.
- 4The group's progress is stored in Kafka itself, in an internal topic called __consumer_offsets, keyed by group, topic and partition
- Scaling rule: throughput scales with consumers up to the partition count and no further
- Reusing a group.id between two different applications makes them steal each other's partitions and offsets
- One broker acts as the group coordinator for each group
// Three instances of this service share the "billing" group:// Kafka splits the topic's partitions between them, so each order is handled once.Properties props = new Properties();props.put("bootstrap.servers", "localhost:9092");props.put("group.id", "billing");props.put("key.deserializer", StringDeserializer.class.getName());props.put("value.deserializer", StringDeserializer.class.getName());KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);consumer.subscribe(List.of("orders"));- line 3 Three instances of this service share the "billing" group: Kafka splits the topic's partitions between them, so each order is handled once.
- line 8 uses KafkaConsumer, the core of Consumer groups
- line 9 uses subscribe, the core of Consumer groups
Read the full explanation (about 2 min)
A consumer group is a set of consumers that share one job. Kafka gives each partition of the topic to exactly one member of the group, so the group reads every event once in total.
A mental model
Think of a checkout area with several cashiers (consumers) and several lanes (partitions). A lane is served by one cashier at a time, never two. If a cashier leaves, their lanes go to the others. Two different shops (groups) can each serve every customer independently.
How it works
Every consumer sets a group.id. All consumers with the same id form one group. Kafka splits the topic's partitions among them: with 6 partitions and 3 consumers, each gets 2.
Different groups are independent. A billing group and an analytics group both read every event of orders, each with its own position. This is how Kafka does publish/subscribe and queueing in one model.
If a consumer crashes or a new one joins, the group rebalances: partitions are redistributed. Members that start with more consumers than partitions leave the extras idle.
The group's progress is stored in Kafka itself, in an internal topic called __consumer_offsets, keyed by group, topic and partition.
Going deeper
Scaling rule: throughput scales with consumers up to the partition count and no further. If a group cannot keep up and already has one consumer per partition, the options are faster processing, batching, or more partitions (planned ahead, because it changes key placement).
Reusing a group.id between two different applications makes them steal each other's partitions and offsets. Changing the id of an existing application makes it start fresh (depending on auto.offset.reset), which can mean reprocessing a whole topic or skipping to the end.
Static membership (group.instance.id) lets a restarting pod rejoin without triggering a rebalance, which matters on Kubernetes where rolling restarts are normal.
Under the hood
One broker acts as the group coordinator for each group. Members send JoinGroup; the coordinator picks a leader consumer, which runs the partition assignor (range, round-robin, sticky or cooperative-sticky) and returns the plan via SyncGroup. Heartbeats keep membership alive, and the session timeout (45 seconds by default in recent versions) is how a dead member is detected.
Common mistakes
- More consumers than partitions and wondering why some do nothing.
- Two different services sharing a group.id by copy and paste.
- Expecting messages to be delivered to every consumer in the same group (that needs different groups).
How Unravel recognises it
When you paste code, Unravel looks for these signs of consumer groups. Spotting them yourself is a good habit when you read unfamiliar code:
- Annotations:
@KafkaListener - Types:
KafkaConsumer - Method calls:
subscribe() - Configuration keys:
group.idgroup.instance.id - Constants:
ConsumerConfig.GROUP_ID_CONFIG
Part of a learning path
- Kafka for Java developers · previous: Kafka consumer and the poll loop · next: Offsets and commits
Related concepts
Builds on this
Frequently asked questions
What is Consumer groups?
A consumer group is a set of consumers that share one job. Kafka gives each partition of the topic to exactly one member of the group, so the group reads every event once in total.
What are the most common mistakes with Consumer groups?
More consumers than partitions and wondering why some do nothing. Two different services sharing a group.id by copy and paste. Expecting messages to be delivered to every consumer in the same group (that needs different groups).
How can I see Consumer groups in real code?
Paste your own Java into Unravel and it will point out where consumer groups appears, explain how it works at the detail level you choose, and draw the flow of the code around it.