Unravel › Java concepts › Messaging and Kafka
Kafka consumer and the poll loop
A consumer reads events from a topic in a loop.
Learn these first
At a glance
A consumer is a reader with a bookmark.
- 1The consumer first subscribes to topicsIt then calls poll(timeout) in a loop. Each call returns a batch of records (possibly empty) from the partitions assigned to this consumer.
- 2Your code loops over the batch, handles each record, and the consumer remembers the positionWhere that position is stored (the commit) decides what happens after a crash or restart.
- 3poll is also how the consumer proves it is aliveIf you do not call it again within max.poll.interval.ms (5 minutes by default), the broker assumes the consumer is stuck and gives its partitions to someone else.
- 4A KafkaConsumer is not thread safeOne thread owns it. To scale, run more consumers (in the same group), not more threads on one consumer.
- Batch size is bounded by max.poll.records (500 by default)
- Typical shutdown: another thread calls consumer.wakeup(), which makes poll throw WakeupException; the loop catches it, commits what is finished and calls close() so the group learns immediately that this member left
- Under poll the client sends Fetch requests to the leader broker of every assigned partition and keeps prefetched data in memory, so the next poll often returns without waiting on the network
- The consumer's position per partition is the offset of the next record to return
consumer.subscribe(List.of("orders"));while (running) { ConsumerRecords<String, String> batch = consumer.poll(Duration.ofMillis(500)); for (var r : batch) handle(r); consumer.commitSync();}- line 1 uses subscribe, the core of Kafka consumer and the poll loop
- line 3 uses ConsumerRecord, the core of Kafka consumer and the poll loop
Read the full explanation (about 2 min)
A consumer reads events from a topic in a loop. Each poll call asks the broker for the next batch after the last position, and your code processes the batch and then reports how far it got.
A mental model
A consumer is a reader with a bookmark. poll means "give me the next pages after my bookmark". Reading does not remove pages. The bookmark (the offset) is what moves forward once you say you have finished.
How it works
The consumer first subscribes to topics. It then calls poll(timeout) in a loop. Each call returns a batch of records (possibly empty) from the partitions assigned to this consumer.
Your code loops over the batch, handles each record, and the consumer remembers the position. Where that position is stored (the commit) decides what happens after a crash or restart.
poll is also how the consumer proves it is alive. If you do not call it again within max.poll.interval.ms (5 minutes by default), the broker assumes the consumer is stuck and gives its partitions to someone else.
A KafkaConsumer is not thread safe. One thread owns it. To scale, run more consumers (in the same group), not more threads on one consumer.
Going deeper
Batch size is bounded by max.poll.records (500 by default). If processing a batch takes longer than max.poll.interval.ms, the consumer is kicked out of the group, the partitions rebalance and the same records get processed again by another consumer. Slow processing therefore causes duplicates, not just delay.
Typical shutdown: another thread calls consumer.wakeup(), which makes poll throw WakeupException; the loop catches it, commits what is finished and calls close() so the group learns immediately that this member left.
Error handling inside the loop is a design decision: skipping a "poison" record loses it, retrying forever blocks the partition. Production code usually retries a few times, then sends the record to a dead-letter topic.
Under the hood
Under poll the client sends Fetch requests to the leader broker of every assigned partition and keeps prefetched data in memory, so the next poll often returns without waiting on the network. A separate heartbeat thread tells the group coordinator that the consumer is alive.
The consumer's position per partition is the offset of the next record to return. It is advanced in memory by poll, and only becomes durable when committed.
Common mistakes
- Doing slow work inside the poll loop and getting kicked from the group.
- Sharing one KafkaConsumer between threads.
- Committing before the work is done (lost events) or never committing (reprocessing everything).
- Forgetting to close the consumer.
How Unravel recognises it
When you paste code, Unravel looks for these signs of kafka consumer and the poll loop. Spotting them yourself is a good habit when you read unfamiliar code:
- Types:
KafkaConsumerConsumerRecordConsumerRecordsConsumerConfig - Method calls:
poll()subscribe()records() - Created with new:
KafkaConsumer
Part of a learning path
- Kafka for Java developers · previous: Kafka producer · next: Consumer groups
Related concepts
Builds on this
Frequently asked questions
What is Kafka consumer and the poll loop?
A consumer reads events from a topic in a loop. Each poll call asks the broker for the next batch after the last position, and your code processes the batch and then reports how far it got.
What are the most common mistakes with Kafka consumer and the poll loop?
Doing slow work inside the poll loop and getting kicked from the group. Sharing one KafkaConsumer between threads. Committing before the work is done (lost events) or never committing (reprocessing everything). Forgetting to close the consumer.
How can I see Kafka consumer and the poll loop in real code?
Paste your own Java into Unravel and it will point out where kafka consumer and the poll loop appears, explain how it works at the detail level you choose, and draw the flow of the code around it.