The consumer kicked out of its group
A hard Java puzzle in the messaging and kafka mistakes family. One line below contains a real bug. Try to spot it first, then read why it fails and how to fix it.
Highlighted: line 9. Play this one as a puzzle without the answer.
What is wrong
If the time between poll calls exceeds max.poll.interval.ms (5 minutes by default), the broker assumes the consumer is dead and rebalances. With a batch of records at two minutes each the loop takes longer, the commit fails with CommitFailedException and the same records are processed again by someone else.
The fix
How to avoid it next time
Keep the time between polls short. Long work belongs in a separate worker with pause and resume, or smaller batches.
Want the background? Read Kafka consumer and the poll loop in the Java concepts library.
About messaging and kafka mistakes
Message handling combines retries, offsets and transactions, and the order of those steps decides whether a failure loses data or processes it twice.
In code review: Trace what happens if the process dies after each line. Commit an offset only after the work is done, and make handlers idempotent.
More puzzles in this family
Frequently asked questions
What is the bug in "The consumer kicked out of its group"?
If the time between poll calls exceeds max.poll.interval.ms (5 minutes by default), the broker assumes the consumer is dead and rebalances. With a batch of records at two minutes each the loop takes longer, the commit fails with CommitFailedException and the same records are processed again by someone else. The faulty line is line 9: heavyReport(r.value()); // takes about 2 minutes per record
How do I fix it?
Replace it with: Lower max.poll.records, raise max.poll.interval.ms, or hand the work to a bounded executor and pause the partitions until it is done.. Keep the time between polls short. Long work belongs in a separate worker with pause and resume, or smaller batches.
How do I catch messaging and kafka mistakes in code review?
Trace what happens if the process dies after each line. Commit an offset only after the work is done, and make handlers idempotent.
Is there a new puzzle every day?
Yes. The daily Spot the Bug puzzle picks one of these snippets for each UTC day, with the difficulty set by the weekday. Your streak is stored only in your browser.