The payment applied twice after a rebalance
A hard Java puzzle in the messaging and kafka mistakes family. One line below contains a real bug. Try to spot it first, then read why it fails and how to fix it.
Highlighted: line 12. Play this one as a puzzle without the answer.
What is wrong
Kafka delivers at least once. After a crash or rebalance records between the last commit and the failure are delivered again, so the same payment is added twice. Without a record of processed event ids the handler is not idempotent.
The fix
How to avoid it next time
Store the event id with the result in the same transaction, or make the operation naturally idempotent.
Want the background? Read Idempotency and duplicate handling in the Java concepts library.
About messaging and kafka mistakes
Message handling combines retries, offsets and transactions, and the order of those steps decides whether a failure loses data or processes it twice.
In code review: Trace what happens if the process dies after each line. Commit an offset only after the work is done, and make handlers idempotent.
More puzzles in this family
Frequently asked questions
What is the bug in "The payment applied twice after a rebalance"?
Kafka delivers at least once. After a crash or rebalance records between the last commit and the failure are delivered again, so the same payment is added twice. Without a record of processed event ids the handler is not idempotent. The faulty line is line 12: accounts.addToBalance(e.accountId(), e.amountCents());
How do I fix it?
Replace it with: if (processed.insertIfAbsent(e.eventId())) { accounts.addToBalance(e.accountId(), e.amountCents()); } // in one transaction. Store the event id with the result in the same transaction, or make the operation naturally idempotent.
How do I catch messaging and kafka mistakes in code review?
Trace what happens if the process dies after each line. Commit an offset only after the work is done, and make handlers idempotent.
Is there a new puzzle every day?
Yes. The daily Spot the Bug puzzle picks one of these snippets for each UTC day, with the difficulty set by the weekday. Your streak is stored only in your browser.