Unravel โบ Java concepts โบ Messaging and Kafka
Retries and dead-letter topics
When a record cannot be processed, a Kafka consumer must not block forever and must not silently drop it.
Learn these first
At a glance
A post office sorting line: a parcel with an unreadable address is tried a couple of times, then put in a "return to sender" bin where a person can look at it later, instead of stopping the whole conveyor belt.
- 1In Spring Kafka a DefaultErrorHandler decides what to do when your listener throwsWith a BackOff such as FixedBackOff(1000, 3) it retries the same record three times, one second apart, before giving up.
- 2A DeadLetterPublishingRecoverer is the "giving up" action: it sends the failed record, with headers describing the exception, to a topic named <original>.DLT by default
- 3@RetryableTopic does the same with retry topics: failed records move through topic-retry-0, topic-retry-1 and finally topic-dlt, so a slow retry does not hold up the records behind it
- 4A person or a separate consumer reads the dead-letter topic, fixes the cause or the data, and republishes the records if they should be processed
- Retrying inside the listener blocks that partition: nothing behind the failing record is processed while it retries
- Not every exception deserves a retry
- The error handler catches the exception from the listener container, seeks the consumer back to the failed record (so the next poll returns it again) and counts attempts per record in memory
@Beanpublic DefaultErrorHandler errorHandler(KafkaTemplate<Object, Object> template) { DeadLetterPublishingRecoverer recoverer = new DeadLetterPublishingRecoverer(template); DefaultErrorHandler handler = new DefaultErrorHandler(recoverer, new FixedBackOff(1000L, 3)); handler.addNotRetryableExceptions(IllegalArgumentException.class); return handler;}- line 2 uses DefaultErrorHandler, the core of Retries and dead-letter topics
- line 3 uses DeadLetterPublishingRecoverer, the core of Retries and dead-letter topics
- line 4 uses DefaultErrorHandler, the core of Retries and dead-letter topics
Read the full explanation (about 2 min)
When a record cannot be processed, a Kafka consumer must not block forever and must not silently drop it. The usual answer is a few retries with a delay, then publishing the record to a separate dead-letter topic for later inspection.
A mental model
A post office sorting line: a parcel with an unreadable address is tried a couple of times, then put in a "return to sender" bin where a person can look at it later, instead of stopping the whole conveyor belt.
How it works
In Spring Kafka a DefaultErrorHandler decides what to do when your listener throws. With a BackOff such as FixedBackOff(1000, 3) it retries the same record three times, one second apart, before giving up.
A DeadLetterPublishingRecoverer is the "giving up" action: it sends the failed record, with headers describing the exception, to a topic named <original>.DLT by default.
@RetryableTopic does the same with retry topics: failed records move through topic-retry-0, topic-retry-1 and finally topic-dlt, so a slow retry does not hold up the records behind it.
A person or a separate consumer reads the dead-letter topic, fixes the cause or the data, and republishes the records if they should be processed.
Going deeper
Retrying inside the listener blocks that partition: nothing behind the failing record is processed while it retries. For failures that need minutes to clear, retry topics are better than a long in-place back-off.
Not every exception deserves a retry. A malformed payload will fail the same way forever, so classify it as not retryable and send it straight to the dead-letter topic; retry only transient problems such as a timeout.
A record that cannot even be deserialized fails before your code runs. Wrap the deserializer in ErrorHandlingDeserializer so the failure becomes a normal error that the handler can route.
Watch the dead-letter topic: a growing DLT is an incident, not a feature. Alert on it, keep the original headers, and make the handler idempotent so a replay does not duplicate work.
Under the hood
The error handler catches the exception from the listener container, seeks the consumer back to the failed record (so the next poll returns it again) and counts attempts per record in memory. When attempts run out it calls the recoverer, then commits past the record.
Common mistakes
- Catching exceptions inside the listener so the error handler never sees them and nothing is retried or dead-lettered.
- Retrying non-transient errors again and again.
- No monitoring of the dead-letter topic.
- Long in-place retries that stall a whole partition.
How Unravel recognises it
When you paste code, Unravel looks for these signs of retries and dead-letter topics. Spotting them yourself is a good habit when you read unfamiliar code:
- Annotations:
@RetryableTopic@DltHandler - Types:
DeadLetterPublishingRecovererDefaultErrorHandlerFixedBackOffExponentialBackOffSeekToCurrentErrorHandlerErrorHandlingDeserializerCommonErrorHandler
Part of a learning path
- Kafka for Java developers ยท previous: Spring Kafka @KafkaListener
Related concepts
Frequently asked questions
What is Retries and dead-letter topics?
When a record cannot be processed, a Kafka consumer must not block forever and must not silently drop it. The usual answer is a few retries with a delay, then publishing the record to a separate dead-letter topic for later inspection.
What are the most common mistakes with Retries and dead-letter topics?
Catching exceptions inside the listener so the error handler never sees them and nothing is retried or dead-lettered. Retrying non-transient errors again and again. No monitoring of the dead-letter topic. Long in-place retries that stall a whole partition.
How can I see Retries and dead-letter topics in real code?
Paste your own Java into Unravel and it will point out where retries and dead-letter topics appears, explain how it works at the detail level you choose, and draw the flow of the code around it.