Idempotency and duplicate handling

Networks and message brokers repeat things: a client retries, a broker redelivers.

🔎 Unravel this example

At a glance

Mental model

A lift button.

What happens, in order

1Some operations are naturally idempotent: set a status to SHIPPED, PUT a whole resource, delete by idOthers are not: add 10 to a balance, create an order, send an email.
2To make the rest safe, the caller sends a unique key (Idempotency-Key header, message id)The server stores the key with the result in a table with a unique constraint; a repeated key returns the stored result without redoing the work.
3In consumers, record the message id in a processed_messages table in the same transaction as the business changeA second delivery hits the unique constraint and is skipped.
4Upserts (INSERT ..ON CONFLICT DO NOTHING, MERGE) and conditional updates (WHERE status = ...) give idempotency at the database level.

Remember

Check-then-insert (if (!exists(id)) insert) is not safe under concurrency: two threads both see "not exists"
The key must be stored in the same transaction as the effect
Idempotency turns at-least-once delivery into effectively-once processing

Watch out

Check-then-act without a unique constraintRetrying non-idempotent POSTs blindlyGenerating a new key per retryStoring the key outside the business transaction

Example (highlighted lines explain it)

@Transactionalpublic void handle(PaymentRequested event) {    try {        processed.save(new ProcessedMessage(event.messageId()));    } catch (DataIntegrityViolationException duplicate) {        return; // already handled: the unique key on message_id rejected the second insert    }    payments.charge(event.orderId(), event.amount());}

line 4 uses ProcessedMessage, the core of Idempotency and duplicate handling
line 6 already handled: the unique key on message_id rejected the second insert

Read the full explanation (about 2 min)

Networks and message brokers repeat things: a client retries, a broker redelivers. An idempotent operation gives the same result no matter how many times it runs, so duplicates are harmless.

A mental model

A lift button. Pressing it five times still calls one lift. A vending machine where each press dispenses another can is the opposite, and a retry there costs money.

How it works

Some operations are naturally idempotent: set a status to SHIPPED, PUT a whole resource, delete by id. Others are not: add 10 to a balance, create an order, send an email.

To make the rest safe, the caller sends a unique key (Idempotency-Key header, message id). The server stores the key with the result in a table with a unique constraint; a repeated key returns the stored result without redoing the work.

In consumers, record the message id in a processed_messages table in the same transaction as the business change. A second delivery hits the unique constraint and is skipped.

Upserts (INSERT ... ON CONFLICT DO NOTHING, MERGE) and conditional updates (WHERE status = ...) give idempotency at the database level.

Going deeper

Check-then-insert (if (!exists(id)) insert) is not safe under concurrency: two threads both see "not exists". Rely on the unique constraint and handle the violation.

The key must be stored in the same transaction as the effect. Recording it before the work risks losing the work; after the work risks repeating it.

Keep keys for at least as long as retries can happen (hours or days), then expire them. Keys should be generated by the client for a logical operation, not per attempt.

Under the hood

Idempotency turns at-least-once delivery into effectively-once processing. The unique index is the arbiter: the database serialises concurrent inserts of the same key, so exactly one transaction wins and the others fail or wait and then read the winner stored response.

Common mistakes

Check-then-act without a unique constraint.
Retrying non-idempotent POSTs blindly.
Generating a new key per retry.
Storing the key outside the business transaction.

How Unravel recognises it

When you paste code, Unravel looks for these signs of idempotency and duplicate handling. Spotting them yourself is a good habit when you read unfamiliar code:

Annotations: @Idempotent
Types: IdempotencyKey IdempotencyRecord IdempotencyService IdempotentReceiver IdempotentRepository ProcessedMessage ProcessedEvent ProcessedMessageRepository DuplicateKeyException IdempotentConsumer
Method calls: existsByIdempotencyKey() existsByMessageId() existsByEventId() alreadyProcessed() isDuplicate() markProcessed()
Configuration keys: Idempotency-Key idempotency-key

Part of a learning path

Reliable messaging and distributed patterns · previous: AWS SQS and SNS messaging · next: Transactional outbox pattern
Structuring an enterprise service · previous: @Transactional · next: Transactional outbox pattern

Related concepts

Transactional outbox patternI/O and data Retry and circuit breakerI/O and data

Frequently asked questions

What is Idempotency and duplicate handling?

Networks and message brokers repeat things: a client retries, a broker redelivers. An idempotent operation gives the same result no matter how many times it runs, so duplicates are harmless.

What are the most common mistakes with Idempotency and duplicate handling?

Check-then-act without a unique constraint. Retrying non-idempotent POSTs blindly. Generating a new key per retry. Storing the key outside the business transaction.

How can I see Idempotency and duplicate handling in real code?

Paste your own Java into Unravel and it will point out where idempotency and duplicate handling appears, explain how it works at the detail level you choose, and draw the flow of the code around it.