BackendBench
Coffee

Unravel › Java concepts › Messaging and Kafka

Kafka

Kafka idempotent producer and transactions

Two features make Kafka writes safer: the idempotent producer removes duplicates caused by retries, and transactions write to several partitions and commit consumer offsets all-or-nothing.

🔎 Unravel this examplePrerequisites (2)

Learn these first

Kafka producerOffsets and commits

At a glance

Mental model

A bank transfer slip with a serial number (idempotence: the clerk ignores a second copy) inside a sealed envelope that is only opened by the reader if the stamp says committed (transactions).

What happens, in order
  1. 1enable.idempotence=true (default since Kafka 3) gives each producer an id and each message a sequence number per partitionThe broker drops a retried duplicate it has already stored.
  2. 2For transactions, set a transactional.id, call initTransactions() once, and wrap the work in beginTransaction(), sends, sendOffsetsToTransaction(offsets, groupMetadata), commitTransaction(); on error call abortTransaction()
  3. 3Consumers must use isolation.level=read_committed to skip records of aborted and still-open transactionsThe default read_uncommitted sees everything.
  4. 4The classic consume, process, produce loop becomes atomic: the output records and the input offsets commit together, so a crash never leaves output without matching offsets or the other way round
Remember
Watch out
Assuming exactly-once covers database or HTTP side effectsConsumers left at read_uncommittedOne transaction per recordSharing a transactional.id across instances
Example (highlighted lines explain it)
producer.initTransactions();while (true) {    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));    if (records.isEmpty()) continue;    producer.beginTransaction();    try {        for (ConsumerRecord<String, String> r : records) {            producer.send(new ProducerRecord<>("enriched", r.key(), enrich(r.value())));        }        producer.sendOffsetsToTransaction(offsetsOf(records), consumer.groupMetadata());        producer.commitTransaction();    } catch (Exception e) {        producer.abortTransaction();    }}
  1. line 1 uses initTransactions, the core of Kafka idempotent producer and transactions
  2. line 5 uses beginTransaction, the core of Kafka idempotent producer and transactions
  3. line 10 uses sendOffsetsToTransaction, the core of Kafka idempotent producer and transactions
  4. line 11 uses commitTransaction, the core of Kafka idempotent producer and transactions
  5. line 13 uses abortTransaction, the core of Kafka idempotent producer and transactions
Read the full explanation (about 2 min)

Two features make Kafka writes safer: the idempotent producer removes duplicates caused by retries, and transactions write to several partitions and commit consumer offsets all-or-nothing. Together they give exactly-once processing inside Kafka.

A mental model

A bank transfer slip with a serial number (idempotence: the clerk ignores a second copy) inside a sealed envelope that is only opened by the reader if the stamp says committed (transactions).

How it works

enable.idempotence=true (default since Kafka 3) gives each producer an id and each message a sequence number per partition. The broker drops a retried duplicate it has already stored.

For transactions, set a transactional.id, call initTransactions() once, and wrap the work in beginTransaction(), sends, sendOffsetsToTransaction(offsets, groupMetadata), commitTransaction(); on error call abortTransaction().

Consumers must use isolation.level=read_committed to skip records of aborted and still-open transactions. The default read_uncommitted sees everything.

The classic consume, process, produce loop becomes atomic: the output records and the input offsets commit together, so a crash never leaves output without matching offsets or the other way round.

Going deeper

Exactly-once covers Kafka to Kafka. A side effect outside Kafka (a database write, an HTTP call, an email) is not part of the transaction and can repeat, so those still need idempotent handling.

The same transactional.id used by two live producers fences the older one (ProducerFencedException). That is intended: it stops zombie instances after a rebalance. Handle it by closing the producer, not retrying.

Transactions add latency (commit markers) and are best used with batches per poll, not one transaction per record.

Under the hood

A transaction coordinator broker keeps a transaction log. On commit it writes commit markers into every partition touched, and read_committed consumers only deliver records below the last stable offset, which is held back by any open transaction.

Common mistakes

  • Assuming exactly-once covers database or HTTP side effects.
  • Consumers left at read_uncommitted.
  • One transaction per record.
  • Sharing a transactional.id across instances.

How Unravel recognises it

When you paste code, Unravel looks for these signs of kafka idempotent producer and transactions. Spotting them yourself is a good habit when you read unfamiliar code:

Part of a learning path

Related concepts

Kafka StreamsKafkaIdempotency and duplicate handlingI/O and data

Frequently asked questions

What is Kafka idempotent producer and transactions?

Two features make Kafka writes safer: the idempotent producer removes duplicates caused by retries, and transactions write to several partitions and commit consumer offsets all-or-nothing. Together they give exactly-once processing inside Kafka.

What are the most common mistakes with Kafka idempotent producer and transactions?

Assuming exactly-once covers database or HTTP side effects. Consumers left at read_uncommitted. One transaction per record. Sharing a transactional.id across instances.

How can I see Kafka idempotent producer and transactions in real code?

Paste your own Java into Unravel and it will point out where kafka idempotent producer and transactions appears, explain how it works at the detail level you choose, and draw the flow of the code around it.