Offsets and commits

An offset is a record's position in a partition.

🔎 Unravel this example Prerequisites (1)

Learn these first

Kafka consumer and the poll loop

At a glance

Mental model

The offset is a bookmark.

What happens, in order

1Every record has an offset: 0, 1, 2, ..within its partition. The committed offset for a group and partition is the position the group will resume from, meaning the offset of the next record to read.
2With enable.auto.commit=true the client commits for you every few seconds (auto.commit.interval.ms), whether or not your code has finished the recordsIt is easy but can lose work: records polled and committed but not yet processed vanish if the app crashes.
3commitSync() commits the latest polled position and waits, retrying on recoverable errorscommitAsync() does not wait, which is faster but can fail silently, so it needs a callback or a final commitSync on shutdown.
4auto.offset.reset says where to start when the group has no committed offset (or it expired): latest means only new events, earliest means from the beginning, none means fail

Remember

Delivery semantics follow from the order of process and commit
seek lets you reposition on purpose (replay from a time, skip a poison record), which is why reprocessing in Kafka is a feature, not an incident
Commits are ordinary records written to the compacted internal topic __consumer_offsets, keyed by (group, topic, partition)

Watch out

Auto-commit combined with slow processing, losing records on a crashCommitting the offset of the record just processed instead of the next one (off by one: one record repeats)Using commitAsync with no callback and no final commitSyncNot making handlers idempotent although at-least-once delivery repeats records

Example (highlighted lines explain it)

for (var r : consumer.poll(Duration.ofSeconds(1))) {  process(r);          // do the work first}consumer.commitSync(); // then save progress: at-least-once

line 2 do the work first
line 4 then save progress: at-least-once

Read the full explanation (about 2 min)

An offset is a record's position in a partition. A commit saves "I have processed up to here" so a restarted consumer continues from that point, which decides whether you lose or repeat events after a failure.

A mental model

The offset is a bookmark. Committing is writing the page number on a sticky note somewhere safe. If you write it before reading the page and then faint, you skip the page. If you write it after and faint before writing, you read the page again.

How it works

Every record has an offset: 0, 1, 2, ... within its partition. The committed offset for a group and partition is the position the group will resume from, meaning the offset of the next record to read.

With enable.auto.commit=true the client commits for you every few seconds (auto.commit.interval.ms), whether or not your code has finished the records. It is easy but can lose work: records polled and committed but not yet processed vanish if the app crashes.

commitSync() commits the latest polled position and waits, retrying on recoverable errors. commitAsync() does not wait, which is faster but can fail silently, so it needs a callback or a final commitSync on shutdown.

auto.offset.reset says where to start when the group has no committed offset (or it expired): latest means only new events, earliest means from the beginning, none means fail.

Going deeper

Delivery semantics follow from the order of process and commit. Process, then commit: at-least-once (duplicates possible, so processing must be idempotent). Commit, then process: at-most-once (loss possible). Exactly-once needs transactions or writing the result and the offset atomically in the same store.

seek lets you reposition on purpose (replay from a time, skip a poison record), which is why reprocessing in Kafka is a feature, not an incident.

Committing inside a per-record loop is safe but slow (one round trip per record). The usual compromise is one commit per batch, or per N records, committing the offset of the last processed record plus one.

Under the hood

Commits are ordinary records written to the compacted internal topic __consumer_offsets, keyed by (group, topic, partition). The coordinator keeps the latest value in memory. Offsets for an inactive group expire after offsets.retention.minutes (7 days by default), after which auto.offset.reset applies again.

Common mistakes

Auto-commit combined with slow processing, losing records on a crash.
Committing the offset of the record just processed instead of the next one (off by one: one record repeats).
Using commitAsync with no callback and no final commitSync.
Not making handlers idempotent although at-least-once delivery repeats records.

How Unravel recognises it

When you paste code, Unravel looks for these signs of offsets and commits. Spotting them yourself is a good habit when you read unfamiliar code:

Types: OffsetAndMetadata Acknowledgment
Method calls: commitSync() commitAsync() seek() seekToBeginning() seekToEnd() committed() acknowledge()
Configuration keys: enable.auto.commit auto.offset.reset
Constants: ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG ConsumerConfig.AUTO_OFFSET_RESET_CONFIG

Part of a learning path

Kafka for Java developers · previous: Consumer groups · next: Rebalancing

Related concepts

Kafka consumer and the poll loopKafka Consumer groupsKafka RebalancingKafka

Builds on this

Retries and dead-letter topicsKafka Kafka idempotent producer and transactionsKafka

Frequently asked questions

What is Offsets and commits?

What are the most common mistakes with Offsets and commits?

Auto-commit combined with slow processing, losing records on a crash. Committing the offset of the record just processed instead of the next one (off by one: one record repeats). Using commitAsync with no callback and no final commitSync. Not making handlers idempotent although at-least-once delivery repeats records.

How can I see Offsets and commits in real code?

Paste your own Java into Unravel and it will point out where offsets and commits appears, explain how it works at the detail level you choose, and draw the flow of the code around it.