Kafka producer

A producer sends events to a topic.

🔎 Unravel this example Prerequisites (2)

Learn these first

Kafka topics, partitions and events Serializers and deserializers

At a glance

Mental model

A producer is a courier who collects parcels at a counter, packs several into one van trip (a batch) and gets a signature on delivery.

What happens, in order

1producer.send(record) does not write to the network immediatelyIt serializes the key and value, chooses a partition, and appends the record to an in-memory batch for that partition. A background sender thread ships full or old-enough batches to the broker.
2send returns a FutureCalling .get() blocks until the broker answers; passing a callback gets the result asynchronously. If you ignore both, a failed send is silently lost.
3acks controls durability: 0 fire and forget, 1 the leader wrote it, all every in-sync replica wrote itFor data you cannot lose, use all, ideally with idempotence enabled.
4flush() pushes everything buffered, and close() flushes and releases resourcesSkipping them on shutdown loses the tail of the buffer.

Remember

Retries can reorder or duplicate events: if batch 1 fails and batch 2 succeeds, a retry of batch 1 lands after batch 2
Tuning knobs: linger.ms (how long to wait to fill a batch), batch.size, compression.type and max.in.flight.requests.per.connection
The client keeps a RecordAccumulator (a deque of batches per partition) and a Sender thread
The default partitioner hashes the key with murmur2 modulo the partition count

Watch out

Ignoring the send result, so failures go unnoticedCreating a new producer per requestForgetting flush() or close() before the program exitsAssuming acks=1 means the data is safe from a broker crash

Example (highlighted lines explain it)

producer.send(new ProducerRecord<>("orders", key, value), (meta, ex) -> {  if (ex != null) log.error("send failed", ex);});

line 1 uses ProducerRecord, the core of Kafka producer

Read the full explanation (about 2 min)

A producer sends events to a topic. It batches them in memory, picks a partition per event and waits for the broker to acknowledge according to the acks setting.

A mental model

A producer is a courier who collects parcels at a counter, packs several into one van trip (a batch) and gets a signature on delivery. Waiting for more parcels (linger) makes trips fuller but slower to leave.

How it works

producer.send(record) does not write to the network immediately. It serializes the key and value, chooses a partition, and appends the record to an in-memory batch for that partition. A background sender thread ships full or old-enough batches to the broker.

send returns a Future. Calling .get() blocks until the broker answers; passing a callback gets the result asynchronously. If you ignore both, a failed send is silently lost.

acks controls durability: 0 fire and forget, 1 the leader wrote it, all every in-sync replica wrote it. For data you cannot lose, use all, ideally with idempotence enabled.

flush() pushes everything buffered, and close() flushes and releases resources. Skipping them on shutdown loses the tail of the buffer.

Going deeper

Retries can reorder or duplicate events: if batch 1 fails and batch 2 succeeds, a retry of batch 1 lands after batch 2. enable.idempotence=true (the default in recent clients) gives each producer an id and sequence numbers so the broker drops duplicates and keeps order within a partition.

Tuning knobs: linger.ms (how long to wait to fill a batch), batch.size, compression.type and max.in.flight.requests.per.connection. More batching means higher throughput and higher latency.

A KafkaProducer is thread safe and meant to be shared. Creating one per message is a classic performance bug, because each owns buffers, threads and connections.

Under the hood

The client keeps a RecordAccumulator (a deque of batches per partition) and a Sender thread. send only touches the accumulator, which is why it is fast and why buffer.memory and max.block.ms matter: when the buffer is full, send blocks.

The default partitioner hashes the key with murmur2 modulo the partition count. For records without a key the modern client uses a sticky strategy, filling one partition's batch before switching, to build bigger batches.

Common mistakes

Ignoring the send result, so failures go unnoticed.
Creating a new producer per request.
Forgetting flush() or close() before the program exits.
Assuming acks=1 means the data is safe from a broker crash.

How Unravel recognises it

When you paste code, Unravel looks for these signs of kafka producer. Spotting them yourself is a good habit when you read unfamiliar code:

Types: KafkaProducer ProducerRecord KafkaTemplate ProducerConfig
Created with new: ProducerRecord KafkaProducer

Part of a learning path

Kafka for Java developers · previous: Serializers and deserializers · next: Kafka consumer and the poll loop
Kafka beyond the basics · next: Kafka idempotent producer and transactions

Related concepts

Kafka topics, partitions and eventsKafka Serializers and deserializersKafka Kafka consumer and the poll loopKafka

Builds on this

Kafka idempotent producer and transactionsKafka

Frequently asked questions

What is Kafka producer?

A producer sends events to a topic. It batches them in memory, picks a partition per event and waits for the broker to acknowledge according to the acks setting.

What are the most common mistakes with Kafka producer?

Ignoring the send result, so failures go unnoticed. Creating a new producer per request. Forgetting flush() or close() before the program exits. Assuming acks=1 means the data is safe from a broker crash.

How can I see Kafka producer in real code?

Paste your own Java into Unravel and it will point out where kafka producer appears, explain how it works at the detail level you choose, and draw the flow of the code around it.