Serializers and deserializers

Kafka stores only bytes.

🔎 Unravel this example

At a glance

Mental model

Kafka is a postal service that carries sealed boxes.

What happens, in order

1The producer is configured with a key serializer and a value serializer; the consumer with the matching deserializersCommon ones: String, Long, byte[], JSON and Avro or Protobuf with a schema registry.
2The format is a contract between producer and consumer teamsChanging a field in the JSON without care breaks consumers that are still running the old code.
3If the consumer cannot deserialize a record, the failure happens inside poll, before your code sees the record, so the same bad record can make the consumer fail forever (a poison pill)

Remember

Schema evolution is the real problem: Avro or Protobuf with a schema registry lets you add optional fields compatibly and rejects incompatible changes before they reach production
Wrap deserializers with ErrorHandlingDeserializer (Spring Kafka) so a bad record becomes a handled error instead of an endless crash loop
The interfaces are tiny: Serializer<T>.serialize(topic, data) returns byte[] and Deserializer<T>.deserialize(topic, bytes) returns T

Watch out

Producer writes JSON, consumer expects Avro (or the reverse)No plan for bad records, so one poison message blocks a partitionChanging field names without versioning the schema

Example (highlighted lines explain it)

Properties props = new Properties();props.put("key.serializer", StringSerializer.class.getName());props.put("value.serializer", StringSerializer.class.getName());// the value is JSON text; the consumer must use a matching StringDeserializerString json = mapper.writeValueAsString(order);producer.send(new ProducerRecord<>("orders", order.getCustomerId(), json));

line 2 uses StringSerializer, the core of Serializers and deserializers
line 3 uses StringSerializer, the core of Serializers and deserializers
line 5 the value is JSON text; the consumer must use a matching StringDeserializer

Read the full explanation (about 1 min)

Kafka stores only bytes. A serializer turns your key and value objects into bytes when producing, and a deserializer turns them back when consuming; both sides must agree on the format.

A mental model

Kafka is a postal service that carries sealed boxes. The serializer is how you pack the box, the deserializer is how the receiver unpacks it. If they use different packing rules, the receiver gets a mess.

How it works

The producer is configured with a key serializer and a value serializer; the consumer with the matching deserializers. Common ones: String, Long, byte[], JSON and Avro or Protobuf with a schema registry.

The format is a contract between producer and consumer teams. Changing a field in the JSON without care breaks consumers that are still running the old code.

If the consumer cannot deserialize a record, the failure happens inside poll, before your code sees the record, so the same bad record can make the consumer fail forever (a poison pill).

Going deeper

Schema evolution is the real problem: Avro or Protobuf with a schema registry lets you add optional fields compatibly and rejects incompatible changes before they reach production. Plain JSON has no such guard.

Wrap deserializers with ErrorHandlingDeserializer (Spring Kafka) so a bad record becomes a handled error instead of an endless crash loop.

Keys matter: the partition is chosen from the serialized key bytes. Changing key serialization changes partition placement and therefore ordering guarantees.

Under the hood

The interfaces are tiny: Serializer<T>.serialize(topic, data) returns byte[] and Deserializer<T>.deserialize(topic, bytes) returns T. The topic argument allows a serializer to pick a schema per topic, which is how schema registry integration works.

Common mistakes

Producer writes JSON, consumer expects Avro (or the reverse).
No plan for bad records, so one poison message blocks a partition.
Changing field names without versioning the schema.

How Unravel recognises it

When you paste code, Unravel looks for these signs of serializers and deserializers. Spotting them yourself is a good habit when you read unfamiliar code:

Types: StringSerializer StringDeserializer LongSerializer LongDeserializer ByteArraySerializer ByteArrayDeserializer JsonSerializer JsonDeserializer ErrorHandlingDeserializer KafkaAvroSerializer KafkaAvroDeserializer Serializer Deserializer Serde and 1 more
Configuration keys: key.serializer value.serializer key.deserializer value.deserializer
Constants: ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG

Part of a learning path

Kafka for Java developers · previous: Kafka topics, partitions and events · next: Kafka producer

Related concepts

Kafka producerKafka Kafka consumer and the poll loopKafka

Builds on this

Kafka StreamsKafka Schema Registry, Avro and ProtobufKafka

Frequently asked questions

What is Serializers and deserializers?

Kafka stores only bytes. A serializer turns your key and value objects into bytes when producing, and a deserializer turns them back when consuming; both sides must agree on the format.

What are the most common mistakes with Serializers and deserializers?

Producer writes JSON, consumer expects Avro (or the reverse). No plan for bad records, so one poison message blocks a partition. Changing field names without versioning the schema.

How can I see Serializers and deserializers in real code?

Paste your own Java into Unravel and it will point out where serializers and deserializers appears, explain how it works at the detail level you choose, and draw the flow of the code around it.