BackendBench
Coffee

Unravel โ€บ Java concepts โ€บ Messaging and Kafka

Kafka

Schema Registry, Avro and Protobuf

A schema registry stores the structure of messages (Avro, Protobuf or JSON Schema) and checks that producers and consumers change it compatibly.

๐Ÿ”Ž Unravel this examplePrerequisites (1)

Learn these first

Serializers and deserializers

At a glance

Mental model

A shared dictionary.

What happens, in order
  1. 1The serializer registers the schema (once) with the registry, gets an id, and writes magic byte + schema id + binary payloadThe deserializer reads the id, fetches the schema (cached) and decodes the payload.
  2. 2Producers and consumers use KafkaAvroSerializer / KafkaAvroDeserializer and set schema.registry.urlWith specific.avro.reader=true you get generated classes, otherwise a GenericRecord (a map-like object).
  3. 3Each topic value has a subject (orders-value) with a version history and a compatibility rule: BACKWARD (new readers read old data), FORWARD, FULL or NONE
  4. 4Safe evolution: add a field with a default, remove a field that had a defaultRenaming or changing a type is incompatible and must go through a new topic or aliases.
Remember
Watch out
Auto-registering schemas from production codeAdding a required field without a defaultReusing field numbers or names with different meaningsDifferent schema subject naming strategies on producer and consumer
Example (highlighted lines explain it)
Properties props = new Properties();props.put("bootstrap.servers", "localhost:9092");props.put("key.serializer", StringSerializer.class.getName());props.put("value.serializer", KafkaAvroSerializer.class.getName());props.put("schema.registry.url", "http://registry:8081");props.put("auto.register.schemas", "false");Producer<String, GenericRecord> producer = new KafkaProducer<>(props);
  1. line 4 uses KafkaAvroSerializer, the core of Schema Registry, Avro and Protobuf
  2. line 7 uses GenericRecord, the core of Schema Registry, Avro and Protobuf
Read the full explanation (about 2 min)

A schema registry stores the structure of messages (Avro, Protobuf or JSON Schema) and checks that producers and consumers change it compatibly. Messages then carry only a small schema id instead of field names.

A mental model

A shared dictionary. Each message carries just a page number, and anyone who reads it looks up the page. The librarian refuses a new edition that would make old readers unable to understand it.

How it works

The serializer registers the schema (once) with the registry, gets an id, and writes magic byte + schema id + binary payload. The deserializer reads the id, fetches the schema (cached) and decodes the payload.

Producers and consumers use KafkaAvroSerializer / KafkaAvroDeserializer and set schema.registry.url. With specific.avro.reader=true you get generated classes, otherwise a GenericRecord (a map-like object).

Each topic value has a subject (orders-value) with a version history and a compatibility rule: BACKWARD (new readers read old data), FORWARD, FULL or NONE.

Safe evolution: add a field with a default, remove a field that had a default. Renaming or changing a type is incompatible and must go through a new topic or aliases.

Going deeper

Turn off auto.register.schemas in production and register schemas from CI. Otherwise any code deployment can change a shared contract at runtime.

Compatibility is checked against the registry, but consumers still need the new code to use new fields. Upgrade consumers first for backward-compatible changes (new readers, old data) and producers first for forward ones.

Without a registry, JSON on Kafka drifts silently: a renamed field becomes null in every consumer, which is found in production.

Under the hood

Avro encodes data without field names, in the exact order of the writer schema, which is why the reader needs the writer schema to decode: it resolves the writer schema against its own reader schema field by field (skipping, defaulting, promoting). Protobuf identifies fields by numbers, which provides similar evolution rules.

Common mistakes

  • Auto-registering schemas from production code.
  • Adding a required field without a default.
  • Reusing field numbers or names with different meanings.
  • Different schema subject naming strategies on producer and consumer.

How Unravel recognises it

When you paste code, Unravel looks for these signs of schema registry, avro and protobuf. Spotting them yourself is a good habit when you read unfamiliar code:

Part of a learning path

Related concepts

Serializers and deserializersKafkaKafka Connect connectorsKafka

Frequently asked questions

What is Schema Registry, Avro and Protobuf?

A schema registry stores the structure of messages (Avro, Protobuf or JSON Schema) and checks that producers and consumers change it compatibly. Messages then carry only a small schema id instead of field names.

What are the most common mistakes with Schema Registry, Avro and Protobuf?

Auto-registering schemas from production code. Adding a required field without a default. Reusing field numbers or names with different meanings. Different schema subject naming strategies on producer and consumer.

How can I see Schema Registry, Avro and Protobuf in real code?

Paste your own Java into Unravel and it will point out where schema registry, avro and protobuf appears, explain how it works at the detail level you choose, and draw the flow of the code around it.