Kafka topics, partitions and events

Kafka is a durable log of events. Producers append events to a named topic, consumers read them in order, and the topic is split into partitions so many machines can share the work.

A mental model

A topic is a notebook that only ever gets new lines at the bottom. Nobody erases or edits a line. Each reader keeps a bookmark and reads forward at their own speed. Partitions are several parallel notebooks for the same subject, so a crowd can write and read at once.

How it works

An event (also called a record or message) has a key, a value, a timestamp and optional headers. The value is usually JSON or Avro describing something that happened: "order 42 was paid".

Events live in a topic, for example orders. A topic is divided into partitions, numbered 0, 1, 2 and so on. Each partition is an ordered, append-only log, and every event in it gets a sequence number called its offset.

The producer chooses the partition. If the event has a key, Kafka hashes the key, so all events with the same key (say the same customer id) go to the same partition and stay in order. Without a key, events are spread across partitions.

Events are not deleted when read. Kafka keeps them for a retention period (7 days by default) or until a size limit, so a new consumer can start from the beginning, and a broken one can read again.

Going deeper

Order is guaranteed only inside one partition, never across a whole topic. If two events must be processed in order, they need the same key. This is the single most common design question with Kafka.

The partition count caps parallelism: a consumer group can have at most one active consumer per partition. Ten partitions means at most ten consumers working at once, and more consumers just sit idle. Adding partitions later changes which partition a key hashes to, which breaks per-key ordering for existing keys.

Each partition is replicated to other brokers (the replication factor, commonly 3). One replica is the leader and takes all reads and writes; followers copy it. If the leader dies, an in-sync follower takes over, so no acknowledged data is lost when the producer is configured with acks=all.

Under the hood

On disk a partition is a directory of segment files (each a few hundred MB by default) with companion index files that map offsets to file positions. Writes are sequential appends, which disks and the page cache handle extremely well, and that is the core reason Kafka is fast.

Consumers fetch by asking the broker for "records from offset N". The broker can hand data from the page cache straight to the network socket with zero-copy (sendfile), so reading does not burn CPU copying bytes through the JVM.

Common mistakes

• Expecting global ordering across partitions.
• Choosing a key with a few hot values, so one partition gets all the traffic.
• Treating a topic like a queue where a read removes the message.