BackendBench
Coffee

Unravel › Java concepts › Messaging and Kafka

Kafka

Kafka Streams

Kafka Streams is a library for processing topics as continuous streams inside your own application: filter, transform, join and aggregate events, with state kept locally and backed up to Kafka.

🔎 Unravel this examplePrerequisites (3)

Learn these first

Kafka topics, partitions and eventsSerializers and deserializersLambdas and functional interfaces

At a glance

Mental model

A sorting line built from a recipe.

What happens, in order
  1. 1You describe a topology with StreamsBuilder: builder.stream("orders") reads a topic as a KStream (every event is a fact), builder.table("customers") reads it as a KTable (the latest value per key)
  2. 2Stateless steps: filter, map, flatMap, branchStateful steps: groupByKey().count(), aggregate, windowed counts (windowedBy(TimeWindows...)) and joins between streams and tables.
  3. 3The result is written with to("output-topic")The application id doubles as the consumer group, so running more instances automatically splits partitions between them.
  4. 4State lives in local RocksDB stores plus a changelog topic in KafkaAfter a crash or move, another instance rebuilds the store by replaying the changelog.
Remember
Watch out
Joining topics that are not co-partitionedChanging the key and forgetting that a repartition topic costs a round tripIgnoring late and out-of-order events in windowsChanging the topology of a running application with state without resetting it
Example (highlighted lines explain it)
StreamsBuilder builder = new StreamsBuilder();KStream<String, String> orders = builder.stream("orders");orders.filter((k, v) -> v != null && !v.isEmpty())    .groupByKey()    .windowedBy(TimeWindows.ofSizeWithNoGrace(Duration.ofMinutes(5)))    .count()    .toStream()    .map((window, count) -> KeyValue.pair(window.key(), String.valueOf(count)))    .to("orders-per-5-minutes");
  1. line 1 uses StreamsBuilder, the core of Kafka Streams
  2. line 2 uses KStream, the core of Kafka Streams
  3. line 4 uses groupByKey, the core of Kafka Streams
  4. line 5 uses windowedBy, the core of Kafka Streams
  5. line 8 uses KeyValue, the core of Kafka Streams
Read the full explanation (about 2 min)

Kafka Streams is a library for processing topics as continuous streams inside your own application: filter, transform, join and aggregate events, with state kept locally and backed up to Kafka.

A mental model

A sorting line built from a recipe. Events flow past, each station (filter, map, count) does its small job, and the running totals are kept in a notebook at the station that is also copied to a safe in the warehouse in case the station burns down.

How it works

You describe a topology with StreamsBuilder: builder.stream("orders") reads a topic as a KStream (every event is a fact), builder.table("customers") reads it as a KTable (the latest value per key).

Stateless steps: filter, map, flatMap, branch. Stateful steps: groupByKey().count(), aggregate, windowed counts (windowedBy(TimeWindows...)) and joins between streams and tables.

The result is written with to("output-topic"). The application id doubles as the consumer group, so running more instances automatically splits partitions between them.

State lives in local RocksDB stores plus a changelog topic in Kafka. After a crash or move, another instance rebuilds the store by replaying the changelog.

Going deeper

Joins need co-partitioning: both topics must have the same number of partitions and the same key strategy, otherwise matching records land on different instances and never meet.

Time matters. Windows use event time from the record timestamp, late events are accepted only within the grace period, and suppress emits a final result per window instead of every update.

Exactly-once is one setting away (processing.guarantee=exactly_once_v2), at the cost of transactions and slightly higher latency. Without it, processing is at-least-once and results can be double counted after a failure.

Under the hood

The topology is split into sub-topologies, each turned into tasks (one per input partition). Tasks are assigned to stream threads in instances through the consumer group protocol. Each task owns its state stores and processes records one at a time on its thread; a repartition topic is inserted automatically when the key changes.

Common mistakes

  • Joining topics that are not co-partitioned.
  • Changing the key and forgetting that a repartition topic costs a round trip.
  • Ignoring late and out-of-order events in windows.
  • Changing the topology of a running application with state without resetting it.

How Unravel recognises it

When you paste code, Unravel looks for these signs of kafka streams. Spotting them yourself is a good habit when you read unfamiliar code:

Part of a learning path

Related concepts

Consumer groupsKafkaKafka idempotent producer and transactionsKafka

Frequently asked questions

What is Kafka Streams?

Kafka Streams is a library for processing topics as continuous streams inside your own application: filter, transform, join and aggregate events, with state kept locally and backed up to Kafka.

What are the most common mistakes with Kafka Streams?

Joining topics that are not co-partitioned. Changing the key and forgetting that a repartition topic costs a round trip. Ignoring late and out-of-order events in windows. Changing the topology of a running application with state without resetting it.

How can I see Kafka Streams in real code?

Paste your own Java into Unravel and it will point out where kafka streams appears, explain how it works at the detail level you choose, and draw the flow of the code around it.