BackendBench
Coffee

Unravel › Java concepts › Testing

Kafka

Testing Kafka code

Kafka code is tested at three levels: unit tests with mock producers and consumers, topology tests for Kafka Streams without a broker, and integration tests against a real broker started by the test (embedded or in a container).

🔎 Unravel this examplePrerequisites (2)

Learn these first

JUnit testsKafka consumer and the poll loop

At a glance

Mental model

Testing a mail system: first with a pretend mailbox on the desk (mock), then with a small real post office built just for the test (embedded or container), never with the real national mail.

What happens, in order
  1. 1MockProducer records everything sent to it, so a unit test asserts producer.history() without any networkMockConsumer is fed records with addRecord and schedulePollTask.
  2. 2TopologyTestDriver runs a Kafka Streams topology in memory: pipe in a TestInputTopic, read results from a TestOutputTopic, and inspect state storesNo broker is needed.
  3. 3@EmbeddedKafka (spring-kafka-test) starts a broker in the same JVM for a Spring test; Testcontainers KafkaContainer starts the real image and is closer to production
  4. 4Integration tests wait for asynchronous results with a bounded timeout (Awaitility) instead of Thread.sleep
Remember
Watch out
Thread.sleep waits in testsShared topics and group ids between testsTesting only the happy pathRelying on mocks for ordering and transaction behaviour
Example (highlighted lines explain it)
MockProducer<String, String> producer = new MockProducer<>(true, new StringSerializer(), new StringSerializer());OrderPublisher publisher = new OrderPublisher(producer);publisher.publish(new Order("o-1", "c-7"));assertEquals(1, producer.history().size());assertEquals("orders", producer.history().get(0).topic());
  1. line 1 uses MockProducer, the core of Testing Kafka code
Read the full explanation (about 2 min)

Kafka code is tested at three levels: unit tests with mock producers and consumers, topology tests for Kafka Streams without a broker, and integration tests against a real broker started by the test (embedded or in a container).

A mental model

Testing a mail system: first with a pretend mailbox on the desk (mock), then with a small real post office built just for the test (embedded or container), never with the real national mail.

How it works

MockProducer records everything sent to it, so a unit test asserts producer.history() without any network. MockConsumer is fed records with addRecord and schedulePollTask.

TopologyTestDriver runs a Kafka Streams topology in memory: pipe in a TestInputTopic, read results from a TestOutputTopic, and inspect state stores. No broker is needed.

@EmbeddedKafka (spring-kafka-test) starts a broker in the same JVM for a Spring test; Testcontainers KafkaContainer starts the real image and is closer to production.

Integration tests wait for asynchronous results with a bounded timeout (Awaitility) instead of Thread.sleep.

Going deeper

Use unique topic names and consumer group ids per test, or a @DirtiesContext, so tests do not read each other leftover messages. Shared brokers are a source of flaky tests.

Test the failure paths on purpose: a poison message that cannot be deserialized, a processing exception, a rebalance. They are the cases that hurt in production.

A mock cannot show broker behaviour such as partition ordering, transactions or compaction. Keep a few container-based tests for those.

Under the hood

The embedded broker is the real Kafka server code started in-process with a temporary log directory. MockProducer implements the Producer interface by storing records in a list and completing futures immediately or on demand, so error callbacks can be triggered with errorNext.

Common mistakes

  • Thread.sleep waits in tests.
  • Shared topics and group ids between tests.
  • Testing only the happy path.
  • Relying on mocks for ordering and transaction behaviour.

How Unravel recognises it

When you paste code, Unravel looks for these signs of testing kafka code. Spotting them yourself is a good habit when you read unfamiliar code:

Part of a learning path

Related concepts

JUnit testsTestingTestcontainers (real databases and brokers in tests)Testing

Frequently asked questions

What is Testing Kafka code?

Kafka code is tested at three levels: unit tests with mock producers and consumers, topology tests for Kafka Streams without a broker, and integration tests against a real broker started by the test (embedded or in a container).

What are the most common mistakes with Testing Kafka code?

Thread.sleep waits in tests. Shared topics and group ids between tests. Testing only the happy path. Relying on mocks for ordering and transaction behaviour.

How can I see Testing Kafka code in real code?

Paste your own Java into Unravel and it will point out where testing kafka code appears, explain how it works at the detail level you choose, and draw the flow of the code around it.