BackendBench
Coffee

Unravel › Java concepts › Messaging and Kafka

Kafka

Kafka Connect connectors

Kafka Connect moves data between Kafka and other systems (databases, search, object storage) using ready-made connectors that you configure instead of coding consumers and producers.

🔎 Unravel this examplePrerequisites (1)

Learn these first

Kafka topics, partitions and events

At a glance

Mental model

Standard pipe fittings.

What happens, in order
  1. 1A connector is configured with JSON (connector.class, tasks.max, connection details, topics) and submitted to a Connect worker over RESTSource connectors read an external system and write to topics; sink connectors read topics and write outward.
  2. 2Work is divided into tasks, spread over a cluster of workersOffsets (a source position such as a database log position) are stored in Kafka, so a restart resumes where it stopped.
  3. 3Converters (Avro, JSON) and single message transforms (rename a field, drop a column, route to a topic by a field) adapt data without code
  4. 4Writing your own connector means implementing SourceConnector plus SourceTask with poll() returning SourceRecords, or SinkTask.put(Collection<SinkRecord>)
Remember
Watch out
No dead-letter strategy for bad recordsSink writes that are not idempotentToo many tasks for too few partitions (extra tasks idle)Putting heavy transformation logic into SMTs instead of a stream processor
Example (highlighted lines explain it)
public class RestSourceTask extends SourceTask {    @Override    public List<SourceRecord> poll() throws InterruptedException {        List<SourceRecord> out = new ArrayList<>();        for (String item : api.fetchSince(lastId)) {            Map<String, ?> partition = Collections.singletonMap("source", "rest");            Map<String, ?> offset = Collections.singletonMap("lastId", lastId);            out.add(new SourceRecord(partition, offset, "api-items", Schema.STRING_SCHEMA, item));        }        return out;    }}
  1. line 1 uses SourceTask, the core of Kafka Connect connectors
  2. line 3 uses SourceRecord, the core of Kafka Connect connectors
  3. line 4 uses SourceRecord, the core of Kafka Connect connectors
  4. line 8 uses SourceRecord, the core of Kafka Connect connectors
Read the full explanation (about 2 min)

Kafka Connect moves data between Kafka and other systems (databases, search, object storage) using ready-made connectors that you configure instead of coding consumers and producers.

A mental model

Standard pipe fittings. A source connector is a fitting that brings water from a tank into the Kafka pipe, a sink connector drains the pipe into another tank. You pick fittings and set the valves; you do not forge metal.

How it works

A connector is configured with JSON (connector.class, tasks.max, connection details, topics) and submitted to a Connect worker over REST. Source connectors read an external system and write to topics; sink connectors read topics and write outward.

Work is divided into tasks, spread over a cluster of workers. Offsets (a source position such as a database log position) are stored in Kafka, so a restart resumes where it stopped.

Converters (Avro, JSON) and single message transforms (rename a field, drop a column, route to a topic by a field) adapt data without code.

Writing your own connector means implementing SourceConnector plus SourceTask with poll() returning SourceRecords, or SinkTask.put(Collection<SinkRecord>).

Going deeper

Plan for bad records. errors.tolerance=all with a dead-letter queue topic keeps the pipeline flowing while parking records that fail conversion; the default stops the task at the first bad record.

Change data capture connectors (Debezium) read the database log rather than polling tables, which captures deletes and does not load the database, but needs log retention and permissions.

Delivery is at-least-once for most sinks; make the target idempotent (upserts by key) so a replay after failure does not duplicate rows.

Under the hood

A worker runs a herder that tracks connectors and tasks in internal topics (connect-configs, connect-offsets, connect-status). Each task gets its own thread and its own Kafka producer or consumer; on worker failure the group protocol reassigns the tasks to other workers.

Common mistakes

  • No dead-letter strategy for bad records.
  • Sink writes that are not idempotent.
  • Too many tasks for too few partitions (extra tasks idle).
  • Putting heavy transformation logic into SMTs instead of a stream processor.

How Unravel recognises it

When you paste code, Unravel looks for these signs of kafka connect connectors. Spotting them yourself is a good habit when you read unfamiliar code:

Part of a learning path

Related concepts

Schema Registry, Avro and ProtobufKafkaTransactional outbox patternI/O and data

Frequently asked questions

What is Kafka Connect connectors?

Kafka Connect moves data between Kafka and other systems (databases, search, object storage) using ready-made connectors that you configure instead of coding consumers and producers.

What are the most common mistakes with Kafka Connect connectors?

No dead-letter strategy for bad records. Sink writes that are not idempotent. Too many tasks for too few partitions (extra tasks idle). Putting heavy transformation logic into SMTs instead of a stream processor.

How can I see Kafka Connect connectors in real code?

Paste your own Java into Unravel and it will point out where kafka connect connectors appears, explain how it works at the detail level you choose, and draw the flow of the code around it.