Transactional outbox pattern

Saving to the database and publishing a message are two separate actions, and a crash between them loses or invents events.

🔎 Unravel this example Prerequisites (1)

Learn these first

@Transactional

At a glance

Mental model

Writing the letter and putting it in the out-tray on your desk in the same movement.

What happens, in order

1In the business transaction: save the order and insert a row into an outbox table (id, aggregate id, type, payload, created at) in the same database transactionEither both commit or both roll back.
2A relay reads unpublished outbox rows in order, publishes them to Kafka or a broker, and marks them published (or deletes them)It can be a scheduled poller, or change data capture reading the database log (Debezium).
3Delivery is at least once: the relay can crash after publishing and before marking the row, so consumers get duplicates and must be idempotent
4The message key is the aggregate id so events for one entity stay in order on one partition

Remember

The naive alternative save(); kafka.send(); has a dual-write bug
Poll-based relays need an index on the unpublished flag and cleanup of old rows, otherwise the outbox table becomes the slowest table in the system
The guarantee comes entirely from the local ACID transaction on one database: the outbox row is just data

Watch out

Dual write of database and broker without an outboxConsumers that are not idempotentNever cleaning up published rowsSeveral relays publishing the same rows

Example (highlighted lines explain it)

@Transactionalpublic Order place(OrderRequest request) {    Order order = orders.save(new Order(request));    outbox.save(new OutboxEvent(        "Order", order.getId().toString(), "OrderPlaced", json.write(new OrderPlaced(order))));    return order;}

line 4 uses OutboxEvent, the core of Transactional outbox pattern

Read the full explanation (about 2 min)

Saving to the database and publishing a message are two separate actions, and a crash between them loses or invents events. The outbox pattern writes the message to a table in the same transaction, and a separate relay publishes it afterwards.

A mental model

Writing the letter and putting it in the out-tray on your desk in the same movement. A mail clerk later empties the tray. If you are interrupted, either both the record and the letter exist, or neither does.

How it works

In the business transaction: save the order and insert a row into an outbox table (id, aggregate id, type, payload, created at) in the same database transaction. Either both commit or both roll back.

A relay reads unpublished outbox rows in order, publishes them to Kafka or a broker, and marks them published (or deletes them). It can be a scheduled poller, or change data capture reading the database log (Debezium).

Delivery is at least once: the relay can crash after publishing and before marking the row, so consumers get duplicates and must be idempotent.

The message key is the aggregate id so events for one entity stay in order on one partition.

Going deeper

The naive alternative save(); kafka.send(); has a dual-write bug. Send inside the transaction and a rollback leaves a message about something that never happened; send after commit and a crash loses it.

Poll-based relays need an index on the unpublished flag and cleanup of old rows, otherwise the outbox table becomes the slowest table in the system. With several relay instances, use SELECT ... FOR UPDATE SKIP LOCKED so they do not take the same rows.

CDC avoids polling latency and load, but ties you to the database log format and needs Kafka Connect to operate.

Under the hood

The guarantee comes entirely from the local ACID transaction on one database: the outbox row is just data. The relay turns a "state change" into a "message" later, and ordering is preserved by processing rows by increasing id (or by reading the commit log, which is already ordered).

Common mistakes

Dual write of database and broker without an outbox.
Consumers that are not idempotent.
Never cleaning up published rows.
Several relays publishing the same rows.

How Unravel recognises it

When you paste code, Unravel looks for these signs of transactional outbox pattern. Spotting them yourself is a good habit when you read unfamiliar code:

Types: OutboxEvent OutboxRepository OutboxMessage Outbox OutboxPublisher OutboxEntity OutboxRelay OutboxService OutboxProcessor DebeziumEngine EmbeddedEngine OutboxEventRepository OutboxRecord
Method calls: saveOutbox() findUnpublished() markPublished()
Configuration keys: outbox outbox_events aggregatetype

Part of a learning path

Reliable messaging and distributed patterns · previous: Idempotency and duplicate handling · next: Sagas and distributed transactions
Structuring an enterprise service · previous: Idempotency and duplicate handling

Related concepts

Idempotency and duplicate handlingI/O and data Sagas and distributed transactionsI/O and data

Frequently asked questions

What is Transactional outbox pattern?

What are the most common mistakes with Transactional outbox pattern?

Dual write of database and broker without an outbox. Consumers that are not idempotent. Never cleaning up published rows. Several relays publishing the same rows.

How can I see Transactional outbox pattern in real code?

Paste your own Java into Unravel and it will point out where transactional outbox pattern appears, explain how it works at the detail level you choose, and draw the flow of the code around it.