The transactional outbox

Save the order, then publish OrderPlaced. Two lines, and between them is every failure that produces an order nobody knows about — or an event for an order that was rolled back. The transactional outbox is the pattern that makes "write the row and tell the world" one atomic act, at the price of a table, a relay, and a guarantee that is at-least-once rather than exactly-once.

#The dual-write problem

@Transactional
public void place(Order order) {
    orders.save(order);                                 // in the transaction
    kafka.send("orders", new OrderPlaced(order.id()));  // NOT in the transaction
}

Four things can happen, and two are wrong. The commit succeeds and the send succeeds: fine. The commit fails and the send was never reached: fine. The send happens and then the commit fails — Kafka has an event for an order that does not exist. Or, reorder the lines, the commit succeeds and the send fails — the network blinked, the broker was rebalancing, the process was killed — and the order exists with no event, so inventory never hears, and the customer's confirmation never goes out. Both are dual writes: two systems, no transaction spanning them, and the second one failing after the first committed.

Retrying the send does not fix it (the process may be gone). Putting the send inside the transaction does not fix it (Kafka is not a participant, and a commit that fails after the send still leaves the event out there). 2PC would, and the last lesson said why not.

#The outbox table

Write the event into the same database, in the same transaction as the row:

CREATE TABLE outbox (
    id           BIGSERIAL PRIMARY KEY,
    aggregate    TEXT NOT NULL,          -- 'order'
    aggregate_id TEXT NOT NULL,          -- '42', also the partition key
    type         TEXT NOT NULL,          -- 'OrderPlaced'
    payload      JSONB NOT NULL,
    created_at   TIMESTAMPTZ NOT NULL DEFAULT now(),
    published_at TIMESTAMPTZ             -- NULL until relayed
);

@Transactional
public void place(Order order) {
    orders.save(order);
    outbox.save(new OutboxRow("order", order.id(), "OrderPlaced", json(order)));   // same transaction
}

Now there is exactly one commit, and it either contains both the order and the event or neither. The event has not reached Kafka yet; it is a row, durable, waiting. A separate relay reads unpublished rows and publishes them.

#The relay: polling, or reading the log

Two ways to move rows from the table to the broker.

Polling. A scheduled job selects unpublished rows in id order, publishes each, and marks it. With SELECT ... FOR UPDATE SKIP LOCKED (from the SQL course's locking lesson) several relay instances can run without publishing the same row twice:

@Scheduled(fixedDelay = 200)
@Transactional
void relay() {
    List<OutboxRow> batch = outbox.lockUnpublished(100);            // FOR UPDATE SKIP LOCKED, ORDER BY id
    for (OutboxRow row : batch) {
        kafka.send(row.topic(), row.aggregateId(), row.payload()).get();   // wait for acks=all
        row.markPublished();
    }
}

Simple, no new infrastructure, and a latency of about the polling interval. The cost is the poll itself against a busy table, and one subtlety: mark the row after the send has been acknowledged, so a crash between the two republishes rather than loses.

Change data capture. Debezium reads PostgreSQL's write-ahead log (the same log replication uses) through a logical replication slot, sees every insert into outbox the moment it commits, and publishes it to Kafka itself — the Debezium outbox event router does the table-to-topic mapping. No polling, sub-second latency, no load on the table, and the relay's state is the replication slot's position, which survives restarts. The cost is Debezium (a Kafka Connect deployment) and the operational rule that comes with logical replication: a slot that is not consumed holds WAL forever, and a forgotten slot is a disk that fills. Rows can then be deleted immediately or never, since the log already carried them.

Polling for a service with one relay and modest volume; CDC when the outbox is the backbone of many services or latency matters.

#At-least-once, and therefore idempotent consumers

Both relays can publish a row twice — the send succeeded and the mark or the slot update did not — and neither can publish it zero times. That is at-least-once, and it is not a flaw to engineer away but the guarantee to design for: every consumer of an outbox event must be idempotent, which the distributed-systems course's idempotency lesson and the Kafka course's idempotent-consumer section both cover. The row's id (or a UUID in the payload) is the deduplication key; a consumer that records processed ids in its own transaction handles the duplicate by doing nothing.

Ordering comes along for free within an aggregate: rows are relayed in id order and keyed by aggregate_id, so all of order 42's events land on one partition in the order they were written. Across aggregates there is no ordering, and there should not need to be.

#Keeping the table small

An outbox that is never cleaned is a table that grows by every event forever. With polling, delete rows after publication in the same batch or with a periodic job that removes rows older than a retention window (keep a day or two for debugging). With CDC, delete immediately after insert in the same transaction — Debezium sees the insert in the log regardless of the delete — so the table is always empty and the index never grows. Either way, put the relay's query on an index over (published_at, id) or the poll becomes a sequential scan of history.