The Transactional Outbox: Reliable Events Without Distributed Transactions

Event-driven architecture can help services evolve independently and respond to business changes without tight, synchronous coupling. But publishing an event reliably is harder than calling a message broker after saving a database record.

Consider an order service that writes a new order to its database and then publishes an OrderCreated event. If the database commit succeeds but the broker is unavailable, downstream services never hear about the order. Reverse the order—publish first, then commit—and consumers may act on an order that was never saved.

This is the dual-write problem: two independent systems must be updated as one logical operation, but neither a normal database transaction nor a message-broker publish can atomically commit both. The transactional outbox pattern addresses this by recording the event in the same database transaction as the business change, then publishing it asynchronously.

The core idea

Instead of writing to the database and publishing directly, the service performs two database writes in one local transaction:

1. Persist the business change, such as creating an order.
2. Insert an event record into an outbox table.
3. Commit both changes together.

A separate relay process reads committed outbox records and publishes them to the broker. After successful publication, it marks the record as delivered or removes it according to the system's retention requirements.

The key guarantee is local atomicity: if the business transaction rolls back, its outbox event rolls back too. If the transaction commits, the event is durably recorded and remains available for delivery, even if the broker is temporarily unavailable.

Request
  |
  v
Database transaction
  ├── Update business tables
  └── Insert outbox event
          |
          v
     Outbox relay
          |
          v
     Message broker
          |
          v
     Idempotent consumers

The outbox does not make the database and broker one transaction. It makes the intent to publish part of the database transaction, then allows delivery to be retried independently.

Design the outbox record for operations

A useful outbox record typically includes:

• A unique event identifier.
• An event type and schema version.
• The aggregate or entity identifier.
• The event payload.
• A creation timestamp.
• Delivery state, attempt count, or last error when using polling.

Store enough information to publish the event without reconstructing it from mutable business data later. For example, if an order address changes after creation, a delayed OrderCreated event should still describe the order as it was when the event was produced.

Keep payloads intentional. Large snapshots increase database, broker, and network costs; overly small payloads can force consumers into synchronous calls to the producer. Include what consumers need for the event's purpose, and use stable identifiers for data that consumers are expected to retrieve separately.

The outbox and business records should be written using the same database transaction and connection. Writing the event to a separate database would reintroduce the dual-write problem the pattern is meant to avoid.

Deliver events with a polling relay

The simplest relay periodically queries for unpublished events, publishes them, and records successful delivery. This is easy to operate with existing infrastructure and can provide adequate latency for many services.

Polling needs careful coordination if multiple relay instances run concurrently. Workers must claim records safely—for example, through database locking or an atomic claim operation—and release or recover claims when a worker crashes. Index the fields used to find pending records, and process bounded batches so one poll does not create unbounded memory or long-running transactions.

Do not hold a database transaction open while waiting on a broker network call. A common approach is to claim a batch in a short transaction, publish outside that transaction, then mark successful events as delivered. Claims need a lease or timeout so work can be recovered after a process failure.

Polling introduces a delay between the database commit and broker publication. Tune the polling interval and batch size based on delivery-latency requirements and database load. Very frequent polling can waste resources; very large batches can increase recovery time and make failures harder to isolate.

Or use change data capture

For higher throughput or lower polling overhead, a change data capture (CDC) pipeline can read committed database changes from the transaction log and route outbox rows to a broker. Because the relay observes committed writes, it preserves the atomic relationship between the business update and its outbox record.

CDC reduces application-managed polling logic but adds infrastructure and operational complexity. Teams need to monitor connector lag, schema changes, offsets, and recovery behavior. The outbox table should still be modeled intentionally; sending every database change downstream is not a substitute for defining stable domain events.

Choose polling when simplicity and moderate delivery latency are acceptable. Consider CDC when the throughput or latency requirements justify operating the additional pipeline. In either case, the application should produce the outbox event transactionally.

Assume delivery can happen more than once

Most practical outbox relays provide at-least-once delivery. A worker may publish an event successfully and crash before recording that it was delivered. When it recovers, it may publish the same event again.

This is expected behavior, not necessarily a defect. Exactly-once effects across a database, broker, and consumer are difficult to guarantee end to end. Design consumers to be idempotent instead:

1. Give each event a stable identifier.
2. Record processed identifiers in the consumer's own database.
3. Apply the consumer's business update and record the identifier in one local transaction.
4. When the same event arrives again, recognize it and avoid applying its effect twice.

The deduplication record must be committed atomically with the consumer's state change. If the consumer records an event as processed before applying the change, a crash can lose the effect; if it applies the change first, a crash can cause it to be applied twice. A single local transaction resolves that gap for database-backed consumers.

For operations that reach another external system, such as charging a payment provider, use that system's idempotency support where available and persist the operation's stable key. Consumer deduplication alone cannot make a separate remote side effect atomic.

Preserve ordering only where it matters

Parallel relay workers can publish events out of order. Global ordering is expensive and often unnecessary. First decide what ordering the business actually requires.

If events for one aggregate must remain ordered, include an aggregate identifier and sequence number, then partition or coordinate delivery by that aggregate. Consumers can detect gaps or stale sequence numbers and apply an explicit policy. Avoid serializing all events globally when only per-entity order matters; that can create a bottleneck.

Also consider the broker's delivery semantics, partition strategy, and consumer concurrency. An ordered publish does not guarantee ordered business effects if consumers process messages concurrently without preserving the relevant key.

Make failure and backlog visible

Retries should be bounded and observable. Use backoff to avoid flooding a recovering broker, retain enough error information to diagnose repeated failures, and define a policy for events that cannot be delivered after repeated attempts. Depending on the business criticality, that may mean a dead-letter queue, an operator alert, or a quarantined outbox record requiring investigation.

Monitor:

• The age of the oldest unpublished event.
• Outbox row count and growth rate.
• Relay throughput, errors, and retry volume.
• Broker publish latency and availability.
• Consumer lag and duplicate-detection rates.

An outbox that is silently accumulating rows is a delayed production incident. Alert on event age and backlog, not only on whether the relay process is alive.

Plan retention and cleanup as well. Delivered records may be deleted or archived after an appropriate period, but preserve enough information to support audit, replay, and incident investigation. Cleanup should not race with active workers or remove undelivered records.

Evolve event schemas deliberately

Events are contracts between independently deployed components. Include a schema version or otherwise establish a compatible evolution policy. Prefer additive changes where possible, and ensure consumers tolerate fields they do not use. Renaming or changing the meaning of an existing field can break older consumers even when the JSON remains valid.

Test serialization and compatibility as part of the producer and consumer workflow. For critical events, maintain representative payloads and verify that supported consumer versions can still interpret them. A durable outbox makes replay possible, so old events may be processed by newer code long after they were first created.

When the outbox is—and is not—the right tool

The transactional outbox is useful when a committed business change must eventually trigger work in another service and losing that notification is unacceptable. It avoids a fragile direct dual write while keeping the core transaction local to one database.

It also adds a table, a relay, retry behavior, monitoring, and cleanup responsibilities. For a low-value best-effort notification, a simpler mechanism may be sufficient. For workflows that require coordinated compensation across multiple services, the outbox can reliably start each step, but it does not replace a saga or other workflow state machine.

The pattern's guarantee is precise: a committed business change has a durable record of the event that should follow it. Reliable systems build on that foundation with retryable delivery, idempotent consumers, explicit ordering requirements, and operational visibility. This turns a hidden dual-write failure into work that can be measured, recovered, and explained.