Change data capture (CDC) is a technique for detecting every insert, update, and delete in a source database and delivering those changes to other systems, usually within seconds. Instead of copying whole tables on a schedule, CDC streams only what changed, so warehouses, search indexes, caches, and other services stay in sync with far less load on the source. The most robust form reads the database's own transaction log, which captures every change in commit order without touching application code.

What is change data capture?

Most operational data lives in relational databases like PostgreSQL or MySQL. Many other systems need a copy: an analytics warehouse, a search index, a cache, a fraud model, or another microservice. The naive approach is a nightly full export, which is slow, heavy on the source, and always out of date.

Change data capture turns the database into a source of events. Each committed row change becomes a record describing what changed, typically with the row before and after the change, the operation type, and metadata such as the transaction position. Downstream consumers apply those records to keep their copy current. CDC is a natural fit for event-driven architecture and for feeding streaming platforms such as Kafka.

Types of change data capture

There are three common ways to capture changes, with very different trade-offs.

Query-based CDC

Query-based CDC periodically runs a query like "give me rows where updated_at is later than the last value I saw". It is easy to set up and needs no special database privileges. But it has real gaps: it cannot see hard deletes, it misses intermediate updates between polls, it depends on every writer setting updated_at correctly, and frequent polling adds load to the source.

Trigger-based CDC

Trigger-based CDC adds database triggers that write each change into a separate change table, which a process then reads. It captures deletes and every individual change. The cost is extra write work inside every transaction, triggers to maintain on each table, and a change table that must be cleaned up.

Log-based CDC

Log-based CDC reads the database's transaction log, which the database already writes for durability and replication. PostgreSQL exposes it through logical decoding and replication slots, MySQL through its row-based binary log, and other databases have their own mechanisms. A connector reads the log, converts entries into change events, and publishes them. This captures every committed change, including deletes, in commit order, with minimal extra load on the source and no changes to application code.

Approach Captures deletes Every intermediate change Source overhead Setup effort
Query-based No (unless soft deletes) No Polling queries Low
Trigger-based Yes Yes Extra writes per transaction Medium
Log-based Yes Yes Low; reads existing log Medium to high

For most production use cases, log-based CDC is the preferred approach.

How does log-based CDC work?

Tools such as Debezium implement log-based CDC as connectors, commonly running on Kafka Connect. Conceptually, a connector does the following:

  1. Initial snapshot. It reads the current contents of the selected tables so consumers start with a complete copy, and records the log position at which the snapshot is consistent.
  2. Streaming. It reads the log from that position forward, turning each committed row change into an event.
  3. Publishing. It writes events to a topic, often one topic per table, keyed by the row's primary key.
  4. Offset tracking. It stores the last processed log position so it can resume after a restart without skipping changes.

A typical change event contains the row state before and after the change, an operation code, and source metadata. A simplified update event might look like this:

{
  "before": {"id": 42, "email": "[email protected]", "status": "active"},
  "after":  {"id": 42, "email": "[email protected]", "status": "active"},
  "source": {"db": "shop", "table": "customers", "lsn": 23984712},
  "op": "u",
  "ts_ms": 1785398400000
}

In Debezium's format, op is c for create, u for update, d for delete, and r for rows read during a snapshot. For deletes, after is null. Some setups also emit a tombstone record so compacted topics can drop the key entirely.

Log-based CDC does require operational care. In PostgreSQL, a replication slot keeps WAL segments until the consumer confirms it has processed them, so a stopped connector can cause WAL to pile up and fill the disk. Monitor slot lag and alert on it.

Ordering and delivery guarantees in CDC

The transaction log is ordered, but order has to be preserved through the rest of the pipeline. Keying events by primary key means all changes to one row go to the same partition, and Kafka guarantees order within a partition. There is no global order across partitions, so consumers should not assume changes to different rows arrive in commit order. The message ordering guide covers this in detail.

Most CDC pipelines provide at-least-once delivery. After a crash, a connector resumes from its last committed offset and may re-emit a few events. Consumers should therefore apply changes idempotently: upsert by primary key, and ignore events whose log position is older than what has already been applied.

How to handle schema changes in CDC

Source schemas change: columns are added, renamed, or dropped. Log-based connectors track schema history so they can decode older log entries correctly, and they usually emit events with the new shape after a change. Downstream, a few practices keep things stable:

  • Use a schema registry with formats like Avro or Protobuf and enforce compatibility rules.
  • Prefer additive changes, such as adding nullable columns, over renames and type changes.
  • Treat the CDC stream of internal tables as a semi-private contract, and coordinate breaking changes with consumers.
  • Land raw change events first, then transform them, so a schema problem does not lose data.

What is the outbox pattern?

Raw table CDC exposes your internal schema to every consumer, which couples services to implementation details. The transactional outbox pattern avoids that. The application writes its business change and a separate event row into an outbox table in the same database transaction. CDC then reads only the outbox table and publishes those events.

Because both writes commit or roll back together, you avoid the dual-write problem where the database update succeeds but the message publish fails, or the reverse. Consumers receive intentional, well-defined domain events such as OrderPlaced rather than raw row diffs. Debezium includes an outbox event router that routes outbox rows to topics based on their fields. Outbox events are still delivered at least once, so consumers should deduplicate by event ID, as described in idempotency.

Common CDC use cases

  • Replicating operational data into a warehouse or lake for near-real-time analytics.
  • Keeping search indexes and caches in sync with the system of record.
  • Publishing domain events from a service through the outbox pattern.
  • Migrating data between databases with minimal downtime.
  • Feeding stream processing jobs, as discussed in batch vs stream processing.

Key takeaways

  • Change data capture streams row-level changes instead of copying whole tables.
  • Log-based CDC reads the transaction log, capturing all changes, including deletes, with low overhead.
  • Key events by primary key to preserve per-row order; do not assume global order.
  • Expect at-least-once delivery and make consumers idempotent.
  • Use the outbox pattern to publish clean domain events without dual writes.

Frequently asked questions

Is CDC the same as database replication?

They use similar mechanisms, since both often read the transaction log. Replication copies data to another instance of the same database, while CDC turns changes into events that any kind of system can consume. CDC targets are usually heterogeneous, such as a warehouse or a search index.

Does CDC slow down the source database?

Log-based CDC adds relatively little load because it reads a log the database already writes. The main risks are the initial snapshot and log retention when a consumer falls behind. Query-based and trigger-based approaches add more direct load.

Can CDC capture deleted rows?

Log-based and trigger-based CDC capture deletes because the delete is recorded as its own change. Query-based CDC cannot see hard deletes, since a deleted row no longer matches any query. Teams using query-based CDC often switch to soft deletes for this reason.

When should I use the outbox pattern instead of table CDC?

Use the outbox pattern when other services consume the events and you want a stable contract independent of your table schema. Table-level CDC is fine for replicating data into analytics systems, where full row history is what you want.