Change Data Capture: The Plumbing That Keeps Distributed Systems in Sync

Most distributed systems eventually run into the same problem: the database holds the authoritative state, but a dozen other things need to know when that state changes. A search index needs to stay current. A cache needs to be invalidated. A downstream analytics warehouse needs fresh rows. Microservices need to react to events they didn’t produce.

The naive answer is polling—query the database every few seconds, check what changed. It works until it doesn’t. Poll too infrequently and your data goes stale; poll too frequently and you’re hammering the database with queries that add no business value. You also can’t detect deletes without keeping a soft-delete column, and you lose the order of changes if multiple rows update in the same polling window.

Change data capture (CDC) is the better answer, and understanding how it works clarifies why it’s become foundational infrastructure rather than a niche integration trick.

Reading the Log, Not the Tables

Every serious transactional database maintains a write-ahead log (WAL) or its equivalent—Postgres calls it the WAL, MySQL has the binlog, MongoDB has the oplog. This log exists primarily for crash recovery and replication, but it also contains a complete, ordered record of every committed change: which rows were inserted, updated, or deleted, and in what sequence.

Log-based CDC taps directly into this stream. Instead of asking “what does the table look like now?”, it asks “what mutations happened since the last position I processed?” The result is a sequence of events—each carrying the before and after state of a row—delivered in commit order with no load on the primary query path.

This is meaningfully different from polling. You get every intermediate state, not just the latest. You get deletes for free. The ordering guarantee lets you reconstruct state deterministically. And because you’re tailing a log rather than running SELECT statements, the database sees traffic comparable to a standby replica, not an analytics workload.

What You Can Build on Top of It

The event stream that CDC produces is general-purpose. Common uses include:

Cache invalidation. Rather than expiring cache entries on a timer or relying on application code to remember to invalidate, a CDC consumer watches for changes to specific tables and evicts or refreshes the corresponding cache keys immediately after the write commits.

Search index synchronization. Keeping Elasticsearch or a similar index in sync with a relational source of truth has historically required dual-write logic scattered through the application. CDC moves that responsibility to the infrastructure layer—the application writes once, the index stays current automatically.

Event-driven integration. Publishing domain events from a database write is the reliable half of the outbox pattern. A CDC connector reads committed outbox rows and publishes them to a message broker, eliminating the dual-write problem between the database and the broker entirely.

Live database migrations. Shadow-writing to a new schema while CDC replays every change from the old one is a well-established path to zero-downtime migrations, including cross-engine moves.

The Tradeoffs Worth Understanding

Log-based CDC isn’t free of complexity. A few issues come up repeatedly in production:

Schema evolution is the most persistent one. The log contains binary or semi-structured data encoded against the schema that existed at write time. If you alter a column and your CDC consumer doesn’t know about it, deserialization breaks. Managing schema registry integration—associating each event with the schema version it was written under—adds operational overhead but is essentially mandatory at scale.

Delivery semantics are at-least-once by default. The consumer tracks its position in the log, and if it crashes between processing an event and committing that position, it will re-process on restart. Downstream systems need to be idempotent, or the pipeline needs a deduplication layer.

The replication slot mechanism in Postgres (the standard way to consume the WAL externally) will hold WAL segments on disk until the slowest consumer acknowledges them. A lagging or dead consumer that holds an active replication slot can cause disk pressure on the primary—monitoring slot lag is an operational necessity, not a nice-to-have.

Why It’s Infrastructure Now

CDC used to be a specialized integration technique, common in large enterprises with ETL pipelines but unfamiliar elsewhere. The combination of event-driven architecture becoming standard, streaming platforms like Kafka becoming ubiquitous, and tools like Debezium making connector deployment straightforward has changed that. Today CDC is closer to a default component in any serious data platform than it is to an advanced technique.

The underlying idea is simple: the database already knows what changed. Reading that signal directly, instead of inferring it from query results, is just sound engineering.