Photo by Taylor Vick on Unsplash

Read-Only Replicas: The Stale Data Consistency Problem


Read-only replicas are everywhere in modern database architectures. They offload read traffic from the primary, enable geographic distribution, and provide a foundation for horizontal scaling. But they introduce a fundamental problem: the data they serve is always slightly behind the primary, and sometimes significantly behind.

This isn’t a bug. It’s an architectural tradeoff baked into asynchronous replication, and it forces application developers to reason carefully about consistency.

How Replication Lag Happens

When a write commits on the primary database, it must be replicated to read replicas. The primary acknowledges the write immediately (after writing to its own WAL and memory), then ships log entries to replicas asynchronously. Replicas apply these changes as fast as they can, but they’re always catching up.

Under normal load, lag might be milliseconds. During bulk operations, network hiccups, or replica catchup after downtime, lag can grow to seconds or even minutes. The replica is always serving a slightly stale snapshot of the data.

This creates visible anomalies. A user updates their profile, then immediately refreshes and sees the old data because the request hit a lagging replica. An item is deleted from inventory on the primary, but a concurrent read on a replica still shows it as available. A payment is recorded, but the account balance query returns the pre-payment amount.

Read-After-Write Consistency

The most common consistency expectation is read-after-write: if I write something, I should see it immediately on subsequent reads. Applications handle this in several ways.

Sticky sessions route all requests from a user to the same replica (or the primary) for a window after a write. This works but ties users to specific servers, complicating load balancing and failover.

Write-to-primary, read-from-primary patterns route critical reads back to the primary for a period after any write. This is simple and effective but sacrifices some of the read scaling benefit of replicas.

Replication position tracking embeds the log position of the write in the session or response. Subsequent reads wait for a replica to catch up to at least that position before serving the query. This is precise but requires application and database cooperation.

Monotonic Reads

A subtler issue: if successive reads hit different replicas with different lag, you can read forward in time, then backward. You see the updated profile, then on the next request you see the old one. This violates monotonic reads, the expectation that time doesn’t flow backward within a session.

Sticky sessions solve this by pinning a user to a single replica. Another approach is to track the highest replication position seen by a session and ensure all subsequent reads use a replica at least that far ahead. This requires coordinating replica selection based on replication lag metadata.

When Stale Reads Are Acceptable

Not all reads need strong consistency. Analytics dashboards, leaderboards, search indexes, and reporting queries often tolerate staleness measured in seconds or minutes. Offloading these to replicas makes sense.

The challenge is distinguishing staleness-tolerant queries from consistency-critical ones. Some systems expose this as an explicit query hint or routing parameter. Others use heuristics: reads immediately after a write go to the primary, background jobs and dashboards go to replicas.

Measuring and Monitoring Lag

Replication lag is measurable. Most databases expose metrics: the time or log position difference between primary and replica. Some provide per-transaction lag visibility, letting you see exactly how far behind each replica is.

Alerting on lag prevents the worst surprises, but deciding on thresholds is hard. A few seconds of lag under high load is normal. Sustained lag indicates a replica struggling to keep up, possibly due to resource contention, slow disk, or a long-running query blocking replication.

Cross-Region Replication

Lag compounds with distance. Cross-region replicas face speed-of-light delays plus network variability. A replica 100ms away will always lag by at least that much, and likely more. This makes read-after-write consistency harder: redirecting to the primary means crossing regions, adding latency.

Some systems use quorum reads or regional primaries to mitigate this, but these sacrifice simplicity. The fundamental tradeoff remains: low-latency local reads or strong consistency.

The Consistency Spectrum

Read replicas sit in the middle of the consistency spectrum. They’re not eventually consistent in the loose sense—replication is reliable and ordered—but they’re not linearizable or serializable either. They offer a weaker guarantee: bounded staleness, where the bound is replication lag.

Understanding this tradeoff is essential for designing systems that use replicas effectively. Route consistency-critical reads carefully, monitor lag, and design UX to tolerate occasional staleness where it’s acceptable. The performance and scaling benefits are substantial, but they come with the responsibility of managing inconsistency.