Replication, Read Replicas and Lag

Copies of the database that serve reads, and the window in which they are wrong.

Overview

The arrangement

One server — the main — accepts each write. It records each change in a log, and streams that log to one or more replicas, which apply the changes to their own copies and serve read queries.

The appeal is that most workloads are overwhelmingly reads. Adding replicas times read room without any change to the data model, and unlike sharding it's reversible: a replica that isn't helping can simply be removed.

Replicas also serve as warm standbys. If the main fails, one is promoted.

Replication, Read Replicas and Lag

This module needs JavaScript: it steps through two transactions rather than showing a finished picture.

Worth knowing

One primary takes the writes and streams its log to replicas, which apply it and serve reads.
Replication lag is the delay between a commit on the primary and its arrival on a replica. It is never zero.
The classic bug: write, redirect, read from a replica, and the user does not see their own change.
Synchronous replication removes the lag and adds a network round trip to every commit. That is the trade, and it cannot be avoided.

Replication, Read Replicas and Lag

How to serve more reads than one machine can. The staleness that comes with it.

How far reads actually scale

"Times read room" has a limit, and the limit is worth seeing because it decides when replicas stop being the answer. each replica serves reads *and* must apply each write the main takes. The write load isn't shared out, it's copied to all of them. So if one machine handles 10,000 ops a second, a replica taking W writes per second has only 10,000 - W left for reads, and N replicas serve N × (10,000 - W):

write rate1 replica3 replicas10 replicas
500/s (5%)9,50028,50095,000
2,000/s (20%)8,00024,00080,000
5,000/s (50%)5,00015,00050,000

At a light write rate reads scale almost linearly. Ten replicas give nearly ten times the read throughput. This Is why read replicas are the standard answer for read-heavy workloads. But look down the columns rather than across: as writes climb, each replica loses that room to apply work, and the whole fleet's read ceiling drops with it. Once writes alone approach what one machine can do, 10,000 - W heads toward zero and adding replicas buys almost nothing — each new machine spends all its room just keeping up with the write stream.

That is the wall replicas hit and sharding is built to get past: sharding splits the writes across machines. It's the tool for a write-bound system just where more replicas is the tool for a read-bound one.

Lag

A change committed on the main isn't instantly present on a replica. It has to be written to the log, sent over a network, received, and applied. The gap is replication lag, and it's never zero.

Under normal rules it's milliseconds. Under load it isn't, and the reasons matter:

A large write. A single statement updating a million rows makes a great deal of log to ship and apply.

Single-threaded apply. Some engines apply the log serially though the main made it with many concurrent links. The replica simply can't keep up with a busy main.

A long query on the replica. Applying a change that conflicts with a running read forces a choice between cancelling the query and pausing replication. Both happen, based on configuration.

Network. A cross-region replica has a floor set by the speed of light.

The bug this causes

The failure mode is exact and extremely common:

1. User updates their profile        -> primary
2. Application redirects to profile  ->
3. Profile page reads                -> replica, 200ms behind
4. User sees their OLD profile

Nothing errored. The user changed a thing, was shown the previous value, and fairly concluded the save failed.

The standard fixes, in rough order of preference:

Read-your-writes routing. After a write, send that user's reads to the main for a short window. Simple and effective.

Route by criticality. any of it the user just affected reads from the main; dashboards and search read from replicas.

Wait for the log spot. Record the main's log spot at commit and have the replica wait until it has applied at least that far. Correct, and needs engine support.

Don't redirect to a read. Render the result from what was just written.

Synchronous replication

The lag can be removed. synchronous_commit in PostgreSQL, semi-sync in MySQL: the main doesn't confirm a commit until at least one replica confirms it has the change.

The cost is unavoidable and appears on each write: a commit now includes a network round trip. Throughput falls, and latency rises by the distance to the replica. If the synchronous replica turns unreachable, writes stall fully unless a fallback is configured.

That "distance to the replica" isn't a figure of speech. It has a floor set by physics, and the floor is worth knowing before choosing where a synchronous replica lives. Light in fibre travels at about two-thirds of c, so the round-trip *can't* be faster than twice the distance over that speed:

synchronous replicaround-trip flooradded to every commit
same datacentre~0 msnegligible
same region (~100 km)~1 mstolerable
across a continent (~5,500 km)~56 msevery write waits 56 ms+

Real fibre paths run about half again longer than the straight-line distance, so the cross-continent figure is nearer 70 ms . That is per commit, and no amount of hardware drops it. It's the reason a synchronous replica is kept nearby and durability across regions is bought with asynchronous copies plus the accepted risk of the section above.

This is a direct instance of the CAP trade: agreement across replicas costs uptime and latency, and no configuration escapes it. The usual compromise is one synchronous replica nearby for durability, and asynchronous replicas further away for read room.

Failover and lost writes

If the main fails while a replica is 200ms behind, promoting that replica loses 200ms of committed writes. They were confirmed to clients and are gone.

Which is why durability needs and replication mode are the same choice. Asynchronous replication means accepting that a failover can lose the most recent writes.

Questions people ask about Replication, Read Replicas and Lag

What is replication lag? The delay between a write on the primary and that write appearing on a replica. It can be milliseconds or much longer under load.

What is a read-your-writes problem? A user writes data, then reads from a replica that has not caught up and does not see their own change. Route that read to the primary.

What is the difference between synchronous and asynchronous replication? Synchronous waits for a replica to confirm before the write returns. Asynchronous returns right away, which is faster but can lose recent writes on failover.

Do read replicas help with write load? No. Every replica must apply every write, so replicas scale reads only.

What happens to writes during failover? If replication was asynchronous, writes the replica had not received are lost when it is promoted.

How can I measure lag? Compare the primary's log position with the replica's, or use the lag figure your database exposes. Alert when it passes your tolerance.

Where it goes wrong with Replication, Read Replicas and Lag

given replicas are current. They aren't, and the window is where the bugs live.

Sending a write to a replica. It's read-only. The error is confusing when routing is silent.

Monitoring lag in bytes only. Bytes behind doesn't translate to seconds behind. Track both.

Using replicas for locking or counters. any of it read-then-write must go to the main, or two clients read the same stale value.

No read-your-writes routing. After a user writes, their next read should go to the main for a few seconds, or they see stale data and file a bug that is hard to trace.

Check yourself

0 of 3

Answer without scrolling back up.

  1. Why does a user sometimes not see their own change after saving?

  2. What does synchronous replication cost?

  3. What happens if a primary fails while a replica is 200ms behind?

Cheat sheet

Replication, Read Replicas and Lag

Set up a primary and a read replica, write to one and read from the other, and measure the lag window where a query returns stale data.

DATABASE · vizlearn.in/database/replication_and_lag.html

About the author

Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.