Replication

The problem: one database, too many reads

Say your app's database answers about 5,000 queries a second, and at busy hours the app sends it 10,000 a second, 9 in 10 of them reads. It's asked twice what it can do. Worse, it's the only copy of your data: when that machine fails, the site is down.

A bigger machine buys time but stays a single point of failure. A cache holds only popular rows, in memory. The fix for both problems is to keep copies of the whole database on several servers and let reads go to any of them. That's replication.

How it works: one leader, many followers

Copies must hold the same data, so the question is who takes the writes. A write sent to whichever copy is least busy never reaches the others. An app that writes to every copy itself breaks when one of those writes fails, or two writes reach the copies in different orders.

The common design is leader and followers (also called primary and replicas):

• One copy, the leader, takes every write and numbers each change in its log: change #6, #7, #8.
• The followers receive the log and apply the same changes in the same order, so each ends up with the same data.
• Reads can go to any copy. Followers that serve reads are often called read replicas.

Built-in replication in PostgreSQL and MySQL works this way, asynchronous by default.

Replication lag, step by step

A change reaches the followers a moment later. The delay is replication lag: usually milliseconds, but seconds when a follower is busy or the network is slow. Say Ana's comment under Bo's and Cy's becomes change #7 on the leader and reaches Follower 1 after 50 ms, Follower 2 after 200 ms. What a read from each copy returns:

time     leader        follower 1    follower 2
0 ms     Bo, Cy, Ana   Bo, Cy        Bo, Cy
50 ms    Bo, Cy, Ana   Bo, Cy, Ana   Bo, Cy
100 ms   Bo, Cy, Ana   Bo, Cy, Ana   Bo, Cy
200 ms   Bo, Cy, Ana   Bo, Cy, Ana   Bo, Cy, Ana

Until 200 ms the copies disagree, and the answer depends on which copy a read hits. When writes pause, every follower catches up: the copies are eventually consistent.

"I posted and my post is gone"

This is the bug every replicated system meets. Ana posts her comment, and her page reloads at 100 ms. The reload reads from Follower 2, which gets change #7 only at 200 ms, so her own comment is missing from her own page. Many people then post it again.

The fix is read-your-writes (or read-after-write consistency): users always see their own changes. Two common ways:

• Read your own recent changes from the leader. For a few seconds after Ana writes, longer than the lag you expect, her reads go to the leader, which always has them. Everyone else's reads still go to followers.
• Read from a follower that has caught up. Remember that Ana's write was change #7, and read from a follower only once it has applied #7.

Sending every read to the leader also works for Ana, but it puts all 10,000 queries a second back on one machine. A related trick, sticky reads, sends each user's reads to the same follower every time. They stop a different bug: Bo reads from Follower 1 at 100 ms and sees Ana's comment, reloads at 150 ms, hits Follower 2, and the comment has vanished. But they don't fix Ana's case: if her follower hasn't got change #7, her comment is still missing.

Most reads, like product pages, public timelines and other people's profiles, are fine a moment old. Read from the leader when the answer must be exact right now, like a bank balance before a transfer, or when it's the user's own fresh change: their saved profile, the order they just placed.

Synchronous or asynchronous?

When should the leader tell the app "saved"?

• Asynchronous: right away, then it sends the change on. Writes are fast, but the last changes may be on no follower yet.
• Synchronous: only once every follower has the change. Every confirmed write is on every copy, but the slowest follower sets the pace: if changes take 2 seconds to reach Follower 2 under load, every write takes about 2 seconds, and a follower that goes down stops all writes.

A common middle ground waits for one follower and copies to the rest asynchronously (MySQL calls it semi-synchronous replication). With Follower 1 at 50 ms, a write waits about 50 ms, and two copies always have every confirmed write.

Failover: when the leader dies

If the leader dies, a follower has to take its place. That's failover:

1. The followers, or a monitoring service, stop hearing from the leader. After a timeout, say 10 seconds, it's declared dead.
2. The follower with the most changes is promoted to be the new leader.
3. The app sends writes to the new leader, and the other followers follow it.

With asynchronous replication, failover can lose data. Dan posts a comment, change #8, at 0 ms. The leader saves it, tells Dan "posted", and crashes at 30 ms, before the change reaches Follower 1 at 50 ms. Follower 1 is promoted with changes up to #7, and Dan's comment is gone although he was told it was saved. When the old leader comes back, it rejoins as a follower; its change #8 clashes with whatever the new leader numbered #8 and is usually thrown away.

One more danger: an old leader that was only cut off, not dead, may keep accepting writes next to the new one. That's split brain, the subject of Consensus and Leader Election.

The numbers: reads spread, writes don't

With 1 leader and 4 followers, and the 9,000 reads a second spread evenly over all 5 copies, each copy serves 1,800 reads. Add the writes each copy applies and it's about 2,800 a second, well under the 5,000 one server answers. With 10 copies, each would serve 900 reads.

Writes don't spread. Every copy must apply every write to stay a full copy, so with 5 copies or 10, each applies all 1,000 writes a second and holds all the data. When writes outgrow one leader (say it takes about 3,000 a second) or the data outgrows one machine, more followers won't help: you split the data across databases. That's sharding.

Where it shows up in design interviews

Most designs are read-heavy, and read replicas (often next to a cache) are the standard answer when the database can't keep up with reads. A strong answer covers three things:

1. The fix, sized: a leader for writes, followers for reads, and how many reads each copy serves.
2. Lag for the user's own writes: read their recent changes from the leader, so a post never vanishes after a reload.
3. What failover loses: with asynchronous replication, the last few writes; one synchronous follower prevents that.

Common interview questions

What's the difference between replication and sharding?

Replication keeps full copies: it spreads reads and survives a failed machine, but every copy applies every write. Sharding splits the data, so each server holds and writes only its part. Large systems do both.

A user saves their profile and still sees the old one. Why?

The read went to a follower that hadn't received the change yet: replication lag. Read that user's own recent changes from the leader for a short window, or from a follower that has applied their write.

What happens when the leader fails?

After a timeout, the most up-to-date follower is promoted and writes go to it. With asynchronous replication, writes no follower had received are lost, and the old leader must stop accepting writes.

Summary

• Leader and followers: the leader takes every write; followers apply its changes in order and serve reads.
• Replication lag: followers are a moment behind, usually milliseconds, seconds under load.
• Read-your-writes: read a user's own recent changes from the leader, or from a follower that has caught up.
• Sync or async: wait for followers (safe, slower) or don't (fast, but a failover can lose the last writes).
• Reads spread, writes don't: 1,800 reads per copy over 5 copies, but every copy applies all 1,000 writes.