Multi-Region and Disaster Recovery

The problem: one region, one bad day

Most systems start in one cloud region: a few data centers in one area, say in the US. Spreading servers and database replicas over those data centers survives one of them failing. It doesn't survive a fault that hits the whole region at once, such as a bad network change rolled out everywhere: every request fails, replicas included, because they sit behind the same broken network. A real disaster could destroy every copy at once.

One region also costs time. A round trip between continents takes about 150 ms, so a US shop's users in Europe wait that long for every trip. Surviving a lost region means keeping a copy elsewhere, and how far you go depends on two numbers.

Two numbers first: RPO and RTO

• RPO (recovery point objective): how much recent data you can afford to lose, counted in time.
• RTO (recovery time objective): how long you can be down before the service is back.

Take the simplest safety net: a nightly backup of the database, kept in another region. If a disaster wipes out the region at 23:00 and the last backup ran at 2:00, everything written in between is gone. Nightly backups give an RPO of up to 24 hours. The RTO is no better: you start new servers in another region and copy the backup onto them first.

step                   number
last backup            2:00
disaster               23:00
writes lost            21 hours
database to copy back  10 TB = 10,000 GB
copy speed             about 1 GB a second (one server's network)
copy time              10,000 s, almost 3 hours

Add starting the servers and checking the data, and a restore takes hours. Keep backups anyway: replication copies mistakes too. A table a bug deletes is gone from every replica within seconds; only a backup brings it back.

The targets come from the business

The closer RPO and RTO get to zero, the more the design costs, so the business sets them, often per system, by what lost data and downtime would cost:

system                          RPO       RTO
a bank's ledger of transfers    0         minutes
likes on a busy photo app       minutes   minutes
a news site                     minutes   minutes
a sales report rebuilt nightly  a day     a day

A ledger can't forget a transfer the customer was told is done. A busy app must come back fast, but a few minutes of lost likes is barely noticed. Anything you can rebuild can wait.

Active-passive: a standby region

Say the targets are an RPO of a few seconds and an RTO of a few minutes. A common design adds a second region, B, in Europe, with its own app servers and database copy. Region A serves every user while B stays up to date and waits: active-passive. How B's copy stays up to date sets the RPO:

• Asynchronous replication: A saves each write, answers the user, then sends the change on. Writes stay fast, and B runs a moment behind, say 2 seconds. That lag is the RPO.
• Synchronous replication: each write waits until B has it, so no confirmed write is lost. But every write waits for a round trip across the ocean, about 150 ms, and a slow link holds up writes. Systems that need an RPO of 0, like a ledger, pay it, often replicating synchronously to a site nearby and asynchronously to one far away.

When A fails, you fail over: notice it's down, promote B's database so it takes writes, and point DNS at region B. Say A takes about 200 writes a second, each reaches B 2 seconds later, and A fails at 12:00:00:

time       what happens                          B has writes up to
11:59:58   normal, B is 2 s behind               11:59:56
12:00:00   region A fails                        11:59:58
12:03:00   declared down after 3 minutes         11:59:58
12:04:00   B promoted, DNS answers B (TTL 60 s)  11:59:58
12:05:00   every saved DNS answer has run out    11:59:58

The writes from 11:59:58 to 12:00:00, about 400, were only in region A: an RPO of 2 seconds. Everyone is back by 12:05:00, an RTO of 5 minutes, most of it spent noticing. If A was only cut off, not dead, it must also stop taking writes, or both regions accept them: split brain, the subject of Consensus and Leader Election.

The cost: a mostly idle second region, and far users still wait 150 ms a round trip.

DNS TTL is part of the RTO

Changing a DNS record only changes what new lookups get: a resolver that looked the name up just before the switch keeps the old address until its TTL runs out. A TTL of 60 seconds adds up to a minute. With a TTL of a day (86,400 seconds), some users keep reaching the dead region for up to a day, and reloading the page doesn't help. Keep the TTL short on names you may fail over, or put a global load balancer with fixed addresses in front of the regions: switching it doesn't wait for caches.

Active-active: both regions serve

In active-active, both regions serve users, each user from the nearest. A user in Europe waits about 20 ms a round trip to a region on their own continent instead of 150 ms, both regions do useful work, and when one fails, its users go to the other, which is already running.

Reads are easy; writes are hard, because two users a continent apart can change the same data at once. Each kind of data needs a rule:

data                  writes go to      a clash is settled by
likes, view counts    both regions      adding the counts
shopping cart         both regions      merging the items
profile bio           both regions      keeping the later edit
account balance       its home region   never happening
username              one region        never happening

Merging works when clashes combine: counts add up, a cart keeps every item. Last-write-wins keeps the later edit and silently drops the other, so it suits data where that's harmless. Data where two yeses break a promise goes to one place that decides: with a $150 balance, each region could approve a $100 payment and spend money that isn't there. A common design gives each user a home region near them, so their own writes stay fast. When a region fails, its users' home moves to the other region, whose copy of their data is a few seconds behind: the same RPO as active-passive.

Test it: failover drills

A failover you never practice tends to break the day you need it: a script that no longer works, a standby too small for the traffic. Teams run drills, failing over on purpose on a calm day, and measure the real RPO and RTO.

Where it shows up in design interviews

When a design has users worldwide or holds money, expect "what happens if a whole region goes down?" Answer in this order:

1. The targets: state the RPO and RTO, because they pick the design.
2. The design: active-passive or active-active, and why.
3. The cost: what a failover loses (the replication lag) and how long it takes.
4. The proof: regular failover drills.

Common interview questions

What's the difference between RPO and RTO?

RPO is how much data you can lose, measured as time before the failure; RTO is how long you can be down after it. Nightly backups give up to 24 hours and hours; an async standby region gives seconds and minutes.

Why not replicate synchronously to the other region?

No confirmed write would be lost, but every write would wait about 150 ms for a round trip across an ocean. Most systems accept an RPO of a few seconds; ledgers pay for zero.

Active-passive or active-active?

Active-passive is simpler: one region takes the writes, so nothing clashes, at the price of an idle region and a failover wait. Active-active serves everyone from nearby, but each kind of data needs a write rule: merge, last-write-wins or one home region.

Summary

• RPO and RTO: data you can lose, time you can be down; the business sets them.
• Backups: up to 24 hours lost, hours to restore.
• Active-passive: a standby region fed by async replication; a failover loses the lag and takes minutes.
• DNS TTL: keep it short, or users keep going to the dead region.
• Active-active: both regions serve their nearest users; writes are merged or go to one home region.
• Drills: an unpracticed failover breaks when you need it.