Safe Deployments

The problem: most outages start with a change

Say your shop runs on 20 app servers behind a load balancer, about 10,000 requests a second. The team ships a new version to all 20 at once. It starts fine and passes its health checks, but it crashes on the first real requests. Every server fails, and the whole shop is down until someone puts the old version back.

This is how systems usually fail: Google's SRE book puts roughly 70% of outages down to changes in a live system. So a safe process assumes a bad change will get through, and does two things: it limits how many users a bad change can hurt (its blast radius), and it undoes the change fast.

Rolling deploys: a few servers at a time

A rolling deploy updates a few servers at a time. Each batch is drained (the balancer stops sending it new requests and lets open ones finish), updated, and put back in the balancer's list once it passes its health check.

With batches of 5, the 15 servers still serving carry all 10,000 requests a second, about 670 each, under the 1,000 one server can handle. Here is the same buggy release:

step     first batch                       the rest
drain    servers 1–5 get no new requests   15 servers, about 670 a second each
update   new version, health check passed  old version
crash    every request to them fails       old version
stop     taken out, rollout stopped        never get the new version

The crashing batch holds 5 of the 20 places in the balancer's list, so 25% of requests fail, for up to 15 seconds, until health checks take those servers out (every 5 seconds, out after 3 misses). The deploy tool sees the batch fail and stops, and servers 1–5 go back to the old version.

Rolling deploys need no extra servers. But a health check only asks "are you alive?": a release that fails 2% of requests, or makes pages slower, passes every check and rolls on to all 20 servers. And undoing a finished rollout means rolling the old version back out, which takes minutes.

Canary releases: a small share first, compared

A canary release sends a small share of real traffic, often 1 to 5%, to the new version while the old one serves the rest. The rollout compares the two at the same time on the numbers your SLO is written in, error rate and latency. If they match, it widens in steps, say 1% → 10% → 50% → 100%; if the canary is worse, it rolls back on its own.

Take the release that fails 2% of its requests. Its canary gets 1% of the traffic for 10 minutes, and the rule is: roll back if the canary's error rate is more than twice the old version's.

canary traffic        1% of 10,000/s = 100 requests a second
canary step           10 minutes = 600 seconds → 60,000 requests
old version errors    0.1% of its requests
canary errors         2%, 20× the old rate → roll back
failed on the canary  2% of 60,000 = about 1,200 requests
same bug, everyone    10,000/s × 600 s × 2% = 120,000 failed requests

The canary cut the damage 100 times. It compares with the old version, not a fixed number, because errors are never zero: a busy hour raises both rates together, so only the difference speaks about the new code. The costs: metrics per version, slower releases, and a stretch when both versions serve users, so both must work with the same data.

Blue-green: two full environments

Blue-green keeps two complete sets of servers. Blue runs the old version and serves everyone; green gets the new version and is tested with no users. Then the balancer switches all traffic to green at once, and blue keeps running.

Rollback is the same switch in reverse, done in seconds because blue never changed. The price is a second full set of servers while both run, here 20 extra, and a bug in green reaches every user at once until you switch back.

way          rollback                      extra servers     first real users
rolling      minutes: roll the old back    none              a batch, 25%
canary       seconds: move the 1% back     1                 1% of traffic
blue-green   seconds: switch back to blue  20, a full set    everyone at once

Feature flags: deploy is not release

A feature flag is an if around new code that asks the flag service whether this user gets it:

def checkout(user):
    if flags.on("new-checkout", user):
        return new_checkout(user)
    return old_checkout(user)

The code ships to every server switched off, so the deploy changes nothing users see. The release comes later, from a settings page, with no deploy: on for staff, then 5% of users, then everyone. Servers keep the flags in memory and pick up changes within seconds, and a user's group comes from a hash of their ID, so nobody flips between checkouts.

If errors jump at 5%, switching the flag off is a kill switch: every server takes the old path within seconds, while rolling back the release would take minutes and undo every other change in it. The catch is cleanup: once a feature is on for everyone, delete its flag and the old code. Flags left in forever pile up into ifs nobody dares remove.

Database changes: expand and contract

During a rollout the old and new versions share one database, and a rollback brings the old version back. So a schema change must work with every version that may run against it. Renaming the column zip to postal_code in one step breaks the old version at once: its queries ask for a column that no longer exists. The safe way is expand and contract, each step shipped on its own:

1. Expand: add the postal_code column. The old version ignores it.
2. Deploy code that writes both columns, still reading zip.
3. Backfill: copy zip into postal_code for the rows that lack it.
4. Deploy code that reads postal_code and still writes both, so a rollback still works.
5. Contract: once no version that might run reads zip, stop writing it and drop it.

The backfill comes after the write-both deploy, or rows written in between would be missed. It's API Versioning's rule again, add and never remove what someone still reads; here the someone is your own old version.

Where it shows up in design interviews

Deployment rarely gets its own deep dive, but interviewers ask how you'd ship changes without taking the site down. One sentence in your design answers it: changes roll out behind feature flags, canary first, with automatic rollback on the SLO. If the design changes a schema, add that migrations are expand and contract. "Tests will catch it" is a weak answer: interviewers expect a plan for the bad change that gets through.

Common interview questions

What's the difference between a canary and blue-green?

A canary sends a small share of traffic to the new version and widens step by step, so a bad version reaches few users. Blue-green switches everyone to a second full environment at once and back just as fast, at the cost of twice the servers.

How do you roll back a bad release quickly?

Make rollback a traffic change instead of a new deploy: move the canary's share back, switch the balancer back to blue, or turn a flag off. Each takes seconds, and the rollout can do it on its own when the canary's errors or latency look worse than the old version's.

Why feature flags if you already have canaries?

A canary tests a version of the code; a flag controls one feature inside it. You release that feature to staff or 5% of users whenever you like, and turn just it off in seconds.

Summary

• Most outages start with a change: limit how many users a bad one hurts, and undo it fast.
• Rolling deploys update a batch at a time and stop when health checks fail: 25% of requests hit, not all of them.
• Canary releases send 1% first and compare with the old version: about 1,200 failed requests instead of 120,000.
• Blue-green switches everyone at once and back in seconds, for a second full set of servers.
• Feature flags split deploy from release and give each feature a kill switch.
• Expand and contract keeps schema changes safe for old and new versions at once.