Scaling Up vs Scaling Out

The problem: one server, three times the traffic

Say your site runs on one app server, and that server answers about 1,000 requests a second. A launch brings 3,000 a second. The server still finishes only about 1,000 each second, so 2,000 more join the line every second: after 10 seconds, 20,000 requests are waiting. Pages get slow for everyone, then requests start to time out.

There are two ways to get more capacity: make the one machine bigger, or add more machines. They're called scaling up (vertical scaling) and scaling out (horizontal scaling).

Scaling up: a bigger machine

Move the app to a machine with more CPU cores and memory. If a machine with 3× the cores answers about 3,000 requests a second, you're done for today, and nothing else changes: same code, same setup, one machine to deploy and watch. That's why scaling up is often the right first move. Its limits show up as the site keeps growing:

• A ceiling. There's a biggest machine you can buy or rent. When the site outgrows it, scaling up is over.
• Paying for the peak. Busy hours run about 5× the average, and one machine can't shrink at night. Sized for a 3,000-a-second peak, it averages about 600 a second: a fifth of what you pay for.
• One point of failure. A crash, a bad deploy or a hardware fault takes the whole site down, and moving to a bigger machine usually means stopping the old one.

Scaling out: more machines

Instead, run the same app on several ordinary servers. At about 1,000 requests a second each, three servers cover 3,000 a second; if traffic grows 10× to 30,000, you add servers until you have about 30. There's no ceiling, one server failing leaves the others running, and you can add servers for busy hours and remove them after.

Scaling out needs two things. First, a load balancer: users know one address, so something at that address must pass each request to one of the servers and stop sending to a server that fails. How it picks (round robin, least connections, health checks) is a lesson of its own. Second, servers that keep nothing only they know, which is where people get caught.

Where the session goes

When Ana logs in, the server that checks her password usually remembers her in a session: a record saying that session id s42 belongs to Ana. Her browser gets a cookie holding only the id and sends it with every request. On one server, that works. With three servers behind a load balancer that takes turns, it breaks:

request      goes to   that server's memory   Ana sees
log in       App 1     s42 = Ana (saved)      she's logged in
my lessons   App 2     nothing                the login page
my lessons   App 3     nothing                the login page
my lessons   App 1     s42 = Ana              her lessons

Only App 1 knows s42, so Ana is logged out on about 2 of every 3 requests. Saving the session to App 1's disk doesn't help: the other servers still can't read it. The fix is to move the session out of the app servers, in one of two common ways:

• A shared session store. Sessions live in one in-memory store, such as Redis or Memcached, and every app server looks up the cookie's id there. One such server answers about 100,000 lookups a second, so 3,000 a second is light work, and each lookup is one round trip inside the data center, about 0.5 ms. To log a session out, delete it.
• A signed token. At login, the server writes who Ana is into a token and signs it with a key only the servers have (a JSON Web Token, or JWT, is a common format). Any server checks the signature and reads the token, with no lookup at all. Ana can read what's inside, since signed isn't encrypted, but she can't change it without breaking the signature. A plain cookie saying user=Ana is the opposite: anyone can edit it to user=Bob.

Either way, the app servers are now stateless: any server can answer any request, so servers can be added, removed or lost and nobody is logged out. The state hasn't gone away; it lives in parts built to hold it.

The tempting fix: sticky sessions

A quicker fix is to have the load balancer remember each user's server (with a cookie of its own, or by IP address) and always send them back there. These sticky sessions hide the problem rather than solve it. When App 1 crashes, every session in its memory is gone and all its users are logged out; a restart doesn't help, because a restarted server starts with empty memory. Load also spreads unevenly: a server you just added only gets new visitors.

What scales out easily, and what doesn't

A part is easy to copy when it's stateless: app servers whose sessions live elsewhere, or servers that send the same page to everyone. Anything that keeps data needs a plan first. A photo saved on one server's disk exists only there, so files go to shared storage; a count kept in one server's memory sees only that server's share; and every copy of the database must hold the same data.

Say each request makes one database lookup and one database server answers about 5,000 a second. At 3,000 requests a second that's fine, but at 6,000 the database is the limit, however many app servers you add. Copying every write to more database servers (replication) and splitting the data between them (sharding) each get their own lesson. The session store holds state too, so run it with a replica: losing it would log everyone out.

Where it shows up in design interviews

Almost every design question gets to "what happens when traffic grows 10×?". The default first move: make the app servers stateless, then add more of them behind a load balancer. In order:

1. Find what runs out first: one app server answers about 1,000 requests a second.
2. Move sessions out of the app servers: a shared store or signed tokens.
3. Put a load balancer in front and add servers: about 30 for 30,000 a second.
4. Name what breaks next: usually the database.

Stateless comes first: add servers before that, and users get logged out. Scaling up is still worth naming as a stopgap, with its ceiling and its single point of failure.

Common interview questions

What's the difference between vertical and horizontal scaling?

Vertical scaling (scaling up) gives one machine more CPU and memory: simple, but it has a ceiling and stays a single point of failure. Horizontal scaling (scaling out) adds machines behind a load balancer: no practical ceiling and it survives one failing, but the servers must be stateless.

How do you keep users logged in across many servers?

Keep the session out of the app servers: in a shared session store every server reads, or in a signed token the client sends with each request. Then any server can answer any request.

Why not just use sticky sessions?

They tie each user to one server, so when that server fails, its users lose their sessions. They also spread load unevenly, and stateless servers make them unnecessary.

Session store or signed token?

A session store costs one quick lookup per request, but you can end a session at once by deleting it. A token needs no lookup, but it stays valid until it expires, so tokens are kept short-lived, or servers check a short list of revoked ones, which brings a lookup back.

Why can't the database scale out the same way?

Every copy must hold the same data, so every write has to reach all the copies, or the data has to be split between servers. That's replication and sharding, each with trade-offs of its own.

Summary

• Scaling up: a bigger machine. Simple, but it has a ceiling, you pay for the peak all day, and it's one point of failure.
• Scaling out: more machines behind a load balancer. No ceiling, and it survives a server failing.
• Stateless servers: the session lives in a shared store or a signed token, never inside one server.
• Sticky sessions: a crash still logs users out.
• Next limit: the database, scaled with replication and sharding.