Latency Numbers

The problem: where the data lives decides the speed

Every request a server answers reads data from somewhere: its own memory, a local SSD or hard disk, or another machine over the network. The code can look the same, yet the time can differ by more than a million times. These rough numbers explain why caches exist, why calls between continents stay off the busy path, and why big services put servers near their users. You don't need exact figures, only each step's order of magnitude.

The numbers

These are the round figures engineers carry around (a famous list goes by the name "latency numbers every programmer should know"). They are rough orders of magnitude that change with hardware, but their order rarely changes.

read from memory (RAM)                100 ns
read a small block from an SSD        100 µs    1,000 × memory
round trip inside one data center     0.5 ms    5,000 × memory
seek on a spinning hard disk           10 ms    100,000 × memory
round trip between continents         150 ms    1.5 million × memory

The units: 1 millisecond (ms) is 1,000 microseconds (µs), and 1 µs is 1,000 nanoseconds (ns). So one disk seek lasts as long as about 100,000 memory reads, or 100 SSD reads.

The surprise: a round trip to another machine in the same data center (0.5 ms) is about 20 times faster than one seek on your own hard disk (10 ms). Moving parts and long distances are what's slow.

A few more, with the same warning: reading 1 MB in a row takes about 0.25 ms from memory, 1 ms from an SSD and 20 ms from a hard disk. A disk is slow at seeking, not at reading in a row: the same 1 MB scattered in 250 pieces of 4 KB needs 250 seeks, about 2.5 seconds, over 100 times slower. That is why databases and logs like to write in sequence.

Making them feel real: memory = 1 second

Nanoseconds are too small to imagine, so stretch time by 10 million: pretend one memory read takes 1 second.

real time                  if a memory read took 1 second
memory read       100 ns   1 second
SSD read          100 µs   about 17 minutes
same data center  0.5 ms   about 1 hour 23 minutes
disk seek          10 ms   about 28 hours, a bit over a day
across an ocean   150 ms   about 17 days, two and a half weeks

On that scale, a disk answers the next day and another continent in two and a half weeks.

How it works: round trips add up

Latencies add up when steps happen one after another. A common mistake: the app servers move to Amsterdam to be near European users, but the database stays in California. One page needs 10 database calls, made one at a time, each waiting for the one before:

call 1       0 →   150 ms
call 2     150 →   300 ms
call 3     300 →   450 ms
...
call 10  1,350 → 1,500 ms

The page waits 10 × 150 ms = 1,500 ms before it can even be sent. A common rule of thumb says about 0.1 seconds feels instant; this page is 15 times over, almost all of it spent on the ocean. The fixes that help cut the number or the length of the trips:

• Send the calls at the same time. If none needs another's answer, the page waits about one trip: 150 ms. If the other 9 need an id that call 1 returns, the best is two trips in a chain: 300 ms. A page waits for its longest chain of calls, not the sum.
• Ask for everything in one call. One query for all 20 comments instead of one per comment: one trip.
• Keep a copy of the data near the app. With a copy in Amsterdam each trip is about 0.5 ms, so even 10 calls in a row take 5 ms, 300 times faster.

Two fixes that barely help: a faster database server (its own work was a tiny part of each call) and more bandwidth. Bandwidth is how much data fits through at once; latency is how long one trip takes. A 1 KB reply takes about 8 µs to send on a 1 Gbps link; the trip takes 150 ms. Distance sets a floor: California to the Netherlands is about 8,800 km, and light in fiber covers about 200,000 km a second, so even a perfectly straight cable needs almost 90 ms there and back. Real routes are longer, and routers add time.

Why a cache in memory wins

Say the copy in Amsterdam keeps its data on a hard disk and each lookup costs one seek, while a cache server next to it keeps the most-read data in memory. Both are one 0.5 ms trip from the app:

cache:  0.5 ms trip + 100 ns read  ≈  0.5 ms
disk:   0.5 ms trip + 10 ms seek   = 10.5 ms   (about 20 × slower)

The gap is about 20 times, not 100,000, because both lookups pay for the trip. A cache inside the app's own memory skips even that: about 100 ns, 5,000 times faster than asking a cache server. On SSDs the database comes close (about 0.6 ms), and the cache's bigger win is load: say one cache server answers about 100,000 lookups a second and one database server about 5,000.

The choices and their trade-offs

• Calls side by side need calls that don't wait for each other's answers; the longest chain sets the time.
• Copies near the users can be a moment behind the original (replication lag), and writes still travel to the main database.
• Memory costs far more per gigabyte than disk and is wiped on a restart, so the database stays the source of truth and the cache holds the hot part.

Where it shows up in design interviews

You rarely compute latency in an interview; you use it to justify decisions with three rules:

1. Keep hot data in memory. Small data read over and over, like a home page's top-10 list, belongs in a cache.
2. Avoid round trips on the busy path. Make independent calls side by side, batch small queries, and keep calls across an ocean off the path every request takes.
3. Put data near the users. Copies of the data in each region, and of images and videos on servers close to viewers (CDNs), turn 150 ms trips into short ones.

Say the number with the decision: "a cache read is about half a millisecond and a disk seek is ten, so the busy path reads from the cache."

Common interview questions

Why is memory so much faster than a hard disk?

A hard disk must move an arm and wait for the platter to turn to the right spot, about 10 ms a seek. Memory has no moving parts and answers in about 100 ns, 100,000 times faster; SSDs sit in between at about 100 µs.

A page is slow for users far from the servers. What do you check first?

Count the round trips that cross the distance: ten in a row at 150 ms is 1.5 seconds. Make independent calls side by side, batch them, or move the data near the users.

How much faster is a cache than a database?

If the database must seek a hard disk, a cache server in the same data center is about 20 times faster: 0.5 ms against 10.5 ms. Not 100,000 times, because both pay the network trip.

Summary

• Memory about 100 ns, SSD about 100 µs, same data center about 0.5 ms, disk seek about 10 ms, between continents about 150 ms: rough, and they change with hardware.
• One disk seek lasts 100,000 memory reads; if memory took 1 second, a trip across an ocean would take about two and a half weeks.
• Round trips add up: 10 in a row at 150 ms is 1.5 seconds. Side by side it's one trip, or the longest chain; next to the data, 5 ms.
• A cache next door answers in about 0.5 ms, a disk lookup in about 10.5 ms.
• The rules: hot data in memory, few round trips on the busy path, data near the users.