Monitoring and SLOs

The problem: slow for some, and nobody knows why

A shop's checkout calls two services, stock and payments, and payments asks a fraud check. Shoppers write in: checkout is slow, but only for some of them. Nobody knows which shoppers, why, or since when.

Without measurements every fix is a guess: more servers don't help a service that is waiting on another. Monitoring means every service records what it does, so you can ask the system: since when, for whom, in which service?

How it works: three signals

• Logs are one line per event, with its details: the time, the service, the order, the error. They answer questions about one event ("why did order 81 fail?"), but at volume they cost the most: 400 checkouts a second writing 10 lines of about 200 bytes each is 800 KB a second, about 70 GB a day (a day has 86,400 seconds).
• Metrics are numbers over time: requests a second, error rate, latency percentiles. They're added up as they're collected, so a metric costs about the same at 10 or 10,000 requests a second: cheap to keep for months, and what dashboards and alerts are built on ("is the error rate up since 14:00?").
• Traces follow one request across services. Each service adds a span, its start and end, and all the spans share one trace id. A trace answers "which of the 4 services makes checkout slow?".

Together: metrics say something is wrong and since when, a trace says where, and that service's logs, found by the trace id in each line, say why.

Percentiles, not averages

Here are 10 checkout times, sorted:

rank      1    2    3    4    5    6    7    8    9     10
time    100  100  100  100  100  100  100  100  100  2,000 ms

average = (9 × 100 + 2,000) ÷ 10 = 290 ms
p50 = rank 5  =   100 ms
p90 = rank 9  =   100 ms
p99 = rank 10 = 2,000 ms

Against a goal of under 300 ms, the average of 290 ms says all is well, yet 1 shopper in 10 waited 2 seconds. A percentile keeps the tail in view: p90 is the time 90% of requests stay at or under. The method here is the nearest rank: sort the times and take the one at rank N% × count, rounded up. With only 10 times p99 is just the slowest; dashboards compute it each minute over many requests.

Another sample, sorted: 90, 100, 100, 110, 120, 130, 150, 400, 1,500, 2,400 ms. Its p50, the 5th, is 120 ms; its p90, the 9th, is 1,500 ms.

The tail reaches more people than it seems. A page that makes 10 calls is slow if any one is: if 1 call in 100 is slow, 1 − 0.99 to the 10th power, about 9.6% of pages, are slow, and with 100 calls 63%. Percentiles also can't be averaged (the p99 of two servers isn't the mean of their p99s), so they're computed from all the requests.

Traces: finding the slow service

The p99 metric showed when it started: at 14:00 it jumped from about 200 ms to about 2 seconds. A trace of one slow checkout shows where:

span           start    end    span   own work
checkout           0  1,900   1,900        100
  stock           20    120     100        100
  payments       150  1,850   1,700        100
    fraud check  200  1,800   1,600      1,600

A span includes the time spent waiting for the calls it makes. Payments looks slow at 1,700 ms, but 1,600 ms of that is waiting for the fraud check: subtract each span's calls and only the fraud check has real work to show.

SLIs, SLOs and SLAs

• An SLI (service level indicator) is a measured number: the share of checkout requests that succeed in under 300 ms, measured where requests come in.
• An SLO (objective) is its target: 99.9% of checkouts good, over 30 days.
• An SLA (agreement) is a contract built on it, with consequences such as money back. It's usually looser than the SLO, so the team hears about trouble before it costs money.

Good SLIs measure what users feel. CPU is a possible cause, not a symptom: a service waiting on another uses almost no CPU while its users wait.

Error budgets

99.9% allows 0.1% to be bad. Thirty days are 43,200 minutes, and 0.1% of that is 43.2 minutes: the error budget. Each extra nine divides it by 10:

SLO       bad share   bad time per 30 days
99%       1%          7.2 hours
99.9%     0.1%        43.2 minutes
99.99%    0.01%       4.3 minutes
99.999%   0.001%      26 seconds

Over a year, 99.9% allows about 8.8 hours. The budget can be counted in requests too: 1 checkout in 10 slow for 200 minutes spends as much as all of them failing for 20.

The budget is there to be spent. With plenty left, ship faster and take risks. Used up, slow down risky releases and put the time into reliability until the 30-day window moves past the bad days. Agreed ahead of time, this turns "how reliable is enough?" into a number; Google's SRE books made the practice well known.

Alerting: page on symptoms

A page rings the on-call engineer's phone, even at 3 a.m. Page for symptoms users feel (errors, slow requests, the budget burning fast), and only when someone must act now. Causes like a server at 80% CPU or a disk filling 1% a day go to dashboards and tickets; a page for each teaches people to ignore pages.

At a steady pace, a 30-day budget lasts 720 hours. With 5% of checkouts failing against a 0.1% budget, it burns 50 times too fast and is gone in about 14 hours. A common rule from Google's SRE workbook pages when one hour spends 2% of the month's budget, a burn 14.4 times the steady pace, and opens a ticket for a slow burn.

The choices and their trade-offs

• Log detail or cost: more lines help debugging but cost storage, so detailed logs are often kept for days and metrics for months.
• Metric labels: each label value is its own time series. Labels by service and status code are fine; a label by user id makes millions of series.
• Trace sampling: tracing everything is costly, so systems often keep a sample, say 1 request in 100, plus the slow and failed ones.

Where it shows up in design interviews

After the design, interviewers often ask how you'd know it works. Name the SLO for the busy path ("99.9% of checkouts succeed in under 300 ms, over 30 days") and the metrics you'd watch per service: requests a second, error rate, p50 and p99 latency. Google's SRE book calls latency, traffic, errors and saturation the four golden signals.

Then expect: "checkout got slow, how do you find out why?" In order: an alert fires; the p99 metric shows since when; a trace of a slow request shows which service; that service's logs show why; after the fix, the same metric shows it worked.

Common interview questions

What's the difference between logs, metrics and traces?

Logs record single events with details, metrics measure over time, and traces follow one request across services: metrics tell you that something is wrong, traces where, logs why.

Why watch p99 instead of the average?

An average blends a few slow requests into many fast ones: nine at 100 ms and one at 2,000 ms average 290 ms. High percentiles show the tail, which many users hit because one page makes many calls.

How much downtime does 99.9% allow?

About 43 minutes per 30 days, or about 8.8 hours a year. 99.99% allows about 4.3 minutes per 30 days, and 99% about 7.2 hours.

Summary

• Logs for one event's details, metrics for numbers over time, traces for one request's path.
• Averages hide the tail: watch p50, p90 and p99.
• A span includes its calls: subtract them to find the slow service.
• SLI: a measured number users feel. SLO: its target, like 99.9% over 30 days. SLA: a contract.
• 99.9% leaves about 43 minutes of error budget per 30 days: spend it, and slow down when it's gone.
• Page on symptoms users feel, only when someone must act now.