Metrics and Alerting

What the interviewer is asking

"Design a metrics and alerting system" means the system behind dashboards and pages: every server reports numbers such as CPU, requests and latency, engineers chart them, and rules page someone when something breaks. Prometheus and Datadog are well-known tools that do this job.

The interviewer listens for a few things: that you notice the system is write-heavy, the opposite of most designs; that you model the data as time series and watch how many a label creates; that you keep a year of history without keeping every point; and that an alert pages one person once. The walkthrough follows the 8 steps of the interview framework.

1. Scope: what version 1 must do

• Collect 100 metrics from each of 10,000 servers every 10 seconds.
• Show them on dashboards of the last hour, day or month.
• Alert on rules such as "error rate > 1% for 5 minutes", paging someone within a minute of the rule holding.

Park the rest, one line each: logs are text, far bigger and searched by words; traces follow one request through every service; anomaly detection learns what normal looks like and can come later.

2. Data: time series and labels

Each point is a metric name, labels such as host=web-1, a time and a value. A time series is all the points with the same name and labels, in time order. Six points from the agents make three series:

series                 values
cpu{host=web-1}        41.5, 41.5
cpu{host=web-2}        12.0, 12.5
requests{host=web-1}   220, 231

A chart asks for one series over a time range, so the store keeps each series' points together, sorted by time: a time-series database, the shape of a wide-column store. The name and labels are kept once per series, in an index; each point needs only its time and value.

The trap is cardinality: every combination of label values is a series of its own. Request latency labelled by host has 10,000 series. Add user_id for 1 million users and it can reach 10,000 × 1 million = 10 billion, while the whole system has about 1 million today (10,000 servers × 100 metrics). Each series costs an index entry and memory, so keep labels to small, fixed sets (host, service, status code) and leave per-user detail to logs.

3. The API

POST /points
[{"metric": "cpu", "labels": {"host": "web-1"},
  "time": 1792843200, "value": 41.5},
 … 99 more points]
→ 202 Accepted

GET /query?metric=cpu&labels=host:web-1&from=12:00&to=13:00&step=1m
→ 60 values, one a minute

Each server runs a small agent that sends its 100 points in one request every 10 seconds; a request per point would be 100 times the requests, each paying for a connection, headers and a reply. 202 Accepted means the agent never waits for the points to be stored. The query names a metric, labels, a time range and a step: the query service turns the points into one value per step.

Push or pull? Many systems pull instead, as Prometheus does: a collector asks every server for its numbers every 10 seconds (scraping), and a server that stops answering stands out at once. Pushing suits short-lived jobs and servers behind firewalls. The points are the same either way.

4. The busy path: ingesting points

Every agent writes every 10 seconds, whether anyone looks or not, while a dashboard reads a few series only when someone opens it. A URL shortener answers about 100 reads for every write; here writes win by far, so the busy path is ingesting points. The rate is flat, with no busy hour: agents report on a timer. Reads are the bursty side: during an incident everyone opens dashboards at once.

5. The numbers

what                    estimate
points a second         10,000 × 100 ÷ 10 s = 100,000
requests a second       10,000 ÷ 10 s = 1,000
series                  10,000 × 100 = 1 million
points a day            100,000 × 86,400 ≈ 8.6 billion
bytes a day             8.6 billion × 16 bytes ≈ 140 GB
a year at full detail   ≈ 50 TB, before 3 copies

A point as sent is 16 bytes: 8 for the time, 8 for the value. The usual slips: forgetting the 10 seconds gives 1 million points a second; counting points as bytes gives about 9 GB a day. The number that drives the design is the year: about 50 TB, almost all of it 10-second detail nobody will look at again.

6. Services: writers, a query service and a stream

• Writers take the agents' batches, check each point and append it to its series: about 1,000 batches a second, all day.
• The query service reads ranges of series, turns points into steps and merges series, such as 500 hosts into one line.
• A points stream sits between the agents and the writers.

Writes and reads are split because a heavy query that slows the writes makes new points late, and the alerts that read them too. The stream is there because the database pauses now and then, for a minute or two while a server restarts. The agents append their batches to the stream, a log on disk, and are done; the writers catch up once the database is back. As in pub/sub, it's split into partitions by series, each writer reads a few and saves its offset, and if one crashes another resumes where it stopped. Agents that retry instead would all retry at once when the database returns.

7. Scale: shards by series

Say one database server takes about 100,000 points a second: all of today's load. Plan for 10× more servers, 10 million series and about 1 million points a second, split over about 10 shards, each kept in 3 copies.

The shard key is a hash of the series, its name and labels: each series lives whole on one shard, and the series spread evenly. Splitting by time sends every write to the current hour's shard while the other nine sit idle; splitting by metric name makes uneven shards, since one name can't be split. A chart across 500 hosts asks every shard that holds one of them and merges the answers: fan out and merge, as slow as the slowest shard.

8. Deep dive: downsampling and retention

A chart of the last month is a few hundred pixels wide, yet one series has 259,200 points in that month. So downsample: summarize old points into rollups, and delete each kind after its time (retention):

kept               for       per series   all series
10-second points   2 days    17,280       ≈ 280 GB
1-minute rollups   30 days   43,200       ≈ 1.4 TB
1-hour rollups     1 year    8,760        ≈ 280 GB
all of it                                 ≈ 2 TB

A rollup is 32 bytes (time, average, min, max), so the year takes about 2 TB instead of about 50 TB, about 25 times less. A background job rolls up each minute; here is one minute of cpu{host=web-7}, in percent:

12:00:00  12:00:10  12:00:20  12:00:30  12:00:40  12:00:50
28        30        29        95        30        28
rollup 12:00: avg 40, min 28, max 95

Keep the max and the min, not only the average: 40% looks calm, but for 10 seconds the server was at 95%, and in an hourly rollup one spike is 1 point of 360. Hourly rollups come from the minute ones: the average of the 60 averages (each minute has 6 points), the min of the mins, the max of the maxes. Keeping every 6th point instead would keep the 28 and drop the 95.

9. Deep dive: compression and layout

A series' points sit together in time order, so an hour of one series is one read of neighboring bytes, and neighbors are alike. Store the first point in full, then only the changes, delta encoding:

as sent             as stored
1792843200  41.5    1792843200  41.5
1792843210  41.5    +10         +0
1792843220  41.7    +10         +0.2
1792843230  41.6    +10         −0.1

Small numbers fit in a few bits, so points shrink to a few bytes instead of 16. Real stores go further: the change of the change is 0 when every gap is 10 seconds, and decimal values are compared bit by bit with the previous one. Facebook's published Gorilla design averaged about 1.37 bytes a point. Don't drop the times, though: agents miss reports, and one gap would shift every later point.

10. Deep dive: alert rules

An alert evaluator runs each rule every 30 seconds on the last 30 seconds of points. For error_rate > 1% for 5 minutes, a check above 1% makes the alert pending; it fires once every check for 5 minutes has been above 1%; a check under 1% resets it, or resolves it if it fired. One host's error rate:

checks              rate       the alert
12:00:00            0.3%       OK
12:00:30            4.0%       pending since 12:00:30
12:01:00–12:01:30   0.3–0.4%   OK again
12:02:00–12:04:00   2.4–2.8%   pending since 12:02:00
12:04:30            0.8%       OK again
12:05:00–12:09:30   2.6–3.4%   pending since 12:05:00
12:10:00–12:12:00   2.7–3.3%   firing: one page
12:12:30–12:13:00   0.4–0.5%   resolved

One page for one real problem. Without "for 5 minutes" the rule would fire at 12:00:30, 12:02:00 and 12:05:00 and resolve in between: flapping, which teaches people to ignore pages. Counting from 12:02:00 would say 12:07:00, but the dip at 12:04:30 reset the count. The wait is on purpose; the pipeline itself is quick: a point is at most 10 seconds old when sent, spends a few seconds in the stream and writers, and is checked within 30 seconds.

Firing alerts go to a notifier. When a bad release hits all 500 of checkout's hosts, the per-host rule fires 500 times: grouping makes it one page ("checkout errors on 500 hosts"), deduplication drops the same alert at every later check, and labels route it to checkout's on-call engineer. Paging at every check would be 120 pages an hour; send once, remind if nobody acknowledges it, and send "resolved" at the end, through a notification system's queues and retries.

The alerting path must not depend on the system it watches: run it on its own servers, alert when points stop arriving, and have something outside watch the watcher, an alert that always fires and pages from elsewhere when it stops.

What breaks first

• A label with too many values: cap the series per metric.
• The incident rush: cap the series per query and cache recent results.
• Writers behind the stream: alerts judge old data; alert on the lag.
• A dead agent: it looks like a quiet server; alert on missing data.

Summary: the order to follow

1. Scope: collect, show, alert; park logs, traces and anomaly detection.
2. Data: a series is a name plus labels; keep label values few.
3. API: POST /points, 100 at a time, answered 202; GET /query with a range and a step.
4. Busy path: ingesting points, read or not.
5. Numbers: about 100,000 points a second, about 140 GB a day.
6. Services: writers and a query service apart, a stream before the writers.
7. Scale: about 10 shards by a hash of the series.
8. Deep dives: rollups with min and max (about 2 TB a year), delta encoding, and alert rules that page once, routed to the right person.