Log Collection

The problem

"Design a log collection system." Every server writes log lines such as 12:00:01 checkout web-7 ERROR timeout calling payments, and during an incident engineers must find checkout's errors with the word timeout from the last hour.

This practice problem reuses the pattern of Design Metrics and Alerting, with a Kafka-like queue in front. What's new: text lines far bigger than metric points, bursts when something breaks, search by words, and 90 days kept. Park charts made from logs and tracing one request across services.

The numbers

From the interviewer: 10,000 servers, each writing about 1,000 lines a second of about 200 bytes. Round a day to 100,000 seconds (86,400 really).

what              estimate
lines a second    10,000 × 1,000 = 10 million
bytes a second    10 million × 200 bytes = 2 GB
a day             2 GB × 100,000 s ≈ 200 TB (about 170 TB with 86,400)
compressed, 10×   about 20 TB a day
one server        1,000 × 200 bytes = 200 KB a second

Metrics took in about 1.6 MB a second: logs are over 1,000 times the bytes. The slips: one line a second per server gives 200 GB a day; forgetting the day, 2 GB.

Getting lines off the server: an agent

Sending each line as it's logged would be 10 million requests a second, and a slow receiver would make logging calls wait or drop lines, slowing the app just when something is wrong.

Instead the app writes to a local file, which never waits on the network. An agent reads new lines, sends about 1,000 at a time, compressed, and saves its place in the file once the indexers confirm. If they're slow, the file just grows (an hour is about 720 MB), and after a restart the agent resumes from its saved place. In all: 10,000 requests a second. An agent buffering in memory would lose its lines on a restart.

Bursts: a Kafka-like queue in front

Say an outage triples the lines, from 2 GB to 6 GB a second, for 10 minutes, while the indexers handle about 3 GB a second. A queue between agents and indexers takes it all, since appending is cheap:

backlog   (6 − 3) GB a second × 600 s = 1.8 TB
drain     1.8 TB ÷ (3 − 2) GB a second spare = 1,800 s = 30 minutes

As in Message Queue (Kafka), it has partitions, offsets and 3 copies, so a crashed indexer resumes where it stopped. Leaving the backlog on the agents' disks keeps it on the failing servers and sets 10,000 agents retrying at once; dropping it loses half the outage's lines.

Floods: drop debug lines first

A bad release turns on debug logging in checkout, which writes 1% of the lines, about 100,000 a second: it jumps to 2 million. Say each service may send twice its usual lines, 200,000 for checkout, so it's 10 times over. Its agents cut by level:

• Keep every line: errors, warnings, audit lines, and a count of the lines dropped.
• Keep a sample: info lines such as GET /cart 200 in 35 ms, 1 in 10 if dropping debug isn't enough, marked as sampled.
• Drop first: debug lines, the 1.9 million new ones a second.

Without debug, checkout is back to about 100,000 a second, within its share, and the count makes any gap a known one.

Search: scan blocks or index every word

Engineers search the last 7 days, mostly one service over the last hour or two: service=checkout level=error timeout. The choice is cost. Say lines compress about 10×, and an index of every word takes about as much room as the lines.

• An inverted index of every word, as in Search Engine: any word over any week comes back fast. But 7 days of lines are 1.4 PB, so the index adds about 1.4 PB to 140 TB of compressed lines, 11 times the storage, and every word of 10 million lines a second must be indexed on arrival.
• Compressed blocks per service and hour, with a small index of which block holds what: 140 TB. A search of checkout's last hour reads only its blocks, 72 GB of lines; say 50 workers each scan about 1 GB a second: under 2 seconds. One word over every service for a week means scanning 1.4 PB: hours.

Most searches name a service and a recent window, so the blocks win. A SQL table with LIKE '%timeout%' fails both ways: 10 million inserts a second is about 2,000 database servers at 5,000 writes each, and a leading % can't use an index.

Retention: hot, cold, deleted

tier               days      where            size
hot, searchable    1–7       the log store    140 TB
cold               8–90      object storage   about 1.7 PB
deleted            from 91

Security wants 90 days; engineers search almost only the last 7. All 90 days searchable would be 1.8 PB on fast disks, about 13 times the hot store. Object storage is far cheaper per TB and needs no search servers; a rare old search loads that service's blocks back first, in minutes.

Same pattern as Metrics and Alerting

Agents send batches; ingest is the busy path, written all day, mostly never read; a stream on disk sits in front of the writers, here the Kafka-like queue; searches run apart from the writers. Metrics rolled old points into averages; text can't be averaged, so logs move old blocks to cheaper storage.

Summary

• Scope: collect every line, search recent ones, keep 90 days.
• Numbers: 10 million lines, about 2 GB, a second; about 200 TB a day.
• Agent: a local file, shipped in batches by an agent that remembers its place.
• Queue: a Kafka-like queue in front of the indexers holds a burst.
• Floods: drop debug first, then sample info; never errors.
• Search: compressed blocks by service and hour, scanned.
• Retention: 7 days searchable, object storage to day 90, then deleted.