CI/CD (GitHub Actions)

What the interviewer is asking

"Design a CI/CD system like GitHub Actions" means: every time someone pushes code, build it, test it and maybe deploy it, then show the result on the commit. CI (continuous integration) builds and tests every change; CD (continuous delivery) ships the changes that pass.

Like an online judge, it runs code it didn't write, so a queue in front of isolated runners is the start. What's new: a pipeline is a graph of jobs that wait for each other, jobs run for minutes with full rights, people watch logs live, and the deploy job holds keys to production. The walkthrough follows the 8 steps of the interview framework: scope, data, API, the busy path, numbers, services, scale and deep dives.

1. Scope: what version 1 must do

• Run the pipeline on every push: a file in the repository lists the jobs (build, test, lint, deploy) and which jobs each one needs.
• Live logs for every job while it runs.
• Pass or fail on the commit: the green check or the red cross.
• Artifacts: keep build outputs, such as the built app or test reports, to download later.
• Deploy from main: the deploy job runs only on the main branch and ships with a canary first, as the Safe Deployments lesson does.

Park: a marketplace of reusable steps, matrix builds (one job on many OS versions), self-hosted runners.

2. Data: runs, jobs and files

record       fields
run          id, repo, commit, branch, status, created
job          id, run, name, needs [job names], status, runner, started, ended
log chunks   logs/r7/build/0001, 0002 …        (object storage)
artifact     artifacts/r7/app.tar.gz, say 200 MB  (object storage)
cache        caches/shop-web/2a7c037f.tar      (object storage)

Runs and jobs are small rows that pages list and the scheduler updates: a database. The pipeline file isn't copied: the CI API reads it from the repository at the pushed commit, so the pipeline is versioned with the code. Logs, artifacts and caches are files, written once and read whole: object storage, with each file's key in its row (the Files and Object Storage lesson).

3. The API: three calls

POST /runs                    (the code host's push webhook)
  {"repo": "shop/web", "commit": "a1b2c3", "branch": "main"}
  → 202 Accepted   {"run": "r7"}

GET /runs/r7
  → 200 OK   {"status": "running", "jobs": [{"name": "build", "status": "passed"}, …]}

GET /runs/r7/jobs/build/log   (server-sent events)
  → the log, chunk by chunk, as the job prints it

The push arrives as a webhook: the code host calls POST /runs the moment a push lands. As in the Webhooks lesson, check its signature, skip repeats by the event's id, and answer 202 at once; the jobs take minutes. Polling millions of repositories would mostly hear "nothing new" and still start runs late. When the run ends, the CI API calls the code host back to set the commit's status.

4. The busy path: from push to check mark

1. The code host sends the push webhook: POST /runs for commit a1b2c3.
2. The CI API reads the pipeline file at that commit and saves run r7 with its 5 jobs.
3. Jobs that need nothing go on the queue: here, only build.
4. Each job that passes lets the jobs that needed it start.
5. When every job is done, the run is marked passed or failed.
6. The CI API sets the green check or the red cross on the commit.

5. The numbers

From the interviewer: 1 million pushes a day, 5 jobs each, about 4 minutes per job.

• Jobs: 1 million × 5 = 5 million jobs a day.
• Work: 5 million × 4 minutes = 20 million job-minutes a day.
• Jobs starting: 5 million ÷ 100,000 seconds (a day is 86,400, rounded) = 50 a second, so about 250 a second at busy hours, 5× the average.
• Jobs running at once: a rate times how long each lasts: 250 a second × 240 s = 60,000 jobs at once at busy hours (about 70,000 with the exact 86,400), about 12,000 on average.
• Logs: say 10 lines a second per running job: 600,000 lines a second at busy hours.
• Artifacts: say one push in ten keeps a 200 MB artifact: 100,000 × 200 MB = 20 TB a day, which is why artifacts expire after a while.

6. Services: the API never runs code

One app that also runs every job would put 60,000 jobs of customers' code next to the API. As in Design LeetCode, split it:

• CI API: checks webhooks, reads the pipeline file, saves runs and jobs, decides which jobs are ready, and sets the commit's status. It never runs customer code.
• Job queue: holds ready jobs until a runner is free, so a rush waits instead of failing (the Message Queues lesson). A job whose runner dies goes back on the queue after a timeout.
• Runners: take a job, clone the repository at the commit, run the job's commands, upload logs, artifacts and caches to object storage, and report the result to the CI API.

7. Scale: a runner pool that follows the queue

About 60,000 jobs run at once at busy hours, about 12,000 on average, far fewer at night. A fixed 60,000 runners pays for about 5× what you use. A fixed 12,000 finishes only 50 jobs a second while 250 arrive at busy hours, so the queue grows by about 12,000 jobs a minute. Instead, autoscale: jobs waiting in the queue is the signal to add runners, idle runners the signal to remove some, and a few spares stay booted because starting a machine takes a while (say 30 seconds). Cap the jobs one customer runs at once, so one huge push can't take the whole pool.

8. Deep dive: the job graph

Each job lists the jobs it needs. Jobs and needs form a directed graph with no cycles, and the CI API starts each job the moment everything it needs has passed:

job         needs                        minutes   runs
build       –                            4         0–4
test-unit   build                        4         4–8
test-e2e    build                        8         4–12
lint        build                        2         4–6
deploy      test-unit, test-e2e, lint    2         12–14

Keep a counter per job: how many of its needs haven't passed. Jobs at 0 are queued; when a job passes, every job that needs it counts down, and any that reach 0 are queued. That's a topological order, the same order as the Course Schedule coding lesson. Run r7 finishes at 14 minutes, against 4 + 4 + 8 + 2 + 2 = 20 one after another. The longest chain, build → test-e2e → deploy, is the critical path: speeding up lint changes nothing. A pipeline whose needs form a loop is rejected when the run is created.

When a job fails, everything that needs it is skipped: if test-unit fails at 8 minutes, deploy never starts, while test-e2e still finishes at 12 so the developer sees every failure from one push. The run is marked failed and the commit gets a red cross. Retrying until it passes would hide real bugs.

9. Deep dive: runners and isolation

Unlike a judge's short sandboxed run, a CI job runs for minutes with full rights: it installs packages, writes anywhere, needs the network and can leave programs running. The next job on that machine may belong to another company. So each job gets a fresh virtual machine, destroyed when the job ends: nothing carries over, one repository can't read another's code or secrets, and every job starts from the same clean image. GitHub's hosted runners give each job a fresh VM. Containers start faster but share the host's kernel, a weaker wall for code running as root. Wiping only the job's folder isn't enough: a job can leave things anywhere.

10. Deep dive: caching dependencies

Since every job starts on a fresh VM, it installs its dependencies again: in a typical job, 2 of its 4 minutes. Save the installed folder in object storage under a key made from a hash of the lockfile, the file that pins every package's exact version. Same lockfile, same key, a cache hit restored in seconds; change a dependency and the key changes. Four pushes of test-unit:

push   change           key        result
1      first push       2a7c037f   miss: install 2 min, save
2      app code only    2a7c037f   hit: restored in seconds
3      adds dayjs       caa2e930   miss: install 2 min, save
4      app code only    caa2e930   hit

Keying by commit never hits, since every push is a new commit; keying by branch restores old packages after a dependency changed. A common extra: let a branch read its own caches and main's, never another branch's, so a pull request can't plant a bad cache that main restores. The savings are large: at 5 million jobs a day, a 90% hit rate saves about 9 million of the 20 million job-minutes.

11. Deep dive: logs and secrets

Writing 600,000 log lines a second as database rows would be over 100 times what one database server handles (about 5,000 simple writes a second). Instead the runner sends the log in chunks, say every 5 seconds: 50 lines per write, about 12,000 writes a second, spread over object storage's many servers. The runner also hands each chunk to the CI API, which pushes it to every browser watching over server-sent events or a WebSocket (the Real-Time Updates lesson); a page opened mid-job reads the stored chunks first, then follows live.

Secrets need three rules:

• Scoped: the cloud key goes only to the deploy job, only on main, never to pull requests from forks (code nobody has reviewed), and never into the repository.
• Masked: any secret's value printed to the log is replaced with ***.
• Short-lived: instead of a long-lived key, the CI system signs a statement of which repository, branch and run this is, and the cloud trades it for credentials that expire within the hour (the Authentication and API Keys lesson). GitHub Actions does this with OpenID Connect.

What breaks first

• A runner dying mid-job: the job returns to the queue and runs again, so a deploy step must be safe to repeat.
• A lost webhook: now and then, ask the code host for recent pushes and start any run that is missing.
• Flaky tests: report the tests that pass only on a retry.
• Storage filling up: a cache size limit per repository (least recently used out first) and artifacts that expire.

Summary: the order to follow

1. Scope: run each push's pipeline, live logs, pass or fail on the commit, artifacts, deploy from main.
2. Data: run and job rows; logs, artifacts and caches in object storage.
3. API: the push webhook calls POST /runs (202), GET /runs/{id}, a live log stream.
4. Busy path: push, run, each job as its needs pass, the mark on the commit.
5. Numbers: 20 million job-minutes a day, about 60,000 jobs at once at busy hours.
6. Services: the CI API never runs code; a job queue feeds runners.
7. Scale: a runner pool that grows and shrinks with the queue.
8. Deep dives: the job graph in topological order; a fresh VM per job; a cache keyed by the lockfile's hash; logs in chunks; secrets scoped, masked and short-lived.

The same steps work for any "design X" question.