Timeouts, Retries, Circuit Breakers

The problem: one slow service takes down the rest

Say a checkout service shows "you may also like" from a recommender service, then takes the payment. It has 200 threads and handles 100 checkouts a second, and each checkout holds a thread until its calls answer. Normally the recommender answers in about 50 ms, so about 5 threads are waiting on it at any moment: threads in use = calls a second × seconds each call waits (Little's law), and 100 × 0.05 = 5.

Then the recommender slows down: each answer takes 30 s. Nothing fails; the answers just don't come back. Every second, 100 more threads start waiting and none returns:

time     waiting   free
before   5         195
0.5 s    50        150
1 s      100       100
2 s      200       0

After 2 seconds every thread is waiting, and new checkouts queue and fail, though the payment service is fine. To carry 30-second waits, checkout would need 100 × 30 = 3,000 threads. A dependency the order doesn't even need has taken down the service that makes money: a cascading failure. Slow is worse than down here: a service that refuses at once frees the thread at once.

Timeouts: decide how long to wait

Every call to another service needs a timeout, set from that service's normal latency. Say half of the recommender's healthy answers come within 50 ms and 99 in 100 within 200 ms (its p99). A timeout of about 300 ms, a bit above the p99, almost never cuts off a healthy call (at most 1 in 100 takes over 200 ms), and a slow call frees its thread after 0.3 s. Now at most 100 × 0.3 = 30 threads wait on the recommender, and 170 stay free.

Too long (30 s) is the pile-up above. Too short (50 ms, the usual time) cuts off half the healthy answers. Down a chain of calls, each timeout should be shorter than the one above it, or the caller gives up while the work below goes on.

Then decide what the caller does on a timeout: its fallback. For the recommender, leave "you may also like" out, or show a cached list or the best sellers, so the page degrades gracefully and the order goes through. Not every call has one: an order can't skip the payment, so there the fallback is a clear error the user can act on ("try again").

Retries: only safe calls, only failures that pass

• Retry timeouts and 503s (and a 429 after the wait it asks for), on reads or on writes sent with an idempotency key.
• Don't retry a 400 or a 404: the same request fails the same way. Don't retry a POST without a key that timed out either: the card may already be charged.

Retries also multiply. If the shopper's app, checkout and payment each make up to 3 tries, one tap on Pay can reach an overloaded payments database 3 × 3 × 3 = 27 times. So retry at one layer only, a couple of times, within a retry budget: a common rule caps retries at about 10% of a client's calls, so at 100 checkouts a second, about 10 retries a second at most.

Backoff with jitter

Backing off (1 s, then 2 s, then 4 s) gives a struggling service time. But say the recommender crashes and restarts, and the 1,000 calls waiting on it fail at the same moment. If every caller waits exactly 1 s, all 1,000 retries land in the same 100 ms. If the restarting service handles about 200 per 100 ms, 800 fail again, back off 2 s together and arrive together again.

The fix is jitter: each caller waits a random time between 0 and its backoff, a choice often called full jitter. One run with 1,000 callers, counting retries per 100 ms after the crash:

slot          no jitter   full jitter
0–0.1 s       0           97
0.1–0.2 s     0           96
0.2–0.3 s     0           99
0.3–0.4 s     0           117
0.4–0.5 s     0           90
0.5–0.6 s     0           85
0.6–0.7 s     0           98
0.7–0.8 s     0           106
0.8–0.9 s     0           107
0.9–1.0 s     1,000       105

Every slot gets between 85 and 117 retries, under the 200 the service handles, so all 1,000 get through on their first retry. In code:

import random

def wait_before_retry(attempt):            # attempt = 1, 2, 3, ...
    backoff = min(30, 2 ** (attempt - 1))  # 1 s, 2 s, 4 s ... at most 30 s
    return random.uniform(0, backoff)      # full jitter

Circuit breakers: stop calling a service that keeps failing

A recommender that's down for minutes still costs every checkout a timeout, and keeps getting 100 calls a second while it restarts. A circuit breaker in the caller counts the results of real calls and has three states:

• Closed: calls go through. If at least half of the calls in the last 10 s failed (with a minimum, say 20 calls, so 1 failure out of 2 doesn't count), it opens.
• Open: for, say, 30 s, every call fails fast, straight to the fallback, without calling the service.
• Half-open: a few test calls (say 3) go through. If all work, it closes; if one fails, it opens for another 30 s.

A run at 100 calls a second, where from 10 s the recommender fails every call right away, and it's back at 60 s:

time   what happens                             state
0 s    calls work                               closed
10 s   every call starts failing                closed
15 s   500 of the last 1,000 calls failed       open
45 s   half-open: the first test call fails     open
60 s   the recommender is back                  open
75 s   half-open: 3 test calls work             closed

A load balancer's health checks call a /health page; a breaker judges by the real calls, and each caller keeps its own. The pattern was popularized by Michael Nygard's book Release It!.

Bulkheads

A bulkhead gives each dependency its own small pool of threads, like the watertight walls of a ship. If at most 40 of checkout's threads may wait on the recommender, a slow recommender fills its 40 in 0.4 s, further calls get the fallback at once, and the other 160 keep serving checkouts.

Choices and trade-offs

• Retries: they hide failures that pass, but make an overload worse. Limit them, add jitter, keep a budget.
• Breaker settings: too sensitive cuts off a healthy service at every blip; too slow lets failures pile up. Open too briefly and it hammers the service; too long and the feature stays off after it recovers.

Where it shows up in design interviews

Every arrow between two services in your design is a call that can be slow or fail, and interviewers often ask "what happens when this service is slow?". For each call, say: a timeout set from the normal latency, retries with backoff and jitter on idempotent calls only, a circuit breaker, and the fallback when it's open. Add bulkheads for a service with several dependencies, and idempotency keys on the payment path.

Common interview questions

What is a cascading failure?

A failure that spreads from one service to the services that call it, usually because their threads or connections all end up waiting on it. Timeouts, bulkheads and circuit breakers stop the spread.

How do you choose a timeout?

From the dependency's normal latency: a bit above its p99, so healthy calls almost never hit it. Then decide the fallback for when it fires.

Why add jitter to retries?

Callers that fail together and wait the same time retry together, and hit the recovering service with the same spike again. A random wait spreads them out.

What are the states of a circuit breaker?

Closed (calls go through), open (calls fail fast for a while after too many failures) and half-open (a few test calls decide whether to close or open again).

Summary

• A slow dependency can hold every thread of its caller: a cascading failure.
• Give every call a timeout just above the normal p99, and a fallback.
• Retry only safe calls and failures that pass, at one layer, within a budget.
• Back off with jitter so retries don't arrive together.
• A circuit breaker stops calls to a failing service, then tests it before closing; bulkheads cap what one dependency can take.