API Gateway

The problem

A company's mobile app calls 20 backend services directly, and each checks sign-ins and rate limits its own way. The interviewer asks for an API gateway: one front door that every request goes through.

Version 1 does, once, the jobs every request needs: route it, check who is calling, apply the rate limit, accept HTTPS. Park the rest: merged replies, caching, a partner portal.

The numbers

Say about 1 billion requests a day across the 20 services, with busy hours at about 5× the average.

what                        estimate
requests a second, average  1 billion ÷ 100,000 s ≈ 10,000
requests a second, busy     10,000 × 5 ≈ 50,000
one gateway server          about 10,000 a second (say)
gateway servers             50,000 ÷ 10,000 = 5, so run 6

A day has 86,400 seconds; 100,000 keeps the arithmetic easy (the exact figure gives about 58,000). The number to notice: the gateway carries the sum of every service's traffic, not one service's share (2,500 a second if all 20 were alike).

Decision 1: route by path

The app knows one address. The gateway keeps a route table of path prefixes in memory:

path starts with   goes to
/orders            orders service
/search            search service
/users             users service

Matching a prefix in memory takes microseconds, and the table changes only when a service is added or moved, so it lives in config that each gateway server reloads. A routing database asked on every request would add a wait to all 50,000 requests a second. Letting the app call services directly defeats the point: no service could move without breaking old apps. Within a service, the gateway spreads calls over its servers, with health checks (the Load Balancing lesson).

Decision 2: check the caller once

Requests carry a signed 15-minute token or a partner's API key (the Authentication and API Keys lesson). The gateway checks what the request alone can answer: the signature and expiry (401 if bad) and the scopes (403 if they don't allow the call). The service checks what needs its own data: does Ana own order 42, can it still be canceled.

The gateway passes the verified user id on in a header, after dropping any such header the app sent, or anyone could claim to be Ana:

(from the app, over HTTPS)
GET /orders/42
Authorization: Bearer eyJhbGciOi…
X-User-Id: admin

(from the gateway to an orders server)
GET /orders/42
X-User-Id: ana

Services accept calls only from the gateway, so they can trust that header.

Decision 3: the order of the checks

1. Decrypt: TLS termination at the gateway, which holds the certificate; otherwise it couldn't read the path or the token.
2. Check the token's signature and expiry: local, no trip.
3. Take a token from the caller's bucket in the counter store.
4. Forward to a server of the service, with the user id, and pass the reply back.

Checking the token first refuses a forged one before it costs a trip or uses up a real caller's limit. Behind the gateway, calls go on the private network, encrypted again (mutual TLS) where rules require it.

Decision 4: stateless and scaled out

Everything passes through the gateway, so it can't be a single point of failure. Run identical gateway servers behind a load balancer, each keeping nothing of its own: routes from config, tokens that check themselves, buckets in the counter store. The gateway is stateless, so any server can take any request. At about 10,000 requests a second each, busy hours need 5; run 6, so losing one still leaves enough, with the load balancers in a pair.

One big server with a standby must still carry 50,000 a second alone. Servers keeping their own buckets bring back the Rate Limiter's bug: over 6 servers, a caller gets up to 6× its limit.

Decision 5: one slow service

Search starts answering in 10 seconds instead of 50 ms. Say each gateway server sends it about 1,000 requests a second and can keep about 10,000 open at once, for all services. Requests open = requests a second × seconds each waits:

search answers in      open search requests, per gateway server
50 ms (normal)         1,000 × 0.05 s = 50
10 s (slow)            1,000 × 10 s = 10,000, every slot
10 s, 300 ms timeout   1,000 × 0.3 s = 300

Unprotected, search takes every slot and orders calls fail too. Give each service a timeout just above its normal p99 (300 ms here), a cap on its open requests, and a circuit breaker: once half its calls in the last 10 seconds fail, the gateway answers 503 at once and tests the service again later (the Timeouts, Retries, Circuit Breakers lesson). Retrying up to 3 times triples the load on a struggling service. More gateway servers only buy room to wait: 6,000 search calls a second × 10 s = 60,000 waiting, every slot of your 6 servers.

Same pattern as the Rate Limiter

A rate limiter is one check on every request, with counts in a shared in-memory store, and a gateway is where it usually runs. What carries over: a token bucket per caller, taken in one step so two gateway servers can't both spend the last token; failing open if the store stops answering; a tight time budget. What's new: routes, the caller checked once, TLS, and protection from slow services.

Summary

• Scope: one front door for 20 services: routing, sign-in checks, rate limits, HTTPS.
• Numbers: about 50,000 requests a second at busy hours, the sum of all services.
• Routing: a table of path prefixes in each gateway server's memory.
• Sign-in checks: signature, expiry and scopes at the gateway, the user id passed on; each service decides what that user may do.
• Order: end TLS, check the token, take from the caller's bucket, forward.
• Scale: 6 stateless gateway servers behind a pair of load balancers.
• Slow services: a timeout and a cap per service, and a circuit breaker.