The System Design Bible

The 6-step attack plan (45-min slot)

1. Requirements — 5–10 min. Functional, non-functional (scale, latency, consistency, availability), explicit scope cuts. Close with: "So we're building X for Y users under Z constraints — agreed?"
2. API & data model — 5 min. 3–6 endpoints, 2–4 core entities. Writing them down forces precision.
3. Capacity estimation — 5 min. QPS, storage, bandwidth. Round aggressively; the goal is justifying choices, not decimals.
4. High-level design — 10–15 min. Full request path on the board: client → edge → services → stores → async. Breadth before depth.
5. Deep dives — 15–20 min. 1–2 hardest components: sharding, consistency, failure modes, hot keys.
6. Wrap-up — 5 min. Bottlenecks, SPOFs, monitoring, "with more time I'd…". Self-critique is a staff signal.

Latency numbers (order of magnitude)

QPS & storage quick formulas

• Seconds/day ≈ 86,400 → round to 100k. 10M requests/day ≈ 100 QPS average.
• Peak ≈ 2–5× average (spiky consumer traffic: use 10×).
• DAU → QPS: DAU × actions/user/day ÷ 100k. 100M DAU × 10 actions = 1B/day ≈ 10k QPS avg, ~30–50k peak.
• Storage: items/day × bytes/item × retention days × replication factor (usually 3). 100M msgs/day × 1 KB × 5 yr ≈ 100 GB/day ≈ ~180 TB raw, ~0.5 PB replicated.
• Bandwidth: QPS × payload size. 10k QPS × 10 KB = 100 MB/s ≈ 0.8 Gbps.
• Cache sizing (80/20): hot set ≈ 20% of daily working set covers ~80% of reads.
• Connections: concurrent users × 1 socket each; 10M concurrent WebSockets ÷ ~200k conns/box ≈ 50–100 gateway nodes.

"If you hear X → reach for Y"

Big decisions — one-liners

Numbers to memorize (order-of-magnitude, not gospel)

• Single Postgres: low tens of thousands of read QPS with tuning/indexes; ~5–10k write TPS; hundreds of GB to low TB comfortable per node.
• Redis: ~100k+ ops/s per node (simple GET/SET), sub-ms latency; cluster for more.
• Kafka/Kinesis partition (shard): single-digit MB/s to tens of MB/s each; Kinesis shard = 1 MB/s in, 2 MB/s out, 1k records/s — scale by partitions.
• One server: ~65k ephemeral ports per (src IP, dst IP, dst port) tuple; ~10k–200k concurrent WebSockets per node depending on tuning.
• DynamoDB partition: 3k RCU / 1k WCU hard cap per partition — hot partitions throttle regardless of table capacity.
• SQS: effectively unlimited throughput (standard); FIFO ~300 msg/s per message group (3k batched).
• S3: 3,500 PUT / 5,500 GET per second per prefix; 5 TB max object.
• Lambda: cold start ~100 ms–1 s+ (worse in VPC/large bundles); 15-min max runtime.
• Availability math: 99.9% = ~8.7 h down/yr; 99.99% = ~52 min; serial dependencies multiply (0.999 × 0.999 ≈ 0.998).
• Payload sanity: tweet ~1 KB, photo ~500 KB–2 MB, 1 min video ~5–50 MB.

The universal skeleton

Nearly every answer starts as a variation of this diagram. Draw it, then mutate it for the question.

Classic pitfalls — top 10

1. Jumping to architecture before requirements — designing the wrong system perfectly.
2. No numbers — choosing Kafka-and-shards for 50 QPS, or one Postgres for 500k QPS.
3. Saying "exactly once" without idempotency — interviewers pounce on this.
4. Cache with no invalidation story, no TTL, and no stampede protection.
5. Single point of failure left on the board (one LB, one DB, one region) and never mentioned.
6. Sharding by a low-cardinality or hot key (date, country, celebrity user ID).
7. Ignoring the write path / async path — only designing the happy read flow.
8. Distributed transactions across services instead of sagas + idempotency.
9. Silence — designing in your head instead of narrating trade-offs out loud.
10. Never self-critiquing: no bottleneck list, no monitoring, no "what breaks first at 10×".

Say this, not that

The 6-step framework (with time-boxes)

Treat the interview like an incident bridge: you own the clock, you narrate state, and you never go silent. The framework below fits a 45-minute slot; stretch each phase ~20% for a 60-minute one. The single biggest failure mode is spending 25 minutes on requirements and never designing anything — the time-boxes exist to prevent exactly that.

• 1. Requirements (5–10 min). Functional + non-functional + explicit scope cuts. End with a spoken summary: "So we're building X for Y users with Z constraints — agreed?"
• 2. API & data model (5 min). 3–6 endpoints and 2–4 core entities. This forces precision: you can't hand-wave a system whose inputs and outputs you've written down.
• 3. Capacity estimation (5 min). QPS, storage, bandwidth — rounded aggressively (Chapter 2). The point is to justify architecture choices, not to be right to two decimals.
• 4. High-level design (10–15 min). Boxes and arrows: clients → edge → services → data stores → async pipelines. Get the whole request path on the board before drilling anywhere.
• 5. Deep dives (15–20 min). Pick the 1–2 hardest components (or let the interviewer pick) and go deep: data model, sharding, consistency, failure modes, hot spots.
• 6. Wrap-up (5 min). Bottlenecks, single points of failure, monitoring, what you'd do with more time. Ending with self-critique is a staff signal in itself.

Exactly which clarifying questions to ask

Don't ask twenty questions; ask the eight that change the architecture. Group them, fire them in one burst, and write the answers down where the interviewer can see them.

Functional

• Who are the users? Consumers, businesses, internal services? A B2B invoicing API and a consumer feed are different planets.
• What are the 2–3 core flows? "Post a tweet, read a timeline, follow a user" — name them and refuse everything else politely.
• Read-heavy or write-heavy? Ask for the read:write ratio. 100:1 (feed) vs 1:1 (chat) vs 1:100 (metrics ingest) selects completely different toolkits.

Non-functional

• Scale: DAU/MAU, items created per day, peak vs average. "100M DAU" is the number the whole design hangs off.
• Latency SLO: p99 target for the hot path — 100 ms feed read vs 1 s report generation changes caching strategy entirely.
• Consistency vs availability: "If the network partitions, do we show stale data or an error?" For payments the answer is usually consistency; for likes, availability.
• Durability: Can we ever lose a write? (Payments: never. Analytics events: 0.01% loss may be fine.)
• Cost/constraints: Greenfield or existing stack? Team size? Any compliance regime (PCI, GDPR data residency)?

Scope cutting

• Explicitly park: auth, billing, admin tooling, ML ranking, abuse — "I'll assume an existing auth service and skip moderation unless you want it."
• Cutting scope is a positive signal: staff engineers descope for a living. Just cut out loud and get a nod.

// Requirements checklist as a type — literally what your whiteboard
// notes column should contain after minute 10.
interface RequirementsNotes {
  functional: {
    actors: string[];              // ["end user", "creator", "internal cron"]
    coreFlows: [string, string, string?];  // force yourself to max 3
    readWriteRatio: `${number}:${number}`; // "100:1" — ask, don't guess
  };
  nonFunctional: {
    dau: number;                   // 100_000_000
    writesPerDay: number;          // 500_000_000
    latencySloMs: { p50: number; p99: number }; // hot path only
    // Discriminated union: forces you to PICK a side of CAP per flow.
    consistency:
      | { mode: 'strong'; why: string }        // "money movement"
      | { mode: 'eventual'; staleness: string } // "feed, <5s stale ok"
      | { mode: 'mixed'; strongFlows: string[] };
    durability: 'zero-loss' | 'best-effort';
    multiRegion: boolean;
    compliance?: ('PCI' | 'GDPR' | 'HIPAA')[];
  };
  outOfScope: string[];            // say these OUT LOUD and get a nod
  assumptions: string[];           // "auth exists", "images via CDN"
}

const tinyUrl: RequirementsNotes = {
  functional: {
    actors: ['anonymous reader', 'registered creator'],
    coreFlows: ['create short link', 'redirect on click', 'view stats'],
    readWriteRatio: '100:1',
  },
  nonFunctional: {
    dau: 10_000_000,
    writesPerDay: 1_000_000,
    latencySloMs: { p50: 20, p99: 100 },   // redirects must be fast
    consistency: { mode: 'eventual', staleness: 'new link live <1s' },
    durability: 'zero-loss',               // losing a link breaks the web
    multiRegion: true,
  },
  outOfScope: ['custom domains', 'abuse scanning', 'billing'],
  assumptions: ['CDN available', 'analytics can lag minutes'],
};

A concrete API sketch beats ten minutes of talk

Two minutes of typed endpoints locks down semantics — idempotency, pagination, and payload shape — that vague talk never does. Fintech habit that transfers everywhere: put the idempotency key in the write path from the first line.

// API sketch: enough to anchor the whole design, small enough for 5 min.
type Cursor = string; // opaque, encodes (shardKey, timestamp, id)

interface CreateLinkRequest {
  longUrl: string;
  customAlias?: string;
  expiresAt?: string;              // ISO 8601
  idempotencyKey: string;          // client-generated UUID — retries are
}                                  // free, duplicates are impossible

interface CreateLinkResponse {
  shortCode: string;               // "x7Kp2Q"
  shortUrl: string;                // "https://sho.rt/x7Kp2Q"
}

interface LinkStatsResponse {
  shortCode: string;
  clicks: number;                  // eventually consistent — say so!
  uniqueVisitors: number;
  window: '24h' | '7d' | 'all';
}

// The contract, as the interviewer sees it on the board:
//   POST /v1/links            -> 201 CreateLinkResponse   (write path)
//   GET  /:shortCode          -> 302 Location: longUrl    (hot path, p99 < 100ms)
//   GET  /v1/links/:code/stats?window=7d -> LinkStatsResponse
//   GET  /v1/links?cursor=... -> paginated list (cursor, not offset —
//                                offset pagination dies at scale)
interface Paginated<T> {
  items: T[];
  nextCursor: Cursor | null;       // null => end of list
}

Drive the conversation — don't be driven

• Narrate transitions: "Requirements are locked; I'll sketch the API next, then estimate." You set the agenda; the interviewer redirects only if they need to.
• Offer forks instead of waiting: "I can deep-dive the fan-out service or the storage layer — which is more interesting to you?" You look collaborative and in control.
• Timestamp yourself: glance at the clock at each phase boundary. Saying "we're 20 minutes in, let me get the full picture up before drilling" is a maturity signal.
• When interrupted, treat it as data: an interviewer's question is a hint about what they score on. Answer it fully, then explicitly return: "…and that closes the cache question; back to the write path."
• Never go silent for more than ~20 seconds. Thinking out loud badly beats thinking silently well — silence is unscoreable.

Senior vs staff: what actually differs

Phrases that signal seniority (use them verbatim)

• "Given ~X QPS and a p99 of Y ms, I'd start with…" — numbers before nouns.
• "The trade-off here is A vs B; because of requirement R, I'm taking A."
• "This is the part I'd prototype first, because it's the riskiest assumption."
• "If this were 10× the scale, I'd change X — but at the stated scale that's over-engineering."
• "The failure mode I'm worried about is… and the mitigation is…"
• "I've built something similar; the thing that bit us in production was…" — scar tissue is gold, if it's true.

Map new questions onto known systems: the archetypes

You cannot prepare every question, but ~90% of prompts are re-skins of a handful of archetypes. Identify the archetype in the first ten minutes and you inherit an entire known-good reference architecture — then spend your effort on what makes this prompt different.

Whiteboard layout strategy

• Divide the board into fixed regions before writing anything: left column = requirements + numbers (never erase it — you'll refer back constantly); top strip = API + entities; center = the architecture diagram; bottom-right = deep-dive scratch space.
• Draw the request path left-to-right: client → CDN/edge → LB → services → stores. Async flows drop below the sync path with dashed arrows.
• Write numbers on arrows ("40k QPS", "~2 KB"), not in the air. A diagram with QPS annotations photographs like a staff engineer's board.
• Never erase the requirements or the high-level diagram. Deep-dive in the scratch region; redraw the one component you're zooming into.
• Virtual interviews: same regions in Excalidraw/Miro; keep boxes coarse (10–15 max on the main diagram) and zoom with the tool, not with clutter.

Latency numbers every engineer should know

These are the physics of your design — updated descendants of Jeff Dean's classic list. Memorize the orders of magnitude, not the digits: the useful fact is that RAM is ~1,000× faster than SSD and a cross-region hop costs a million times an L1 hit.

Powers of two and data sizes

• What fits in memory: commodity cache nodes carry 64–512 GB RAM; a modest Redis cluster holds 1–5 TB. If your hot set is <1 TB, it fits in memory — say that out loud, it kills whole classes of complexity.
• What fits on one machine: tens of TB on disk, ~10–50k QPS for a tuned Postgres on simple queries, ~100k+ QPS for Redis. "Does it fit on one box?" is always estimation question zero.
• Useful cheat: 2¹⁰ ≈ 10³, so 2³² ≈ 4 billion (IDs), 2⁶⁴ ≈ 1.8 × 10¹⁹ (never exhausts).

The four formulas

QPS math

• Seconds per day ≈ 86,400 — round to 100,000 (10⁵). The single most useful rounding in interviews: it's 14% off, and 14% never changes an architecture.
• Average QPS = DAU × actions/user/day ÷ 10⁵. Example: 100M DAU × 10 reads/day = 10⁹/10⁵ = 10,000 QPS.
• Peak QPS = 2–5× average. Use 2× for globally-spread traffic, 3× as default, 5× for lunchtime/prime-time spikes in one geography, 10×+ only for event-driven bursts (flash sale, Super Bowl). Design capacity for peak, cost for average.

Storage math

• Raw/day = items/day × bytes/item. Then multiply by replication factor (usually 3) and retention (days, or ×365×years). Add ~20–30% for indexes and overhead if you want a flourish.

Bandwidth math

• Egress = QPS × response bytes. 10k QPS × 2 KB = 20 MB/s = 160 Mbps — trivial. 10k QPS × 2 MB images = 20 GB/s — that's a CDN requirement, discovered in one line of math.
• Sanity anchors: a 10 Gbps NIC ≈ 1.25 GB/s; a single cache node serving 500 MB/s is near its practical ceiling.

Memory / cache math

• Pareto rule: ~20% of data serves ~80% of reads. Cache size ≈ 0.2 × hot dataset. Refine with TTLs: cache = QPS × bytes × TTL for pass-through caches.

// estimation.ts — the whole back-of-envelope toolkit, typed.
// Everything rounds aggressively: envelope math wants 1 significant figure.

const SECONDS_PER_DAY = 100_000; // 86,400 rounded — the interview classic

export function qps(dau: number, actionsPerUserPerDay: number): number {
  return (dau * actionsPerUserPerDay) / SECONDS_PER_DAY;
}

export function peakQps(avgQps: number, spikeFactor: 2 | 3 | 5 | 10 = 3): number {
  return avgQps * spikeFactor;
}

interface StorageOpts {
  replicationFactor?: number; // default 3 — say why: survive AZ loss
  retentionDays?: number;     // default 5 years
  indexOverhead?: number;     // default 1.25 (25% for indexes/metadata)
}

export function storageTotalBytes(
  itemsPerDay: number,
  bytesPerItem: number,
  { replicationFactor = 3, retentionDays = 5 * 365, indexOverhead = 1.25 }: StorageOpts = {},
): number {
  return itemsPerDay * bytesPerItem * retentionDays * replicationFactor * indexOverhead;
}

export function bandwidthMBps(qpsVal: number, bytesPerResponse: number): number {
  return (qpsVal * bytesPerResponse) / 1e6;
}

export function cacheSizeBytes(hotFraction: number, datasetBytes: number): number {
  return hotFraction * datasetBytes; // classic 80/20 => hotFraction 0.2
}

const fmt = (b: number): string =>
  b >= 1e15 ? `${(b / 1e15).toFixed(1)} PB`
  : b >= 1e12 ? `${(b / 1e12).toFixed(1)} TB`
  : b >= 1e9  ? `${(b / 1e9).toFixed(1)} GB`
  : `${(b / 1e6).toFixed(1)} MB`;

// --- Worked run: Twitter-scale feed ---
const DAU = 200e6;
const readQps  = qps(DAU, 20);            // 200M × 20 timeline loads / 1e5
const writeQps = qps(DAU, 1);             // ~1 tweet/user/day average
console.log(readQps);                     // 40,000 avg read QPS
console.log(peakQps(readQps));            // 120,000 peak read QPS
console.log(writeQps);                    // 2,000 avg write QPS

const tweetBytes = 300;                   // 140–280 chars + metadata + ids
const perDayRaw = DAU * 1 * tweetBytes;   // 60 GB/day of raw tweet text
console.log(fmt(storageTotalBytes(DAU * 1, tweetBytes)));
// ≈ 411 TB over 5y with 3× replication + indexes — text alone is
// "big but boring"; media is 100–1000× larger and goes to object storage.

Worked examples — do these out loud

1. Twitter-scale feed storage

• 200M DAU, 1 tweet/user/day avg, ~300 bytes/tweet (text + metadata) → 60 GB/day raw.
• × 3 replication × 5-year retention (~1,800 days) ≈ ~330 TB, call it 400 TB with indexes — sharded but unremarkable.
• 10% of tweets carry a ~1 MB image: 20M × 1 MB = 20 TB/day of media → object storage + CDN, never the database. The math just partitioned your architecture for you.
• Read side: 40k avg / 120k peak QPS — no relational DB serves that raw; you've just justified the feed cache and fan-out-on-write.

2. Concurrent connections for a chat app

• 50M DAU, users online ~2 h/day → avg concurrent = 50M × (2/24) ≈ ~4M concurrent connections; peak (evening skew) ~2× → 8–10M.
• A tuned WebSocket gateway holds ~100k–500k idle connections (memory-bound: ~10 KB/conn ≈ 1 GB per 100k). At 200k/box → ~50 gateway nodes at peak, plus headroom → ~75. Small fleet; the hard part is the connection registry, not the count.
• Messages: 50M × 40 msgs/day = 2×10⁹/day → 20k avg / ~60k peak msg/s — comfortably one Kafka cluster's territory.

3. Cache size for 20% hot data

• Product catalog: 500M items × 2 KB = 1 TB total.
• 80/20 rule → hot set = 0.2 × 1 TB = 200 GB → fits in one large Redis node's RAM, but run 3–4 nodes anyway: at 100k lookup QPS a single node saturates CPU/NIC before RAM, and you want failure isolation.
• Expected hit rate ~80%+ → DB sees only 20k QPS of misses instead of 100k — this line is what justifies the cache to the interviewer.

Rounding rules and sanity checks

• One significant figure, powers of ten. 86,400 → 10⁵; 2.34 TB → "a few TB". Precision signals you're missing the point.
• Round in a consistent direction for capacity: round traffic up, hardware capability down — errors then land on the safe side.
• Sanity-check against knowns: Google Search is ~100k QPS; if your URL shortener estimate says 5M QPS, re-derive. Whole-of-Twitter writes are ~5–10k TPS; whole-of-Visa is ~2k–10k TPS average.
• Dimensional analysis out loud: "users/day × bytes/user = bytes/day" — narrating units catches 90% of slips before the interviewer does.
• State the conclusion, not just the number: every estimate must end in a design decision — "…therefore one Postgres is fine for year one" or "…therefore this must shard from day one."

Where latency accumulates: budget the request path

SLOs are budgets. Walk the path, add the costs, and see what's left — this diagram is worth drawing in almost any interview when someone says "p99 under 200 ms".

• The network to the user dwarfs everything inside your datacenter. 40 ms of last-mile RTT vs 1 ms of Redis: edge caching and payload size beat micro-optimizing service code.
• Hops compound at the tail: five sequential internal calls at p99 5 ms each is 25 ms — and the p99 of the composition is far worse than 25 ms because tails multiply. Parallelize fan-out calls; cap sequential depth.
• The p99 is set by the miss path, not the hit path. Always quote both.

// latency-budget.ts — compose a request path and see where the SLO goes.
interface Hop {
  name: string;
  p50Ms: number;
  p99Ms: number;
  parallel?: boolean; // parallel hops contribute max(), not sum()
}

function budget(sloMs: number, hops: Hop[]): void {
  const seq = hops.filter(h => !h.parallel);
  const par = hops.filter(h => h.parallel);
  const p50 = seq.reduce((s, h) => s + h.p50Ms, 0) + Math.max(0, ...par.map(h => h.p50Ms));
  // Crude but honest tail model: sequential p99s do NOT simply add in
  // reality (tails multiply); summing is a lower bound — say so aloud.
  const p99 = seq.reduce((s, h) => s + h.p99Ms, 0) + Math.max(0, ...par.map(h => h.p99Ms));
  console.log(`p50 ${p50}ms | p99 ≥ ${p99}ms | slack vs SLO: ${sloMs - p99}ms`);
}

budget(200, [
  { name: 'client→edge RTT', p50Ms: 40, p99Ms: 90 },
  { name: 'edge→region',     p50Ms: 10, p99Ms: 25 },
  { name: 'app logic',       p50Ms: 5,  p99Ms: 15 },
  { name: 'cache (20% miss→db)', p50Ms: 1, p99Ms: 12 },
  { name: 'user service',    p50Ms: 3,  p99Ms: 10, parallel: true },
  { name: 'ranking service', p50Ms: 8,  p99Ms: 30, parallel: true },
]);
// => p50 64ms | p99 ≥ 172ms | slack 28ms
// Conclusion you say in the interview: "the SLO is met only if ranking
// stays parallel and we allow at most ONE retry on the edge hop."

Vertical vs horizontal scaling

• Vertical (scale up): bigger box. Zero code changes, no distributed-systems problems, strong single-node consistency for free. Limits: a hardware ceiling (~a few hundred cores / a few TB RAM), price grows super-linearly, and it's still one blast radius — the box dies, you're down.
• Horizontal (scale out): more boxes. Near-linear cost, no ceiling, failure of one node is survivable. Costs: you just bought load balancing, state distribution, partial failures, and consistency questions. Complexity is the price, paid forever.
• Staff-level default: scale up until the numbers say otherwise. A single modern server handles ~10–50k QPS of real app logic and Postgres on NVMe handles low-tens-of-thousands of simple TPS — that's bigger than most businesses ever get.
• The asymmetry that shapes everything: stateless things scale out trivially; stateful things (databases, sessions, caches) are where all the hard work lives. Hence the prime directive: push state to the edges of your architecture and keep the middle stateless.

The canonical evolution: eight stages

Tell this as a story driven by symptoms: each stage exists because the previous one hit a measurable wall. Approximate capacity signposts are per-stage guides, not laws.

• Stage order is negotiable, walls are not. Some systems jump straight to Stage 4 (consumer apps behind a CDN from day one); nobody skips the statelessness prerequisite of Stage 3.
• Stages 1–5 are configuration; Stages 6–8 are architecture. Replicas and caches bolt on; sharding and multi-region change your data model and your team's life. Delay them until forced — then do them deliberately.

Statelessness: the enabler of everything

• Why: if any app node can serve any request, then adding nodes is pure capacity, deploys are rolling restarts, and a node death loses nothing. Every horizontal-scaling benefit assumes this.
• Externalize the state: sessions → Redis/DynamoDB; uploaded files → S3; locks/leases → the database or a coordination store. The app process holds only per-request scratch.
• Sticky sessions are the anti-pattern: pinning a user to a node via LB cookies "works" until a deploy, a scale-in event, or a node crash logs everyone out. It also skews load and blocks autoscaling from actually removing nodes. Acceptable only as a short-lived migration crutch or for genuinely connection-oriented protocols (WebSockets — and even then the state lives elsewhere; only the socket is pinned).
• JWTs vs server-side sessions: JWTs remove the session store read but make revocation hard (you end up adding a denylist — a session store with extra steps). Server-side sessions in Redis are the boring, correct default; say the trade-off either way.

// session.ts — externalizing session state (express-style middleware).
// The node keeps NOTHING between requests; kill it mid-flight and the
// user never notices. That property is what "stateless" buys you.
import type { Redis } from 'ioredis';

interface SessionData {
  userId: string;
  roles: ('user' | 'admin')[];
  createdAt: number;            // epoch ms
  lastSeenAt: number;
}

const SESSION_TTL_S = 60 * 60 * 24;      // 24h sliding window

function key(sessionId: string): string {
  return `sess:${sessionId}`;            // namespacing: ops can SCAN sess:*
}

export function sessionMiddleware(redis: Redis) {
  return async (req: any, res: any, next: () => void): Promise<void> => {
    const sid: string | undefined = req.cookies?.sid;
    if (!sid) { req.session = null; return next(); }

    const raw = await redis.get(key(sid));      // ~1ms in-region
    if (!raw) { req.session = null; return next(); }

    const session = JSON.parse(raw) as SessionData;
    session.lastSeenAt = Date.now();

    // Sliding expiry: refresh TTL on activity. Fire-and-forget —
    // losing one refresh costs nothing, so don't block the request.
    void redis.set(key(sid), JSON.stringify(session), 'EX', SESSION_TTL_S);

    req.session = session;
    next();
  };
}

export async function createSession(
  redis: Redis, sessionId: string, userId: string, roles: SessionData['roles'],
): Promise<void> {
  const now = Date.now();
  const data: SessionData = { userId, roles, createdAt: now, lastSeenAt: now };
  await redis.set(key(sessionId), JSON.stringify(data), 'EX', SESSION_TTL_S);
}
// Sizing check (Ch. 2 math): 10M sessions × ~200B ≈ 2GB — one small
// Redis with a replica. Session stores are rarely the scaling problem.

Read scaling vs write scaling — two different toolkits

• Interview reflex: the moment you learn the read:write ratio (Ch. 1), you know which toolkit dominates the design. A 100:1 system is a caching problem; a 1:10 system is a partitioning problem.

The scale cube (AKF)

• X-axis — clone it: identical stateless copies behind an LB (Stage 3). Cheapest; solves throughput, not data size.
• Y-axis — split by function: decompose into services (orders, payments, users). Scales teams and isolates failure domains as much as it scales load.
• Z-axis — split by data: shard by customer/key (Stage 6). Solves data size and write throughput; the most expensive axis.
• Use the cube as a vocabulary check: "we've maxed X, this bottleneck is data volume, so it's a Z-axis problem" is a crisp staff sentence.

When NOT to scale

• "You are not Google": 99% of systems live and die below 10k QPS — inside one beefy Postgres and a handful of app nodes. Kubernetes-and-Kafka for a 200-QPS product is résumé-driven design, and interviewers penalize it.
• Premature optimization has compounding cost: every layer (shards, queues, regions) adds failure modes, on-call load, and slower feature work — permanently.
• Profile before scaling: most "we need to scale" incidents are a missing index, an N+1 query, or an unbounded payload. A 10× win from a query plan is cheaper than any architecture change.
• The staff move: design the seams (stateless app, data access behind a repository, IDs that don't assume one DB) so future scaling is possible — then don't build it yet. Say exactly that in the interview.

Capacity headroom and autoscaling basics

• Headroom: run stateless tiers at ~40–60% utilization at peak; stateful tiers with ~2× headroom (they scale slowly and painfully). Headroom is what absorbs the spike while autoscaling reacts.
• Scale on what metric? Target-tracking on CPU (~50–60%) is the default for CPU-bound services; for I/O-bound or queue-fed services use requests-per-target or queue depth ÷ processing rate. Never scale on memory for GC'd runtimes (it lies) and never on p99 latency alone (too noisy, reacts too late).
• Physics of reaction time: metric period (60s) + evaluation + instance boot (60–300s) means autoscaling answers in minutes. Sub-minute spikes are handled by headroom and load-shedding, not by scaling.
• Scale up fast, down slow: aggressive scale-out, conservative scale-in with cooldowns — flapping wastes more than idle capacity does.

// autoscaler.ts — a target-tracking policy evaluator (what ASG/HPA do).
interface AutoscalePolicy {
  metric: 'cpuUtilization' | 'requestsPerTarget' | 'queueDepthPerWorker';
  target: number;            // e.g. 55 (% CPU) or 800 (req/target)
  minInstances: number;
  maxInstances: number;
  scaleInCooldownS: number;  // slow down; flapping is worse than idle
  scaleOutCooldownS: number; // speed up; user pain is worse than cost
}

interface FleetState {
  instances: number;
  metricValue: number;       // current observed value of policy.metric
  lastScaleOutAt: number;    // epoch s
  lastScaleInAt: number;
}

type Decision =
  | { action: 'scale-out'; to: number; reason: string }
  | { action: 'scale-in';  to: number; reason: string }
  | { action: 'hold';      reason: string };

export function evaluate(p: AutoscalePolicy, s: FleetState, nowS: number): Decision {
  // Core of target tracking: desired = current × (observed / target).
  // 8 instances at 80% CPU targeting 55% => ceil(8 × 80/55) = 12.
  const desiredRaw = s.instances * (s.metricValue / p.target);
  const desired = Math.min(p.maxInstances, Math.max(p.minInstances, Math.ceil(desiredRaw)));

  if (desired > s.instances) {
    if (nowS - s.lastScaleOutAt < p.scaleOutCooldownS)
      return { action: 'hold', reason: 'scale-out cooldown active' };
    return { action: 'scale-out', to: desired,
             reason: `${p.metric}=${s.metricValue} above target ${p.target}` };
  }
  // Deadband: only scale in when meaningfully under target (<90% of it),
  // otherwise metric noise makes the fleet oscillate.
  if (desired < s.instances && s.metricValue < p.target * 0.9) {
    if (nowS - s.lastScaleInAt < p.scaleInCooldownS)
      return { action: 'hold', reason: 'scale-in cooldown active' };
    const to = Math.max(desired, s.instances - 1);   // step down gently
    return { action: 'scale-in', to, reason: 'sustained under target' };
  }
  return { action: 'hold', reason: 'within deadband' };
}

const decision = evaluate(
  { metric: 'cpuUtilization', target: 55, minInstances: 4, maxInstances: 40,
    scaleInCooldownS: 600, scaleOutCooldownS: 120 },
  { instances: 8, metricValue: 82, lastScaleOutAt: 0, lastScaleInAt: 0 },
  1_000,
);
// => { action: 'scale-out', to: 12, reason: 'cpuUtilization=82 above target 55' }

Decision flowchart: "the system is slow — scale what?"

Never answer "what would you scale?" with a component. Answer with this procedure: profile first, then match the bottleneck to its toolkit.

The layer models: OSI 7 vs TCP/IP 4

OSI is the vocabulary; TCP/IP is what actually ships. Interviewers use OSI numbers as shorthand ("L4 vs L7 load balancer"), so you need the mapping cold — but in practice engineers only touch four layers.

Encapsulation: how bytes physically travel

• Your app writes an HTTP request; the kernel slices it into TCP segments (each stamped with ports + sequence number).
• Each segment is wrapped in an IP packet (source/destination IP, TTL) — routers only ever read this header.
• Each packet is wrapped in an Ethernet frame (MAC addresses, checksum) for the current hop; the frame is rewritten at every router, the IP packet survives end-to-end.
• Typical MTU is 1,500 bytes, so a 100 KB response is ~70 packets — each one an independent unit that can be lost, reordered, or duplicated. TCP's whole job is hiding that.

DNS: the first round trip

Before any connection exists, the client must turn api.example.com into an IP. The strong answer narrates the full cold path and then explains why it's almost never taken.

• 1. Stub → recursive resolver. The OS asks its configured resolver (ISP, 8.8.8.8, or the VPC's Route 53 Resolver at 10.0.0.2). If cached and TTL-fresh: done in <1 ms.
• 2. Root servers. Cold path: the recursive resolver asks a root server, which replies "ask the .com TLD servers" (an NS referral, not the answer).
• 3. TLD servers. The .com server refers to example.com's authoritative nameservers — for AWS-hosted zones, four Route 53 nameservers.
• 4. Authoritative answer. Route 53 returns the A/AAAA record (or an alias to an ALB/CloudFront) plus a TTL. Every hop caches the result for that TTL.
• TTL trade-off: 300 s is the pragmatic default — low enough to fail over in minutes, high enough that >99% of lookups hit cache. TTL 60 s or less is for records you actively fail over; TTL 86,400 s means a bad record haunts you for a day.
• Route 53 isn't just a phone book: latency-based, geolocation, weighted, and failover routing policies make DNS your global load balancer (details in chapter 6).

TCP: reliable delivery over an unreliable network

• 3-way handshake (1 RTT before any data): client sends SYN (with a random initial sequence number), server replies SYN-ACK, client sends ACK — and can attach data to it. Both sides now agree on sequence numbers.
• Reliability: every byte is sequenced; receiver ACKs; sender retransmits on timeout or 3 duplicate ACKs. Lost packet = stall + retransmit, not corruption.
• Flow control (protect the receiver): the receiver advertises a window ("I can buffer 64 KB") in every ACK; a slow consumer shrinks it, possibly to zero — this is backpressure at L4.
• Congestion control (protect the network): slow start begins with ~10 packets (initcwnd) and doubles the congestion window every RTT until loss, then backs off (CUBIC/BBR). Consequence: a new connection can't use your bandwidth for the first several RTTs — the #1 reason connection reuse matters.
• Teardown: FIN/ACK in each direction; the closer parks in TIME_WAIT for ~60 s to absorb stragglers. Servers that open one connection per request exhaust ports on TIME_WAIT — another argument for pooling.

UDP vs TCP — when each

QUIC and HTTP/3

• QUIC rebuilds TCP's reliability per-stream in userspace over UDP — chosen precisely because middleboxes and kernels ossified TCP; UDP passes through everywhere and Google could iterate without waiting for OS upgrades.
• Transport + TLS 1.3 handshake are fused: 1 RTT cold, 0-RTT resumed (send the request with the first packet).
• A lost packet stalls only its own stream — kills TCP's head-of-line blocking, which is why HTTP/3 shines on lossy mobile networks.
• Connection ID instead of the 4-tuple: switching Wi-Fi → LTE migrates the connection instead of resetting it.

TLS: the handshake tax

• TLS 1.2: 2 RTTs — hello/cipher negotiation, then certificate + key exchange, then finished. Only then application data.
• TLS 1.3: 1 RTT — client guesses the key-exchange group and sends its key share in the first flight; server responds with cert + finished. Weak ciphers (RSA key exchange, CBC) are simply removed, and forward secrecy is mandatory.
• Session resumption: a returning client presents a session ticket (PSK) and skips certificate verification; TLS 1.3 even allows 0-RTT early data — with a catch: 0-RTT requests are replayable, so only idempotent GETs belong there. Never a payment POST.
• Certificate chain validation, OCSP, and SNI all ride inside the handshake — SNI is why one IP can serve many hostnames, and why the ALB/CloudFront picks a cert before seeing HTTP.

HTTP/1.1 vs HTTP/2 vs HTTP/3

• HTTP/1.1: one in-flight request per connection (pipelining is dead in practice). Browsers hack around it with 6 parallel connections per host; ops hack around it with domain sharding and sprite sheets. Application-level head-of-line blocking: a slow response blocks the connection.
• HTTP/2: one TCP connection, many interleaved binary streams (multiplexing) + header compression (HPACK). Kills app-level HOL and the 6-connection hack — but inherits TCP-level HOL: one lost packet stalls all streams, because TCP delivers bytes in order regardless of which stream they belong to.
• HTTP/3: same semantics over QUIC — multiplexing without TCP HOL, faster handshakes, connection migration. Semantics (methods, status codes, headers) are identical across all three; only the framing changes.
• Backend reality: your CDN speaks h2/h3 to browsers and typically plain HTTP/1.1 over warm keep-alive connections to origin — the client-side wins matter most on high-latency last miles.

Why is something fast vs slow?

This is the section to internalize. Almost every "why is it slow" answer reduces to: count the round trips, then check payload size, then check server time — in that order.

• Latency ≠ bandwidth. Latency is how long one bit takes to arrive (physics: ~1 ms per 100 km in fiber; NYC↔Sydney ~200 ms RTT and no product manager can fix the speed of light). Bandwidth is how many bits per second flow once moving. Upgrading a 1 Gbps to 10 Gbps link does nothing for a 2 KB API call — its cost is pure RTTs.
• RTT accounting for a cold HTTPS request: DNS (1 RTT to resolver, more if cold) + TCP handshake (1) + TLS 1.3 (1; TLS 1.2: 2) + HTTP request/response (1) ≈ 4 RTTs minimum. At 150 ms RTT that's 600 ms before the first byte of your 5 ms backend response. Same request on a reused connection: 1 RTT = 150 ms. Same request to an edge 15 ms away, warm: 15 ms. That's a 40× spread with identical code.
• Connection reuse is the highest-leverage fix: keep-alive, HTTP/2 multiplexing, and client-side pooling skip DNS + TCP + TLS and keep the congestion window warm (a fresh connection is also slow-start throttled). Lambda→RDS without a pool re-pays this tax on every invocation — hence RDS Proxy.
• Nagle's algorithm: the kernel batches small writes until the previous ACK returns; combined with delayed ACKs it injects up to ~40–200 ms stalls into chatty request/response protocols. Every serious server/DB client sets TCP_NODELAY (Node's socket.setNoDelay()).
• TCP head-of-line blocking: ordered delivery means one lost packet at position n quarantines every received packet after it. On a 1%-loss mobile link this dominates tail latency — the p99 killer that HTTP/3 exists to fix.
• Nearby edge beats fatter pipes: for small payloads, total time ≈ RTT × round-trips. A CDN edge 15 ms away that terminates TLS and holds warm origin connections converts a 4-RTT × 150 ms cold start into 4 × 15 ms + one warm origin hop. This is why chapter 5 exists.

// Typed fetch wrapper: conditional requests with ETag + 304 handling.
// Demonstrates the caching headers from the table below, in code.
interface CacheEntry<T> {
  etag: string;
  value: T;
  storedAt: number;
}

type FetchResult<T> =
  | { source: "network"; status: 200; value: T }
  | { source: "cache";   status: 304; value: T }; // revalidated, body skipped

class ConditionalClient {
  private cache = new Map<string, CacheEntry<unknown>>();

  async getJson<T>(url: string): Promise<FetchResult<T>> {
    const cached = this.cache.get(url) as CacheEntry<T> | undefined;
    const headers = new Headers({ accept: "application/json" });
    // If we have a prior ETag, ask the server "only send the body if it changed".
    if (cached) headers.set("if-none-match", cached.etag);

    const res = await fetch(url, { headers });

    if (res.status === 304) {
      if (!cached) throw new Error("304 without a cached entry — server bug or evicted cache");
      // Server confirmed our copy is fresh: zero bytes of body crossed the wire.
      return { source: "cache", status: 304, value: cached.value };
    }
    if (!res.ok) throw new Error(`GET ${url} failed: ${res.status}`);

    const value = (await res.json()) as T;
    const etag = res.headers.get("etag");
    if (etag) this.cache.set(url, { etag, value, storedAt: Date.now() });
    return { source: "network", status: 200, value };
  }
}

// Usage: second call sends If-None-Match and typically pays only ~1 RTT + headers.
const api = new ConditionalClient();
interface Price { symbol: string; usd: number }
const first  = await api.getJson<Price>("https://api.example.com/price/BTC"); // 200, full body
const second = await api.getJson<Price>("https://api.example.com/price/BTC"); // 304, cached body
console.log(first.source, second.source); // "network" "cache"

The headers that actually matter

Caching

Content negotiation

Auth & cookies

CORS

• CORS is a browser-enforced read guard, not a server firewall — curl ignores it entirely. The server opts pages from other origins in.
• "Simple" requests (GET/POST with vanilla headers) fly directly; the browser just checks Access-Control-Allow-Origin on the response before releasing it to JS.
• Anything else — Authorization header, Content-Type: application/json, PUT/DELETE — triggers a preflight: an OPTIONS request asking permission. That's a whole extra RTT; cache the verdict with Access-Control-Max-Age: 86400.
• Key response headers: Access-Control-Allow-Origin (echo one origin, not *, when using credentials), -Allow-Methods, -Allow-Headers, -Allow-Credentials, -Expose-Headers.

Security & infrastructure

Cookies vs tokens

• Cookies: browser-managed, auto-attached, HttpOnly shields them from XSS — but auto-attach means you must defend CSRF (SameSite + anti-CSRF tokens). Natural fit for same-site web apps and server-rendered sessions.
• Bearer tokens (JWT): code-attached (CSRF-immune), work for mobile/M2M/cross-domain, self-contained claims avoid a session-store lookup — but JS-readable storage is XSS-stealable, and stateless tokens can't be revoked before expiry (mitigate with short-lived access tokens ~5–15 min + refresh-token rotation).
• Pragmatic hybrid for SPAs: JWT delivered in an HttpOnly; Secure; SameSite=Lax cookie — XSS-resistant storage, CSRF-resistant attribute, stateless verification.

Firewalls, security groups, NAT

• Stateless filtering evaluates each packet alone against rules — so you must open both directions, including high-numbered ephemeral return ports. Stateful filtering tracks connections: allow the inbound request and the response is automatically permitted.
• WAF is L7: it reads the HTTP itself — SQLi/XSS signatures, bot fingerprints, per-IP rate rules, geo blocks. AWS WAF attaches to CloudFront, ALB, or API Gateway. It complements, never replaces, SGs.
• NAT & private subnets: app servers and DBs live in private subnets with no public IPs — unroutable from the internet by construction. Outbound calls (package repos, Stripe) traverse a NAT gateway that rewrites source addresses and statefully maps returns. One NAT GW per AZ; NAT data-processing charges are a classic surprise line item (VPC endpoints for S3/DynamoDB dodge them).

Networking in code

// Tiny TCP echo server + client with node:net — the handshake made tangible.
// `nc localhost 4000` works against it too.
import net from "node:net";

const server = net.createServer((socket: net.Socket) => {
  // This callback fires only AFTER the kernel completed SYN → SYN-ACK → ACK.
  // The 3-way handshake already cost 1 RTT before we see any bytes.
  socket.setNoDelay(true); // disable Nagle: don't batch our small writes
  console.log(`established: ${socket.remoteAddress}:${socket.remotePort}`);

  socket.on("data", (chunk: Buffer) => {
    // TCP is a BYTE STREAM, not a message stream: one client write() may
    // arrive as several 'data' events, or several writes as one. Real
    // protocols add framing (length prefix / newline) on top.
    socket.write(chunk); // echo back
  });
  socket.on("end", () => socket.end()); // peer sent FIN; send ours (teardown)
});
server.listen(4000);

// --- client ---
const client = net.createConnection({ port: 4000 }, () => {
  // 'connect' = our ACK is sent; connection is ESTABLISHED.
  client.setNoDelay(true);
  client.write("ping over a fresh connection\n");
});
client.on("data", (buf: Buffer) => {
  console.log("echo:", buf.toString().trim());
  client.end(); // FIN → server ACK+FIN → our ACK; we hold TIME_WAIT ~60s
});

// Connection pooling with undici: pay DNS + TCP + TLS + slow-start ONCE,
// then multiplex thousands of requests over warm connections.
import { Pool } from "undici";

const upstream = new Pool("https://payments.internal.example.com", {
  connections: 32,          // cap concurrent sockets to this origin
  pipelining: 1,            // 1 = safe request/response; h2 would multiplex
  keepAliveTimeout: 30_000, // hold idle sockets 30s — must be BELOW the
                            // server/LB idle timeout (ALB default 60s), or
                            // you'll write into a socket the peer just closed
  connectTimeout: 3_000,
  bodyTimeout: 5_000,
});

interface ChargeRequest { orderId: string; amountMinor: number; currency: "USD" | "EUR" }
interface ChargeResponse { chargeId: string; status: "captured" | "declined" }

export async function charge(req: ChargeRequest): Promise<ChargeResponse> {
  const { statusCode, body } = await upstream.request({
    path: "/v1/charges",
    method: "POST",
    headers: {
      "content-type": "application/json",
      "idempotency-key": req.orderId, // safe retries over flaky networks (ch. 12)
    },
    body: JSON.stringify(req),
  });
  if (statusCode >= 500) throw new Error(`upstream ${statusCode}`); // retryable
  if (statusCode >= 400) throw new Error(`charge rejected: ${statusCode}`); // don't retry
  return (await body.json()) as ChargeResponse;
}
// Warm-pool p50 ≈ 1 RTT + server time. Cold socket per request ≈ 3-4 RTTs extra
// — at 40ms RTT that's ~120ms of pure, avoidable handshake tax per call.

Putting it together: life of a request

Why CDNs: you can't patch the speed of light

• Light in fiber travels ~200,000 km/s → ~1 ms per 100 km one way, before adding router hops and congestion. São Paulo → Virginia is ~140 ms RTT on a good day.
• A cold HTTPS fetch costs ~4 RTTs (DNS + TCP + TLS + HTTP). At 140 ms that's ~560 ms; from an edge 15 ms away it's ~60 ms. Same bytes, ~10× faster — no code changed.
• A CDN is thousands of edge locations (CloudFront: 600+) that (a) serve cached responses locally and (b) terminate connections locally even when they can't cache.
• Secondary wins: origin offload (90%+ of static traffic never reaches you), DDoS absorption at the edge, and a global anycast front door.

Pull vs push CDNs

Cache keys and normalization

The cache key is the identity of a cached object — by default roughly host + path (+ selected query strings/headers/cookies). Getting it wrong causes the two classic CDN failures:

• Key too broad → serving user A's private response to user B (cache poisoning territory). E.g. caching /me without keying on the session.
• Key too narrow → fragmentation: ?utm_source=twitter vs ?utm_source=email become different objects, hit ratio collapses, origin melts.
• Normalize aggressively: strip marketing params, sort query strings, lowercase where safe, collapse Accept-Encoding variants to gzip/br buckets. Include only what changes the response (CloudFront cache policies express exactly this).
• Vary (ch. 4) is the origin's way to add request headers into the key — use surgically; Vary: Cookie ≈ cache disabled.

TTLs and the freshness ladder

• Cache-Control: s-maxage=600, max-age=60 — CDN keeps 10 min, browsers 1 min. s-maxage exists because you can purge shared caches but not browsers: give the tier you control the longer TTL.
• stale-while-revalidate=30 — serve the stale copy instantly while refetching in the background: users never pay the miss penalty on refresh; latency stays flat across TTL expiry.
• stale-if-error=86400 — origin down? Serve yesterday's copy rather than a 502. Free resilience; your origin can burn for a day while the site stays up.
• Pattern: short-ish s-maxage + generous stale-* directives beats a long TTL — freshness within minutes, availability in hours.

Invalidation: version, don't purge

• Versioned URLs / content hashing (preferred): app.3f2a91.js — new deploy → new hash → new cache key. Old and new coexist (no mid-deploy mixed pages), "invalidation" is instant and atomic, and assets become Cache-Control: public, max-age=31536000, immutable. Every modern bundler does this for free.
• Purge/invalidation API (fallback): for URLs you can't version — /, /sitemap.xml, an urgent legal takedown. CloudFront invalidations take seconds-to-minutes to propagate and cost after the first 1,000 paths/month; wildcard /* is the sledgehammer.
• Surrogate keys / cache tags (Fastly, Cloudflare): tag responses (product-42), purge by tag when that product changes — the precision tool for cached APIs.
• Rule of thumb: if you're purging on every deploy, your URLs should have been versioned.

What to cache (more than static files)

• Obvious: JS/CSS/images/fonts — hashed, immutable, year-long TTL.
• API responses — yes, really: anything shared across users and tolerant of brief staleness: product catalogs, search suggestions, FX rates, config/flags, leaderboards. Even s-maxage=1 with request collapsing turns 10,000 req/s into ~1 origin req/s per key — a 5-second-stale price list is fine; a melted origin is not.
• Video: HLS/DASH segments are just small immutable files — this is how Netflix-scale delivery works. Keep manifests short-TTL (live) and segments long-TTL; protect them with signed URLs/cookies.
• Never: per-user personalized responses (unless keyed per-user at edge — usually not worth it), anything with Set-Cookie, payments/auth endpoints. Mark them Cache-Control: private, no-store explicitly.

Dynamic acceleration: CDN value with zero caching

• Even for no-store APIs, the edge terminates TCP + TLS ~15 ms from the user — the 3–4 cold handshake RTTs happen locally instead of across an ocean.
• Edge → origin rides warm, pooled, long-lived connections over the provider's backbone (often better routes than the public internet); slow-start is already paid.
• Result: 20–40% TTFB cuts on fully dynamic traffic. This is CloudFront-in-front-of-ALB, or AWS Global Accelerator when you want pure L4 anycast without HTTP awareness.
• Origin shield: one designated regional cache between all edges and your origin. N edges no longer each miss independently — misses funnel through the shield, collapsing duplicate fetches. Origin sees ~1 fetch per object per TTL instead of hundreds; also the fix for "every PoP hammers my origin at TTL expiry".

Edge compute: running code at the PoP

• Canonical edge jobs: auth at edge (verify JWT signature, reject before origin), A/B assignment (sticky cookie + cache-key variant), geo-routing (CloudFront-Viewer-Country → origin or redirect), header hygiene (normalize before the cache key is computed — order matters: viewer-request functions run before cache lookup).

// Lambda@Edge-style viewer-request handler: sticky A/B assignment.
// Runs BEFORE cache lookup, so the variant cookie can join the cache key
// (configure the cache policy to include the "ab" cookie) — each variant
// caches independently and stays fast.
import type { CloudFrontRequestEvent, CloudFrontRequest } from "aws-lambda";

const EXPERIMENT = { name: "checkout_v2", variants: ["control", "treatment"] as const };
type Variant = (typeof EXPERIMENT.variants)[number];

function readAbCookie(req: CloudFrontRequest): Variant | undefined {
  const raw = req.headers["cookie"]?.map(h => h.value).join("; ") ?? "";
  const m = raw.match(/(?:^|;\s*)ab=([^;]+)/);
  return EXPERIMENT.variants.find(v => v === m?.[1]);
}

export const handler = async (event: CloudFrontRequestEvent): Promise<CloudFrontRequest> => {
  const request = event.Records[0].cf.request;

  // Sticky: returning users keep their bucket (stable UX + valid stats).
  let variant = readAbCookie(request);
  if (!variant) {
    // 50/50 assignment; deterministic hashing on a user id is even better.
    variant = Math.random() < 0.5 ? "control" : "treatment";
    // Normalize the cookie header so the cache key sees exactly one form.
    request.headers["cookie"] = [{ key: "Cookie", value: `ab=${variant}` }];
  }

  // Route treatment to a different origin path — same URL for the user.
  if (variant === "treatment" && request.uri.startsWith("/checkout")) {
    request.uri = `/v2${request.uri}`;
  }
  return request; // continue to cache lookup → origin on miss
};

Private content: signed URLs and signed cookies

• Problem: paid video / private docs must flow through the CDN (offload!) but not be shareable links.
• Signed URL: URL + policy (expiry, optional IP/path constraints) + signature from your private key; the edge verifies with the public key — no call to your origin. One URL per object; ideal for a single download.
• Signed cookies: same trust model, but the policy rides in cookies covering a path pattern (/videos/course-42/*) — one grant for thousands of HLS segments without rewriting every URL in the manifest.
• Keep expiries short (minutes–hours) and combine with S3 Origin Access Control so nobody can bypass CloudFront and hit the bucket directly.

// CloudFront signed-URL generator (canned policy): time-boxed access to a
// private object, verified AT THE EDGE with zero origin involvement.
import { createSign } from "node:crypto";

interface SignedUrlInput {
  url: string;            // e.g. https://d111abc.cloudfront.net/videos/ep1/seg-001.ts
  keyPairId: string;      // public key id registered with the distribution
  privateKeyPem: string;  // from Secrets Manager — never bundle in code
  expiresInSeconds: number;
}

// CloudFront's URL-safe base64: + → -, = → _, / → ~
const cfSafe = (b64: string) => b64.replace(/\+/g, "-").replace(/=/g, "_").replace(/\//g, "~");

export function signUrl(input: SignedUrlInput): string {
  const expires = Math.floor(Date.now() / 1000) + input.expiresInSeconds;

  // Canned policy: this exact resource, valid until `expires`. Custom
  // policies add IP ranges and start times — same signing flow.
  const policy = JSON.stringify({
    Statement: [{
      Resource: input.url,
      Condition: { DateLessThan: { "AWS:EpochTime": expires } },
    }],
  });

  const signature = createSign("RSA-SHA1") // CloudFront's required scheme
    .update(policy)
    .sign(input.privateKeyPem, "base64");

  const sep = input.url.includes("?") ? "&" : "?";
  return `${input.url}${sep}Expires=${expires}` +
         `&Signature=${cfSafe(signature)}` +
         `&Key-Pair-Id=${input.keyPairId}`;
}

// 10-minute grant: plenty for a player to fetch the segment, useless to share.
const url = signUrl({
  url: "https://d111abc.cloudfront.net/videos/ep1/seg-001.ts",
  keyPairId: "K2JCJMDEHXQW5F",
  privateKeyPem: process.env.CF_SIGNING_KEY!,
  expiresInSeconds: 600,
});

Decision: should this go through the CDN / be cached at edge?

L4 vs L7: what each layer can see and do

Balancing algorithms — and when each wins

• Round robin: rotate through targets. Fine when requests are uniform and instances identical. Degrades when one request is 100× another.
• Weighted round robin: RR with per-target weights — mixed instance sizes, or canaries (99/1 traffic splits, ALB weighted target groups).
• Least connections: pick the target with fewest in-flight requests — self-corrects for slow requests and slow instances; the sane default for variable workloads (long polls, reports next to fast reads).
• Least response time: least-conns plus latency awareness — routes around a degrading node before health checks notice. More state to track; supported by Nginx Plus/Envoy.
• IP hash: same client IP → same target. Crude stickiness with zero cookies — but mobile NATs put thousands of users behind one IP, and any target change reshuffles everyone.
• Consistent hashing: hash(key) → position on a ring; adding/removing a node remaps only ~1/N of keys. Essential when targets hold per-key state (caches, WebSocket hubs, shard routers) — full treatment in chapter 11.
• Reality check: with health checks, connection reuse, and autoscaling, algorithm choice matters less than people think — except under heterogeneous load, where least-connections visibly beats RR.

Health checks, draining, slow start

• Active checks: the LB probes /healthz every N seconds; fail K-in-a-row → out of rotation. Make the endpoint check the instance's own dependencies-critical state — not its downstreams, or one flaky dependency fails your whole fleet at once (cascading self-inflicted outage).
• Passive checks / outlier ejection: watch real traffic; a target throwing 5xx or timeouts gets ejected temporarily (Envoy's outlier detection). Catches "healthy /healthz, broken /checkout" — combine both.
• Connection draining (deregistration delay): on deploy/scale-in, stop sending new requests but let in-flight ones finish (ALB default 300 s). Skipping this is why deploys "randomly" 502.
• Slow start: ramp a freshly registered target from 0→full weight over e.g. 60 s — cold JITs, empty local caches, and unfilled DB pools don't get instantly firehosed.

Sticky sessions — and why to avoid them

• Mechanism: LB sets a cookie (ALB: AWSALB) pinning a client to one target.
• Costs: load skews (hot targets can't shed), scale-in/deploys log users out, autoscaling effectiveness drops, and one instance's death is now user-visible.
• Staff answer: externalize the state instead — sessions in Redis/DynamoDB, or a signed cookie/JWT — so any instance can serve any user. Stickiness is a migration crutch for legacy in-memory-session apps, not a design choice.

Reverse proxy chores (the unglamorous 80%)

• TLS termination: decrypt once at the proxy; plain HTTP (or lighter mTLS) inside the VPC. Centralizes certs/ciphers (ACM on ALB = zero-touch rotation) and offloads handshake CPU from app nodes.
• Compression: gzip/brotli at the proxy so app code never thinks about Accept-Encoding.
• Buffering slow clients: the killer feature nobody mentions — a mobile client trickling a response at 3G speeds would pin an app worker for 30 s; the proxy slurps the app's response in ms, frees the worker, and spoon-feeds the client from its own cheap buffers. This is why Node/Python apps sit behind Nginx/ALB even single-instance.
• Plus: request normalization, header hygiene (set X-Forwarded-For, strip hop-by-hop), static files, and a single choke point for logging/metrics.

API gateway vs service mesh

• API gateway = north–south (edge, client-facing): authN/Z (JWT/OIDC validation, API keys), per-client rate limiting and quotas, routing/versioning (/v1 → service A), request/response transforms, canary routing, developer portal. One policy enforcement point so 30 services don't reimplement auth 30 ways.
• Service mesh = east–west (service-to-service): sidecars (Envoy/Istio) or ambient handling mTLS between services, retries/timeouts/circuit breaking, and traffic shifting — uniformly, in infrastructure, out of app code.
• They coexist: gateway at the front door, mesh in the hallways. Don't let an interviewer catch you rate-limiting internal traffic at the gateway or exposing the mesh to the internet.

// Build-your-own mini API gateway: a typed middleware pipeline.
// auth → rate limit → route → proxy, composed like ALB/Kong stages.
interface GatewayContext {
  req: { method: string; path: string; headers: Record<string, string>; body?: string };
  clientId?: string;                    // set by auth middleware
  upstream?: string;                    // set by router
  res?: { status: number; body: string };
}
// Each middleware either responds (short-circuit) or calls next().
type Middleware = (ctx: GatewayContext, next: () => Promise<void>) => Promise<void>;

const compose = (mws: Middleware[]) => (ctx: GatewayContext): Promise<void> => {
  const run = (i: number): Promise<void> =>
    i < mws.length ? mws[i](ctx, () => run(i + 1)) : Promise.resolve();
  return run(0);
};

const auth: Middleware = async (ctx, next) => {
  const token = ctx.req.headers["authorization"]?.replace(/^Bearer /, "");
  const claims = token ? verifyJwt(token) : undefined;   // signature + exp check
  if (!claims) { ctx.res = { status: 401, body: "unauthenticated" }; return; }
  ctx.clientId = claims.sub;
  await next();
};

// Token bucket per client: `rate` tokens/sec, burst up to `capacity`.
const buckets = new Map<string, { tokens: number; last: number }>();
const rateLimit = (rate: number, capacity: number): Middleware => async (ctx, next) => {
  const key = ctx.clientId!;
  const now = Date.now();
  const b = buckets.get(key) ?? { tokens: capacity, last: now };
  b.tokens = Math.min(capacity, b.tokens + ((now - b.last) / 1000) * rate);
  b.last = now;
  if (b.tokens < 1) { ctx.res = { status: 429, body: "rate limit exceeded" }; return; }
  b.tokens -= 1;
  buckets.set(key, b);
  await next();
};

const routes: Array<{ prefix: string; upstream: string }> = [
  { prefix: "/orders", upstream: "http://orders.internal:8080" },
  { prefix: "/users",  upstream: "http://users.internal:8080" },
];
const router: Middleware = async (ctx, next) => {
  const match = routes.find(r => ctx.req.path.startsWith(r.prefix));
  if (!match) { ctx.res = { status: 404, body: "no route" }; return; }
  ctx.upstream = match.upstream;
  await next();
};

const proxy: Middleware = async (ctx) => {
  const r = await fetch(ctx.upstream! + ctx.req.path, {
    method: ctx.req.method,
    headers: { ...ctx.req.headers, "x-client-id": ctx.clientId! },
    body: ctx.req.body,
  });
  ctx.res = { status: r.status, body: await r.text() };
};

export const gateway = compose([auth, rateLimit(50, 100), router, proxy]);
declare function verifyJwt(token: string): { sub: string } | undefined;

// Balancer strategies behind one interface — swap algorithms without
// touching call sites (exactly how Envoy/Nginx model lb_policy).
interface Target { id: string; url: string; weight: number; inflight: number }
interface BalancingStrategy { pick(targets: Target[]): Target }

// Smooth weighted round robin (Nginx's algorithm): spreads a {5,1,1}
// weighting as A A B A C A A — no bursts of the same target.
class SmoothWeightedRR implements BalancingStrategy {
  private current = new Map<string, number>();
  pick(targets: Target[]): Target {
    let best: Target | undefined;
    let total = 0;
    for (const t of targets) {
      const cur = (this.current.get(t.id) ?? 0) + t.weight; // earn credit
      this.current.set(t.id, cur);
      total += t.weight;
      if (!best || cur > (this.current.get(best.id) ?? 0)) best = t;
    }
    this.current.set(best!.id, this.current.get(best!.id)! - total); // spend it
    return best!;
  }
}

// Power-of-two-choices least-connections: sample 2 random targets, take the
// less loaded — near-optimal spread without scanning the whole fleet.
class LeastConnections implements BalancingStrategy {
  pick(targets: Target[]): Target {
    const a = targets[Math.floor(Math.random() * targets.length)];
    const b = targets[Math.floor(Math.random() * targets.length)];
    return a.inflight <= b.inflight ? a : b;
  }
}

class LoadBalancer {
  constructor(private targets: Target[], private strategy: BalancingStrategy) {}
  async forward(path: string): Promise<Response> {
    const healthy = this.targets.filter(t => t.weight > 0); // ejected = weight 0
    if (healthy.length === 0) throw new Error("no healthy upstreams (503)");
    const target = this.strategy.pick(healthy);
    target.inflight++;
    try { return await fetch(target.url + path); }
    finally { target.inflight--; }
  }
}

const lb = new LoadBalancer(
  [
    { id: "a", url: "http://10.0.1.10:8080", weight: 5, inflight: 0 },
    { id: "b", url: "http://10.0.1.11:8080", weight: 1, inflight: 0 }, // canary
  ],
  new SmoothWeightedRR(), // or: new LeastConnections()
);

ALB vs NLB vs API Gateway vs CloudFront

They stack: CloudFront → (WAF) → ALB → services is the canonical web path; API Gateway → Lambda the canonical serverless path; NLB fronts databases, brokers, and anything PrivateLink-exposed.

Decision flowchart

Global load balancing

• DNS-based: Route 53 answers differently per client — latency (lowest-RTT region), geolocation (compliance: EU users → eu-central-1), weighted (gradual region migration), failover (primary/secondary gated on health checks). Caveat: failover speed is bounded by TTL + resolvers that ignore TTLs — think minutes, not seconds.
• Anycast: the same IP announced from every edge via BGP; the internet routes each user to the nearest one. Failover is route reconvergence — seconds, no client cache to expire. This is CloudFront's and Global Accelerator's mechanism.
• Staff pattern: anycast front door (Global Accelerator/CloudFront) for fast failover + Route 53 policies for coarse geography/compliance, health checks deep enough to detect a sick region, not just a live LB.

REST done properly

REST is not "JSON over HTTP." It is resource-oriented design that exploits HTTP's built-in semantics — caching, safety, idempotency — instead of fighting them.

• Resources are nouns, plural: /payments, /payments/{id}, /payments/{id}/refunds. Never verbs in paths (/createPayment is RPC wearing a REST costume).
• Verbs carry semantics: GET is safe (no side effects) and cacheable; PUT and DELETE are idempotent by definition; POST is neither — which is exactly why it needs idempotency keys (below). PATCH is partial update (use JSON Merge Patch or JSON Patch, and say which).
• Sub-resources for containment, query params for filtering: /accounts/{id}/transactions?status=settled&from=2026-01-01. Don't nest deeper than two levels — link instead.
• Use HTTP's machinery: ETag/If-Match for optimistic concurrency, Cache-Control for freshness, Location header on 201, Retry-After on 429/503.
• Actions that don't map to CRUD: model the action as a resource. Cancelling a payment is POST /payments/{id}/cancellations — you get an auditable record and idempotency for free. Fintech loves this: every state change becomes a first-class object.

The ~15 status codes engineers actually use

The 4xx/5xx split is a contract about blame and retry behavior: 4xx means "your request is wrong, retrying the same bytes won't help"; 5xx means "we failed, retry with backoff is reasonable." Clients build retry logic on this — mislabeling a validation error as 500 triggers pointless retry storms.

Richardson maturity, briefly

• Level 0: one URL, POST everything (SOAP-style tunnel).
• Level 1: resources get their own URLs.
• Level 2: HTTP verbs + status codes used correctly. This is where 95% of good production APIs live — Stripe, GitHub, Twilio.
• Level 3 (HATEOAS): responses embed hypermedia links driving state transitions. Rarely worth it for machine clients; a links object on paginated responses is the pragmatic subset everyone actually ships.

REST vs GraphQL vs gRPC vs WebSockets

These are not competitors on one axis — they optimize different edges of the system. Staff-level answer: pick per edge, and it's normal to run all four in one platform.

• Typical fintech layout: REST for the public/partner API (Stripe-style), gRPC between internal services (ledger ↔ payments ↔ risk), GraphQL at the BFF layer feeding mobile + web dashboards, WebSockets for live transaction feeds.
• GraphQL's tax: you inherit query-cost analysis, depth limiting, and the N+1 resolver problem (DataLoader batching). Don't pay it unless client shape diversity is real.
• gRPC's tax: load balancing needs L7 awareness (long-lived HTTP/2 connections pin to one backend — see ch6), and debugging binary frames needs tooling (grpcurl).

Idempotency keys: the payment-grade deep dive

The problem: POST /payments succeeds server-side, but the response is lost — client timeout, LB reset, mobile network blip. The client cannot distinguish "never happened" from "happened, ack lost." Its only rational move is retry — and a naive retry charges the customer twice. GET/PUT/DELETE are naturally idempotent; POST is where money gets duplicated.

• The fix: client generates a unique key (UUIDv4) per logical operation — not per HTTP attempt — and sends it as Idempotency-Key: 3f2a… on every retry of that operation.
• Server-side dedup store: keyed by (api_key, idempotency_key), three states — in-flight, completed (with stored response), absent. Completed → replay the stored status + body byte-for-byte. In-flight → return 409 (or block briefly): a concurrent duplicate must not execute twice.
• Fingerprint the body: hash the request payload and store it. Same key + different body = client bug → 422. Never silently replay a response for a different request.
• TTL: Stripe keeps keys 24h; that bounds store size while covering realistic retry windows. Persist in Redis (with fallback to DB for audit) or the primary DB inside the same transaction as the side effect — the truly bulletproof version writes the idempotency record and the payment row atomically.
• Reserve first, execute second: insert the in-flight record with a unique constraint before doing the work. The unique-violation on concurrent insert is your mutex.

// Idempotency middleware (Express-style) with a typed pluggable store.
import { createHash } from "node:crypto";

type IdempotencyState =
  | { status: "in_flight"; fingerprint: string }
  | { status: "completed"; fingerprint: string; httpStatus: number; body: string };

interface IdempotencyStore {
  /** Atomically claim the key. Returns existing record if already present. */
  reserve(key: string, fingerprint: string, ttlSec: number): Promise<
    { claimed: true } | { claimed: false; existing: IdempotencyState }
  >;
  complete(key: string, rec: Extract<IdempotencyState, { status: "completed" }>): Promise<void>;
  release(key: string): Promise<void>;          // free the slot if handler crashed
}

const fingerprintOf = (method: string, path: string, body: unknown): string =>
  createHash("sha256").update(`${method} ${path} ${JSON.stringify(body)}`).digest("hex");

export function idempotency(store: IdempotencyStore, ttlSec = 86_400) {
  return async (req: Req, res: Res, next: Next): Promise<void> => {
    if (req.method !== "POST") return next();          // GET/PUT/DELETE already idempotent
    const rawKey = req.header("Idempotency-Key");
    if (!rawKey) return next();                        // or 400 if your API mandates keys
    const key = `${req.apiKeyId}:${rawKey}`;           // scope per caller — never global!
    const fp = fingerprintOf(req.method, req.path, req.body);

    const claim = await store.reserve(key, fp, ttlSec);
    if (!claim.claimed) {
      const ex = claim.existing;
      if (ex.fingerprint !== fp)                       // same key, different payload => bug
        return res.status(422).json({ type: "idempotency_key_reuse",
          detail: "Idempotency-Key was used with a different request body." });
      if (ex.status === "in_flight")                   // concurrent duplicate racing us
        return res.status(409).set("Retry-After", "1")
          .json({ type: "request_in_flight", detail: "Original request still processing." });
      // Completed: replay the recorded response byte-for-byte.
      return res.status(ex.httpStatus).set("Idempotent-Replayed", "true").send(ex.body);
    }

    // We own the slot: capture the response the handler produces.
    const send = res.send.bind(res);
    res.send = (body: string) => {
      const done = res.statusCode < 500
        ? store.complete(key, { status: "completed", fingerprint: fp,
                                httpStatus: res.statusCode, body })
        : store.release(key);                          // 5xx: let the client retry for real
      void done.catch(() => { /* log; replay degrades to re-execution */ });
      return send(body);
    };
    try { next(); } catch (e) { await store.release(key); throw e; }
  };
}

• Why 5xx releases the key: if the handler failed before committing, replaying a stored 500 forever would brick the operation. Release lets the retry actually re-execute. If the failure happened after the charge committed, your handler itself must be idempotent internally (unique constraint on payment_intent_id) — defense in depth.
• Redis reserve is SET key value NX EX ttl; DB reserve is INSERT … ON CONFLICT DO NOTHING RETURNING. Both give you the atomic claim.

Pagination: offset vs cursor

• Offset (?limit=20&offset=10000): simple, supports "jump to page 47." But the DB must scan and discard 10,000 rows per query — O(offset) cost, so p99 degrades with depth. Worse, it's unstable: a row inserted (or deleted) before your offset while you paginate shifts everything — you see duplicates or skip rows. On an append-heavy transactions table this is not an edge case, it's every request.
• Cursor / keyset (?limit=20&cursor=eyJ…): the cursor encodes the sort position of the last item — WHERE (created_at, id) < ($1, $2) ORDER BY created_at DESC, id DESC LIMIT 20. Index seek, O(limit) regardless of depth, and stable under concurrent inserts (new rows land before your cursor, never inside your next page). Cost: no random page jumps, and the cursor must include a unique tiebreaker (id) or equal timestamps cause skips.
• Rule of thumb: cursor for any feed/list that grows (transactions, events, logs); offset only for small, mostly-static admin tables where "page 3 of 9" UX matters.

// Opaque keyset cursor: encode the sort position, verify it on the way back in.
interface TxnCursor { createdAt: string; id: string }   // (sort key, unique tiebreaker)

const encodeCursor = (c: TxnCursor): string =>
  Buffer.from(JSON.stringify(c)).toString("base64url");  // opaque to clients — the shape
                                                         // is NOT part of your API contract
const decodeCursor = (raw: string): TxnCursor => {
  const c: unknown = JSON.parse(Buffer.from(raw, "base64url").toString());
  if (typeof c !== "object" || c === null ||
      typeof (c as TxnCursor).createdAt !== "string" ||
      typeof (c as TxnCursor).id !== "string")
    throw new ApiError(400, "invalid_cursor");           // never trust decoded input
  return c as TxnCursor;
};

async function listTransactions(accountId: string, limit: number, raw?: string) {
  const cur = raw ? decodeCursor(raw) : undefined;
  const rows = await db.query<TxnRow>(
    `SELECT id, created_at, amount_minor, currency
       FROM transactions
      WHERE account_id = $1
        AND ($2::timestamptz IS NULL OR (created_at, id) < ($2, $3))
      ORDER BY created_at DESC, id DESC
      LIMIT $4`,                                          // fetch limit+1 to detect hasMore
    [accountId, cur?.createdAt ?? null, cur?.id ?? null, limit + 1]);

  const page = rows.slice(0, limit);
  const last = page.at(-1);
  return {
    data: page,
    hasMore: rows.length > limit,
    nextCursor: last ? encodeCursor({ createdAt: last.created_at, id: last.id }) : null,
  };
}

Filtering, sorting, versioning, errors, rate limits

Filtering & sorting conventions

• Flat params for equality: ?status=settled&currency=USD. Bracket or suffix operators for ranges: ?amount[gte]=1000 or ?created_after=… — pick one style and never mix.
• Sorting: ?sort=-created_at,amount (leading - = descending). Whitelist sortable fields — every sort you allow is an index you must maintain.
• Reject unknown query params with 400 rather than silently ignoring them; a typo in staus= that returns everything is a data-leak class of bug.

Versioning

• URL (/v2/payments): explicit, cache-friendly, easy to route/deprecate. Coarse-grained; forks docs and SDKs.
• Header (Stripe-Version: 2026-06-01): date-pinned versions per account, gateway translates old shapes to the current internal model. Most operable at scale; invisible in logs unless you add it.
• Field evolution (no versions): only additive changes, forever. Works if you enforce the compat rules below with tooling.
• Pragmatic default: URL major version (rarely bumped, v2 is a big deal) + additive evolution within it.

Backwards-compatibility rules (additive-only)

• Safe: adding optional request fields, adding response fields, adding enum values only if clients were told to treat unknown values as a default, adding endpoints.
• Breaking: removing/renaming fields, changing types or semantics (major: switching amount from float dollars to integer cents), tightening validation, changing error codes clients branch on, reordering meaning of defaults.
• Clients must be tolerant readers: ignore unknown fields, never exhaustively match enums from the wire.

Error contract: problem+json style

// RFC 9457 (problem+json) flavored, with a machine-readable discriminant.
// Clients branch on `type` (stable), humans read `detail` (free to change).
interface ApiProblem {
  type: string;          // stable code/URI: "https://api.orbitx.dev/errors/insufficient-funds"
  title: string;         // short human summary, stable per type
  status: number;        // mirrors HTTP status
  detail?: string;       // instance-specific explanation — NEVER parse this
  instance?: string;     // path or trace id for support: "/payments/pay_9f3"
  errors?: Array<{ field: string; code: string; message: string }>; // 422 field errors
}

type PaymentError =
  | { type: "insufficient_funds";  status: 422; balanceMinor: number }
  | { type: "card_declined";       status: 402; declineCode: string }
  | { type: "idempotency_key_reuse"; status: 422 }
  | { type: "rate_limited";        status: 429; retryAfterSec: number };

// Discriminated union => exhaustive, compiler-checked client handling:
function handle(e: PaymentError): ClientAction {
  switch (e.type) {
    case "insufficient_funds":    return promptTopUp(e.balanceMinor);
    case "card_declined":         return showDecline(e.declineCode);
    case "idempotency_key_reuse": return reportClientBug();
    case "rate_limited":          return backoff(e.retryAfterSec);
  } // adding a variant breaks compilation here — that is the point
}

Rate-limit headers

• Return on every response: RateLimit-Limit: 100, RateLimit-Remaining: 12, RateLimit-Reset: 30 (IETF draft standard; legacy X-RateLimit-* still common). On 429 add Retry-After.
• Well-behaved SDKs throttle before hitting 429 by watching Remaining — publishing headers turns your limits from a wall into a signal. Algorithms (token bucket etc.) are in ch12.

OpenAPI-first workflow

• Spec is the source of truth, reviewed in PRs before code: design review happens on the contract, not the implementation.
• Generate TS types + clients (openapi-typescript, orval), validate requests at runtime against the same spec (zod schemas generated from it), and run contract diff in CI (oasdiff) to mechanically block breaking changes.
• Same spec drives docs, mocks (Prism), and SDK generation — one artifact, five consumers.

Webhooks are your API in reverse — you become the client calling consumer-owned endpoints, and idempotency, retries, ordering, and signature verification all flip direction. Full treatment in ch15; the one rule that belongs here: every webhook delivery carries an event id, and consumers dedupe on it — exactly the idempotency-key pattern with roles swapped.

The cache hierarchy

Every serious system is a stack of caches; a request is a ball dropping through layers until one can answer. Each layer trades freshness and capacity for latency. Know the numbers — interviewers listen for them.

• Browser cache (~0 ms): Cache-Control, ETags, service workers. Free capacity you control only via headers.
• CDN edge (~5–30 ms): static assets, and increasingly API GETs with short TTLs. Absorbs the geographic tax.
• API gateway / edge cache (~1–5 ms in-region): whole-response caching keyed on path + params.
• Service in-memory (~100 ns–1 µs): per-instance Map/LRU. Fastest possible, but N instances = N stale copies — good for config, feature flags, hot reference data.
• Distributed cache — Redis/Memcached (~0.3–1 ms in-VPC): the shared workhorse. One coherent copy, network hop per read.
• DB buffer pool (~0.1–1 ms for a cached page): Postgres shared_buffers / InnoDB pool. You get this for free — a "cache miss" that hits a warm buffer pool is far cheaper than a cold disk read (~1–10 ms SSD).

Write/read strategies

Eviction policies

• LRU — evict least recently used. Default choice; exploits temporal locality. Weakness: one big scan (batch job reading everything once) flushes the hot set.
• LFU — evict least frequently used. Scan-resistant, but slow to forget yesterday's celebrity; production LFU decays counts over time (Redis allkeys-lfu uses a probabilistic 8-bit counter with decay).
• FIFO — evict oldest inserted. Simple, ignores access patterns; fine for uniform access.
• TTL — time-based expiry. Not really eviction (capacity) but freshness control; usually combined with LRU/LFU.
• Random — surprisingly close to LRU on skewed workloads, O(1), no metadata; Redis offers allkeys-random.

The classic LRU trick: a JS/TS Map preserves insertion order, so delete + re-insert moves a key to the "most recent" end — giving O(1) LRU without hand-rolling a linked list.

// O(1) LRU cache exploiting Map's insertion-order iteration.
// Oldest key = first key in iteration order; "touch" = delete + re-set.
export class LruCache<K, V> {
  private readonly map = new Map<K, { value: V; expiresAt: number }>();

  constructor(
    private readonly capacity: number,
    private readonly defaultTtlMs = Infinity,
  ) {
    if (capacity < 1) throw new RangeError("capacity must be >= 1");
  }

  get(key: K): V | undefined {
    const entry = this.map.get(key);
    if (!entry) return undefined;
    if (entry.expiresAt <= Date.now()) {         // lazy TTL expiry on read
      this.map.delete(key);
      return undefined;
    }
    this.map.delete(key);                         // touch: move to MRU end
    this.map.set(key, entry);
    return entry.value;
  }

  set(key: K, value: V, ttlMs = this.defaultTtlMs): void {
    if (this.map.has(key)) this.map.delete(key);  // re-insert at MRU end
    else if (this.map.size >= this.capacity) {
      const lru = this.map.keys().next();         // first key = least recent
      if (!lru.done) this.map.delete(lru.value);
    }
    this.map.set(key, { value, expiresAt: Date.now() + ttlMs });
  }

  delete(key: K): boolean { return this.map.delete(key); }
  get size(): number { return this.map.size; }
}

// Interview note: the textbook variant is HashMap + doubly-linked list —
// the DLL gives O(1) "unlink node, splice to head" explicitly, which is
// what Map is doing for you under the hood. Be ready to code the DLL
// version by hand: node {key, value, prev, next} + head/tail sentinels.
// LFU sketch: Map<key, node> + Map<freq, DLL of nodes> + minFreq pointer;
// on access, move node from freq bucket f to f+1 — O(1) (Leetcode 460).

Cache invalidation

"There are only two hard things in Computer Science: cache invalidation and naming things" — Phil Karlton. The joke survives because invalidation is a distributed consistency problem wearing a performance costume.

• TTL: bound staleness by time. Simplest, no infrastructure, and the fallback safety net under every other strategy. Pick TTL = maximum staleness the business tolerates, not a random 300.
• Event-driven: DB change → CDC stream (Debezium, DynamoDB Streams) or pub/sub → consumers delete/refresh affected keys. Near-real-time freshness; cost is a pipeline plus handling event loss/reorder (which is why you still keep a TTL).
• Versioned keys: never mutate — write user:42:v7, store the current version pointer, let old versions age out via TTL/eviction. Sidesteps races entirely at the price of extra memory and a pointer lookup; also the trick behind cache-busting asset URLs.

Consistency between cache and DB

• Invalidate, don't update, and do it after the DB write. Write-then-invalidate (a.k.a. the "Scaling Memcache at Facebook" pattern) beats write-then-update because two racing writers can update the cache out of order and strand a stale value forever; a delete is order-insensitive.
• The read-modify race (why even invalidation isn't airtight): reader misses → reads DB (old value) → pauses; writer updates DB → invalidates; reader wakes and sets the cache to the old value, which now lives until TTL. Rare (needs a miss interleaved with a write) but real at scale.
• Mitigations: short TTL as backstop; compare-and-set with a version/timestamp in the cached value; or delayed double delete — delete on write, then delete again after ~500 ms–1 s to sweep up any stale set from an in-flight reader.
• Read replicas widen the race: the reader may fetch from a lagging replica after invalidation. Either invalidate off the replica's CDC stream or read-your-writes from primary for cache fills triggered by writes.

The four classic failure modes

1. Cache stampede / thundering herd

• A hot key expires; 10,000 concurrent requests all miss and all hit the DB with the same query. The DB, sized for a 99% hit rate, receives 100× load and folds — which slows the rebuild, which prolongs the stampede.
• Mitigations: (a) single-flight locking — one caller rebuilds, everyone else awaits the same promise (per-instance) or a Redis SET NX lock (cross-fleet), serving stale meanwhile; (b) probabilistic early refresh (XFetch, Vattani et al.) — each reader independently decides to refresh slightly before expiry with probability rising as expiry nears, so one reader statistically refreshes and the key never publicly expires; (c) never letting hot keys expire at all — background refresh loop.

2. Hot key

• One key (celebrity profile, BTC-USD price) takes 500k reads/s. A single Redis shard caps out around 100k–1M simple ops/s — one key can saturate one node while the rest of the cluster idles, since a key lives on exactly one shard.
• Mitigations: per-instance local cache in front of Redis with a tiny TTL (1–5 s) — absorbs almost all reads at the cost of seconds of staleness; key splitting — write N replicas price:btc:0..N-1, readers pick one at random, spreading load across shards; read replicas of the cache tier.

3. Cache penetration

• Requests for keys that don't exist in the DB either (misspelled ids, or an attacker enumerating ids). Every request misses the cache and misses the DB, so nothing ever gets cached — the cache is bypassed entirely.
• Mitigations: negative caching — cache the "not found" result with a short TTL (30–60 s); a Bloom filter of all valid ids in front of the cache — "definitely absent" answers never touch cache or DB (false positives just fall through; internals in ch16); input validation to reject impossible ids before any lookup.

4. Cache avalanche

• Many keys expire simultaneously — a deploy warmed a million keys with identical TTLs, or the cache node restarts cold — and the whole read load lands on the DB at once. A stampede is one key; an avalanche is the entire keyspace.
• Mitigations: jitter every TTL (one-liner below) so expiries spread; warm caches before shifting traffic after deploys/restarts; Redis HA (replica promotion) so "node died" isn't "cache is cold"; circuit breakers + load shedding on the DB path as the last line.

// Jittered TTL — the avalanche one-liner (±spread around the base TTL):
export const jitteredTtl = (baseMs: number, spread = 0.1): number =>
  Math.round(baseMs * (1 - spread + Math.random() * 2 * spread));

// XFetch: probabilistic early refresh (Vattani, Chierichetti & Lowenstein,
// "Optimal Probabilistic Cache Stampede Prevention", VLDB 2015).
// Recompute early with probability that rises as expiry approaches:
// refresh when  now - delta * beta * ln(rand())  >=  expiresAt
interface XEntry<V> { value: V; deltaMs: number; expiresAt: number }
//                              ^ deltaMs = how long the last recompute took

export function xfetchShouldRefresh<V>(e: XEntry<V>, beta = 1.0): boolean {
  // -ln(U) is Exp(1)-distributed; beta > 1 refreshes earlier/more eagerly.
  return Date.now() - e.deltaMs * beta * Math.log(Math.random()) >= e.expiresAt;
}

export async function xfetchGet<V>(
  cache: Map<string, XEntry<V>>,
  key: string,
  ttlMs: number,
  recompute: () => Promise<V>,
): Promise<V> {
  const hit = cache.get(key);
  if (hit && !xfetchShouldRefresh(hit)) return hit.value;   // fresh enough

  const started = Date.now();
  const value = await recompute();                           // one lucky reader pays
  cache.set(key, { value, deltaMs: Date.now() - started,
                   expiresAt: Date.now() + jitteredTtl(ttlMs) });
  return value;
}

// Generic cache-aside helper with single-flight dedup: concurrent callers
// for the same key share ONE loader call instead of stampeding the DB.
interface RemoteCache {
  get(key: string): Promise<string | null>;
  set(key: string, value: string, ttlMs: number): Promise<void>;
}

const inFlight = new Map<string, Promise<unknown>>();  // per-instance dedup

export async function cacheAside<T>(
  cache: RemoteCache,
  key: string,
  ttlMs: number,
  loader: () => Promise<T>,          // the expensive DB/API call
): Promise<T> {
  const cached = await cache.get(key);
  if (cached !== null) return JSON.parse(cached) as T;   // hit

  const pending = inFlight.get(key);                     // someone already loading?
  if (pending) return pending as Promise<T>;             // piggyback on their flight

  const flight = (async (): Promise<T> => {
    try {
      const value = await loader();
      // Jitter here too — uniform TTLs are how avalanches are born.
      await cache.set(key, JSON.stringify(value), jitteredTtl(ttlMs));
      return value;
    } finally {
      inFlight.delete(key);          // ALWAYS clear, or a thrown loader
    }                                // poisons the key until process restart
  })();

  inFlight.set(key, flight);
  return flight;
}
// Cross-fleet version: swap the in-memory map for a short-TTL Redis lock
// (SET lock:key token NX PX 3000) and serve the stale value while locked.

Redis vs Memcached — and what not to cache

What NOT to cache:

• Account balances and anything money-authoritative — the ledger is the source of truth; a stale cached balance authorizing a withdrawal is an incident, not a bug. Cache derived displays of it with an explicit staleness label if you must.
• Data read once (no reuse = pure overhead + a copy to secure), write-heavy data with read-back-immediately semantics (hit rate near zero, race surface maximal).
• Secrets, tokens, PII in shared caches without encryption/scoping — the cache rarely inherits the DB's access controls, encryption or audit logging.
• Results of queries whose inputs are unbounded/user-crafted (cache-key cardinality explosion evicts your real hot set).
• Anything where serving stale violates correctness rather than just freshness: inventory holds, auth entitlements right after revocation, idempotency records.

Which caching strategy? (decision flow)

9.1 Anatomy of a database

Every serious disk-based database — Postgres, MySQL, Oracle, SQL Server — shares the same skeleton. Internalize it once and the rest of the chapter is variations on a theme.

• Pages are the atom of I/O. Tables and indexes are arrays of fixed-size pages — 16 KB in InnoDB, 8 KB in Postgres. The database never reads "a row"; it reads the whole page containing it. Row layout, free space, and a small header live inside each page.
• Heap files. In Postgres, table rows live in an unordered heap; indexes point at (page, offset) tuples (TIDs). InnoDB is different — the table is a B+tree ordered by primary key (see 9.6).
• Buffer pool. A big in-RAM cache of pages (often 60–80% of DB host memory). Reads hit RAM if the page is cached; writes dirty the in-memory page first. An LRU-ish policy (with midpoint insertion to survive scans) decides eviction. A cold buffer pool is why a freshly restarted replica is slow.
• WAL / redo log. Before any dirty page is written back, the change is appended to the write-ahead log and fsynced. Appends are sequential I/O — hundreds of MB/s even on modest disks and friendly to SSD internals — whereas flushing dirty pages is random I/O. So commit latency is bounded by one sequential append, not by rewriting scattered pages.
• Checkpointing. Periodically the DB flushes dirty pages and records "everything up to LSN X is on disk", letting old WAL segments be recycled. Aggressive checkpoints cause write spikes; lazy ones lengthen recovery.
• Crash recovery. On restart: replay WAL from the last checkpoint (redo), then roll back uncommitted transactions (undo). Durability comes from the log, not from data files being current.

9.2 Storage engines: B-tree vs LSM-tree

B+tree — the read-optimized default

• Structure: a balanced tree where internal nodes hold only keys (routing) and leaves hold data (or row pointers), with leaves linked for range scans. Each node is one page.
• Fan-out is the magic. A 16 KB page holds ~400–1000 keys. Fan-out 500 means 500³ = 125 million rows in 3 levels, ~62 billion in 4. And the top two levels are always in the buffer pool — a point lookup on a billion-row table is typically 1 disk read.
• Page splits: inserting into a full leaf splits it in two and pushes a separator key up — this is why random-key inserts (UUIDv4 PKs) cause fragmentation and write amplification, while sequential keys append to the rightmost leaf.
• Update in place: writes dirty existing pages → random I/O, but reads are predictable. Used by InnoDB, Postgres, SQL Server, Oracle.

LSM-tree — the write-optimized challenger

• Write path: append to WAL → insert into an in-RAM sorted memtable (skip list). When the memtable fills (~64 MB), it's flushed as an immutable sorted file — an SSTable. All disk writes are sequential.
• Read path: check memtable, then SSTables newest→oldest. Each SSTable carries a Bloom filter (skip files that definitely lack the key) and a sparse index (find the block within the file). Deletes are tombstone writes.
• Compaction merges overlapping SSTables, drops overwritten values and expired tombstones. Leveled compaction (RocksDB default) keeps each level 10× larger with non-overlapping files — better read/space amplification, more compaction I/O. Size-tiered (Cassandra default) merges similar-sized files — cheaper writes, worse space amplification (same key in many files).
• The three amplifications: LSM trades read amplification (check many files) for low write amplification on ingest; compaction adds background write amplification; size-tiered adds space amplification. Tuning an LSM store is choosing which one to pay.

// Toy LSM tree: WAL + memtable + SSTable flush + Bloom-ish skip + compaction.
type Value = string | null;                        // null = tombstone (delete marker)
interface Entry { key: string; value: Value; }

/** Immutable sorted run on disk. Real SSTables add a sparse index + true Bloom filter. */
class SSTable {
  private readonly keySet: Set<string>;      // stand-in for a Bloom filter
  constructor(readonly entries: Entry[]) {         // entries pre-sorted by key
    this.keySet = new Set(entries.map(e => e.key));
  }
  mightContain(key: string): boolean { return this.keySet.has(key); }
  get(key: string): Value | undefined {            // binary search: sorted file, O(log n)
    let lo = 0, hi = this.entries.length - 1;
    while (lo <= hi) {
      const mid = (lo + hi) >> 1;
      if (this.entries[mid].key === key) return this.entries[mid].value;
      this.entries[mid].key < key ? (lo = mid + 1) : (hi = mid - 1);
    }
    return undefined;
  }
}

class LsmTree {
  private memtable = new Map<string, Value>(); // real impl: skip list, kept sorted
  private wal: string[] = [];                      // sequential append-only log
  private sstables: SSTable[] = [];                // newest first
  constructor(private readonly memtableLimit = 4) {}

  put(key: string, value: Value): void {
    this.wal.push(JSON.stringify({ key, value })); // 1) durability: WAL append first
    this.memtable.set(key, value);                 // 2) then the in-RAM structure
    if (this.memtable.size >= this.memtableLimit) this.flush();
  }
  delete(key: string): void { this.put(key, null); } // deletes are just writes

  get(key: string): Value | undefined {
    if (this.memtable.has(key)) return this.memtable.get(key); // newest wins
    for (const t of this.sstables) {               // newest → oldest
      if (!t.mightContain(key)) continue;          // Bloom filter skips most files
      const v = t.get(key);
      if (v !== undefined) return v;               // may be null (tombstone) = deleted
    }
    return undefined;
  }

  /** Memtable full → one immutable sorted run. Sequential disk I/O only. */
  private flush(): void {
    const run = [...this.memtable.entries()]
      .sort(([a], [b]) => a.localeCompare(b))
      .map(([key, value]) => ({ key, value }));
    this.sstables.unshift(new SSTable(run));
    this.memtable.clear();
    this.wal = [];                                 // flushed data no longer needs the WAL
    if (this.sstables.length > 4) this.compact();  // too many runs → reads degrade
  }

  /** Merge all runs; newest value per key wins; tombstones finally dropped. */
  private compact(): void {
    const merged = new Map<string, Value>();
    for (const t of [...this.sstables].reverse())  // oldest first so newer overwrites
      for (const e of t.entries) merged.set(e.key, e.value);
    const live = [...merged.entries()]
      .filter(([, v]) => v !== null)               // physical delete happens HERE
      .sort(([a], [b]) => a.localeCompare(b))
      .map(([key, value]) => ({ key, value }));
    this.sstables = [new SSTable(live)];           // size-tiered-style full merge
  }
}

9.3 Indexes — the fine print

• Clustered vs secondary. A clustered index is the table (InnoDB: rows live in the PK B+tree). Secondary indexes store the indexed columns plus the PK; a secondary lookup therefore costs two B-tree descents — index → PK → row. Fat PKs (36-byte UUID strings) bloat every secondary index. Postgres has no clustered index; all indexes point into the heap.
• Composite indexes & leftmost prefix. An index on (user_id, status, created_at) serves WHERE user_id=?, user_id=? AND status=?, and the full triple — but is nearly useless for WHERE status=? alone. Order columns: equality filters first, then the range/sort column. A range on a middle column stops use of columns after it.
• Covering indexes / index-only scans. If every selected column is in the index, the engine never touches the table — often a 10× win. In Postgres INCLUDE (col) adds payload columns without making them key columns.
• Partial indexes: CREATE INDEX ... WHERE status = 'PENDING' — tiny, hot, perfect for queue-like tables where 99% of rows are terminal.
• When indexes hurt: every index is another B-tree updated per write (write amplification), and low-selectivity indexes (e.g. on a boolean) are ignored by the planner anyway — a full scan is cheaper than millions of random heap fetches.

EXPLAIN ANALYZE
SELECT o.id, o.total_cents
FROM   orders o
WHERE  o.user_id = 42 AND o.status = 'PAID'
ORDER  BY o.created_at DESC
LIMIT  20;

-- GOOD plan, with index (user_id, status, created_at DESC):
--   Limit (actual rows=20)
--     -> Index Scan using idx_orders_user_status_created on orders
--          Index Cond: (user_id = 42 AND status = 'PAID')
--   No Sort node: the index already emits rows in ORDER BY order,
--   so LIMIT 20 stops after reading 20 index entries. O(20), not O(rows).

-- RED FLAGS to hunt in any plan:
--   Seq Scan on a large table          → missing or unusable index
--   Rows Removed by Filter: 2,481,904  → index not selective / wrong column order
--   Sort Method: external merge Disk   → ORDER BY not covered; spilling to disk
--   Nested Loop, outer rows=1.2M       → stale statistics; run ANALYZE
--   estimated rows=1 vs actual=500000  → planner misestimate → wrong join choice

9.4 Joins — how the optimizer executes them

There are only three join algorithms. The planner picks per-join based on row estimates, available indexes, sort order, and memory — which is why stale statistics silently flip a fast plan into a disaster.

type Row = Record<string, unknown>;

/** NESTED LOOP: for each outer row, find matches in inner.
 *  Naive O(N·M); with an index on the inner key, O(N·log M).
 *  Planner picks it when the outer side is SMALL and the inner side is indexed. */
function nestedLoopJoin(outer: Row[], inner: Row[],
                        match: (o: Row, i: Row) => boolean): Row[] {
  const out: Row[] = [];
  for (const o of outer)                 // small driving table
    for (const i of inner)               // real DBs: index probe, not a scan
      if (match(o, i)) out.push({ ...o, ...i });
  return out;
}

/** HASH JOIN: build a hash table on the smaller input, probe with the larger.
 *  O(N + M) time, O(min(N, M)) memory; spills to disk partitions if too big.
 *  Planner's choice for large unsorted equijoins. Equality predicates ONLY. */
function hashJoin(build: Row[], probe: Row[], key: string): Row[] {
  const table = new Map<unknown, Row[]>();
  for (const b of build) {                          // phase 1: build (small side)
    const bucket = table.get(b[key]) ?? [];
    bucket.push(b); table.set(b[key], bucket);
  }
  const out: Row[] = [];
  for (const p of probe)                            // phase 2: probe (big side)
    for (const b of table.get(p[key]) ?? []) out.push({ ...b, ...p });
  return out;
}

/** SORT-MERGE JOIN: both inputs sorted on the key → one interleaved pass, O(N + M).
 *  Wins when inputs are ALREADY sorted (index order) or must be sorted anyway. */
function mergeJoin(left: Row[], right: Row[], key: string): Row[] {
  const out: Row[] = []; let i = 0, j = 0;
  const lt = (a: unknown, b: unknown) => String(a) < String(b);
  while (i < left.length && j < right.length) {
    if (lt(left[i][key], right[j][key])) i++;
    else if (lt(right[j][key], left[i][key])) j++;
    else {                                          // equal keys: emit cross-group
      const k = left[i][key]; const jStart = j;
      for (; i < left.length && left[i][key] === k; i++)
        for (j = jStart; j < right.length && right[j][key] === k; j++)
          out.push({ ...left[i], ...right[j] });
    }
  }
  return out;
}

Distributed joins: when tables live on different shards, the engine either broadcasts the small table to every node holding the big one (cheap if it fits in RAM everywhere) or shuffles both tables across the network partitioned by join key so matching rows land on the same node (expensive but scales). Data warehouses (Redshift, Spark, BigQuery) spend most of their optimizer effort avoiding shuffles — and co-locating tables on the same distribution key eliminates the network step entirely.

9.5 Concurrency: MVCC, isolation, locks

• MVCC core idea: writers create new versions instead of overwriting; readers pick the version visible to their snapshot. Result: readers never block writers and writers never block readers — no read locks at all.
• Postgres: an UPDATE writes a whole new row version in the heap; old versions stay until VACUUM reclaims them (hence table bloat on hot-update tables). Each tuple carries xmin/xmax transaction IDs.
• InnoDB: updates in place, but stashes the before-image in the undo log; readers reconstruct old versions by walking undo chains. Long-running transactions pin undo history — the InnoDB analog of bloat.
• Locks still exist for write-write conflicts: shared (S) vs exclusive (X); row-level vs table-level; intention locks to make them compatible. InnoDB adds gap locks under REPEATABLE READ to lock the space between index entries and stop phantom inserts.
• Deadlocks arise from acquiring rows in different orders; engines detect cycles in the waits-for graph and kill one victim. Fix: touch rows in a consistent order, keep transactions short, retry on deadlock errors (they are normal, not bugs).

/** Simplified Postgres-style tuple visibility — the heart of MVCC.
 *  Every row version carries the txid that created it (xmin) and,
 *  if superseded/deleted, the txid that did so (xmax). An UPDATE is
 *  "set xmax on old version + insert new version with fresh xmin". */
interface TupleVersion<T> {
  xmin: number;               // creator transaction id
  xmax: number | null;        // deleter/updater txid, null = still live
  data: T;
}
interface Snapshot {
  xid: number;                // my transaction id
  inProgress: Set<number>;    // txids still running when snapshot was taken
}

function isVisible<T>(v: TupleVersion<T>, s: Snapshot): boolean {
  // A txid's effects are visible if it's me, or it committed before my snapshot.
  const sees = (xid: number) =>
    xid === s.xid || (xid < s.xid && !s.inProgress.has(xid));
  if (!sees(v.xmin)) return false;                 // creator invisible → row not born yet
  if (v.xmax !== null && sees(v.xmax)) return false; // visible tx deleted it → gone
  return true;   // decided with pure arithmetic — no lock was ever taken to read
}
// Old versions where xmax is globally visible are garbage → VACUUM reclaims them.

9.6 MySQL / InnoDB mini-tour

• The table is the PK B+tree (clustered): rows physically ordered by primary key. Sequential PKs (auto-increment, ULID, UUIDv7) append to the last page; random UUIDv4 keys splatter inserts across the tree, causing page splits, half-empty pages, and buffer-pool churn.
• Redo log (WAL) gives durability; undo log gives MVCC and rollback; the doublewrite buffer protects against torn 16 KB pages on 4 KB-atomic disks.
• Buffer pool with midpoint-insertion LRU so a full table scan can't evict the hot working set.
• Adaptive hash index: InnoDB watches access patterns and builds an in-RAM hash over hot B-tree pages, turning repeated point lookups into O(1).
• Binlog is a separate logical log (row events) used for replication and CDC (Debezium tails it); internally each commit is a two-phase commit between the redo log and the binlog so replicas never diverge from crash-recovered state.

9.7 MongoDB mini-tour

• BSON documents — binary JSON with real types (dates, decimals, binary), 16 MB cap per document.
• WiredTiger engine: document-level optimistic concurrency (no page or collection locks on writes), MVCC snapshots for reads, periodic checkpoints plus a journal (WAL) between them, and on-disk compression (snappy/zstd) by default.
• _id / ObjectId: 12 bytes = 4-byte timestamp + 5-byte random + 3-byte counter — roughly time-ordered, so inserts append rather than splatter (the same sequential-key argument as InnoDB).
• Secondary indexes are B-trees, same leftmost-prefix rules as SQL; multikey indexes fan out array fields into multiple entries.
• Replica sets: single primary, followers tail the oplog (an idempotent logical change stream, itself a capped collection); automatic failover by election. Tunable writeConcern/readConcern trade latency vs durability per operation.
• Embed vs reference rule of thumb: embed what you always load together and that grows boundedly (order + its line items); reference what is shared, unbounded, or independently queried (user ↔ orders). The document model wins when your access pattern is "fetch the whole aggregate by key"; relational wins when you query the same data along many different axes.

9.8 Redis mini-tour — why is Redis so fast?

The interview classic. Four compounding reasons, none of which is "it's written in C":

• Everything lives in RAM. A memory access costs ~100 ns vs ~100 µs for an SSD read — a ~1000× floor advantage before any cleverness. Persistence is asynchronous and off the hot path.
• Single-threaded command execution. One thread runs all commands, so there are zero locks, no context switches, no cache-line ping-pong. At in-memory speeds the CPU is rarely the bottleneck — the network is.
• I/O multiplexing (epoll/kqueue): one event loop watches tens of thousands of sockets, servicing whichever is ready — the same trick as Nginx/Node.
• Data structures engineered per size class: O(1) hash-table core; SDS strings (length-prefixed, no strlen scans, binary-safe); small collections stored as listpacks (one contiguous cache-friendly blob) that transparently upgrade to hash tables / skip lists when they grow; sorted sets = hash table + skip list, giving O(1) score lookup and O(log n) rank/range ops; sets of small integers use packed intsets.

• Modern threading: since Redis 6, I/O threads parse and write socket buffers in parallel — but command execution is still one thread, preserving the no-locks model. (Background threads have long handled fsync, big-key frees, etc.)
• Clustering: Redis Cluster hashes keys into 16384 slots (CRC16 mod 16384) spread across primaries; multi-key ops must share a slot (hash tags {user:42}). Sentinel provides monitoring + automatic failover for non-clustered setups.
• What Redis is NOT: a primary system of record (even AOF everysec can lose ~1s), a relational store, or a place for datasets that exceed RAM economics. Treat it as a cache/coordination layer whose loss degrades latency, not correctness.

$ redis-cli
127.0.0.1:6379> RPUSH tiny a b c
127.0.0.1:6379> OBJECT ENCODING tiny     # "listpack"  — small list packed in one blob
127.0.0.1:6379> ZADD board 100 alice 90 bob
127.0.0.1:6379> OBJECT ENCODING board    # "listpack"  — while under 128 entries...
#   ...past zset-max-listpack-entries it upgrades to "skiplist" for O(log n) ranges
127.0.0.1:6379> SET n 12345
127.0.0.1:6379> OBJECT ENCODING n        # "int"       — stored as a machine integer
127.0.0.1:6379> DEBUG SLEEP 5            # NEVER in prod: blocks the ONE thread —
#   every client stalls. Same reason KEYS * and huge LRANGEs are dangerous.

ACID — what each letter actually guarantees

ACID is a per-database, per-transaction contract. Each property is narrower than most people think:

• Atomicity — all-or-nothing on abort: if the transaction fails midway, partial writes are rolled back. Misreading: it says nothing about concurrency. Two atomic transactions can still interleave badly — that is Isolation's job.
• Consistency — the database moves from one valid state to another with respect to declared constraints (FKs, checks, uniqueness). Misreading: the "C" is mostly the application's responsibility; it is not CAP-consistency, and the DB cannot enforce invariants you never declared (e.g. "sum of ledger entries = 0" unless you write it as a constraint or check it in the txn).
• Isolation — concurrent transactions behave as if serial — but only at SERIALIZABLE. Misreading: most databases default to weaker levels (Postgres: READ COMMITTED, MySQL/InnoDB: REPEATABLE READ), so out of the box you are exposed to a specific menu of anomalies (next section).
• Durability — once committed, the write survives a crash, via WAL/fsync. Misreading: durable on one node. If the disk dies before replication catches up, "durable" data is gone — durability across nodes is a replication setting (ch11), not an ACID given.

BASE — and why it exists

BASE (Basically Available, Soft state, Eventually consistent) is not the opposite of ACID — it is what you accept when you shard/replicate for scale and refuse to pay cross-node coordination costs on every write. Reads may be stale, state converges over time, and the system stays up through partitions.

Isolation anomalies — concrete interleavings

Staff-level answers show the interleaving, not the definition. T1/T2 are concurrent transactions; time flows down.

-- DIRTY READ (blocked at READ COMMITTED and above)
T1: UPDATE accounts SET balance = balance - 100 WHERE id = 'A';
T2:                     SELECT balance FROM accounts WHERE id = 'A';  -- sees uncommitted -100
T1: ROLLBACK;           -- T2 acted on money that never moved

-- NON-REPEATABLE READ (possible at READ COMMITTED)
T1: SELECT balance FROM accounts WHERE id = 'A';   -- 500
T2: UPDATE accounts SET balance = 300 WHERE id = 'A'; COMMIT;
T1: SELECT balance FROM accounts WHERE id = 'A';   -- 300: same row, two answers in one txn

-- PHANTOM READ (possible below SERIALIZABLE; range re-query changes)
T1: SELECT count(*) FROM payments WHERE user_id = 42 AND day = today;  -- 3
T2: INSERT INTO payments (user_id, ...) VALUES (42, ...); COMMIT;
T1: SELECT count(*) FROM payments WHERE user_id = 42 AND day = today;  -- 4: a phantom row appeared

-- LOST UPDATE (read-modify-write race)
T1: SELECT balance FROM accounts WHERE id = 'A';   -- 500
T2: SELECT balance FROM accounts WHERE id = 'A';   -- 500
T1: UPDATE accounts SET balance = 600 WHERE id = 'A'; COMMIT;  -- +100
T2: UPDATE accounts SET balance = 550 WHERE id = 'A'; COMMIT;  -- +50, silently erases T1's +100

-- WRITE SKEW (survives SNAPSHOT ISOLATION; needs SERIALIZABLE or explicit locks)
-- Invariant: at least one on-call doctor. Both read a consistent snapshot, write DIFFERENT rows.
T1: SELECT count(*) FROM doctors WHERE on_call;    -- 2, "safe for me to leave"
T2: SELECT count(*) FROM doctors WHERE on_call;    -- 2, "safe for me to leave"
T1: UPDATE doctors SET on_call = false WHERE id = 'alice'; COMMIT;
T2: UPDATE doctors SET on_call = false WHERE id = 'bob';   COMMIT;  -- invariant broken: 0 on call

Optimistic vs pessimistic concurrency control

• Pessimistic: take the lock before touching the data — SELECT ... FOR UPDATE. Correct by construction, but locks are held for the transaction's lifetime: under contention you get queueing, and across services it's a deadlock factory. Use when conflict is likely (hot balance row) and the critical section is short.
• Optimistic: read freely, then commit with a compare-and-swap on a version column; on conflict, retry. No locks held during "think time" — ideal when conflict is rare (a user's own profile, most account rows), and the only option when the read happens in one HTTP request and the write in another.
• SELECT ... FOR UPDATE also has NOWAIT (fail fast) and SKIP LOCKED (the standard trick for building a job queue on Postgres: workers grab unlocked rows without blocking each other).

// Optimistic locking with a version column: compare-and-swap UPDATE + retry loop.
interface Account {
  id: string;
  balanceMinor: bigint;   // integer minor units -- never floats for money
  version: number;        // bumped on every successful write
}

class OptimisticLockError extends Error {}

class AccountRepo {
  constructor(private readonly db: Pool) {}

  async load(id: string): Promise<Account> {
    const { rows } = await this.db.query(
      'SELECT id, balance_minor, version FROM accounts WHERE id = $1', [id]);
    if (rows.length === 0) throw new Error(`account ${id} not found`);
    return { id: rows[0].id, balanceMinor: BigInt(rows[0].balance_minor), version: rows[0].version };
  }

  // Compare-and-swap: succeeds only if nobody else committed since our read.
  async save(a: Account, expectedVersion: number): Promise<void> {
    const res = await this.db.query(
      `UPDATE accounts
          SET balance_minor = $1, version = version + 1
        WHERE id = $2 AND version = $3`,          // <-- the CAS predicate
      [a.balanceMinor.toString(), a.id, expectedVersion]);
    if (res.rowCount === 0) throw new OptimisticLockError('version moved; retry');
  }
}

// Generic retry wrapper: reload -> mutate -> CAS, with jittered exponential backoff.
async function withOptimisticRetry<T>(
  attempt: () => Promise<T>, maxRetries = 5,
): Promise<T> {
  for (let i = 0; ; i++) {
    try {
      return await attempt();
    } catch (err) {
      if (!(err instanceof OptimisticLockError) || i >= maxRetries) throw err;
      const backoffMs = Math.min(200, 10 * 2 ** i) * (0.5 + Math.random()); // full jitter
      await new Promise((r) => setTimeout(r, backoffMs));
    }
  }
}

// Usage: debit an account. Under contention this retries instead of blocking.
async function debit(repo: AccountRepo, id: string, amountMinor: bigint): Promise<void> {
  await withOptimisticRetry(async () => {
    const acct = await repo.load(id);
    if (acct.balanceMinor < amountMinor) throw new Error('insufficient funds'); // not retryable
    acct.balanceMinor -= amountMinor;
    await repo.save(acct, acct.version);
  });
}

Distributed transactions: 2PC and why it blocks

Once the debit and the credit live in different databases (two services), a single ACID transaction no longer exists. Two-Phase Commit is the classic answer: a coordinator asks every participant to prepare (write redo/undo to disk, hold locks, vote), and only if all vote yes does it broadcast commit.

• Why it blocks: a prepared participant has promised "I can commit" and must hold row locks until it learns the outcome. If the coordinator crashes after collecting votes, participants are stuck in doubt — aborting could contradict a commit the coordinator already logged. Locks pile up; throughput collapses to zero on hot rows.
• Other costs: latency of the slowest participant × 2 round trips on every transaction; coordinator is a single point of failure; heterogeneous participants (Kafka? a REST API? Stripe?) don't speak XA at all.
• 3PC adds a pre-commit phase to remove blocking under crash-stop failures, but it assumes bounded network delay (no partitions) — an assumption real networks violate — so it's a mention, not a tool. Modern systems that need atomic cross-partition commit (Spanner, CockroachDB) run 2PC over Paxos/Raft groups, making each "participant" itself fault-tolerant.

Sagas: the industry's answer

A saga replaces one distributed transaction with a sequence of local transactions, each atomically committed in its own service, plus a compensating transaction for each step to semantically undo it if a later step fails. You trade isolation for availability and loose coupling.

// Orchestrated saga: reserve funds -> capture -> ledger write, with typed compensations.
type StepName = 'reserve' | 'capture' | 'ledger';

interface PaymentCtx {
  paymentId: string;          // doubles as the idempotency key for every step
  fromAccount: string;
  toAccount: string;
  amountMinor: bigint;
  reservationId?: string;     // filled in by the reserve step
  captureId?: string;
}

interface SagaStep {
  name: StepName;
  // Both calls MUST be idempotent: retries after timeouts will replay them.
  execute(ctx: PaymentCtx): Promise<void>;
  compensate(ctx: PaymentCtx): Promise<void>;
}

type SagaResult =
  | { status: 'completed' }
  | { status: 'compensated'; failedStep: StepName; cause: Error }
  | { status: 'compensation_failed'; failedStep: StepName; stuckStep: StepName }; // page a human

const steps: SagaStep[] = [
  {
    name: 'reserve',
    async execute(ctx) {                       // local txn in the funds service
      ctx.reservationId = await fundsSvc.reserve({
        idempotencyKey: `${ctx.paymentId}:reserve`,
        account: ctx.fromAccount, amountMinor: ctx.amountMinor,
      });
    },
    async compensate(ctx) {                    // semantic undo: release the hold
      if (ctx.reservationId) await fundsSvc.release(ctx.reservationId);
    },
  },
  {
    name: 'capture',
    async execute(ctx) {
      ctx.captureId = await fundsSvc.capture(ctx.reservationId!,
        { idempotencyKey: `${ctx.paymentId}:capture` });
    },
    async compensate(ctx) {                    // money already moved: undo = refund
      if (ctx.captureId) await fundsSvc.refund(ctx.captureId);
    },
  },
  {
    name: 'ledger',
    async execute(ctx) {                       // append double-entry rows; unique on paymentId
      await ledgerSvc.postEntries({
        idempotencyKey: ctx.paymentId,
        debit: ctx.fromAccount, credit: ctx.toAccount, amountMinor: ctx.amountMinor,
      });
    },
    async compensate() { /* last step: nothing after it can fail */ },
  },
];

async function runPaymentSaga(ctx: PaymentCtx): Promise<SagaResult> {
  const done: SagaStep[] = [];
  for (const step of steps) {
    try {
      await sagaLog.record(ctx.paymentId, step.name, 'started'); // durable state machine
      await step.execute(ctx);
      await sagaLog.record(ctx.paymentId, step.name, 'done');
      done.push(step);
    } catch (cause) {
      // Unwind completed steps in REVERSE order; compensations must be retried, not skipped.
      for (const prev of done.reverse()) {
        try {
          await withOptimisticRetry(() => prev.compensate(ctx));
          await sagaLog.record(ctx.paymentId, prev.name, 'compensated');
        } catch {
          await sagaLog.record(ctx.paymentId, prev.name, 'compensation_failed');
          return { status: 'compensation_failed', failedStep: step.name, stuckStep: prev.name };
        }
      }
      return { status: 'compensated', failedStep: step.name, cause: cause as Error };
    }
  }
  return { status: 'completed' };
}

Saga pitfalls you must name

• Compensations can fail too. A refund API can be down. Compensations must be idempotent + retried from durable state; when retries exhaust, escalate to a manual-ops queue. There is no "rollback of the rollback".
• Semantic vs system rollback. You cannot un-send an email or un-settle a wire; you send a correction / issue a refund / post a reversing ledger entry. Compensation restores the business invariant, not the exact prior bytes.
• Isolation is gone. Between step 1 and step 3, other transactions can observe the intermediate state (funds reserved but ledger empty) — a saga-level dirty read. Countermeasures: semantic locks (mark the payment PENDING and make readers treat pending specially), commutative operations, and re-reading state before each step.
• The saga log itself must be durable and the trigger for each step must be atomic with the local write — which is exactly the transactional outbox pattern: write the business row and the "next step" event in one local transaction, then relay it. Full treatment in ch14.
• Idempotency is the glue. Every step and compensation will eventually be delivered twice (timeout → retry). Key every side effect on paymentId:step and make handlers no-ops on replay. Without this, a saga is a duplicate-payment generator.

Why replicate

• High availability — a node dies (and it will: disks, AZs, deploys), a replica takes over. This is the non-negotiable reason.
• Read scale — most workloads are 10:1 to 100:1 read-heavy; fan reads out to followers.
• Geo-latency — put copies near users: Sydney reads from Sydney (~1 ms) instead of Virginia (~200 ms RTT). Physics is not negotiable.

Leader–follower replication

One node (leader) accepts writes and streams its change log to followers. The key dial is when the leader acknowledges the client:

Failover dangers

• Split brain: the old leader wasn't dead — just partitioned or GC-pausing. A new leader is promoted; now two nodes accept writes and the dataset forks. Guard rails: quorum-based leader election (Raft), fencing tokens (STONITH — "shoot the other node in the head"), and generation numbers on every write.
• Lost writes: with async replication, the promoted follower may be seconds behind; the old leader's tail of acknowledged writes vanishes. GitHub's 2018 incident is the canonical war story — MySQL failover across a 43-second partition left inconsistent writes on both sides.
• Failover timing: too aggressive a health check ⇒ flapping and unnecessary failovers during load spikes; too lazy ⇒ minutes of downtime. There is no clean answer, only tuned timeouts.

Multi-leader and leaderless

• Multi-leader (each region accepts writes): great write latency, but concurrent writes to the same key will conflict. Last-write-wins (LWW) resolution is dangerous: clock skew decides the winner and the "loser" is silently deleted — data loss by design (Cassandra's default). Alternatives: keep siblings and make the app merge (Riak), or use CRDTs — data types (counters, sets, maps) whose merge operation is mathematically commutative/associative/idempotent, so replicas converge without coordination regardless of delivery order. Redis Enterprise CRDBs and collaborative editors use them; they're the one paragraph you should be able to say fluently, not implement.
• Leaderless / quorum (Dynamo-style: Cassandra, Riak): any replica accepts writes. With N replicas, write to W, read from R; if W + R > N the read set intersects the write set, so at least one replica in every read has the latest value (typical: N=3, W=2, R=2). Stale replicas get fixed by read repair and anti-entropy (Merkle-tree sync). Sloppy quorum: during a partition, writes land on any W reachable nodes — even non-home nodes — which hold the data with a "hinted handoff" note and forward it when the home node returns. Availability up, but W+R>N no longer guarantees reading your write.

Replication lag and the guarantees to name-drop

Async followers run 10 ms – seconds behind. Three user-visible anomalies, each with a named guarantee that fixes it:

• Read-your-writes: user updates their profile, refreshes, sees the old value — reads route to a lagging follower. Mitigations: pin the user's reads to the leader for N seconds after their own write (sticky-after-write), route by timestamp ("read from a replica only if it has applied LSN ≥ my last write"), or read-own-data always from leader.
• Monotonic reads: two successive reads hit different replicas and time appears to go backwards (comment visible, then gone). Mitigation: hash each user to a fixed replica.
• Consistent prefix: observer sees the answer before the question because two causally-related writes replicated through different partitions out of order. Mitigation: write causally related data to the same partition, or use causal-consistency machinery.
• Lag-aware routing: production readers export replica lag as a metric and the routing tier drops replicas whose lag exceeds a threshold (e.g., 500 ms) from the read pool.

Partitioning / sharding

Replication copies the whole dataset; partitioning splits it so each shard holds a subset — needed when data or write throughput exceeds one machine. Two families:

Consistent hashing — the algorithm to whiteboard

Naive hash(key) mod N is a trap: adding one node changes N and remaps ~all keys — a full-cluster cache stampede. Consistent hashing maps both nodes and keys onto a ring; a key belongs to the first node clockwise. Adding/removing a node only moves ~1/N of keys. Virtual nodes (each physical node gets 100–200 ring positions) smooth out variance and let heterogeneous nodes take proportional load.

// Consistent hash ring with virtual nodes.
// Ring = sorted array of vnode hash positions; lookup = binary search clockwise.
import { createHash } from 'crypto';

interface VNode { position: number; nodeId: string; }

function hashToUint32(input: string): number {
  // First 4 bytes of md5 as an unsigned 32-bit ring position (fine for placement,
  // NOT for security). Production: murmur3/xxhash for speed.
  return createHash('md5').update(input).digest().readUInt32BE(0);
}

export class ConsistentHashRing {
  private ring: VNode[] = [];              // kept sorted by position
  private readonly nodes = new Set<string>();

  constructor(private readonly vnodesPerNode = 150) {}

  addNode(nodeId: string): void {
    if (this.nodes.has(nodeId)) return;
    this.nodes.add(nodeId);
    for (let i = 0; i < this.vnodesPerNode; i++) {
      this.ring.push({ position: hashToUint32(`${nodeId}#vn${i}`), nodeId });
    }
    this.ring.sort((a, b) => a.position - b.position);
  }

  removeNode(nodeId: string): void {
    this.nodes.delete(nodeId);
    this.ring = this.ring.filter((v) => v.nodeId !== nodeId);
    // Keys on the removed vnodes' arcs now fall through to the next
    // clockwise vnode automatically -- only ~1/N of keys move.
  }

  /** Owner of a key: first vnode clockwise from hash(key). */
  lookup(key: string): string {
    if (this.ring.length === 0) throw new Error('empty ring');
    const h = hashToUint32(key);
    let lo = 0, hi = this.ring.length - 1, idx = 0; // binary search: first pos >= h
    if (h > this.ring[hi].position) {
      idx = 0;                                      // wrap past 12 o'clock
    } else {
      while (lo <= hi) {
        const mid = (lo + hi) >> 1;
        if (this.ring[mid].position >= h) { idx = mid; hi = mid - 1; }
        else lo = mid + 1;
      }
    }
    return this.ring[idx].nodeId;
  }

  /** N distinct nodes clockwise -- the replica set for a key (Dynamo-style). */
  lookupN(key: string, n: number): string[] {
    const out: string[] = [];
    if (this.ring.length === 0) return out;
    let i = this.ring.findIndex((v) => v.position >= hashToUint32(key));
    if (i === -1) i = 0;
    while (out.length < Math.min(n, this.nodes.size)) {
      const id = this.ring[i % this.ring.length].nodeId;
      if (!out.includes(id)) out.push(id);          // skip vnodes of already-chosen nodes
      i++;
    }
    return out;
  }
}

Resharding strategies

• Fixed partition count (Kafka, Cassandra, Elasticsearch): create far more partitions than nodes up front (e.g., 1,024); rebalancing moves whole partitions between nodes — keys never re-hash. Simple and predictable; the cost is choosing the count right, since Kafka can't split a partition later without breaking key ordering.
• Dynamic splitting (HBase, DynamoDB, MongoDB ranges): a partition that exceeds a size/throughput threshold splits in two. Adapts to skew automatically; the cost is split/migration churn and brief hot moments during splits.

Hot partitions & the celebrity problem

• Hashing spreads keys, not load: if one key is Taylor Swift's timeline, its whole partition melts while others idle.
• Key salting: split the hot key into key#0..key#15, writes pick a random suffix, reads fan out and merge over 16 sub-keys. Trade: write scale × read amplification.
• Dedicated shards: detect celebrities (top-K by traffic) and route them to isolated, over-provisioned capacity with their own cache tier.
• Request coalescing: thousands of identical concurrent reads for the hot key collapse into one backend fetch (singleflight) + aggressive edge caching — often the read-side celebrity fix.

Secondary indexes on sharded data

• Local (document-partitioned) index: each shard indexes only its own rows. Writes stay single-shard (fast); a query by the indexed field must scatter-gather every shard — latency is the p99 of the slowest shard, and cost grows with shard count.
• Global (term-partitioned) index: the index itself is sharded by the indexed value; reads hit exactly one index shard, but a single row write now touches multiple shards ⇒ the index is updated async (DynamoDB GSIs are eventually consistent for exactly this reason).
• Rule of thumb: local index for write-heavy + rare cross-shard queries; global index for read-heavy lookups on that field.

Shard routing tier

• Someone must answer "which shard owns key K": (1) shard-aware clients pulling the ring/partition map (Cassandra drivers), (2) a routing proxy (Redis Cluster MOVED redirects, mongos, Vitess vtgate), or (3) a coordination service holding the map (ZooKeeper/etcd) that routers watch.
• The map changes during rebalancing — routers cache it and must handle "wrong shard, here's the new owner" redirects gracefully.

Choosing a shard key

CAP, stated precisely

• The theorem: when a network partition (P) occurs, a distributed system must choose between consistency (every read sees the latest write — linearizability) and availability (every request to a live node gets a non-error response). You cannot have both during the partition.
• P is not a choice. Networks partition — switch failures, AZ isolation, GC pauses that look like partitions. So "CA distributed system" is not a real option; a single-node Postgres is "CA" only in the trivial sense that it isn't distributed.
• CP means: during a partition, the minority side refuses reads/writes (errors or blocks) to avoid serving stale/conflicting data. AP means: both sides keep serving and you reconcile divergence later.
• Common misconceptions to preempt: (1) CAP applies only during partitions, not in normal operation; (2) C and A are binary in the theorem but graded in real systems — most systems are "mostly-C" or "tunably-A"; (3) CAP's C is linearizability, not ACID's C; (4) systems aren't globally CP or AP — individual operations can differ (DynamoDB offers both read modes on the same table).

PACELC — the more useful framing

Partitions are rare; latency is every request. PACELC (Abadi): if Partition, choose A or C; Else, choose Latency or Consistency. The "else" clause is the daily trade-off: synchronously coordinating replicas on every write buys consistency at the price of latency — the reason you'd run Cassandra at ONE even on a healthy network.

The consistency spectrum

• Linearizability is what CAP's "C" means and what Raft/Paxos-backed reads give you; it is the model you pay for with quorum round trips.
• Session guarantees (read-your-writes, monotonic reads) are the pragmatic middle: cheap to implement, and they fix 90% of user-visible weirdness (ch11 covers mitigations).
• Tunable consistency: Cassandra lets each query pick ONE / QUORUM / ALL (with W+R > N for overlap); DynamoDB strongly-consistent reads read from the leader and cost 2× the RCUs of eventually-consistent reads and can't be served from a global table's remote regions — consistency is literally a line item on the bill.

Choosing consistency per feature

Never answer "what consistency does your system need?" with one word — decompose by feature:

• Money movement / ledger / inventory decrement: strong (linearizable or single-leader serialized). A stale read here is a double-spend.
• Social likes, view counters, analytics: eventual. Nobody sues over a like-count that's 3 seconds behind; coordination would cost more than the data is worth.
• Shopping cart / user session / drafts: causal or session consistency — a user must see their own actions in order; other users don't need to.
• Config, feature flags, leader election: strong (that's why etcd/ZooKeeper exist).

// Session consistency ("sticky after write"): route a user's reads to the
// leader for a window after their own write; otherwise use cheap replicas.
type ReadPreference = 'leader' | 'replica';

interface StickyStore {
  // e.g. Redis with TTL, or a signed lastWriteAt claim inside the session cookie
  // (cookie survives multi-node API tiers with no shared state).
  getLastWriteAt(userId: string): Promise<number | null>;
  setLastWriteAt(userId: string, epochMs: number): Promise<void>;
}

export class SessionConsistentRouter {
  constructor(
    private readonly sticky: StickyStore,
    private readonly leader: DbClient,
    private readonly replicas: DbClient[],
    private readonly stickyWindowMs = 5_000,  // > p99 replication lag, with margin
  ) {}

  /** Call after every successful write for this user. */
  async recordWrite(userId: string): Promise<void> {
    await this.sticky.setLastWriteAt(userId, Date.now());
  }

  private async prefer(userId: string): Promise<ReadPreference> {
    const last = await this.sticky.getLastWriteAt(userId);
    const recentlyWrote = last !== null && Date.now() - last < this.stickyWindowMs;
    return recentlyWrote ? 'leader' : 'replica';
  }

  /** Reads see the user's own writes; everyone else enjoys replica latency. */
  async read<T>(userId: string, q: Query<T>): Promise<T> {
    const pref = await this.prefer(userId);
    if (pref === 'leader') return this.leader.query(q);
    const replica = this.replicas[Math.floor(Math.random() * this.replicas.length)];
    return replica.query(q);
    // Hardening: fall back to leader if the chosen replica reports lag > threshold.
  }
}

SQL vs NoSQL — an access-pattern decision, not a religion

The question is never "SQL or NoSQL?"; it's four questions: (1) do you need multi-row ACID transactions and joins? (2) do you know your access patterns up front? (3) what scale of writes/data? (4) how flexible must ad-hoc querying be? Relational wins on 1 and 4; key-value/wide-column wins on 2 and 3.

Polyglot persistence

Real systems map each feature to the store whose model fits, connected by CDC/events (ch14) — one logical system, many stores, each a projection of the source of truth:

Why async at all

Synchronous request/response couples your availability, latency, and capacity to every downstream dependency. A queue breaks that coupling.

• Decouple availability: the producer succeeds even when the consumer is down, deploying, or slow. Payment API accepts the order; the email service being on fire is not the customer's problem.
• Absorb bursts: traffic is spiky (flash sale: 50k writes/s for 3 minutes), workers are steady (2k/s). The queue is the buffer; depth grows, then drains. You provision workers for the average, not the peak.
• Smooth / shape load: protect a fragile downstream (legacy core-banking API rate-limited to 100 TPS) by letting the queue meter delivery.
• Retry as a first-class citizen: failures become redeliveries instead of user-facing 500s.
• The cost: eventual consistency, duplicate deliveries, no ordering guarantees by default, and an ops surface (lag monitoring, DLQ triage) you now own.

Queue vs stream: the key mental model

This distinction drives every technology choice in this chapter. Get it wrong and you pick the wrong tool for years.

• Queue (SQS, RabbitMQ): a message is a task. Competing consumers each grab different messages; a consumed+acked message is gone. Scaling out = add workers, throughput rises linearly. Natural fit: jobs, work distribution, decoupling one producer from one logical consumer.
• Stream (Kafka, Kinesis): a message is a fact in a replayable log. Consumers track their own offset; the data stays for the retention period (hours to forever). Multiple independent consumer groups each read everything. Natural fit: event distribution, fan-out to many teams, replay, analytics, CDC.
• Litmus test: "Could a second team want to consume the same data next quarter?" or "Would I ever want to re-process history?" → stream. "This is a job to be done once by whoever's free" → queue.

Delivery semantics: at-most / at-least / "exactly" once

• At-most-once: fire and forget; ack before processing. Message may be lost, never duplicated. OK for metrics ticks, presence pings — things where a gap is cheaper than a dup.
• At-least-once: ack only after processing succeeds. Crash between processing and ack → redelivery → duplicates are guaranteed to happen eventually. This is the default of every serious system (SQS standard, Kafka default, Kinesis).
• Exactly-once: impossible as a transport guarantee across arbitrary systems (Two Generals). What everyone actually ships is effectively-once = at-least-once delivery + idempotent consumer. Dedup on a stable message/event id, or make the operation naturally idempotent (SET balance = 500 vs ADD 100).
• Kafka's "exactly-once" caveat: Kafka transactions (idempotent producer + read_committed consumers) give exactly-once within a Kafka-to-Kafka pipeline (e.g., Kafka Streams). The moment your consumer touches an external system — Postgres, Stripe, an email API — you are back to at-least-once and need idempotency there. Saying this unprompted is a staff-level signal.

Ordering & consumer groups

• Global ordering doesn't scale. One totally-ordered sequence implies one writer/one lane — a serial bottleneck. Every scalable system offers ordering only within a partition (Kafka/Kinesis) or message group (SQS FIFO).
• Partition by the entity that needs ordering: key = accountId means all events for one account are ordered relative to each other; different accounts interleave freely. That is almost always all the ordering you actually need.
• Consumer groups: Kafka assigns each partition to exactly one consumer in a group. Max parallelism = partition count — pick partition count for target throughput and future consumer scaling (repartitioning later breaks key→partition mapping). More consumers than partitions → idle consumers.
• Rebalancing: when a consumer joins/dies, partitions are reassigned. Classic "stop-the-world" (eager) rebalances pause the whole group; cooperative/incremental rebalancing (Kafka ≥2.4) moves only affected partitions. A crash-looping consumer causes rebalance storms — lag spikes with no traffic increase is the tell.
• Hot partition hazard: keying by merchantId when one merchant is 40% of traffic serializes that merchant onto one partition/consumer. Mitigate with composite keys where per-entity ordering can be relaxed.

Backpressure, DLQs & poison pills

• Backpressure = telling upstream to slow down instead of drowning. Unbounded queues turn overload into a latency disaster (a 10M-deep queue means hours of staleness) and eventually OOM/em disk-full. Bound everything.
• Slow-consumer strategies: (1) scale consumers on queue depth/age (SQS depth → auto scaling; Kafka lag → KEDA); (2) shed load — reject or sample low-value messages when depth exceeds a threshold; (3) push back — block or 429 producers; (4) degrade — skip enrichment steps under load.
• Poison pill: a message that deterministically crashes the consumer (bad schema, edge-case payload). Under at-least-once it redelivers forever, wedging the queue — for FIFO groups it blocks everything behind it.
• Dead-letter queue (DLQ): after maxReceiveCount attempts (SQS redrive policy, typically 3–5), the broker moves the message to a DLQ. Alert on DLQ depth > 0, keep payloads for forensics, and build a redrive path to replay after the bug fix (SQS has one-click DLQ redrive).

Visibility timeout (SQS) vs offset commit (Kafka)

Retries done right: exponential backoff + full jitter

Naive fixed-interval retries synchronize failing clients into thundering herds that re-kill the recovering dependency. AWS's own analysis shows full jitter (uniform random between 0 and the exponential cap) gives the best contention profile.

interface RetryPolicy {
  maxAttempts: number;   // total tries including the first
  baseDelayMs: number;   // e.g. 100
  maxDelayMs: number;    // cap, e.g. 30_000
}

/** Full-jitter exponential backoff: sleep U(0, min(cap, base * 2^attempt)). */
function backoffDelay(attempt: number, p: RetryPolicy): number {
  const exp = Math.min(p.maxDelayMs, p.baseDelayMs * 2 ** attempt);
  return Math.floor(Math.random() * exp);
}

async function withRetry<T>(
  fn: () => Promise<T>,
  p: RetryPolicy,
  isRetryable: (e: unknown) => boolean = () => true,
): Promise<T> {
  let lastErr: unknown;
  for (let attempt = 0; attempt < p.maxAttempts; attempt++) {
    try {
      return await fn();
    } catch (e) {
      lastErr = e;
      // Never retry non-retryable errors (validation, 4xx): retrying a
      // deterministic failure just burns budget and delays the DLQ.
      if (!isRetryable(e) || attempt === p.maxAttempts - 1) break;
      await new Promise((r) => setTimeout(r, backoffDelay(attempt, p)));
    }
  }
  throw lastErr;
}

A typed idempotent consumer (SQS-style)

This is the shape interviewers want on the whiteboard: envelope with a stable id, dedup store checked before work and recorded atomically with (or after) work, bounded retries, DLQ handoff.

/** Envelope every producer must wrap payloads in. messageId is the dedup key. */
interface Envelope<T> {
  messageId: string;        // UUID minted by the ORIGINAL producer (survives redelivery)
  type: string;             // e.g. "payment.captured"
  occurredAt: string;       // ISO-8601
  payload: T;
}

interface QueueMessage { receiptHandle: string; body: string; approxReceiveCount: number; }

interface QueueClient {
  receive(max: number): Promise<QueueMessage[]>;
  delete(receiptHandle: string): Promise<void>;          // ack
  sendToDlq(body: string, reason: string): Promise<void>;
}

/** Dedup store: DynamoDB with conditional put + TTL, or Redis SETNX + EXPIRE. */
interface ProcessedStore {
  /** Atomically record id; returns false if already present (duplicate). */
  markIfNew(id: string, ttlSeconds: number): Promise<boolean>;
  unmark(id: string): Promise<void>;                     // roll back claim on failure
}

class IdempotentConsumer<T> {
  constructor(
    private queue: QueueClient,
    private processed: ProcessedStore,
    private handler: (e: Envelope<T>) => Promise<void>,
    private opts = { maxReceives: 5, dedupTtlSec: 7 * 24 * 3600 },
  ) {}

  async poll(): Promise<void> {
    for (const msg of await this.queue.receive(10)) {
      let env: Envelope<T>;
      try {
        env = JSON.parse(msg.body) as Envelope<T>;
        if (!env.messageId || !env.type) throw new Error("bad envelope");
      } catch {
        // Poison pill: unparseable — retrying will never help. Straight to DLQ.
        await this.queue.sendToDlq(msg.body, "malformed");
        await this.queue.delete(msg.receiptHandle);
        continue;
      }

      // Idempotency gate: claim the id; a duplicate delivery just gets acked.
      if (!(await this.processed.markIfNew(env.messageId, this.opts.dedupTtlSec))) {
        await this.queue.delete(msg.receiptHandle);
        continue;
      }

      try {
        await withRetry(() => this.handler(env), {
          maxAttempts: 3, baseDelayMs: 200, maxDelayMs: 5_000,
        });
        await this.queue.delete(msg.receiptHandle);   // ack ONLY after success
      } catch (err) {
        await this.processed.unmark(env.messageId);   // release claim for redelivery
        if (msg.approxReceiveCount >= this.opts.maxReceives) {
          await this.queue.sendToDlq(msg.body, String(err));
          await this.queue.delete(msg.receiptHandle);
        }
        // else: do nothing — visibility timeout expires, broker redelivers.
      }
    }
  }
}

Bounded in-memory queue with backpressure

type OverflowPolicy = "block" | "reject" | "drop-oldest";

/** Bounded queue: the producer FEELS the pressure instead of hiding it. */
class BoundedQueue<T> {
  private items: T[] = [];
  private waiters: ((v: T) => void)[] = [];

  constructor(private capacity: number, private policy: OverflowPolicy) {}

  /** Returns false when rejected — caller must shed/retry/429 upstream. */
  async offer(item: T): Promise<boolean> {
    const waiter = this.waiters.shift();
    if (waiter) { waiter(item); return true; }        // hand directly to a blocked consumer
    if (this.items.length >= this.capacity) {
      switch (this.policy) {
        case "reject":      return false;             // shed load explicitly
        case "drop-oldest": this.items.shift(); break; // keep freshest (metrics-style)
        case "block":                                  // simple spin-wait for slot
          while (this.items.length >= this.capacity) await new Promise(r => setTimeout(r, 5));
      }
    }
    this.items.push(item);
    return true;
  }

  async take(): Promise<T> {
    const head = this.items.shift();
    if (head !== undefined) return head;
    return new Promise<T>((resolve) => this.waiters.push(resolve)); // park consumer
  }

  get depth(): number { return this.items.length; }   // export as a metric + alarm
}

Choosing the technology

Events vs commands vs queries

Mixing these up is the root cause of "event soup". The grammar matters:

• The tell in a design review: a topic named SendWelcomeEmail is a command wearing an event costume — the producer is secretly orchestrating the consumer. Publish UserRegistered; let the email service decide to react.
• Events are immutable facts. If a fact was wrong, you emit a compensating fact (OrderCancelled), you don't edit history.

Pub/sub and fan-out

One producer, one topic, N independent consumers — each with its own queue, retries, DLQ, and pace. The producer's blast radius stops at the topic.

Choreography vs orchestration

Same question as sagas (ch. 8), seen from the event angle: who owns the workflow?

• Staff-level answer: they compose. Orchestrate the core transactional flow (payment → reserve → ship); choreograph the periphery (emails, analytics, search indexing) off the events the flow emits.

The transactional outbox — flagship pattern

The dual-write problem

Your handler must (a) commit the order to Postgres and (b) publish OrderPlaced to Kafka. These are two systems with no shared transaction. Whatever order you pick, there's a crash window:

• Commit → crash → publish never happens: order exists, downstream never hears. Inventory never reserved. Silent data loss — the worst kind.
• Publish → crash → commit never happens: downstream reacts to an order that doesn't exist. Ghost email, ghost reservation.
• 2PC across DB and broker is the theoretical fix and a practical non-starter (Kafka/SNS aren't XA participants; and see ch. 8 on why 2PC is avoided anyway).

The fix: make the event part of the local transaction

• Write the business row and an outbox row in the same DB transaction — atomicity comes free from the database you already have.
• A relay gets events out: either a poller (SELECT unsent rows, publish, mark sent — simple, adds ~poll-interval latency) or CDC — Debezium tailing the WAL/binlog and pushing outbox rows to Kafka (lower latency, no polling load, more moving parts).
• Resulting guarantee: at-least-once publish. The relay can crash after publishing but before marking sent → duplicate publish. Consumers must be idempotent — which they must be anyway (ch. 13).

// -- Outbox schema (Postgres) -------------------------------------------
// CREATE TABLE outbox (
//   id uuid PRIMARY KEY, aggregate_type text, aggregate_id text,
//   event_type text, payload jsonb, created_at timestamptz DEFAULT now(),
//   sent_at timestamptz NULL);

interface OutboxRow {
  id: string;               // event id — consumers dedup on this
  aggregateType: "order";
  aggregateId: string;
  eventType: "OrderPlaced";
  payload: OrderPlacedEvent;
}

interface OrderPlacedEvent {
  orderId: string; customerId: string;
  totalCents: number; currency: "USD" | "EUR" | "INR";
}

/** Business write + event write in ONE transaction. This is the whole trick. */
async function placeOrderWithOutbox(db: Db, cmd: PlaceOrderCommand): Promise<string> {
  return db.transaction(async (tx) => {
    const orderId = crypto.randomUUID();
    await tx.query(
      `INSERT INTO orders (id, customer_id, total_cents, status)
       VALUES ($1, $2, $3, 'PLACED')`,
      [orderId, cmd.customerId, cmd.totalCents],
    );
    const event: OutboxRow = {
      id: crypto.randomUUID(), aggregateType: "order", aggregateId: orderId,
      eventType: "OrderPlaced",
      payload: { orderId, customerId: cmd.customerId,
                 totalCents: cmd.totalCents, currency: cmd.currency },
    };
    await tx.query(
      `INSERT INTO outbox (id, aggregate_type, aggregate_id, event_type, payload)
       VALUES ($1, $2, $3, $4, $5)`,
      [event.id, event.aggregateType, event.aggregateId, event.eventType,
       JSON.stringify(event.payload)],
    );
    return orderId; // both rows commit or neither does
  });
}

/** Relay: poll unsent rows, publish, mark sent. Run as a single instance or
 *  use FOR UPDATE SKIP LOCKED so multiple relays don't double-claim rows. */
async function relayLoop(db: Db, bus: EventBus): Promise<never> {
  for (;;) {
    const rows = await db.query<OutboxRow>(
      `SELECT * FROM outbox WHERE sent_at IS NULL
       ORDER BY created_at LIMIT 100 FOR UPDATE SKIP LOCKED`,
    );
    for (const row of rows) {
      // Publish keyed by aggregateId so per-order ordering holds downstream.
      await bus.publish(row.eventType, row.aggregateId, row.payload);
      await db.query(`UPDATE outbox SET sent_at = now() WHERE id = $1`, [row.id]);
      // Crash between publish and UPDATE => duplicate publish on restart.
      // That's fine: delivery is at-least-once; consumers dedup on row.id.
    }
    if (rows.length === 0) await sleep(200);
  }
}

Idempotent consumers close the loop

• Every event carries the outbox row's id. Consumers keep a processed_events(event_id PRIMARY KEY) table and insert into it in the same transaction as their side effect — duplicate arrives, insert conflicts, transaction no-ops. This is effectively-once end to end, built from two local transactions and at-least-once glue.

Event sourcing

Instead of storing current state and emitting events as a byproduct, make the event log the source of truth: state = fold(apply, initial, events). Current state is a cache you can throw away and rebuild.

• Wins: perfect audit trail (regulators love it — this is why ledgers are the canonical fintech example), temporal queries ("what was the balance on March 3?"), rebuildable projections, natural fit with CQRS.
• Costs: schema evolution forever (you must be able to read year-old events), replay time (mitigated by snapshots: persist state every N events, replay only the tail), eventual consistency of read models, unfamiliar mental model for most teams.
• When NOT to event-source: CRUD apps with no audit need, teams new to the pattern, anywhere you'd immediately ask "can I just update the row?" It's a per-aggregate decision — event-source the ledger, CRUD the user-profile table.

/** Discriminated union of everything that can happen to an account. */
type AccountEvent =
  | { type: "AccountOpened";  id: string; ownerId: string; at: string }
  | { type: "MoneyDeposited"; id: string; amountCents: number; at: string }
  | { type: "MoneyWithdrawn"; id: string; amountCents: number; at: string }
  | { type: "AccountFrozen";  id: string; reason: string; at: string };

interface AccountState {
  ownerId: string; balanceCents: number; frozen: boolean; version: number;
}

/** Pure transition function: no I/O, trivially testable. */
function apply(state: AccountState, event: AccountEvent): AccountState {
  switch (event.type) {
    case "AccountOpened":
      return { ownerId: event.ownerId, balanceCents: 0, frozen: false, version: 1 };
    case "MoneyDeposited":
      return { ...state, balanceCents: state.balanceCents + event.amountCents,
               version: state.version + 1 };
    case "MoneyWithdrawn":
      return { ...state, balanceCents: state.balanceCents - event.amountCents,
               version: state.version + 1 };
    case "AccountFrozen":
      return { ...state, frozen: true, version: state.version + 1 };
  }
}

/** State is a fold over history. Snapshot = persisted (state, version);
 *  rehydrate = load snapshot, replay only events with version > snapshot. */
function rehydrate(events: AccountEvent[], snapshot?: AccountState): AccountState {
  return events.reduce(apply, snapshot ?? ({} as AccountState));
}

// Command handling = validate against current state, append new fact:
function withdraw(state: AccountState, amountCents: number): AccountEvent {
  if (state.frozen) throw new Error("account frozen");
  if (state.balanceCents < amountCents) throw new Error("insufficient funds");
  return { type: "MoneyWithdrawn", id: crypto.randomUUID(),
           amountCents, at: new Date().toISOString() };
  // Append with optimistic concurrency: expectedVersion = state.version,
  // so two concurrent withdrawals can't both pass the balance check.
}

CQRS: separate write and read models

• Write side: normalized, validated, optimized for commands and invariants. Read side: denormalized projections shaped exactly like each screen/query — a DynamoDB table per access pattern, an Elasticsearch index for search, a Redis leaderboard.
• Projections are built by consuming the event stream. They are eventually consistent — typically tens of milliseconds behind, occasionally seconds. Design the UX for it (optimistic UI, read-your-writes via routing the author's read to the write side or a version-check).
• Killer feature: rebuild a projection from scratch by replaying events into a fresh table, flip a pointer, delete the old one. Bugs in read models become non-events.
• CQRS doesn't require event sourcing (CDC from a normal DB works), but they pair naturally.

Event schema evolution & event styles

• Additive-only is the golden rule: add optional fields, never rename/remove/retype. Consumers must tolerate unknown fields.
• Version the schema (schemaVersion in the envelope). For breaking changes: upcasters — pure functions v1→v2→v3 applied at read time so handlers only ever see the latest shape. Essential for event sourcing, where 2019 events must still parse in 2026.
• Schema registry (Confluent, AWS Glue, EventBridge Schema Registry) enforces compatibility at publish time — a broken producer fails CI, not production consumers at 3am.
• Notification events vs event-carried state transfer (ECST): a thin OrderPlaced {orderId} forces every consumer to call back to the order service (re-coupling, thundering reads); a fat event carrying the full order snapshot lets consumers act autonomously at the cost of size and staleness. Default: carry what most consumers need, keep a fetch path for the rest.

The five delivery mechanisms

• Rule of thumb: server→browser one-way → SSE; bidirectional → WebSocket; server→another company's backend → webhook; nothing else justified → polling (never be embarrassed to propose polling for a 30s-freshness requirement — it's the cheapest thing that works).
• Long polling mechanics: client sends GET, server holds the request until data exists or a ~25s timeout, responds, client immediately re-polls. It's polling where the wait happens server-side — latency of push, statelessness of HTTP. Keep the hold below LB idle timeout.

Flagship: a production webhook system (provider side)

You're Stripe/GitHub/OrbitX: customers register URLs; you must deliver events to thousands of endpoints you don't control — some slow, some down, some hostile. The design pillars:

1. Registration & verification

• Customer registers an HTTPS URL + chosen event types. Verify ownership before sending real data: send a challenge (random token the endpoint must echo back) or require a verification handshake — prevents using your webhook system to spray traffic at arbitrary URLs (SSRF-by-proxy).
• Issue a per-endpoint signing secret at registration; support secret rotation with an overlap window (sign with both, consumer accepts either).
• Guard your egress: resolve and reject private/link-local IPs (169.254.x, 10.x, metadata endpoints), and re-check on redirect — classic SSRF hole.

2. Signing: HMAC-SHA256 + timestamp

• Consumers must be able to prove the POST came from you and isn't a replay. Sign timestamp + "." + body with the shared secret; consumer recomputes, compares in constant time, and rejects stale timestamps (>5 min) to kill replay attacks. This is exactly Stripe's Stripe-Signature scheme.

import { createHmac, timingSafeEqual } from "node:crypto";

interface WebhookHeaders { "x-webhook-timestamp": string; "x-webhook-signature": string; }

/** PROVIDER: sign timestamp.body so neither can be tampered independently. */
function signWebhook(secret: string, rawBody: string, nowMs = Date.now()): WebhookHeaders {
  const ts = Math.floor(nowMs / 1000).toString();
  const sig = createHmac("sha256", secret).update(`${ts}.${rawBody}`).digest("hex");
  return { "x-webhook-timestamp": ts, "x-webhook-signature": `v1=${sig}` };
}

/** CONSUMER: verify over the RAW body (before any JSON parsing/re-serialization —
 *  re-stringified JSON reorders keys and breaks the HMAC). */
function verifyWebhook(
  secret: string, rawBody: string, headers: WebhookHeaders,
  toleranceSec = 300,
): boolean {
  const ts = parseInt(headers["x-webhook-timestamp"], 10);
  if (!Number.isFinite(ts)) return false;
  // Replay defense: reject anything outside the tolerance window.
  if (Math.abs(Date.now() / 1000 - ts) > toleranceSec) return false;

  const expected = createHmac("sha256", secret)
    .update(`${ts}.${rawBody}`).digest();
  const given = Buffer.from(
    headers["x-webhook-signature"].replace(/^v1=/, ""), "hex");
  // timingSafeEqual: a byte-by-byte === leaks match length via timing.
  return given.length === expected.length && timingSafeEqual(given, expected);
}

3. Delivery pipeline: fan-out via queue, never inline

• Never POST to customer endpoints from your request path — a slow endpoint would block your API. Events flow: business event → outbox (ch. 14) → per-delivery jobs on a queue → dispatcher worker pool → customer endpoints.
• One delivery job per (event, endpoint) so one dead endpoint retries independently without delaying anyone else. Isolate slow endpoints (per-endpoint concurrency caps or sharded queues) so a customer with a 30s handler can't consume your worker pool.
• Success = any 2xx within a short timeout (5–10s). 3xx/4xx/5xx/timeout = failure → schedule retry.

4. Retry policy & giving up

• Exponential backoff + jitter across attempts spanning days (Stripe retries for up to 3 days). After exhaustion: dead-letter the delivery, surface it in a customer-facing dashboard with payloads, response codes, and a manual "resend" button, alert the customer, and auto-disable endpoints failing for N days (with notice).
• Ordering is explicitly NOT guaranteed: retries and parallel dispatchers reorder events (a retried invoice.created can land after invoice.paid). Contract: each event carries an id and the object's latest state/sequence; consumers reconcile — treat webhooks as "something changed, here's a snapshot", and fetch-on-receipt when in doubt.

5. Consumer-side contract (put this in your docs)

• Verify the signature over the raw body; reject stale timestamps.
• Return 200 immediately, process async — enqueue the event and ack in <1s. Slow handlers cause provider timeouts, which cause retries, which cause duplicates.
• Idempotency by event id: at-least-once delivery means duplicates will arrive (ch. 13's processed-ids table).
• Don't trust event payload for money decisions — verify by fetching the object from the provider's API (payload could be stale even when authentic).

WebSockets at scale: connection state is the problem

HTTP servers are stateless and interchangeable; WebSocket servers are neither. User 42's socket lives in the memory of exactly one node — so "send a message to user 42" becomes a routing problem.

• The core pattern: a connection registry (Redis: userId → set of (nodeId, socketId)) plus pub/sub between nodes. Any node that needs to reach user 42 publishes to a channel; the node actually holding the socket delivers. Alternative: every node subscribes to a broadcast channel and filters locally (simpler, fine until fan-in cost bites).
• Sticky vs registry: LB stickiness (by cookie/IP) only ensures reconnects land on the same node — it does not solve server-initiated sends to a user connected "somewhere". You need the registry/pub-sub regardless.
• Heartbeats: ping/pong every 15–30s. Detects half-open connections (mobile network drops without FIN) and keeps NAT/LB idle timeouts from severing quiet sockets.
• Reconnect protocol: client reconnects with exponential backoff + jitter (a region blip must not become a reconnect stampede), then resyncs — either resume from a last-seen sequence number or refetch state via REST. Design the resync path first; it's the hard part.
• Backpressure on slow clients: a phone on 2G can't drain your sends; the per-socket buffer grows and OOMs the node. Watch bufferedAmount/socket high-water marks: coalesce updates (send latest state, not every tick), drop intermediate frames, or disconnect with a resume token.
• Scale math: a tuned node handles ~100k–1M mostly-idle connections (memory-bound: ~10–30KB each with TLS). 10M connections ≈ 20–100 nodes — the hard part isn't sockets, it's deploys (every deploy disconnects everyone → drain gradually + jittered reconnect), registry cleanup on node death (TTL entries), and fan-out amplification (1 celebrity message → 1M deliveries).

interface PubSub {
  publish(channel: string, msg: string): Promise<void>;
  subscribe(channel: string, onMsg: (msg: string) => void): Promise<void>;
}
interface Registry {                       // Redis hash/sets with TTL refresh
  register(userId: string, nodeId: string, socketId: string): Promise<void>;
  unregister(userId: string, socketId: string): Promise<void>;
  nodesFor(userId: string): Promise<string[]>;
}

/** Runs on EVERY WS node: local sockets + cross-node routing via pub/sub. */
class ConnectionManager {
  private local = new Map<string, Set<WebSocketLike>>(); // userId → sockets on THIS node

  constructor(private nodeId: string, private registry: Registry, private bus: PubSub) {}

  async start(): Promise<void> {
    // Each node listens on its own channel; peers publish here to reach our sockets.
    await this.bus.subscribe(`node:${this.nodeId}`, (raw) => {
      const { userId, payload } = JSON.parse(raw) as { userId: string; payload: string };
      this.deliverLocal(userId, payload);
    });
  }

  async onConnect(userId: string, ws: WebSocketLike): Promise<void> {
    (this.local.get(userId) ?? this.local.set(userId, new Set()).get(userId)!).add(ws);
    await this.registry.register(userId, this.nodeId, ws.id); // with TTL; heartbeat refreshes
    ws.onClose(() => this.onDisconnect(userId, ws));
  }

  private async onDisconnect(userId: string, ws: WebSocketLike): Promise<void> {
    this.local.get(userId)?.delete(ws);
    await this.registry.unregister(userId, ws.id); // TTL also reaps entries if node dies
  }

  /** Send to a user wherever they're connected — possibly multiple devices/nodes. */
  async sendToUser(userId: string, payload: string): Promise<void> {
    this.deliverLocal(userId, payload);                     // fast path: same node
    for (const nodeId of await this.registry.nodesFor(userId)) {
      if (nodeId !== this.nodeId) {
        await this.bus.publish(`node:${nodeId}`, JSON.stringify({ userId, payload }));
      }
    }
  }

  private deliverLocal(userId: string, payload: string): void {
    for (const ws of this.local.get(userId) ?? []) {
      // Backpressure: don't buffer unbounded for a slow client.
      if (ws.bufferedAmount > 1_000_000) { ws.close(1013, "slow consumer"); continue; }
      ws.send(payload);
    }
  }
}

Server-Sent Events: the underrated one

• One long-lived HTTP response, Content-Type: text/event-stream, messages framed as data: lines. Works through ordinary HTTP infra, no Upgrade dance.
• Built-in resilience the browser gives you free: EventSource auto-reconnects and sends Last-Event-ID — set id: on each event and keep a short replay buffer server-side, and clients survive blips without missing events. With WebSockets you hand-build all of that.
• Limits: server→client only; on HTTP/1.1 browsers cap ~6 connections per origin (HTTP/2 multiplexing fixes it); text-only. This is what LLM token streaming (including the OpenAI/Anthropic APIs) uses.

import type { Request, Response } from "express";

interface SseEvent { id: string; type: string; data: unknown; }
const replayBuffer: SseEvent[] = [];        // ring buffer of recent events

export function sseHandler(req: Request, res: Response): void {
  res.writeHead(200, {
    "Content-Type": "text/event-stream",
    "Cache-Control": "no-cache",
    Connection: "keep-alive",
    "X-Accel-Buffering": "no",              // tell nginx not to buffer the stream
  });

  // Resume: browser sent Last-Event-ID after auto-reconnect — replay the gap.
  const lastId = req.header("Last-Event-ID");
  const start = lastId ? replayBuffer.findIndex((e) => e.id === lastId) + 1 : 0;
  for (const e of replayBuffer.slice(start)) write(res, e);

  const onEvent = (e: SseEvent) => write(res, e);
  eventEmitter.on("event", onEvent);
  const heartbeat = setInterval(() => res.write(": ping\n\n"), 15_000); // comment frame
  req.on("close", () => { clearInterval(heartbeat); eventEmitter.off("event", onEvent); });
}

function write(res: Response, e: SseEvent): void {
  // id line powers Last-Event-ID resume; event line routes to addEventListener(type).
  res.write(`id: ${e.id}\nevent: ${e.type}\ndata: ${JSON.stringify(e.data)}\n\n`);
}

Choosing

The core trade: exactness for memory

Exact answers to "have I seen this before?" or "how many distinct things?" require storing every element (a hash set) — O(n) memory. Probabilistic structures answer the same questions in constant or near-constant memory by accepting a small, one-sided error:

• Bounded memory: you choose the size up front (e.g. 1.2 GB for 1B URLs at 1% error vs ~40+ GB for an exact set).
• O(1) or O(log) operations: insert and query touch k cells, independent of n.
• One-sided error: the structure can be wrong in exactly one direction, and you design the system so that direction is cheap to recover from (e.g. a false positive just costs an extra disk read; a false negative would silently lose data — so the structure never produces one).
• Tunable: the error rate is a knob. More memory → less error, with a closed-form formula.

Bloom filter deep dive

A Bloom filter is a bit array of m bits plus k independent hash functions.

• Insert(x): compute k hashes of x, each mapping to a position in [0, m). Set all k bits to 1.
• Query(x): compute the same k positions. If any bit is 0 → x was definitely never inserted. If all k bits are 1 → x was probably inserted (or you got unlucky and other elements happened to set those bits — a false positive).
• No false negatives, ever: inserting x sets its bits, and bits are never cleared, so a later query on x always finds them set.
• No deletes: clearing a bit might also clear it for other elements that share it — that would create false negatives. Deletion requires a counting variant (below) or rebuilding the filter.
• No enumeration: you can't list members; the filter only answers membership queries. It also can't store values — pair it with the real store.

The math you should know cold

With m bits, k hashes, n inserted elements, the false-positive probability is approximately:

• p ≈ (1 − e^−kn/m)^k — the chance all k probed bits are already set.
• Optimal k = (m/n) · ln 2 ≈ 0.693 · m/n — balances "more hashes catch more zeros" against "more hashes fill the array faster".
• Required m = −n · ln p / (ln 2)² ≈ 1.44 · n · log₂(1/p) — memory needed for n elements at target error p.
• Rule of thumb: ~10 bits (1.25 bytes) per element buys 1% error, regardless of element size. A 200-byte URL costs the same 10 bits as a 16-byte hash.

TypeScript implementation

Production trick: you don't need k independent hash functions. Double hashing — g_i(x) = h1(x) + i·h2(x) mod m — gives k derived hashes from two base hashes with no measurable loss in accuracy (Kirsch & Mitzenmacher, 2006).

/** Bloom filter over strings: Uint8Array bit ops + double hashing. */
class BloomFilter {
  private readonly bits: Uint8Array;
  private readonly m: number;      // total bits
  private readonly k: number;      // number of hash probes
  private count = 0;               // inserts seen (for fpr estimate)

  /** Size for expected n elements at target false-positive rate p. */
  constructor(expectedItems: number, targetFpr: number) {
    const ln2 = Math.LN2;
    this.m = Math.ceil((-expectedItems * Math.log(targetFpr)) / (ln2 * ln2));
    this.k = Math.max(1, Math.round((this.m / expectedItems) * ln2));
    this.bits = new Uint8Array(Math.ceil(this.m / 8)); // zero-filled
  }

  /** FNV-1a — fast, decent-distribution 32-bit hash; seed decorrelates h1/h2. */
  private fnv1a(value: string, seed: number): number {
    let h = (0x811c9dc5 ^ seed) >>> 0;
    for (let i = 0; i < value.length; i++) {
      h ^= value.charCodeAt(i);
      h = Math.imul(h, 0x01000193) >>> 0; // 32-bit FNV prime multiply
    }
    return h >>> 0;
  }

  /** Double hashing: probe i is (h1 + i*h2) mod m — k probes, 2 hashes. */
  private probes(value: string): number[] {
    const h1 = this.fnv1a(value, 0x12345678);
    let h2 = this.fnv1a(value, 0x9e3779b9);
    if (h2 % this.m === 0) h2 += 1;  // h2 must not be ≡ 0 mod m
    const out: number[] = new Array(this.k);
    for (let i = 0; i < this.k; i++) {
      out[i] = (h1 + i * h2) % this.m;
    }
    return out;
  }

  add(value: string): void {
    for (const p of this.probes(value)) {
      this.bits[p >>> 3] |= 1 << (p & 7);  // set bit p
    }
    this.count++;
  }

  /** false → definitely absent. true → probably present. */
  has(value: string): boolean {
    for (const p of this.probes(value)) {
      if ((this.bits[p >>> 3] & (1 << (p & 7))) === 0) return false;
    }
    return true;
  }

  /** Current estimated false-positive rate: (1 - e^(-kn/m))^k. */
  estimatedFpr(): number {
    return Math.pow(1 - Math.exp((-this.k * this.count) / this.m), this.k);
  }
}

// 1M usernames at 1% error → ~1.2 MB instead of tens of MB for a Set
const taken = new BloomFilter(1_000_000, 0.01);
taken.add("sedhu");
taken.has("sedhu");        // true (guaranteed — no false negatives)
taken.has("free_handle");  // false → skip the DB entirely

Where Bloom filters live in real systems

• LSM engines (RocksDB, Cassandra, HBase, LevelDB): each SSTable carries a Bloom filter; a point read consults filters to skip SSTables that definitely don't contain the key — turning a read that could touch 10 files into 1–2 disk seeks. This is why LSM reads are viable at all (ch09).
• CDN "cache on second hit": Akamai found ~75% of objects are requested exactly once ("one-hit wonders"). A Bloom filter of recently-seen URLs gates cache admission — only cache on the second sighting, saving huge amounts of cache churn and disk writes.
• "Username taken" pre-check: answer most availability checks from memory; only hit the DB when the filter says "maybe taken".
• Web crawler visited-set: 10B URLs at 0.1% error ≈ 15 GB of filter vs terabytes for exact storage; a false positive just means one page is mistakenly skipped.
• Safe-browsing / malware URL lists: browsers historically shipped a Bloom-style local filter of bad URLs; "maybe bad" triggers a call to the full server-side list.
• Distributed joins & caches: ship a Bloom filter of join keys to avoid moving non-matching rows; check "does the cache cluster even have this?" before a network hop.

Variants: counting Bloom filters and cuckoo filters

• Counting Bloom filter: replace each bit with a small counter (4 bits is standard — overflow probability is negligible). Insert increments k counters, delete decrements them, query checks all k are > 0. Costs 4× the memory of a plain filter but supports deletion — this is what Cassandra-style compaction-friendly designs and some caches use when membership churns.
• Cuckoo filter: stores short fingerprints in a cuckoo hash table (two candidate buckets per item, evict-and-relocate on collision). Supports deletion natively, gives better space efficiency than Bloom below ~3% target error, and has better cache locality (2 memory accesses vs k). The trade: inserts can fail when the table nears capacity (~95% load), so you must size it honestly. Increasingly the modern default where deletes matter.

Count-Min Sketch — "how many times have I seen X?"

A Bloom filter answers membership; a Count-Min Sketch (CMS) answers frequency. It's a d×w matrix of counters with one hash per row. Increment: bump one counter per row. Query: take the minimum across rows — collisions only inflate counters, so the estimate overestimates, never underestimates. Error is bounded: with w = ⌈e/ε⌉ and d = ⌈ln(1/δ)⌉, the estimate exceeds the truth by more than ε·N with probability at most δ.

/** Count-Min Sketch: fixed memory frequency estimation (overestimates only). */
class CountMinSketch {
  private readonly table: Uint32Array;   // d rows × w cols, flattened
  private readonly w: number;
  private readonly d: number;
  private readonly seeds: number[];

  /** epsilon: additive error factor (as fraction of total count N).
   *  delta: probability the error bound is exceeded. */
  constructor(epsilon = 0.001, delta = 0.01) {
    this.w = Math.ceil(Math.E / epsilon);        // e.g. 2719 cols
    this.d = Math.ceil(Math.log(1 / delta));     // e.g. 5 rows
    this.table = new Uint32Array(this.w * this.d);
    this.seeds = Array.from({ length: this.d }, (_, i) => 0x9e3779b9 * (i + 1));
  }

  private hash(value: string, seed: number): number {
    let h = seed >>> 0;
    for (let i = 0; i < value.length; i++) {
      h = (Math.imul(h ^ value.charCodeAt(i), 0x01000193)) >>> 0;
    }
    return h % this.w;
  }

  increment(value: string, by = 1): void {
    for (let row = 0; row < this.d; row++) {
      this.table[row * this.w + this.hash(value, this.seeds[row])] += by;
    }
  }

  /** Estimated count: true count ≤ estimate ≤ true + ε·N (w.p. 1−δ). */
  estimate(value: string): number {
    let min = Number.MAX_SAFE_INTEGER;
    for (let row = 0; row < this.d; row++) {
      min = Math.min(min, this.table[row * this.w + this.hash(value, this.seeds[row])]);
    }
    return min;
  }
}

// Trending hashtags: ~54 KB of counters instead of a per-tag hashmap
const trends = new CountMinSketch(0.0005, 0.01);
trends.increment("#superbowl");
trends.estimate("#superbowl"); // ≥ true count, tightly bounded above

• Heavy hitters / trending topics: pair a CMS with a small min-heap of the current top-K — the classic "trending hashtags" design.
• Rate-abuse detection: "has this IP made > 10k requests this hour?" across millions of IPs in fixed memory (overestimation errs on the side of throttling — usually acceptable).
• Why min works: every row's counter ≥ truth (collisions only add); the least-collided row is the best estimate.

HyperLogLog — "how many DISTINCT things?"

Counting distinct elements exactly requires remembering every element. HyperLogLog (HLL) estimates cardinality at planet scale in ~12 KB with a standard error of ~0.81% (Redis's 16384-register implementation), whether you're counting 100 or 10 billion distinct items.

• Intuition — the coin-flip trick: hash each element to a uniform random bitstring. Track the longest run of leading zeros seen. Seeing "0001…" is like flipping 3 tails then a head — a 1-in-16 event. If your record run of leading zeros is r, you've probably seen ~2^r distinct elements — just as the longest run of heads at a casino table tells you roughly how many rounds were played. Duplicates hash identically, so they never move the record — that's why it counts distinct elements for free.
• Making it stable: one max is horribly noisy (a single lucky hash doubles the estimate). HLL splits the hash space into m=16384 buckets using the first 14 bits, keeps the max-leading-zero count per bucket, and combines them with a harmonic mean plus bias correction. Standard error = 1.04/√m ≈ 0.81%.
• Mergeable: union two HLLs by taking the per-bucket max — so each app server keeps a local HLL and you merge for the global count. This is the killer feature for distributed analytics.
• Redis: PFADD visitors:2026-07-16 user123, PFCOUNT visitors:2026-07-16, PFMERGE weekly d1 d2 ... d7 — daily unique visitors, mergeable into weekly/monthly, 12 KB per key regardless of traffic.
• Also in: BigQuery/Athena approx_distinct, Druid, Presto, Elasticsearch cardinality aggregation.

Quantile sketches (t-digest) in one paragraph

"What's our p99 latency?" cannot be answered by averaging averages, and storing every sample is prohibitive at millions of requests/sec. t-digest (Dunning) and DDSketch keep a compressed set of centroids — dense near the extreme quantiles where precision matters, sparse in the middle — so you get accurate p99/p999 in a few KB, and sketches from many hosts merge correctly. This is what backs Datadog/Prometheus-style percentile aggregation and Elasticsearch percentile aggregations; the staff-level point is that percentiles don't compose but sketches do — never average per-host p99s (ch on observability).

Choosing the right sketch

Why limit at all

• Protection from abuse: credential stuffing, scraping, OTP/SMS pumping (a fintech classic — attackers burn your SMS budget), brute-force on login and card-check endpoints.
• Protection from yourself: a buggy client loop, a mobile release that retries without backoff, or a batch job hammering an internal API can take you down as effectively as any attacker. Most "DDoS" incidents are self-inflicted.
• Fairness: one tenant's burst shouldn't starve everyone else — per-key limits are how multi-tenant SaaS keeps noisy neighbors contained.
• Cost control: every request has a marginal cost (DynamoDB RCUs, LLM tokens, SMS). Limits turn "unbounded liability" into "bounded budget", and paid tiers into enforceable products (usage plans).
• Downstream contracts: your payment processor allows 100 TPS; a limiter in front of that call is how you stay a good citizen and avoid being limited by them.

Where to enforce: defense in depth

Enforce at multiple layers, cheapest first. Each layer stops a different class of traffic and protects the layers behind it.

• Edge (WAF/CDN): coarse per-IP rules, blocks volumetric floods before they cost you compute.
• Gateway: per-API-key limits — this is where "your plan allows 1,000 req/min" lives.
• Per-service: business-aware limits the gateway can't express — "3 OTP sends per phone number per hour", "5 failed logins per account". These need app context and usually shared state (Redis).

The four algorithms

1. Token bucket — the industry default

A bucket holds up to capacity tokens, refilled at refillRate tokens/sec. Each request takes a token; no token → rejected. Allows bursts up to capacity while enforcing the average rate.

interface RateLimitDecision {
  allowed: boolean;
  remaining: number;       // tokens left after this decision
  retryAfterMs?: number;   // only set when rejected
}

/** Token bucket: burst up to `capacity`, sustained `refillRate`/sec.
 *  Lazy refill: no timers — tokens are computed on access. */
class TokenBucket {
  private tokens: number;
  private lastRefillMs: number;

  constructor(
    private readonly capacity: number,      // burst size
    private readonly refillRate: number,    // tokens per second
    now: number = Date.now(),
  ) {
    this.tokens = capacity;                 // start full
    this.lastRefillMs = now;
  }

  tryConsume(cost = 1, now: number = Date.now()): RateLimitDecision {
    // Refill based on elapsed time, capped at capacity.
    const elapsedSec = (now - this.lastRefillMs) / 1000;
    this.tokens = Math.min(this.capacity, this.tokens + elapsedSec * this.refillRate);
    this.lastRefillMs = now;

    if (this.tokens >= cost) {
      this.tokens -= cost;
      return { allowed: true, remaining: Math.floor(this.tokens) };
    }
    // How long until enough tokens accumulate?
    const deficit = cost - this.tokens;
    return {
      allowed: false,
      remaining: 0,
      retryAfterMs: Math.ceil((deficit / this.refillRate) * 1000),
    };
  }
}

2. Leaky bucket — smoothing the output

Requests enter a FIFO queue (the bucket); a drain processes them at a fixed rate. Overflow is rejected. Where token bucket shapes admission and permits bursts, leaky bucket shapes output — downstream sees a perfectly smooth stream. Use it when the protected resource genuinely can't absorb bursts (a partner API with strict TPS, a hardware writer). Cost: queued requests add latency, and the queue itself needs bounding. In practice most "leaky bucket" implementations (e.g. nginx limit_req) are the meter variant, equivalent to a token bucket with burst=queue size.

3. Fixed window — simple but flawed at the boundary

Count requests per discrete window (e.g. per minute, keyed user:42:12:05); reject above the limit; counter resets each window. One counter, trivially cheap — but it has the boundary-burst flaw:

• Limit: 100/min. A client sends 100 requests at 12:00:59 and 100 more at 12:01:01.
• Both windows are individually legal — but that's 200 requests in 2 seconds, twice the intended rate, concentrated exactly where it hurts.
• Adversaries find window edges deliberately; well-behaved cron jobs find them accidentally (everything fires at :00).

4. Sliding window log & sliding window counter

Sliding log: store a timestamp per request (Redis ZSET), drop entries older than the window, count what remains. Perfectly accurate, but O(limit) memory per key — expensive at high limits. Sliding window counter keeps fixed-window cheapness while killing the boundary burst: keep counts for the current and previous windows and weight the previous one by how much of it still overlaps the sliding window. Cloudflare runs this at edge scale; their analysis showed ~0.003% of requests wrongly allowed/blocked — a fine trade for two counters per key.

/** Sliding window counter: prev-window count weighted by overlap.
 *  estimate = prevCount * overlapFraction + currCount */
class SlidingWindowCounter {
  private windows = new Map<number, number>(); // windowStartMs -> count

  constructor(
    private readonly limit: number,
    private readonly windowMs: number,
  ) {}

  tryConsume(now: number = Date.now()): boolean {
    const currStart = Math.floor(now / this.windowMs) * this.windowMs;
    const prevStart = currStart - this.windowMs;

    const currCount = this.windows.get(currStart) ?? 0;
    const prevCount = this.windows.get(prevStart) ?? 0;

    // Fraction of the previous window still inside the sliding window.
    const overlap = 1 - (now - currStart) / this.windowMs;   // 1 → 0 across the window
    const estimated = prevCount * overlap + currCount;

    if (estimated >= this.limit) return false;

    this.windows.set(currStart, currCount + 1);
    // GC: anything older than prev is dead weight.
    for (const key of this.windows.keys()) {
      if (key < prevStart) this.windows.delete(key);
    }
    return true;
  }
}

Distributed rate limiting

The race problem: with N gateway instances behind a load balancer, per-instance limiters multiply the effective limit by N, and naive shared counters race: two instances read count=99 (limit 100), both allow, both write — 101 requests slipped through. Multiply by concurrency and the limit is fiction. Options:

• Centralized atomic check (Redis + Lua): the read-compute-write of the bucket runs as one atomic script server-side. Single round trip, no race. This is the standard answer.
• Local cache + async sync: each node limits locally against its share and periodically reconciles with a global counter. Cheap and Redis-outage-tolerant, but temporarily inaccurate (can overshoot between syncs). Fine for cost-protection limits; wrong for security limits.
• Sticky routing: consistently hash the rate-limit key (user id) to one node, which owns that key's bucket in memory — no shared store at all. Trade: rebalancing on node failure resets buckets, and hot keys hot-spot a node. This is how some in-memory gateway clusters (and Envoy's per-host mode) work.

-- ratelimit.lua: atomic token bucket in Redis.
-- KEYS[1] = bucket key
-- ARGV: capacity, refill_rate (tokens/sec), now_ms, cost
local capacity = tonumber(ARGV[1])
local rate     = tonumber(ARGV[2])
local now_ms   = tonumber(ARGV[3])
local cost     = tonumber(ARGV[4])

local state  = redis.call('HMGET', KEYS[1], 'tokens', 'ts')
local tokens = tonumber(state[1]) or capacity
local ts     = tonumber(state[2]) or now_ms

-- lazy refill, capped at capacity
tokens = math.min(capacity, tokens + ((now_ms - ts) / 1000) * rate)

local allowed = 0
if tokens >= cost then
  tokens = tokens - cost
  allowed = 1
end

redis.call('HMSET', KEYS[1], 'tokens', tokens, 'ts', now_ms)
-- expire idle buckets: time to fully refill, plus slack
redis.call('PEXPIRE', KEYS[1], math.ceil(capacity / rate * 1000) * 2)
return { allowed, math.floor(tokens) }

import { createClient } from 'redis';

interface DistributedLimitResult { allowed: boolean; remaining: number; }

class RedisTokenBucket {
  private scriptSha: string | null = null;

  constructor(
    private readonly redis: ReturnType<typeof createClient>,
    private readonly capacity: number,
    private readonly refillRate: number,
    private readonly luaScript: string,          // contents of ratelimit.lua
  ) {}

  async tryConsume(key: string, cost = 1): Promise<DistributedLimitResult> {
    // Load once, then EVALSHA — avoids resending the script every call.
    this.scriptSha ??= await this.redis.scriptLoad(this.luaScript);
    try {
      const [allowed, remaining] = (await this.redis.evalSha(this.scriptSha, {
        keys: [`rl:${key}`],
        arguments: [
          String(this.capacity), String(this.refillRate),
          String(Date.now()), String(cost),
        ],
      })) as [number, number];
      return { allowed: allowed === 1, remaining };
    } catch (err) {
      // FAIL-OPEN policy: Redis down must not take the API down.
      // Flip to fail-closed for abuse-sensitive keys (login, OTP).
      console.error('rate limiter unavailable, failing open', err);
      return { allowed: true, remaining: 0 };
    }
  }
}

The HTTP contract: what to return

import type { Request, Response, NextFunction } from 'express';

type KeyExtractor = (req: Request) => string;

/** Express middleware wiring the distributed token bucket + proper headers. */
function rateLimit(limiter: RedisTokenBucket, keyOf: KeyExtractor, limit: number) {
  return async (req: Request, res: Response, next: NextFunction) => {
    const key = keyOf(req);                       // user id > API key > IP
    const result = await limiter.tryConsume(key);

    res.setHeader('X-RateLimit-Limit', String(limit));
    res.setHeader('X-RateLimit-Remaining', String(Math.max(0, result.remaining)));

    if (!result.allowed) {
      const retryAfterSec = 1; // or derive from bucket deficit
      res.setHeader('Retry-After', String(retryAfterSec));
      res.setHeader('X-RateLimit-Reset', String(Math.ceil(Date.now() / 1000) + retryAfterSec));
      return res.status(429).json({
        error: 'rate_limited',
        message: 'Too many requests. Retry after the indicated delay.',
      });
    }
    next();
  };
}

// Dimension by the most specific stable identity available:
const keyOf: KeyExtractor = (req) =>
  (req as any).user?.id ?? req.header('x-api-key') ?? req.ip ?? 'anon';
// app.use('/api', rateLimit(limiter, keyOf, 100));

Dimensioning: per-user vs per-IP vs per-API-key

• Per-user (authenticated id): the fairest and most precise — use it wherever auth exists. Immune to NAT problems.
• Per-API-key: the product/billing dimension — maps to plans and tiers; a customer's many servers share one quota.
• Per-IP: the only option pre-auth (login, signup, OTP) — but corporate NATs and CGNAT put thousands of users behind one IP (limit too low → collateral damage) while attackers rotate IPs (limit too high → useless). Use it as a coarse outer ring, combined with per-account and per-phone-number limits inside.
• Compound keys for abuse: fintech OTP endpoints want per-phone AND per-IP AND per-device limits simultaneously — any one tripping blocks.

Throttling vs load shedding vs backpressure

Client-side behavior completes the loop: on 429/503, retry with exponential backoff + full jitter (sleep = random(0, min(cap, base·2^attempt)) — the AWS-recommended variant, because un-jittered backoff synchronizes clients into waves), always respect Retry-After when present, and cap total retries with a budget so retries can't exceed a fraction of live traffic. Adaptive limits go further: instead of static numbers, adjust limits from observed signals (latency gradients, error rates) — Netflix's concurrency-limits library and TCP-Vegas-style algorithms shrink the window when latency rises, giving you limits that track real capacity as it changes with deploys and hardware.

Which limiter where?

Why auto-increment breaks at scale

• Single point of coordination: one database hands out the next number; every writer serializes through it. Shard the database and two shards will happily both issue id 1000001.
• Leaks business volume: sequential order IDs let anyone diff two IDs a day apart and read your daily order count off your URL — a real competitive-intel technique (the "German tank problem" applied to your invoices).
• Can't pre-generate or generate offline: clients and edge services must round-trip to the DB before they even have an identity for the object; no batching writes, no offline-first apps.
• Ties identity to one storage engine: migrating off the database means migrating the ID authority too.

Requirements matrix — ask these before choosing a scheme:

• Unique — always. Globally or per-shard?
• Sortable? — roughly time-ordered IDs make B-tree PKs happy and give free "recent first" ordering.
• Opaque? — must the ID hide volume/timing from outsiders? (Sortable and opaque are in tension.)
• Size — 64-bit fits in a BIGINT and is cheap to index; 128-bit is collision-proof without coordination.
• Generation locality — can any node mint IDs with zero coordination, or is a call to a service acceptable?

UUIDv4 vs UUIDv7/ULID: randomness is index-hostile

• UUID v4: 122 random bits. Collision-proof, zero coordination, fully opaque — and terrible as a B-tree primary key. Each insert lands at a uniformly random leaf page: every page is always "hot", the working set is the whole index, page splits fragment constantly, and cache hit rates collapse. In InnoDB (clustered PK) this shreds insert throughput and bloats the table — see ch09 for the B-tree mechanics.
• UUID v7 (RFC 9562): 48-bit Unix millisecond timestamp prefix + random tail. Inserts append to the "right edge" of the index like auto-increment, keeping pages warm and splits rare, while staying 128-bit and coordination-free. The modern default answer.
• ULID: same idea predating v7 — 48-bit timestamp + 80 random bits, encoded as 26 chars of Crockford base32 (lexicographically sortable strings). Practically interchangeable with UUIDv7; pick whichever your ecosystem supports natively.
• Caveat: time-prefixed IDs leak creation time and cluster writes at the index tail — a single hot shard in range-partitioned stores like DynamoDB (which is why DynamoDB wants high-cardinality, unordered partition keys — see AWS callout).

Snowflake IDs: 64 bits, k-sorted, coordination-free

Twitter's Snowflake packs a 64-bit integer: 41 bits of milliseconds since a custom epoch (~69 years) + 10 bits of machine id (1024 generators) + 12 bits of per-millisecond sequence (4096 IDs/ms/node ≈ 4M IDs/sec/node). Time-ordered across the fleet (roughly — "k-sorted"), fits a BIGINT, and each node mints locally. The only coordination is assigning machine ids (config, ZooKeeper, or a coordination-free hash of instance identity with collision checks).

/** Snowflake generator: 41-bit ms timestamp | 10-bit machine | 12-bit seq. */
class SnowflakeGenerator {
  // Custom epoch keeps 41 bits useful for ~69 years from here.
  private static readonly EPOCH = 1735689600000n;      // 2025-01-01T00:00:00Z
  private static readonly MACHINE_BITS = 10n;
  private static readonly SEQ_BITS = 12n;
  private static readonly MAX_SEQ = (1n << SnowflakeGenerator.SEQ_BITS) - 1n; // 4095

  private lastMs = -1n;
  private sequence = 0n;

  constructor(private readonly machineId: bigint) {
    if (machineId < 0n || machineId >= 1n << SnowflakeGenerator.MACHINE_BITS) {
      throw new RangeError('machineId must fit in 10 bits (0..1023)');
    }
  }

  next(): bigint {
    let nowMs = BigInt(Date.now());

    // Clock regression guard: NTP stepped us backwards. Never reuse a
    // past millisecond — that risks duplicate IDs. Refuse (or wait it out).
    if (nowMs < this.lastMs) {
      const driftMs = this.lastMs - nowMs;
      if (driftMs > 10n) {
        throw new Error(`clock moved back ${driftMs}ms; refusing to generate`);
      }
      nowMs = this.lastMs;               // small drift: pin to last seen ms
    }

    if (nowMs === this.lastMs) {
      this.sequence = (this.sequence + 1n) & SnowflakeGenerator.MAX_SEQ;
      if (this.sequence === 0n) {
        // 4096 IDs already minted this ms — spin until the next tick.
        while (BigInt(Date.now()) <= this.lastMs) { /* busy-wait ~≤1ms */ }
        nowMs = BigInt(Date.now());
      }
    } else {
      this.sequence = 0n;
    }
    this.lastMs = nowMs;

    return (
      ((nowMs - SnowflakeGenerator.EPOCH) <<
        (SnowflakeGenerator.MACHINE_BITS + SnowflakeGenerator.SEQ_BITS)) |
      (this.machineId << SnowflakeGenerator.SEQ_BITS) |
      this.sequence
    );
  }

  /** Decode for debugging/audit: when and where was this ID minted? */
  static decode(id: bigint): { timestampMs: number; machineId: number; seq: number } {
    return {
      timestampMs: Number((id >> 22n) + SnowflakeGenerator.EPOCH),
      machineId: Number((id >> 12n) & 0x3ffn),
      seq: Number(id & 0xfffn),
    };
  }
}

Ticket servers (Flickr style): before Snowflake, Flickr ran two MySQL "ticket" servers using REPLACE INTO on a single-row table with auto_increment_increment=2 and offsets 1 and 2 — one serves odd IDs, one even, so either can die. Simple, battle-tested, strictly-ordered-ish 64-bit IDs — but every ID mint is still a network round trip, and throughput caps at what two MySQL boxes can serve. It's the right answer when you already run MySQL and need modest scale; mention it as history, choose Snowflake/UUIDv7 today.

Clocks lie

• Wall clock (Date.now()): tells you what time it is — and gets adjusted: NTP steps it forwards or backwards, VMs pause and jump, leap seconds happen. Two calls to Date.now() can return a smaller value the second time.
• Monotonic clock (performance.now(), process.hrtime.bigint()): tells you how much time elapsed — guaranteed never to go backwards, but its absolute value is meaningless and incomparable across machines (or even process restarts).
• NTP reality: ordinary servers drift ~10–200 ppm (seconds/day) and sync to within single-digit milliseconds on a good LAN — tens of ms or worse over WAN. Google/AWS "smear" leap seconds across the day rather than replaying a second.
• Consequence — last-write-wins by timestamp silently loses data: if node A's clock is 30 ms ahead, A's earlier write can carry a later timestamp than B's subsequent write, and LWW (Cassandra-style) will discard the genuinely newer value — no error, no log line, just a lost update. Kleppmann's DDIA treats this as a headline hazard.

// WRONG: wall clock for durations — NTP can step Date.now() backwards
const t0 = Date.now();
await chargeCard(paymentIntent);
const elapsedWrong = Date.now() - t0;      // can be NEGATIVE or wildly off

// RIGHT: monotonic clock for durations — never goes backwards
const start = process.hrtime.bigint();     // ns, monotonic (Node)
await chargeCard(paymentIntent);
const elapsedMs = Number(process.hrtime.bigint() - start) / 1e6;

// Browser equivalent: performance.now() (monotonic ms since page load)
// Rule: wall clock for "when did it happen" (audit logs, certificates);
//       monotonic clock for "how long did it take" (latency, timeouts, leases).

Logical clocks: ordering without trusting time

If physical time can't order events, count causality instead. Lamport clocks capture happened-before: each process keeps a counter; increment on every local event; stamp outgoing messages; on receive, jump to max(local, received) + 1. Guarantee: if a → b (a causally precedes b), then L(a) < L(b).

/** Lamport clock: total-orderable timestamps that respect causality. */
class LamportClock {
  private time = 0;

  /** Rule 1: tick before any local event (write, state change). */
  tick(): number { return ++this.time; }

  /** Rule 2: stamp every outgoing message with the post-tick value. */
  stampSend(): number { return this.tick(); }

  /** Rule 3: on receive, jump past the sender then tick. */
  onReceive(remoteTime: number): number {
    this.time = Math.max(this.time, remoteTime) + 1;
    return this.time;
  }

  now(): number { return this.time; }
}

// Tie-break with node id for a TOTAL order (used by distributed locks,
// e.g. Lamport's mutual exclusion): compare [time, nodeId] tuples.
type LamportStamp = readonly [time: number, nodeId: string];
const before = (a: LamportStamp, b: LamportStamp): boolean =>
  a[0] !== b[0] ? a[0] < b[0] : a[1] < b[1];

The limitation: the converse doesn't hold. L(a) < L(b) does not mean a happened before b — they may be concurrent (neither could have known about the other). Lamport clocks can order events but cannot detect conflicts. For that you need vector clocks.

Vector clocks: detecting concurrency

Keep one counter per node. Now comparison yields three outcomes: before, after, or concurrent — and concurrent means a real conflict that needs resolution (merge, CRDT, or ask the user). This is the Dynamo shopping-cart design: two clients modify the cart against different replicas during a partition; on read, the replicas return sibling versions whose vector clocks are concurrent, and the app merges (union of items) rather than silently dropping one.

type NodeId = string;
type Ordering = 'before' | 'after' | 'concurrent' | 'equal';

/** Vector clock: per-node counters; detects concurrency, not just order. */
class VectorClock {
  constructor(private readonly clock: Map<NodeId, number> = new Map()) {}

  /** Local event on `node`: bump that node's slot. */
  increment(node: NodeId): VectorClock {
    const next = new Map(this.clock);
    next.set(node, (next.get(node) ?? 0) + 1);
    return new VectorClock(next);
  }

  /** On receive: element-wise max, then bump the receiver's slot. */
  merge(other: VectorClock, receiver: NodeId): VectorClock {
    const next = new Map(this.clock);
    for (const [node, t] of other.clock) {
      next.set(node, Math.max(next.get(node) ?? 0, t));
    }
    next.set(receiver, (next.get(receiver) ?? 0) + 1);
    return new VectorClock(next);
  }

  /** before: every entry ≤ other's, at least one strictly <.
   *  concurrent: each side is ahead somewhere → true conflict. */
  compare(other: VectorClock): Ordering {
    let selfAhead = false, otherAhead = false;
    const nodes = new Set([...this.clock.keys(), ...other.clock.keys()]);
    for (const node of nodes) {
      const a = this.clock.get(node) ?? 0;
      const b = other.clock.get(node) ?? 0;
      if (a > b) selfAhead = true;
      if (b > a) otherAhead = true;
    }
    if (selfAhead && otherAhead) return 'concurrent';
    if (selfAhead) return 'after';
    if (otherAhead) return 'before';
    return 'equal';
  }
}

// Dynamo cart: two writes during a partition
let cartA = new VectorClock().increment('replicaA');   // {A:1} add "book"
let cartB = new VectorClock().increment('replicaB');   // {B:1} add "pen"
cartA.compare(cartB);   // 'concurrent' → return BOTH siblings; app merges

• Version vectors vs vector clocks, one line: version vectors are the same structure but bumped only on writes to the object (not every event) — the right tool for replica conflict detection; vector clocks track causality of all events.
• Cost: O(nodes) per stamp; Dynamo prunes old entries, Riak popularized "dotted version vectors" to bound growth.

Hybrid logical clocks and TrueTime

HLC combines a physical timestamp with a logical counter: timestamps stay close to wall time (readable, indexable, bounded drift from NTP time) while updating with Lamport's max-rule so causality is never violated even when physical clocks disagree — best of both. CockroachDB uses HLCs for transaction ordering, and MongoDB's causal-consistency sessions ride on a cluster-wide HLC (its ClusterTime). Google TrueTime takes the hardware route: GPS + atomic clocks in every datacenter expose time as a bounded interval [earliest, latest] (typically < 7 ms wide). Spanner commits a transaction, then waits out the uncertainty ("commit wait") before making it visible — so timestamp order provably matches real-time order, giving external consistency (strict serializability) across the planet. The lesson to quote: you can't remove clock uncertainty, but if you can bound it, you can wait it out.

Choosing an ID scheme

Monolith vs modular monolith vs microservices

The honest framing: microservices trade development velocity at small scale for organizational and deployment independence at large scale. The dominant cost driver is team count, not traffic.

• Don't start with microservices. A startup with 8 engineers and 40 services is paying Google-scale coordination tax with a garage-scale org. Start with a modular monolith; split when a boundary is proven stable and a team needs deploy independence.
• Modular monolith is the staff-level default answer for new products: enforce module boundaries in-process (separate packages, no cross-module DB access, internal APIs), so extraction later is a mechanical move, not an archaeology dig.
• Legitimate split triggers: a team is blocked on another team's release train; a component needs a different scaling profile, language, or compliance boundary (e.g., PCI scope isolation in fintech — put card handling in its own service and shrink the audit surface).

Service boundaries: bounded contexts and own-your-data

• Bounded contexts (DDD): split along business capabilities, not technical layers. "Payments", "Ledger", "KYC", "Notifications" — not "API layer service" and "DB layer service". A good boundary is one where the vocabulary changes: "account" means different things to Ledger and to KYC, and that's fine because they're separate contexts.
• Own-your-data rule: a service's database is private. Other services get data only via its API or its published events — never by querying its tables.
• Database-per-service, and why shared DBs couple teams: a shared schema is a shared contract with no versioning. Team A can't add a column, change a type, or reindex without coordinating with every team that reads the table. You've re-created the monolith's release train at the schema layer — with none of the monolith's transactional benefits. Shared DB also destroys independent scaling (one service's table scan starves another's OLTP) and makes ownership of data quality ambiguous.
• Corollary: cross-service "joins" become API composition or a read model built from events (CQRS, ch14). If you find yourself needing constant cross-service joins, the boundary is wrong — merge the services.
• Litmus test for a boundary: can this service be meaningfully described in one sentence, deployed alone, and tested with stubs of its neighbors? If a change routinely fans out to three services, you have a distributed monolith (see below).

Sync vs async coupling — the availability math

Every synchronous hop makes your availability the product of the chain. Five services at 99.9% each, called in series:

• 0.999⁵ ≈ 0.995 → 99.5% — you silently spent 4x your error budget (43 min/month → ~3.6 h/month of downtime) just on topology.
• Latency compounds too: chain p99s don't add nicely — the chain's tail is dominated by the worst hop, and fan-outs make hitting some slow replica nearly certain (ch20's percentile math).
• Slow is worse than down. A dead dependency fails fast; a slow one holds your threads, connections, and memory hostage until the whole fleet is wedged. Every resilience pattern below exists mostly to convert "slow" into "fast failure".
• Async decoupling (queue/stream between services, ch13) breaks the availability product: the producer succeeds if the broker accepts the message, even while the consumer is down. Cost: eventual consistency and harder reasoning about "is it done yet". Rule of thumb: sync for queries the user is waiting on; async for state changes that can be acknowledged and processed later (order placed → email, ledger posting, analytics).

The resilience toolkit

Timeouts and deadline propagation

• Every remote call needs a timeout. The default in most HTTP clients is infinite or minutes — that's how one slow dependency consumes your entire connection pool.
• Budget propagation: the edge sets a total deadline (e.g., 2s); each hop passes the remaining budget downstream (gRPC does this natively; over HTTP use a header). Without it, a service 3 hops deep happily works for 5s on a request whose caller gave up 4s ago — wasted work at best, retry amplification at worst.
• Set timeouts from the dependency's observed p99 + headroom, not from vibes. Timeout > p99.9 means you almost never cut off legitimate work; timeout ≈ p50 means you time out half your traffic.

// Deadline propagation: pass remaining budget down the call chain.
interface Deadline {
  readonly expiresAt: number;              // epoch ms
  remainingMs(): number;
  expired(): boolean;
}

function deadlineIn(totalBudgetMs: number): Deadline {
  const expiresAt = Date.now() + totalBudgetMs;
  return {
    expiresAt,
    remainingMs: () => Math.max(0, expiresAt - Date.now()),
    expired:     () => Date.now() >= expiresAt,
  };
}

async function callWithDeadline<T>(
  url: string,
  deadline: Deadline,
  minUsefulMs = 25,                        // not worth dialing if less remains
): Promise<T> {
  const budget = deadline.remainingMs();
  if (budget < minUsefulMs) throw new Error('DeadlineExceeded: no budget left');

  const res = await fetch(url, {
    signal: AbortSignal.timeout(budget),   // hard cap on this hop
    headers: { 'x-request-deadline-ms': String(budget) }, // downstream reads
  });
  if (!res.ok) throw new Error(`Upstream ${res.status}`);
  return res.json() as Promise<T>;
}

// Edge handler: one budget governs the whole request tree.
// const dl = deadlineIn(2_000);
// const risk = await callWithDeadline<RiskScore>(riskUrl, dl);
// const quote = await callWithDeadline<Quote>(quoteUrl, dl); // gets what's left

Retries done right

• Only retry idempotent operations (GET, PUT with idempotency keys — ch17). Retrying a non-idempotent POST after an ambiguous timeout is how customers get charged twice.
• Exponential backoff + full jitter: without jitter, all clients that failed together retry together — synchronized waves that re-kill the recovering server. Full jitter (AWS's recommendation): sleep random(0, min(cap, base·2^attempt)).
• Retry budgets, not just per-call caps: cap retries as a fraction of total traffic (e.g., retries ≤ 10% of requests). Per-call "3 retries" through a 3-deep chain is 3×3×3 = 27x amplification at the bottom — a self-inflicted DDoS called a retry storm. Retry at one layer (usually the closest to the user), not every layer.
• Only retry retryable errors: timeouts, 503, connection reset. Never retry 400/422 — the request is wrong and will be wrong again.

// Exponential backoff + full jitter + a fleet-level retry budget.
class RetryBudget {
  private tokens: number;
  constructor(private readonly max: number, private readonly refillPerSec: number) {
    this.tokens = max;
    setInterval(() => { this.tokens = Math.min(this.max, this.tokens + refillPerSec); }, 1_000);
  }
  tryAcquire(): boolean {
    if (this.tokens < 1) return false;     // budget exhausted: fail, don't storm
    this.tokens -= 1;
    return true;
  }
}

const budget = new RetryBudget(100, 10);   // ~10 retries/sec fleet-wide per client

async function withRetries<T>(
  op: () => Promise<T>,
  isRetryable: (e: unknown) => boolean,
  { attempts = 3, baseMs = 100, capMs = 2_000 } = {},
): Promise<T> {
  for (let attempt = 0; ; attempt++) {
    try {
      return await op();
    } catch (err) {
      const lastTry = attempt >= attempts - 1;
      if (lastTry || !isRetryable(err) || !budget.tryAcquire()) throw err;
      const ceiling = Math.min(capMs, baseMs * 2 ** attempt);
      const sleepMs = Math.random() * ceiling;          // FULL jitter
      await new Promise((r) => setTimeout(r, sleepMs));
    }
  }
}

Circuit breaker

When a dependency is failing, stop calling it. Fail fast locally, give it room to recover, and probe cautiously before resuming. Three states: closed (normal, counting failures) → open (all calls rejected instantly for a cool-down) → half-open (let a few probes through; success closes, failure re-opens).

// Circuit breaker with typed states and half-open probe logic.
type BreakerState =
  | { kind: 'closed'; consecutiveFailures: number }
  | { kind: 'open'; openedAt: number }
  | { kind: 'half-open'; probesInFlight: number; probeSuccesses: number };

interface BreakerOptions {
  failureThreshold: number;   // consecutive failures to trip (e.g. 5)
  openMs: number;             // cool-down before probing (e.g. 30_000)
  probeCount: number;         // successes needed to close (e.g. 3)
  maxProbes: number;          // concurrent probes allowed in half-open
}

class CircuitOpenError extends Error {
  constructor(public readonly retryAfterMs: number) { super('circuit open'); }
}

class CircuitBreaker {
  private state: BreakerState = { kind: 'closed', consecutiveFailures: 0 };
  constructor(private readonly opts: BreakerOptions) {}

  async exec<T>(op: () => Promise<T>): Promise<T> {
    const s = this.state;
    if (s.kind === 'open') {
      const elapsed = Date.now() - s.openedAt;
      if (elapsed < this.opts.openMs) {
        throw new CircuitOpenError(this.opts.openMs - elapsed);  // fail FAST
      }
      this.state = { kind: 'half-open', probesInFlight: 0, probeSuccesses: 0 };
    }
    if (this.state.kind === 'half-open') {
      if (this.state.probesInFlight >= this.opts.maxProbes) {
        throw new CircuitOpenError(0);       // only a trickle of probes
      }
      this.state.probesInFlight++;
    }
    try {
      const result = await op();
      this.onSuccess();
      return result;
    } catch (err) {
      this.onFailure();
      throw err;
    }
  }

  private onSuccess(): void {
    if (this.state.kind === 'half-open') {
      this.state.probeSuccesses++;
      if (this.state.probeSuccesses >= this.opts.probeCount) {
        this.state = { kind: 'closed', consecutiveFailures: 0 };  // recovered
      } else {
        this.state.probesInFlight--;
      }
    } else {
      this.state = { kind: 'closed', consecutiveFailures: 0 };
    }
  }

  private onFailure(): void {
    if (this.state.kind === 'half-open') {
      this.state = { kind: 'open', openedAt: Date.now() };        // re-trip
    } else if (this.state.kind === 'closed') {
      this.state.consecutiveFailures++;
      if (this.state.consecutiveFailures >= this.opts.failureThreshold) {
        this.state = { kind: 'open', openedAt: Date.now() };      // trip
      }
    }
  }
}
// Production nuance: count failures over a sliding window with a rate
// threshold (e.g. >50% of last 20 calls) instead of consecutive count,
// and treat timeouts as failures — slow IS failing.

Bulkheads

• Isolate resource pools per dependency — separate connection pools, thread pools, or concurrency limits for each downstream. A slow recommendations service can then exhaust its 20 permits, while payments' pool stays untouched.
• Without bulkheads, one slow dependency + one shared pool = every request handler blocked waiting on the same wedge, and your service is down even though 90% of its dependencies are healthy.
• Also applies at infra level: separate ASGs/target groups for critical vs bulk traffic, per-tenant quotas, dedicated DB replicas for analytics.

// Bulkhead: a semaphore capping concurrent calls per dependency.
class Bulkhead {
  private inFlight = 0;
  private readonly waiters: Array<() => void> = [];
  constructor(
    private readonly maxConcurrent: number,  // e.g. 20 for recs, 100 for payments
    private readonly maxQueue: number,       // bounded! unbounded queues just hide the wedge
  ) {}

  async run<T>(op: () => Promise<T>): Promise<T> {
    if (this.inFlight >= this.maxConcurrent) {
      if (this.waiters.length >= this.maxQueue) {
        throw new Error('BulkheadRejected: dependency saturated'); // shed, don't queue forever
      }
      await new Promise<void>((resolve) => this.waiters.push(resolve));
    }
    this.inFlight++;
    try {
      return await op();
    } finally {
      this.inFlight--;
      this.waiters.shift()?.();              // admit next waiter, if any
    }
  }
}

// One bulkhead per downstream — pools must not be shared.
const pools = {
  payments: new Bulkhead(100, 50),
  recs:     new Bulkhead(20, 10),            // slow recs can only wedge 20 slots
};
// await pools.recs.run(() => fetchRecommendations(userId));

Fallbacks & graceful degradation

• When the breaker is open or the bulkhead rejects, have a plan B: serve stale cache (last known FX rates with a "as of 10:02" stamp), defaults (generic top-10 instead of personalized recommendations), or degraded UX (hide the widget entirely).
• Rank features by criticality: checkout must work; the "customers also bought" rail is sacrificial. Wire breakers so sacrificial features fail silently and critical paths fail loudly.
• Fintech caution: never fall back on correctness-critical data — a stale risk score or balance is worse than an error. Fallbacks are for optional enrichments, not for the ledger.

Load shedding & admission control

• Under overload, serving some requests well beats serving all requests badly (everyone times out, throughput collapses to zero — congestion collapse). Reject excess early with 429/503 + Retry-After.
• Shed by priority: drop crawler/analytics traffic first, interactive user traffic last; per-customer quotas stop one tenant from starving the rest (rate limiting mechanics in ch16).
• Good signals for "I'm overloaded": queue depth, event-loop lag, p99 latency vs SLO — not CPU alone.
• Admission control at the front door (API gateway / LB) is worth 10 resilience patterns inside: work rejected in 1ms costs nothing downstream.

Service discovery

Instances come and go (autoscaling, deploys, failures); callers need a live answer to "where is payments right now?".

Service mesh

• Sidecar model: an Envoy proxy runs next to every service instance; all inbound/outbound traffic flows through it. A control plane (Istio, App Mesh) pushes config to all sidecars.
• What you get without touching app code: mTLS everywhere (identity + encryption between services — big for PCI/SOC2), retries/timeouts/circuit breaking as config, traffic shifting (1% canary to v2), and uniform metrics/traces for every hop.
• The pitch in one line: the resilience toolkit above, implemented once in infrastructure instead of N times in N languages.
• When it's overkill: < ~10–15 services, single language (a shared library is cheaper), or a team with no platform capacity — a mesh is a distributed system you now operate underneath your distributed system. Sidecar-less alternatives (VPC Lattice, Cilium/eBPF, Istio ambient mode) reduce the per-pod tax.

Strangler-fig migration

Never rewrite big-bang. Put a router in front of the monolith and peel routes off one at a time: new service takes /payments/*, everything else still hits the monolith. Each slice ships independently, is reversible with a route flip, and shrinks the monolith until it "withers".

• Order of extraction: start with a leaf capability (few inbound deps), high change-rate (most benefit), clean data ownership. Not the gnarliest core module.
• Data comes with the code: give the new service its own store; sync from the monolith DB during transition (dual-write with reconciliation, or CDC via outbox — ch14), then cut over reads, then writes, then delete the old tables.
• Keep a comparison window: shadow-read both paths and diff results before trusting the new service (especially for money paths).

The distributed monolith anti-pattern

• All the operational cost of microservices, none of the independence: services that must be deployed together (lockstep releases), share a database, chat over chains of synchronous calls, and share fragile internal libraries with breaking changes.
• Smells: a "release train" spreadsheet coordinating 12 services; integration environments where nothing works unless everything is on matching versions; one team's schema migration breaking three other teams.
• Root cause is almost always boundaries drawn along technical layers or org-chart lines instead of business capabilities — the split happened before the domain was understood.
• The fix is usually to merge services back, not add more. Fewer, better-bounded services beat many entangled ones.

The three pillars — and the real test

• Logs: discrete, timestamped events with arbitrary detail. Answer "what exactly happened to request X?" Highest fidelity, highest cost per unit of insight.
• Metrics: pre-aggregated numbers over time (counters, gauges, histograms). Answer "is the system healthy? how healthy? trending which way?" Cheap to store and query, cheap to alert on — but aggregation destroys the individual request.
• Traces: the life of one request across services — a tree of timed spans. Answer "where in this 9-hop call tree did the time or the error go?"
• The pillars are lenses on the same events, and the workflow chains them: a metric alert fires (p99 up) → an exemplar trace shows the slow span is the ledger DB call → logs for that trace ID reveal a lock-wait on a specific account. Any pillar alone leaves you stuck.
• The staff-level test isn't "do you emit all three" — it's "can you go from symptom to root cause for a failure mode you never anticipated, using only what's already emitted?" That requires high-cardinality, correlated telemetry, not three disconnected silos.

Structured logging

• JSON, one event per line. logger.info({ event: 'payment.captured', amountMinor: 4210, currency: 'INR', paymentId }) — machine-parseable, queryable by field. String-interpolated prose ("Payment 123 captured for ₹42.10") is grep-only archaeology.
• Levels used correctly: error = broken and needs action (a human or an alert should care); warn = surprising but handled (fallback served, retry succeeded); info = business-significant state changes (order placed, payout initiated); debug = developer detail, off in prod by default. If everything is error, nothing is.
• What to log: event name, entity IDs, outcome, duration, error class/code, and always the correlation IDs. What never to log (fintech especially): full PANs/card data (PCI-DSS scope explosion — one logged PAN makes your log pipeline in-scope for audit), CVVs (never storable, period), passwords/tokens/API keys, full PII (mask: a****@orbitxpay.com), full request bodies on money paths. Enforce with a redaction layer in the logger, not developer discipline.
• Sampling high-volume logs: keep 100% of errors and slow requests, sample the happy path (e.g., 1%) — errors are rare and precious, successes are statistically redundant.
• Cost control: log verbosity per environment (debug in dev, info in prod), retention tiers (hot/searchable 7–30 days → archive to S3/Glacier for compliance years), and drop-filters for known-noisy events. Log bills quietly reaching 6 figures is a rite of passage — cardinality and volume budgets are real engineering.

Correlation IDs are the keystone. Assign (or accept) a request ID at the edge, propagate it through every service hop and queue message, and auto-attach it to every log line. Node's AsyncLocalStorage makes this invisible to business code — this is the pattern interviewers want to see you actually know:

// Request context + logger that auto-attaches requestId/traceId to every line.
// AsyncLocalStorage carries the context across awaits, callbacks, and promises
// without threading a `ctx` parameter through every function signature.
import { AsyncLocalStorage } from 'node:async_hooks';
import { randomUUID } from 'node:crypto';

interface RequestContext {
  requestId: string;        // ours, minted at the edge if absent
  traceId?: string;         // W3C trace ID, links logs <-> traces
  userId?: string;
  route?: string;
}

const als = new AsyncLocalStorage<RequestContext>();

type Level = 'debug' | 'info' | 'warn' | 'error';
const REDACT = /"(pan|cvv|password|token|ssn)"\s*:\s*"[^"]*"/gi;   // last line of defense

function emit(level: Level, event: string, fields: Record<string, unknown> = {}): void {
  const ctx = als.getStore();               // undefined outside a request (cron, startup)
  const line = JSON.stringify({
    ts: new Date().toISOString(),
    level,
    event,                                  // machine-stable name: 'payment.captured'
    requestId: ctx?.requestId,
    traceId: ctx?.traceId,
    userId: ctx?.userId,
    route: ctx?.route,
    ...fields,
  });
  process.stdout.write(line.replace(REDACT, '"$1":"[REDACTED]"') + '\n');
}

export const log = {
  debug: (e: string, f?: Record<string, unknown>) => emit('debug', e, f),
  info:  (e: string, f?: Record<string, unknown>) => emit('info', e, f),
  warn:  (e: string, f?: Record<string, unknown>) => emit('warn', e, f),
  error: (e: string, f?: Record<string, unknown>) => emit('error', e, f),
};

// Express middleware: open a context per request; everything downstream inherits it.
export function requestContext(req: Req, res: Res, next: Next): void {
  const ctx: RequestContext = {
    requestId: req.header('x-request-id') ?? randomUUID(),  // accept or mint
    traceId: parseTraceparent(req.header('traceparent'))?.traceId,
    route: req.path,
  };
  res.setHeader('x-request-id', ctx.requestId);   // echo so clients can report it
  als.run(ctx, next);                             // scope ctx to this request's async tree
}

// Outbound calls forward the IDs — the chain must not break at hop 2:
//   fetch(url, { headers: { 'x-request-id': als.getStore()!.requestId, traceparent } })
// Business code stays clean: log.info('payout.initiated', { payoutId, amountMinor });
// ...and every line carries requestId/traceId automatically.

Metrics: RED, USE, and why averages lie

• Instrument types: counter (monotonic: requests_total, errors_total — rates come from deltas), gauge (point-in-time: queue depth, in-flight requests, memory), histogram (distribution in buckets: request duration — the only honest way to get percentiles).
• RED method — for every service: Rate (req/s), Errors (failed req/s), Duration (latency distribution). If you dashboard nothing else per service, dashboard RED.
• USE method — for every resource (CPU, disk, pool, queue): Utilization (busy %), Saturation (queued/waiting work — the leading indicator), Errors. RED is the customer's view; USE is the machine's view. Saturation rises before utilization pegs — watch it.
• Averages lie. Mean latency blends a fast majority with a catastrophic tail and reports "fine". At 1M requests/day, p99 = 10,000 requests/day having a terrible time — and heavy users (your best customers) hit more requests, so they hit the tail more often.

• Tail latency amplification: fan out to N services in parallel and the response waits for the slowest. P(all N respond fast) = 0.99^N if each is fast with probability 0.99 — for N = 10, only 90% of user requests avoid every p99; for N = 100, just 37%. Your users experience your dependencies' p99 at your p50. This is why Google's "Tail at Scale" strategies exist: hedged requests, tied requests, cutting off slow replicas.
• Percentiles don't average: the mean of five servers' p99s is meaningless. Aggregate histograms, then compute the percentile.

// RED metrics with a histogram — typed thin wrapper over prom-client.
import { Counter, Histogram, register } from 'prom-client';

const httpRequests = new Counter({
  name: 'http_requests_total',
  help: 'Requests by route/method/status',
  labelNames: ['route', 'method', 'status'] as const,   // low cardinality ONLY —
});                                                     // never userId/requestId as labels

const httpDuration = new Histogram({
  name: 'http_request_duration_seconds',
  help: 'Latency distribution',
  labelNames: ['route', 'method'] as const,
  // Buckets shaped around the SLO (300ms) so burn is readable straight off the chart:
  buckets: [0.025, 0.05, 0.1, 0.2, 0.3, 0.5, 1, 2, 5],
});

export function redMetrics(req: Req, res: Res, next: Next): void {
  const stop = httpDuration.startTimer({ route: req.route?.path ?? 'unmatched', method: req.method });
  res.on('finish', () => {
    stop();
    httpRequests.inc({ route: req.route?.path ?? 'unmatched', method: req.method,
                       status: String(res.statusCode) });
  });
  next();
}
// GET /metrics -> register.metrics() for Prometheus/ADOT scrape.
// p99 in PromQL: histogram_quantile(0.99,
//   sum by (le, route) (rate(http_request_duration_seconds_bucket[5m])))

SLI, SLO, SLA — and error budgets

• SLI (indicator): a measured ratio of good events / total events. "Proportion of checkout requests under 300ms returning non-5xx."
• SLO (objective): the internal target on that SLI over a window. "99.9% over 30 days." Set by engineering + product; drives alerts and priorities.
• SLA (agreement): the external, contractual promise — looser than the SLO, with money attached (service credits). You breach the SLO before the SLA, on purpose: the SLO is the tripwire.
• Error budget = 1 − SLO: the failure you're allowed. 99.9% monthly = 43.8 minutes of full downtime (or the equivalent in partial failure). The budget is a shared currency: deploys, migrations, and chaos experiments all spend it.
• Budgets gate release velocity: budget healthy → ship fast, take risks; budget burned → feature freeze, reliability work only. This turns "dev wants to ship vs ops wants stability" from a culture war into arithmetic. It also caps over-engineering: if you're under-spending the budget, you're probably shipping too slowly.

Distributed tracing

• Model: a trace = one request end-to-end; a span = one timed operation (an HTTP handler, a DB query, a queue publish) with attributes, status, and a parent span — forming a tree whose waterfall shows exactly where time went.
• Context propagation — W3C traceparent: 00-<trace-id 16 bytes>-<parent-span-id 8 bytes>-<flags>, e.g. 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01. Every outbound call forwards it (HTTP headers, message attributes on SQS/Kafka); the receiver continues the same trace. One service dropping the header cuts the trace in half — propagation discipline is the whole game.
• Sampling: tracing everything at scale is unaffordable. Head sampling decides at trace start (cheap, simple — e.g. keep 1%) but is blind: it discards errors and slow traces at the same rate as boring ones. Tail sampling buffers spans and decides at trace end ("keep all errors, all >1s, 1% of the rest") — exactly the traces you want, at the cost of a stateful collector tier. Staff answer: head-sample at a base rate + tail-sample errors/slow traces in the collector.
• OpenTelemetry (OTel) is the vendor-neutral standard for all three signals: instrument once, export anywhere (X-Ray, Jaeger, Datadog, Honeycomb) by swapping the exporter. Auto-instrumentation patches HTTP/Express/pg/redis clients so you get spans and propagation with near-zero code; manual spans add business context.

// OpenTelemetry: typed manual span around an outbound call.
// Auto-instrumentation (@opentelemetry/auto-instrumentations-node, loaded via
// --require before the app) already creates spans for inbound HTTP, Express,
// pg, redis, and injects/extracts `traceparent` on fetch — manual spans are
// for business operations you want visible in the waterfall by name.
import { trace, SpanStatusCode, SpanKind, type Span } from '@opentelemetry/api';

const tracer = trace.getTracer('payments-service', '1.4.0');

interface CaptureResult { captureId: string; status: 'captured' | 'declined'; }

export async function capturePayment(paymentId: string, amountMinor: number): Promise<CaptureResult> {
  // startActiveSpan makes this span the parent of everything awaited inside —
  // the PSP HTTP call below auto-nests under it in the waterfall.
  return tracer.startActiveSpan(
    'payments.capture',
    { kind: SpanKind.CLIENT, attributes: {
        'payment.id': paymentId,               // IDs are fine as span attributes
        'payment.amount_minor': amountMinor,   // (high cardinality OK here,
        'payment.currency': 'INR',             //  unlike metric labels)
    }},
    async (span: Span): Promise<CaptureResult> => {
      try {
        const res = await pspClient.capture(paymentId, amountMinor); // traceparent auto-injected
        span.setAttribute('psp.response_code', res.code);
        if (res.status === 'declined') {
          span.addEvent('psp.declined', { 'decline.reason': res.reason }); // event, not error
        }
        span.setStatus({ code: SpanStatusCode.OK });
        return res;
      } catch (err) {
        span.recordException(err as Error);
        span.setStatus({ code: SpanStatusCode.ERROR, message: (err as Error).message });
        throw err;                              // observe, never swallow
      } finally {
        span.end();                             // unended spans never export
      }
    },
  );
}

Alerting philosophy, dashboards, runbooks

• Alert on symptoms (SLOs), not causes. Page on "checkout SLI burning error budget 14x too fast", not on "CPU > 80%". Causes without user impact are noise; user impact without a matching cause-alert still deserves a page. Burn-rate alerts (fast-burn: page; slow-burn: ticket) are the SRE-book pattern.
• Page vs ticket: a page means "a human must act now or users suffer"; everything else is a ticket for business hours. Every page must be actionable and urgent — if the responder's move is "shrug, it self-healed", delete or demote the alert.
• Alert fatigue is a reliability risk: a team paged 20x/night stops reading pages, and the real one dies in the noise. Review every page weekly; each is either actioned, tuned, or deleted.
• Dashboards that matter: one per-service RED dashboard, same layout everywhere — top row: request rate, error rate (with SLO line), p50/p95/p99 duration; second row: saturation (pool usage, queue depth, event-loop lag); third row: dependency view (RED of each downstream this service calls). Uniformity is the feature — at 3am nobody wants to learn a bespoke dashboard.
• Runbooks: every page links to a runbook — symptom, triage queries (pre-built log/trace links), remediation, escalation. Written before the incident; full incident-response and postmortem practice in ch21.

Deployment strategies

Every deploy is a controlled experiment: "does the new version behave in production?" The strategies differ in blast radius, rollback speed, and cost.

• Rolling: replace instances N at a time behind the LB. Cheap, no extra fleet — but two versions run simultaneously (your API and schema must tolerate that), and rollback means rolling forward again through the same slow process.
• Blue-green: stand up a full copy (green), flip traffic at the LB/DNS, keep blue warm. Instant rollback — flip back. Cost: ~2× fleet during the deploy window. Watch for state: in-flight jobs, sticky sessions, and DB schema shared by both colors.
• Canary: shift 1% → 5% → 25% → 100% of traffic to the new version, gated on metrics (error rate, p99 latency, business KPIs). Automated rollback on regression is the point — a canary without a metric gate is just a slow rolling deploy.
• Feature flags: deploy the code dark, release by flipping a flag. Decouples deploy (engineering event) from release (business event), enables per-user targeting and instant kill switches — but flags rot into permanent config forks if you don't delete them.

Canary rollout with a metric gate

Typed feature-flag client

// Minimal typed feature-flag client with targeting rules.
// Real systems (LaunchDarkly, AppConfig) add streaming updates + analytics.
interface FlagContext {
  userId: string;
  country?: string;
  plan?: 'free' | 'pro' | 'enterprise';
}

type Rule =
  | { kind: 'allowList'; userIds: string[] }
  | { kind: 'attribute'; attr: keyof FlagContext; equals: string }
  | { kind: 'percentage'; rollout: number };          // 0–100

interface FlagDefinition {
  key: string;
  enabled: boolean;               // master kill switch
  rules: Rule[];                  // first match wins
  defaultValue: boolean;
}

// Deterministic bucketing: same user always lands in the same bucket,
// so a 5% rollout doesn't flicker per-request for the same person.
function bucket(userId: string, flagKey: string): number {
  let h = 2166136261;                       // FNV-1a
  for (const c of `${flagKey}:${userId}`) {
    h = (h ^ c.charCodeAt(0)) * 16777619 >>> 0;
  }
  return h % 100;
}

export function isEnabled(flag: FlagDefinition, ctx: FlagContext): boolean {
  if (!flag.enabled) return false;          // kill switch beats everything
  for (const rule of flag.rules) {
    switch (rule.kind) {
      case 'allowList':
        if (rule.userIds.includes(ctx.userId)) return true;
        break;
      case 'attribute':
        if (ctx[rule.attr] === rule.equals) return true;
        break;
      case 'percentage':
        if (bucket(ctx.userId, flag.key) < rule.rollout) return true;
        break;
    }
  }
  return flag.defaultValue;
}

Graceful shutdown & health checks

Deploys, autoscaling, and spot reclaims all kill processes constantly. A service that drops in-flight requests on SIGTERM turns every routine deploy into a micro-outage. The sequence is always the same:

• 1. Receive SIGTERM (orchestrator asks nicely; SIGKILL follows after the grace period — typically 30s).
• 2. Fail readiness / stop accepting: flip the readiness probe to failing so the LB stops routing new traffic. Wait a few seconds for LB connection draining (deregistration delay) to take effect — the LB and your process must agree on who's still sending traffic.
• 3. Drain in-flight requests with a deadline shorter than the grace period.
• 4. Close resources in dependency order: HTTP server → queue consumers → DB/Redis pools.
• 5. Exit 0. Exiting non-zero or timing out gets you SIGKILLed mid-write.

// Graceful shutdown for a Node HTTP service with typed, ordered hooks.
import http from 'node:http';

type ShutdownHook = {
  name: string;
  timeoutMs: number;
  fn: () => Promise<void>;
};

class GracefulShutdown {
  private hooks: ShutdownHook[] = [];
  private shuttingDown = false;

  /** Hooks run in registration order: server first, pools last. */
  register(hook: ShutdownHook): void {
    this.hooks.push(hook);
  }

  get isShuttingDown(): boolean { return this.shuttingDown; }

  listen(): void {
    for (const sig of ['SIGTERM', 'SIGINT'] as const) {
      process.on(sig, () => void this.run(sig));
    }
  }

  private async run(signal: string): Promise<void> {
    if (this.shuttingDown) return;          // idempotent — signals can repeat
    this.shuttingDown = true;               // readiness probe now returns 503
    console.log(`${signal} received; draining...`);

    // Give the LB time to observe failing readiness and stop sending traffic
    // (must be >= probe interval; pairs with ALB deregistration delay).
    await new Promise(r => setTimeout(r, 5_000));

    for (const hook of this.hooks) {
      const timer = new Promise<never>((_, rej) =>
        setTimeout(() => rej(new Error(`${hook.name} timed out`)), hook.timeoutMs));
      try {
        await Promise.race([hook.fn(), timer]);
        console.log(`closed: ${hook.name}`);
      } catch (err) {
        console.error(`shutdown hook failed: ${hook.name}`, err); // keep going
      }
    }
    process.exit(0);
  }
}

const shutdown = new GracefulShutdown();
const server = http.createServer((req, res) => {
  if (req.url === '/health/ready') {
    res.writeHead(shutdown.isShuttingDown ? 503 : 200).end();
    return;
  }
  if (req.url === '/health/live') { res.writeHead(200).end(); return; }
  res.end('ok');
});

// 1. Stop accepting + drain in-flight (server.close waits for open sockets)
shutdown.register({
  name: 'http-server', timeoutMs: 20_000,
  fn: () => new Promise((res, rej) => server.close(e => e ? rej(e) : res())),
});
// 2. Then release downstream resources
shutdown.register({ name: 'pg-pool', timeoutMs: 5_000, fn: () => pgPool.end() });

server.listen(8080, () => shutdown.listen());

Liveness vs readiness vs startup

Why a failing dependency must NOT fail liveness: if your DB has a blip and every pod's liveness check pings the DB, the orchestrator restarts your entire healthy fleet simultaneously — converting a database blip into a full outage plus a thundering-herd reconnect storm. Restarting the app cannot fix the database. Readiness may reflect dependencies; liveness reflects only the process itself.

Zero-downtime database migrations

The cardinal rule: never make a change that old code can't survive, because during any deploy old and new code run against the same schema. The expand–migrate–contract pattern makes every step backward-compatible. Example: splitting full_name into first_name/last_name on a 500M-row table:

• 1. Expand — add nullable columns. ALTER TABLE users ADD COLUMN first_name text NULL. Nullable + no default means metadata-only in Postgres 11+ — no table rewrite, no lock storm. Old code ignores the columns entirely.
• 2. Dual-write. Deploy code that writes both old and new columns on every insert/update. Old rows are still stale — that's fine.
• 3. Backfill old rows in small batches (5–10k rows, sleep between batches, order by PK) so you never hold long locks or saturate replication. For 500M rows this runs for hours-to-days as a background job — monitor replica lag and throttle.
• 4. Verify, then switch reads to the new columns — ideally behind a feature flag so the read path can be flipped back instantly. Add constraints (NOT NULL, indexes) only now, using NOT VALID + VALIDATE CONSTRAINT and CREATE INDEX CONCURRENTLY.
• 5. Contract — stop dual-writing, drop the old column after a full release cycle (and after checking nothing else — reports, ETL, other services — reads it).