Failure modes and resilient design
Partitions, cascading failure, metastable states, retries as amplifiers, and the levers: timeouts, budgets, shedding, isolation.
The outages that make the news are rarely a component dying. They are a small failure that the system's own reactions turned into a large one: a slow dependency that filled every thread pool, a retry policy that tripled the load on a struggling database, a cache that expired and let a stampede through, a partition that produced two leaders. This lesson is the catalogue of those shapes — the ways a distributed system amplifies failure — and the levers that damp it, each of which earlier courses introduced and this one puts in one place.
Partitions
A partition is a network that delivers messages between some nodes and not others, for a while. It is not rare: a switch reboot, a misconfigured firewall rule, a cloud availability zone losing its link, a saturated NIC dropping packets. The CAP lesson gave the choice a partition forces — answer from the side you can reach and risk divergence, or refuse and lose availability — and the failure-detection lesson gave the mechanism that makes the choice safe (a quorum, fencing). The design question is per operation: a product read can serve stale data from the minority side; a payment cannot. Write down which operations are which before the partition, because during one nobody is thinking clearly.
The partition's cousin is asymmetric failure: A can reach B, B cannot reach A, and each side's detector says something different. Health checks that only test one direction miss it; the symptom is requests that arrive and responses that do not.
Cascading failure
One instance of a service slows. Its callers' threads wait on it; their pools fill; they slow; their callers' pools fill. Within a minute, services that never talk to the slow one are down, because they share a caller with something that does. The microservices course's gateway project reproduces this on purpose, and the mechanism is always the same: a resource shared by unrelated work — a thread pool, a connection pool, a queue, a CPU — is filled by the failing work, and the healthy work starves behind it.
The second form is capacity collapse: a service at 80% load loses one of five instances; the remaining four get 100% each, exceed their capacity, slow, fail health checks, get removed, and the remaining three get 133% each. The load balancer's health check, meant to protect users from a bad instance, removes healthy instances that are merely overloaded, and each removal makes the next one likelier. A service with no headroom does not degrade; it collapses.
Metastable states
The nastiest shape, named in a 2021 paper (Bronson et al.) because it kept recurring: a system that is stable under normal load enters, through a trigger (a spike, a deploy, a partition), a state where a sustaining effect keeps it overloaded even after the trigger is gone. The classic: a cache node restarts, every request misses and hits the database, the database slows, requests time out, clients retry, the cache cannot fill because the database is too slow to answer, and the system stays in that state indefinitely — at a load it handled fine an hour earlier — until someone sheds load by hand. The retries are the sustaining effect; the trigger was a single restart.
The tell of a metastable state is that removing the cause does not fix it. The fix is always to break the feedback loop: stop the retries, shed the load, warm the cache from a replica, restart with a fraction of the traffic. Design-time, the question to ask of every feedback path is "if this fires under overload, does it add load?" — and if so, it needs a limit.
Amplifiers
The mechanisms that turn a small failure into a large one, each of which is also a reasonable feature on its own:
- Retries. Three retries turn one failed request into four; three services each retrying three times turn it into sixty-four at the bottom. The retry storm lesson measured it. Retries against an overloaded dependency are the single most common sustaining effect.
- Timeouts that are too long. A 30-second timeout holds a thread for 30 seconds per failed call; a pool of 200 threads is gone after 200 slow calls, which at 100 requests a second is two seconds. The nginx and gateway lessons: every timeout adds up along the chain and must be shorter than the caller's.
- Health checks that include dependencies. A liveness check that fails when the database is slow restarts every pod during a database incident, and the restart storm is added to the incident. The Kubernetes lesson's rule: liveness checks the process, readiness checks the dependencies.
- Synchronous fan-out. A request that calls five services in series has five chances to be slow and a latency that is their sum; in parallel, the maximum — and a thread per call, which is the pool problem again.
- Unbounded queues. A queue that never rejects grows until memory is gone, and then everything is gone; the backpressure lesson.
- Thundering herds. Every client with the same 60-second cache TTL expires at the same second; every consumer restarted by the same deploy rejoins at the same moment; every retry with the same fixed delay lands together. Jitter exists for this.
- Cache dependence. A cache with a 99% hit rate hides a database that can serve 1% of the traffic. When the cache goes, the database sees a hundredfold.
Levers
Each amplifier has a damper, and the resilient design is the set of them applied before the incident:
| Lever | Damps | The rule |
|---|---|---|
| Timeouts | held threads, latency stacking | every call has one; each shorter than its caller's; measured, not guessed |
| Retry budgets | retry storms | retries as a fraction of traffic (10%), with exponential backoff and jitter, only on idempotent calls, never on a 4xx |
| Circuit breakers | calling something that is down | open after a failure rate, half-open to probe, a fallback that is honest about being one |
| Bulkheads | one dependency taking a shared pool | a pool or semaphore per dependency, sized to its capacity |
| Load shedding | capacity collapse | reject early at a queue-depth or latency threshold, cheapest work first, with a 429/503 and Retry-After |
| Backpressure | unbounded growth | bounded queues that reject, propagated to the producer |
| Isolation | blast radius | cells, shards, or regions that fail independently; a deploy that touches one at a time |
| Degradation | all-or-nothing | a stale read, a smaller page, a disabled recommendation, chosen per feature in advance |
| Jitter | synchronised herds | on every TTL, delay and schedule |
| Headroom | collapse under a lost instance | N+1 or N+2 capacity, and autoscaling that does not outrun a slow start |
None of them is free: a timeout is a request you gave up on, shedding is a user you turned away, a breaker's fallback is a worse answer, a bulkhead is capacity you reserved for a dependency that may be idle. The system design course's trade-off lesson applies — say the cost — and the production course's capacity lesson is where the numbers come from.
Designing for it, then proving it
The design method: for every dependency, write down what the service does when it is slow (not just down — slow is worse, because it holds resources), what the timeout is, what the fallback is, and what the user sees. For every shared resource, name what shares it and whether one tenant can exhaust it. For every feedback path — retries, health-check-triggered restarts, cache refills, autoscaling — ask whether it adds load under overload. Then the chaos engineering lesson: inject the slow dependency, the lost instance, the partition, in a controlled way, and watch whether the levers held. A resilience design that has never been exercised is a hypothesis, and the incident is a bad time to test it.