What breaks

The back half of Thursday’s review is a hunt. Sam puts the architecture on the screen and the room goes looking for the things there is only one of.

Anything whose death takes the whole system with it is a single point of failureConcept · lights on your mapsingle point of failureAny component whose failure takes the whole system down: the one balancer, the one database, the one engineer who understands the billing code. SPOF in writing, and every architecture review hunts them box by box., SPOF on the whiteboard, and the hunt is explicitly box by box. The registrations database is one. The timing-ingest entry point is one. And, in a moment of honesty that changes the roadmap more than any technology, the room agrees that the person who understands the results-export code alone is one too. Single points of failure are not always machines.

The insurance

The cure is deliberate duplication: redundancyConcept · lights on your mapredundancyHaving more than one of everything that matters: a second balancer, a live database replica, machines across zones. Deliberate duplication as insurance, priced as the cost of the copy against the cost of the outage.. The registrations database has had a standby twin all along, a live copy holding everything, receiving every write moments after the original, waiting for a day it is usually never needed. The twin costs real money every month. The outage it prevents costs the race, and that arithmetic is the whole of the insurance conversation.

The switch itself is failoverConcept · lights on your mapfailoverThe switch from a failed component to its redundant copy, ideally automatic: health checks notice the death, the standby is promoted, traffic shifts. Seconds of blip instead of hours of outage, and the honest question is always whether it has been tested., and Sam’s question to the room is the one that separates resilience from theatre: when did we last actually pull the plug and watch the twin take over? The answer is Tuesday, during the freeze, on purpose, with the room watching. Nine seconds. The review moves on.

Where the twin lives

The twin’s address matters as much as its existence. Cloud providers carve the world into regionConcept · lights on your mapregionA geographic area where a cloud provider operates infrastructure. Choosing one affects latency, law, resilience, and cost: a product decision wearing a dropdown.s, and inside each region sit isolated availability zoneConcept · lights on your mapavailability zoneAn isolated infrastructure location within a region: separate power, cooling, and facilities, connected to its siblings by fast links. Spreading across two is the standard insurance against one building’s bad day.s: separate buildings, separate power, separate cooling, fast links between. Traversal’s database and its twin live in the same region and different zones, which is the standard answer to a specific nightmare: one building having one bad day. Two buildings, one truth, nine seconds apart. Harborview’s data lives in two places, and now you know exactly why.

That settles what breaks. Before Friday can say where all these machines actually are, the review owes Act II three words.