Single points of failure, redundancy, and failover

Here is how experienced architects actually read a diagram.

The question is not just “what does each box do?”, which you learned in Module 9.

It is also “which box, dying, kills everything?”

You have been circling this question for three modules now. Module 8 split the buildings, Module 9 counted the arrows, and last lesson watched one mighty machine fail as a unit. Today the question gets its name, and its two-part cure.

What is a single point of failure?

A single point of failureConcept · lights on your mapsingle point of failureAny piece of the system without an understudy, so its failure is the system’s failure. Machines qualify, but so do people, certificates, and shared credentials. Abbreviated SPOF in docs and postmortems, and hunted deliberately in every architecture review., written SPOF, is any component with no understudy. If it fails, the system fails with it.

The classic examples:

• The single load balancer all traffic enters through
• The single database holding the truth
• The single third-party payment provider, Module 6’s dependency at its scariest

Notice that the definition says component, not machine. Some of the most damaging SPOFs are not hardware at all:

• The one engineer who understands the billing system, a SPOF who takes vacations
• The one TLS certificate that quietly expires at midnight, a paperwork SPOF behind many famous outages
• The one API key or credential that everything shares

Hunting SPOFs is a formal activity. Architecture reviews literally walk the diagram box by box, asking the same question at each stop: “and what if this dies?”

Single points of failure hide

The dangerous SPOFs are rarely the obvious ones.

A system can look redundant on the whiteboard while sharing fate underneath:

• Two load balancers… both reached through one DNS entry that misconfigures once.
• Instances across three availability zones… all reading configuration from one store.
• Two “independent” services… both secretly calling the same internal account service on every request.

This is Module 9’s arrow lesson wearing its most serious clothes. A dependency shared by every path is a single point of failure, no matter how many copies of everything else exist.

A useful principle is that redundancy is measured along the request’s whole path, not box by box.

What is redundancy?

The cure’s first half is redundancyConcept · lights on your mapredundancySpares, on purpose: a second of whatever the system cannot afford to lose. Insurance arithmetic decides how much, weighing the cost of the copy against the cost of the outage it prevents. The catch is shared fate, since copies that can fail together count as one., the deliberate duplication of everything that matters.

It means two load balancers, or Module 9’s managed one, which already runs as several under one logo. It means a live replica of the database, Module 7’s recognition word, promoted to a starring role next lesson. And it means instances across availability zones, Module 8’s two-buildings paranoia revealed as this principle wearing regional clothes.

Redundant copies come in two arrangements worth naming:

• Active-active: both copies work all the time, sharing the load. Losing one means the other simply carries more.
• Active-passive: one copy works while the standby waits, ready. Losing the active one means the standby takes over.

Redundancy is insurance, and like insurance it is pure cost until the day it is everything. Whether to buy it is lesson one’s tension arriving on schedule, availability wrestling cost. The deciding question is how much an outage of this component would matter.

And the fine print from the hiding section applies. Copies only count if they do not share fate. Two replicas in the same rack are one flood away from being one replica.

What is failover?

A spare tire in the trunk does not steer the car.

The copy only helps if the system can switch to it, and that switching is failoverConcept · lights on your mapfailoverThe switch from a failed component to its understudy, ideally before users notice. Easy to draw and hard to trust, which is why the grown-up question is not whether a standby exists but when the switch was last practiced..

Watch one happen:

Failover can be automatic (machinery detects and switches on its own) or manual, where a paged human flips the switch. Automatic is faster; manual is simpler and less likely to switch by mistake. Critical systems commonly automate it and page the human anyway.

Have we practiced it?

Here is the veteran’s question that cuts through redundancy theater, the same question Module 7 asked about backups:

“Have we ever practiced the failover?”

An untested failover is a hope with a diagram. The replica may be misconfigured, the promotion may take minutes instead of seconds, and the applications may not reconnect. And nobody finds out until the real failure, the worst possible moment for a rehearsal.

Companies that mean it practice deliberately. Some go further and kill their own components on purpose, in production, to prove the switches work, a practice called chaos engineering. Yes, that is a real job, and its existence tells you how seriously the industry takes the gap between drawn redundancy and working redundancy.

A useful principle is that redundancy you have not tested is a diagram, while redundancy you have tested is a capability.

The mental model to remember

A single point of failure is any component (machine, service, provider, credential, or person) whose failure takes the system down.

SPOFs hide behind apparent redundancy as shared dependencies. Measure redundancy along the request’s whole path, not box by box.

Redundancy is deliberate duplication (active-active or active-passive) bought at real cost, justified by what the outage would cost instead.

Failover is the switch to the copy: detect, promote, redirect. Done well, seconds; untested, unknown.

The architect’s walk is three questions per box: what if this dies? Where is its copy? Have we practiced the switch?

You should now be able to take any architecture diagram and perform that walk, including on the boxes drawn with someone else’s logo.