After the fire: debugging, root causes, and blameless truth

It is daylight now. Checkout has been stable since the 3:26am rollback. Now comes the harder, calmer work of finding out what actually happened.

This is where the curriculum’s oldest skills reconvene.

What is debugging?

DebuggingConcept · lights on your mapdebuggingWorking out why software is doing the wrong thing. A disciplined loop of recreating the failure, ruling out suspects, and confirming the culprit, guided by whatever the system recorded and by asking what changed. is the systematic hunt for why software misbehaves. Reproduce the problem, narrow the suspects, verify the culprit.

It runs on evidence, the logs and metrics from earlier in this module, and on the eternal first question, what changed?

What is a trace?

Debugging’s power tool in distributed systems is the traceConcept · lights on your maptraceA record that follows a single request through every service that handled it, showing how long each stop took. Where a log is one service’s account, a trace is the whole route, so it points straight at the slow or failing hop.: the linked record of one request’s complete journey, stitched across every service it touched, with timing at each hop.

A log tells one service’s side of the story. But a trace tells the whole conversation, which makes it observability’s third pillar, joining logs and metrics.

Mini-capstone A taught you to trace a request by hand. Production tooling does it automatically, for every request, and last night’s traces show exactly where order after order went to die.

What is a bug?

The quarry of the hunt is often a bugConcept · lights on your mapbugA mistake in code that makes the program misbehave. One common answer to why an incident happened, but only one of several, since configuration mistakes, capacity limits, and outside dependencies cause incidents at least as often., the code-level flaw itself, Module 3’s word, now met at its most consequential.

But seasoned investigators stay open-minded. Every incident traces back to something, and that something is often not code at all. It may be a config mistake, a capacity ceiling, or a dependency’s bad day.

What is root-cause analysis?

The investigation’s discipline is root-cause analysisConcept · lights on your maproot-cause analysisThe practice of asking why repeatedly, each answer becoming the next question, until the chain reaches something whose fix would keep the problem from coming back., asking “why?” past the symptoms until you hit something worth fixing.

Here is last night, interrogated.

What is the root cause?

Each “why” in that chain is a candidate fix. The root causeConcept · lights on your maproot causeThe underlying condition that allowed a failure, as distinct from the immediate trigger. Correcting it removes the entire category of incident, whereas patching the trigger buys only a reprieve. is the deepest one, the cause that, corrected, prevents the whole class of failure.

Here, the root cause is not the typo but the fact that a pool-halving typo could survive review and reach production unvalidated.

Fix the typo, and you have prevented one recurrence. Fix the validation gap, and you have retired the entire species of failure.

What is a postmortem?

All of it gets written into the postmortemConcept · lights on your mappostmortemA written review of an incident recording what happened, who was affected, why it happened, and what will change. Composed without assigning blame so that people report the truth.: the incident’s formal written account.

It has four standard sections.

• The timeline, what happened minute by minute
• The impact, who was affected and how badly
• The root cause, the why-chain written down
• The action items, the real payload, concrete changes that make recurrence harder

The blameless rule

The postmortem is governed by the culture’s most humane engineering convention, blamelessness. The document says “the config format allowed a silent typo.” It never says “Dave broke it.”

The point is not kindness for its own sake but accuracy. People who fear blame edit the truth, and edited truth writes sequels.

You have seen this muscle before. Module 11’s retro was the same practice at low stakes, and incidents are just retros with adrenaline.

Two report cards from Module 10

The postmortem’s action items close the loop on two words from Module 10, both making their final appearance.

Graceful degradationConcept · lights on your mapgraceful degradationFailing partially instead of completely, so the essentials keep working when a dependency or component fails. Incidents test a system’s degradation design for real, and postmortems grade it. is the idea that when one part fails, the rest should keep working in a reduced form rather than failing along with it. Last night it gets its report card, and the grade is poor. When payments slowed, checkout did not slow down with it. It stopped entirely.

So an action item asks the design question. Could checkout have accepted the order, told the customer the charge would follow, and queued the payment for when the provider recovered?

Module 9’s queue was built for exactly that, and the postmortem is the moment someone notices it was not used.

ResilienceConcept · lights on your mapresilienceA system’s ability to take damage and keep serving, measured after the fact by how quickly problems are noticed, contained, and kept from returning. Fewer incidents is not the measure. Cheaper ones is. is how well the system as a whole absorbs trouble, and after an incident it stops being philosophy and becomes three numbers.

• How fast was it detected? Last night, three minutes from the first failure to the alert, with no customer involved.
• How fast was it mitigated? Twenty-six minutes from alert to a stable checkout.
• Did it ever happen again? That number is decided by whether the action items get done.

Read that way, resilience is not the absence of incidents but their declining cost. A resilient team is not one that never gets paged. It is one whose incidents get shorter, cheaper, and rarer.

The mental model to remember

Debugging is the evidence-powered hunt (reproduce, narrow, verify) and the trace is its power tool, one request’s whole journey timed at every hop.

Root-cause analysis asks “why?” past the symptoms. The root cause is the deepest fix, the one that retires the species, not the specimen. The quarry is often a bug, but config, capacity, and dependencies star regularly too.

The postmortem writes it all down (timeline, impact, cause, action items) blamelessly, because blame edits truth and edited truth writes sequels.

Graceful degradation gets its report card; resilience becomes numbers. They measure how fast the incident was detected, how fast it was mitigated, and whether it ever recurred.

You should now be able to sit in a postmortem meeting and know exactly what belongs in the document, and what, and whom, doesn’t.