When the pager goes off

At 3:07am, the error-rate alert crosses its threshold, and somewhere a phone detonates on a nightstand.

This lesson is about what happens next, the vocabulary of the emergency and the humans it belongs to.

What is an incident?

An incidentConcept · lights on your mapincidentThe formal name for a live problem affecting users. Calling it one is a deliberate act that starts a response with assigned roles and a dedicated channel, and its severity level sets how many people wake up and how quickly. is any unplanned event degrading service for users, anything from “checkout is slow for some customers” to “everything is down.” It is the industry’s word for production is hurting users, now.

The word is declared out loud (“we have an incident”) because saying it activates a process. Roles get assigned, a dedicated channel opens, urgency rules take over.

Incidents are graded by severity. SEV1 is the worst, meaning drop everything. SEV2 is serious, and SEV3 and below are progressively calmer. The grade decides who gets woken and how fast.

What is an outage?

An incident’s severest form is the outageConcept · lights on your mapoutageA total loss of service, as opposed to a slowdown or partial failure. The most serious kind of incident, and the kind that ends up in news headlines.: down, unavailable, not merely degraded. It is the “is it down for everyone or just me?” headline event.

Module 10’s redundancy (the replicas, the zones, the failovers) exists precisely to make outages rare and short.

Who is the on-call engineer?

The phone on the nightstand belongs to the on-call engineerConcept · lights on your mapon-call engineerWhoever currently holds the duty of answering pages for a service, on a schedule the team shares. Not a permanent role but a turn everyone takes, and the person a 3am alert reaches first.. It is a rotation, not a job title. The pager passes around the whole team, typically a week at a time. This week, it is Mira’s.

While carrying it, she is the first to be woken and expected to respond within minutes. This is Module 11’s “code ownership” revealing its 3am meaning. The owning team carries the pager for what it owns.

What is a runbook?

Mira’s first companion is the runbookConcept · lights on your maprunbookA step-by-step guide for handling a specific alert, prepared before any emergency so a tired responder can follow it without thinking. A stale one is worse than none, so good teams rehearse theirs.: the service’s break-glass manual, written in advance by calm daytime engineers for their panicked future selves.

Its shape is nearly always the same. Symptom, checks, remedies.

Runbooks carry the same warning Module 7 attached to backups. One that has never been tested may not work when it is needed. Systems change, commands go stale, and a team that never rehearses its runbook finds that out at 3am.

What is escalation?

If the runbook doesn’t cover it, Mira escalatesConcept · lights on your mapescalationBringing in additional people, usually a specialist, the owning team, or leadership, once an incident outgrows the responder who has it. Expected practice, and treated as a skill rather than a failure.. She wakes the database specialist, the payments owner, whoever the incident demands.

No shame attaches to this. Knowing when to escalate is a listed competency on many engineering job ladders. If shame attaches to anything, it is the opposite, riding a SEV1 alone at 4am out of pride.

Who is the incident commander?

When an incident grows teeth, structure arrives. The incident commanderConcept · lights on your mapincident commanderThe single person running a serious incident, coordinating and communicating rather than fixing. Their value is in holding the big picture while everyone else keeps their head in the details. is one designated person who coordinates. They track the whole picture, assign workstreams, and make the calls, starting with rollback or fix forward. That call is Module 12’s fork, now under fire.

Two things the commander is not. They are not necessarily the most senior person in the room, and they are explicitly not the one typing fixes. Their hands stay off the keyboard so their head stays on the incident.

They also own communication, the steady drumbeat of status updates that keeps stakeholders informed and away from the responders. The role is borrowed from actual emergency services because it works.

Who is the service owner?

Behind every service stands its service ownerConcept · lights on your mapservice ownerThe team accountable for a service over its whole life, including its on-call duty, its documentation, and its upkeep. The answer to the question of whom to call.: the team whose name is on the tin, answerable for its health long-term, its pager, its runbooks, its debt, its capacity planning.

Within minutes, nearly every incident asks the org-chart question. “Wait, who owns payments?” Mira could answer that one at 3:14, which is why Sam was awake by 3:18. If nobody can answer, that silence is itself a finding.

The choreography of 3am

The whole response follows one sequence.

• Detect. The alert fires. (Not a customer email, if Module 13 has gone well.)
• Triage. How bad is it, and who is affected? The severity grade comes from here.
• Mitigate. Stop the harm. Flag off, roll back, scale up.
• Communicate. Status updates on a cadence, so the responders can work.

One word is missing from that list. Understand. The 3am job is to stop the harm, not to explain it. Understanding is tomorrow’s job, done in daylight, and it is the next lesson.

Here is the whole choreography, as the incident channel records it.

The mental model to remember

An incident is production hurting users, declared out loud because the word activates a process, and graded by severity. An outage is its severest form, down rather than degraded.

The on-call engineer is this week’s pager carrier. The runbook is the break-glass manual written for them in advance. Escalating (early, unashamed) is a competency, not a confession.

The incident commander coordinates and communicates without touching the keyboard. The service owner is the team answerable for the service between fires.

The sequence is detect → triage → mitigate → communicate. Stop the harm at 3am; understand it in daylight.

You should now be able to read “SEV1 — Sam is IC, Mira’s rolling back, updates every 15” and see it for what it is, not chaos but a rehearsed choreography, mid-performance.