The four signals (and the numbers on the contract)

Of all the numbers a system could report, four earn permanent spots on the dashboard. You already met each one as an idea. This lesson is where Module 10’s vocabulary becomes Module 13’s needles.

What is the error rate?

The error rateConcept · lights on your maperror rateThe share of requests that came back as failures, tracked over time. A jump in this number means users are already being hurt, which is why most alerts watch it and why anyone checks it first. is the percentage of requests failing, Module 4’s status-code families graphed. What fraction of the system’s answers are 5xx, the server-side-failure family?

It is the bluntest of the four signals. The others hint that something may be going wrong. A spike here means requests are failing for real users right now, so it is not a warning sign of the problem but the problem itself. “What’s the error rate?” is the first question of every incident.

Latency, read at the percentiles

LatencyConcept · lights on your maplatencyHow long a single request takes to be answered. Reported at percentiles rather than as an average, since p99 (the slowest one in a hundred) reveals the experiences an average hides, and at scale that one in a hundred is a crowd. (Module 10’s tradeoff word) becomes a live needle here. For the coffee app it is the wait between a customer tapping Place order and the confirmation appearing. Engineers rarely watch its average because Module 10 warned that averages hide the worst experiences.

Instead they watch its percentiles.

• p50 is the typical request. Half are faster, half slower.
• p99 is the time the slowest 1% of requests exceed, the unluckiest users.

Both numbers exist at every moment. The p99 never “happens”; it only reads higher or lower. On a healthy afternoon the coffee app might show a p50 of 200 milliseconds and a p99 of 400.

Now let the p99 drift to four seconds while the p50 stays put. Most customers notice nothing, but one in every hundred is staring at a spinner long enough to give up. At scale, one percent is a stadium full of people. Hence the p99 obsession.

So the p99 gets its own alert threshold. When it trips, the first question is what changed. Most production problems follow a change, so if new code was deployed minutes before the number climbed, that deploy is the prime suspect and rolling it back is the quickest fix.

If nothing was deployed, the second question is where the slow requests spend their time. Two lessons from now, a trace shows exactly that, one hop at a time.

The fix is usually one you already know from Modules 7 and 10, such as a missing index, a saturated dependency, or a cache that stopped hitting.

Throughput, read against its rhythm

ThroughputConcept · lights on your mapthroughputHow much work the system is handling per unit of time, such as requests or orders per second. Judged against what is normal for that hour, so a sudden quiet can be as telling as a sudden surge. is the volume needle, whether requests per second, orders per minute, or messages drained from the queue. Where latency measures how long one request takes, throughput measures how many are getting through.

This one is read against its usual daily rhythm. A coffee app’s throughput curve is dinner-rush shaped, every single day, so a curve that stays flat at dinner time is as alarming as any spike.

Nothing is erroring and nobody is ordering, which means the trouble sits before your servers. Customers cannot reach checkout at all, because the mobile app is crashing on launch, the front door from Module 9 is misrouting, or the payment provider’s page is failing before the order is ever sent.

The other direction matters too. A surge well above the usual rhythm, from a promotion or a viral post, is not a problem by itself, but it is the moment to glance at the fourth signal, because every extra order is drawing on the same finite resources.

What is saturation?

SaturationConcept · lights on your mapsaturationHow close each resource is to its ceiling, whether CPU, memory, connections, or queue backlog, expressed as a percentage. It moves before users feel anything, which makes it the early-warning needle. (first met in Module 10) measures how full each resource is. CPU, memory, database connections, and queue depth are each shown as a live percentage of their limit.

Every resource has a ceiling, and the ceiling is where things break. A processor at 97% has no room left for the next request, so requests begin to wait. A full connection pool means new queries queue behind old ones. A queue that grows every minute means the workers draining it have fallen behind.

None of those is a failure yet, but each is a failure a few minutes away.

That is why it is the clairvoyant of the four. Error rate and latency tell you the system is already hurting; saturation warns that it is about to.

And because it moves first, it is the signal you can act on before customers notice. The usual moves are adding capacity where the gauge is highest (Module 10’s scaling) or shedding the least important work until the rush passes.

Here are all four on one wall.

The promises above the needles

Above the four live signals sit the promises.

UptimeConcept · lights on your mapuptimeThe fraction of a period during which the service was reachable, usually shown as a percentage with a string of nines. Public status pages report it, and contracts are written against it. and downtimeConcept · lights on your mapdowntimeTime the service was unreachable. The other side of the uptime percentage, and the part customers remember. turn availability into a scoreboard. The percentages are Module 10’s nines, now kept in public. Many companies publish theirs on a status page any customer can visit and watch stay green.

Two recognition terms name the two flavors of promise. They sound alike and differ where it counts.

• An SLORecognition — just know it existsSLOService-level objective. A target a team sets for itself, such as a latency or availability number it aims to hold. Missing it draws internal attention, not penalties. (service-level objective) is the team’s internal target, such as “99.9% of checkouts complete in under two seconds.”
• An SLARecognition — just know it existsSLAService-level agreement. A commitment made to customers in a contract, usually an uptime figure, with credits or refunds owed if it is broken. The version of a target that carries financial consequences. (service-level agreement) is the same shape made into an external contract, with refunds or penalties if it’s missed.

The O is a goal; the A is a promise with money on it. When an enterprise deal “requires a four-nines SLA,” Module 10’s cost curve has just entered the sales negotiation, and now you can follow both sides of that table.

The mental model to remember

Error rate is the fraction of requests failing, the alarm itself, and the first question of every incident.

Latency is read at the percentiles, p50 for the typical user and p99 for the unluckiest 1%, a stadium full of people at scale.

Throughput is volume read against its expected rhythm, where flat at dinner time is a warning in its own right.

Saturation is how full the tanks are, the clairvoyant signal, worth acting on before the other three turn.

Uptime is the public scoreboard; an SLO is the internal target; an SLA is the external promise with money on it.

You should now be able to glance at the four charts and read them like an operator, including the afternoon when three are green and the fourth is quietly telling the future.