Logs, metrics, and the wall of dashboards
Last lesson said teams build senses for the system they cannot see, and this lesson is those senses, one instrument at a time.
They come in two textures. One is written records of what happened, and the other is numbers tracking how the system is doing. The screens and bells are built on top of both.
What are logs?
LogsConcept · lights on your maplogA running text record of what a program did, written by the program itself, one stamped entry for each thing that happened. Kept for reconstruction rather than live reading, and searched by ID when a single order or user needs explaining. are the running commentary code writes about itself as it runs, one timestamped line per event, millions of lines per hour. Think of them as the system’s diary, a record it keeps of its own day.
You can read one cold:
14:02:11 INFO order ord_1042 created customer=cus_8814:02:12 INFO charge ch_9931 succeeded amount=145014:02:40 WARN payment retry attempt=2 order=ord_104314:03:29 ERROR payment failed order=ord_1043 err=timeoutLogs are the instrument for questions about individuals. What exactly happened to order 1043? Search the diary for its ID, and the lines that mention it come back, the whole biography, line by timestamped line.
At production volume, that search needs an industrial tool. The famous one is Splunk, a search engine for logs. When an engineer says “check Splunk,” they mean go interrogate the diary.
What is a metric?
The diary cannot answer every question. You can’t read a million lines to learn how the afternoon is going.
For that, systems keep metricConcept · lights on your mapmetricOne number about the system, recorded again and again on a fixed clock so it can be graphed. Cheap to keep, quick to compare, and blind to any single user, which is the trade it makes for seeing the whole.s: pure numbers about the whole system, sampled continuously over time.
- Requests per second
- Percentage of requests failing
- Queue depth (Module 9’s word, now on a graph)
- Memory in use
That is the division of labor. Logs tell stories about individuals and are testimony; metrics draw curves about populations and are statistics.
What is a dashboard?
Both textures feed the dashboardConcept · lights on your mapdashboardA screen of graphed metrics laid out so a healthy system and a sick one look different from across the room. The place engineers check before they check anything else.: a wall of live charts showing the vital signs at a glance. It is the actual decor of every engineering floor, and the first thing an engineer looks at when anyone asks “is something wrong?”
A healthy dashboard reads as boring wavy lines. Trouble reads as a line doing something lines shouldn’t.
What is an alert?
Nobody stares at dashboards at 3am. So systems ring bells.
An alertConcept · lights on your mapalertA rule that turns a metric into a page. When the number breaks a limit for long enough, someone is notified. Tuning those limits is the hard part, since a bell that rings too often stops being heard. fires automatically when a metric crosses a threshold. Error rate above 2% for five minutes means wake a human.
Waking a human is literal. Engineers call it getting paged, a word left over from the pagers doctors carried, and it means a phone call or a push notification built to get through do-not-disturb. It goes to whichever engineer is on call that week, and the expectation is an acknowledgment within minutes, at whatever hour the threshold crossed.
At 3am that phone lights up on a nightstand, and someone sits up, opens a laptop, and starts reading the same charts you just saw. The next two lessons follow that person.
The craft is calibration, choosing thresholds worth waking someone for. Page on everything and pages become wallpaper. Engineers learn to ignore them, and the one that mattered scrolls past unread. The failure mode has a name, alert fatigue (the boy who cried wolf, automated).
What is a health check?
The humblest instrument gets the last word. A health checkConcept · lights on your maphealth checkA URL a service exposes so other machines can ask whether it is working, usually answering with nothing more than an OK. Load balancers, pipelines, and monitors poll it constantly, and a failed answer is how a sick instance gets pulled from rotation. is a tiny endpoint, often /health, whose only job is to answer “I’m alive and functioning.”
The monitoring, the deploy pipeline, and Module 9’s load balancer all ping it constantly. When the load balancer skipped the frozen instance and sent your request to a healthy one, it was calling this, all along.
The mental model to remember
Logs are the diary, one timestamped line per event, searched (via Splunk) when an individual’s story needs explaining.
Metrics are the vital signs, numbers over time, drawing curves about the whole population.
The dashboard is the wall of those curves, arranged so trouble jumps out at a glance.
An alert is the bell (a threshold crossed, a human notified), calibrated carefully, because too many bells teaches people to ignore bells.
A health check is the heartbeat endpoint underneath everything, pinged constantly by machines that need to know “are you alive?”
You should now be able to pick the right instrument for a question. Logs answer for one order’s biography, metrics and dashboards answer for the afternoon’s health, and alerts cover the middle of the night.
Support reports: “A customer says their 2:03pm order charged them but shows no confirmation.” Which instrument does the investigating engineer reach for first, and why?
▼ answer the check to continue ▼