The suspects

Suspect one is Sunday’s incident, and it has an alibi. The 10:52 call degraded the spectator map, and only the map: every event that reached the pipeline was handled perfectly all morning, and the trace confirms it. Every mat Case crossed after 10:52 reported cleanly. The 25K mat sits before the bridge anyway, and Case crossed it near 10:43, minutes before the room made any call at all.

Suspect two is the ingest, and here the logs produce a twist. During Sunday’s retries, one timing event really did arrive twice: bib 7733’s 30K crossing, delivered, retried, and delivered again. Two rows claiming one crossing is duplicate dataConcept · lights on your mapduplicate dataMultiple rows or records standing for the same real-world entity or event: one customer with two accounts, one event imported twice by a retry. Finding and merging them is deduplication, and it is hard because two records that merely look similar may be two different truths., the mirror image of Case’s problem. Too many rows instead of too few. And the system caught it: the ingest checks each event’s identity against what is already stored, saw the second 7733 for what it was, and dropped it. The defence worked. Which raises Finn’s uncomfortable question. If the machinery notices an event that arrives twice, why did nothing notice an event that arrived zero times?

Why the alarm never rang

The answer is in what checking actually checks. data validationConcept · lights on your mapdata validationChecking a value at several layers before it becomes stored truth. The client side helps the user fix mistakes before submitting, the backend authoritatively enforces the API contract and the product’s rules, and the database’s own constraints protect the stored structure. guards every gate an event passes through, and Sunday it did its job on a quarter-million of them: well-formed chip codes, plausible timestamps, mats that exist. Every test in that stack interrogates a value in hand. An event that never arrives never presents a value, passes no gate, and fails no check. Validation is a doorman, and a doorman cannot bounce a guest who never came.

So absences need a different kind of watcher, one that asks the completeness question on purpose: every runner past 20K should eventually show a 25K. Traversal, it turns out, has no such check. Nothing asked, so nothing noticed, and the hole sat in an official table for a day until its detector arrived by email, polite, baffled, and correct. Finn writes the check into the backlog before lunch. It will not explain this hole, but it will make the next one announce itself.

Two suspects down. The incident is innocent, the ingest is innocent and even mildly heroic, and before anyone drives out to inspect a mat, Priya asks the question every investigation reaches eventually. What about the backup?