Wrong, with confidence

Tuesday, 8:40 a.m., the support queue produces exhibit A. A runner asks for the two-year entry deferral with a doctor’s note, “as your assistant described,” and attaches the chat. The assistant’s answer is warm, specific, well written, and describes a policy Harborview has never had.

Nobody programmed that answer, and nothing malfunctioned to produce it. It is a hallucinationConcept · lights on your maphallucinationA model generating confident, fluent, false output: invented facts, citations, policies, API fields. Not a malfunction. The same likelihood-driven generation that produces correct answers, producing an incorrect one with identical confidence. Managed, never fully eliminated., and last night’s slow-motion view already explained it. Deferral was not in the pasted excerpt, likelihood alone chose every fragment, and a deferral policy is exactly the kind of thing that is likely to exist. The model did what it always does, and it was fluent. Fluency and truth are different axes.

A distribution, not a value

Worse for the tidy-minded: ask the assistant the deferral question five times and you get five differently worded answers, some of them correct refusals. That is probabilistic behaviorConcept · lights on your mapprobabilistic behaviorThe same input can produce different outputs across runs, since the model samples from likelihoods rather than following fixed rules. The break from all software before it: the same function call returned the same fee forever, and “ask the assistant” is a distribution, not a value., and it quietly breaks the whole testing world Act II built. Act II’s tests asserted that a known input yields the known output, forever. Here no single expected string exists, and any single run proves almost nothing. You do not ask whether the model is right. You ask how often.

Asking how often has a discipline: evaluationConcept · lights on your mapevaluationAI’s version of the test suite: run the feature against hundreds of representative cases, score the outputs with rubrics, checkers, sometimes a second model as grader, and track the rates. Re-run on every prompt change, exactly like CI. “Evals” in every AI team’s daily speech., and the data team’s evals are what turn both mysteries into engineering. The assistant: correct on the covered questions 96 times in a hundred, inventive on the uncovered ones. The projection: within six minutes for most of the field, with a thin, measured tail of nonsense, and the 3:58 is simply the tail, arriving on schedule, for one runner in 25,000. Ask the projection twice and it says 3:58 twice. Its spread is across runners, not across runs. Nobody can open the model and delete the wrong answer. There is no line to fix. There are rates, and the rates were known.

What you do about it

What was missing is the deterministic fence around the probabilistic middle: guardrailsConcept · lights on your mapguardrailsDeterministic checks wrapped around probabilistic output: validate claims against real data, filter unsafe content, constrain format, refuse out-of-scope requests. Validation’s old rule, never trust input, extended to its successor: never trust output either. Verify what can be verified.. A projection that beats the runner’s own halfway arithmetic is checkable with a division, and from this week it is checked: implausible projections are clamped before anyone’s family sees them. The assistant gets its fence Wednesday, and the deepest fence of all is already running upstairs. human reviewConcept · lights on your maphuman reviewKeeping people in the loop where errors are costly or irreversible: the model drafts, a human approves. The design question is never “is the AI good enough to remove people?” but “which decisions deserve a person, given the error rates we measured?” is why the vision model that matches race photos to bibs sends every tag it is unsure of to a queue for human eyes before the photo reaches a finisher’s page. Wrong tag, wrong runner, real feelings. That decision deserves a person.

Rates measured, fences built, people placed. One question left from the deferral mess. If likelihood invents what it does not know, the cure is making the truth the likeliest thing in the window, and that is Wednesday’s work.