When AI is wrong (and why testing it is strange)

Ask the coffee assistant for a gift-card balance and it might answer, fluently and specifically: “$23.50, expiring March 3rd”, for a gift card that does not exist.

There is no error, no hesitation, and the grammar is perfect.

What is a hallucination?

That answer is a hallucinationConcept · lights on your maphallucinationA model producing confident, fluent, false output, such as invented facts, citations, balances, or API fields. Not a malfunction, since the same likelihood-driven generation behind every correct answer produces the wrong ones with identical polish. Teams contain it, and nothing removes it entirely.: confident, fluent, false.

Last lesson armed you to understand why it isn’t a bug in the normal sense. Generation picks likely tokens, and nothing in the loop checks facts. The model has read thousands of balance-shaped sentences, so producing another one is easy. A plausible balance is likely-shaped whether or not it is true.

Nothing inside the model could have checked the claim. It holds no gift-card records and no account database, only the frozen numbers, so the figure came out of the same guessing that produced the rest of the sentence.

The machinery that writes brilliant summaries and the machinery that invents gift cards are the same machinery, having the same kind of moment. That is why hallucination is managed, never fully eliminated.

Same input, different outputs

Underneath sits the deeper property: probabilistic behaviorConcept · lights on your mapprobabilistic behaviorThe same input can produce different outputs across runs; the model samples from likelihoods rather than following fixed rules. The break from all software before it: delivery_fee(19, True) is 0 forever; “ask the assistant” is a distribution, not a value.. Each fragment of the answer is drawn from a weighted list rather than settled in advance. Run the same prompt twice, and the draws can land differently, so the answers can too.

Last lesson’s loop looked like it always took the top candidate, and it usually does. The percentages are not just rankings, though, but odds. A fragment scored at 92% wins the draw roughly 92 times in a hundred, and a 5% fragment takes its five, so a lower pick is never an override, only the odds paying out.

That is deliberate, since a model that always took the safest word would read as stilted and repeat itself.

Why AI testing had to change

That distribution collides with the way testing has worked since Module 12. A test names one input and demands one exact answer, as assert delivery_fee(19.00, True) == 0 does, and it passes or fails identically until the code changes. Every test in the pyramid stands on that certainty.

Now try to write it for the assistant. There is no “the” answer to assert, only a spread of answers, most of them excellent and a few invented. A test demanding one exact reply fails tomorrow’s differently worded, equally good one, so the assertion has nothing to grab.

Loosening the test does not save it either. A single run is one draw from the spread, so passing once proves almost nothing. A feature that invents a gift card four times in a hundred will pass five test runs in a row and ship, and the odds collect from real customers instead.

What is evaluation?

So the industry built a statistical replacement. Enter evaluationConcept · lights on your mapevaluationAI’s answer to the test suite. Hundreds of realistic cases get scored, by rubric, by automated checker, sometimes by a grading model. Quality lands as percentages held against thresholds and is rerun whenever the prompt moves, the way CI reruns tests. “Evals” in every AI team’s daily speech., “evals” in the hallway.

An eval runs the feature against hundreds of representative cases and scores every output. Rubrics handle some of the scoring, automated checkers handle more of it, and sometimes a second model serves as grader.

A grader model sounds circular, but judging is the easier job. The grader holds the case’s expected answer and only checks whether the output agrees, and graders are themselves spot-checked against human judges. What comes out the far end is a set of rates.

It is Module 12’s pyramid rebuilt for distributions: run on every prompt change, gating releases, graphed over time like Module 13’s dashboards.

What are guardrails?

GuardrailsConcept · lights on your mapguardrailsPlain deterministic code wrapped around the model’s replies. It checks claims against the database, forces a parseable shape, filters unsafe content, and turns away requests beyond the feature’s scope. Output now earns the suspicion input has always gotten. wrap around the model itself. They are everyday deterministic code, doing what it has always done.

Take the gift card from the opening. The model claims the balance is $23.50, and the guardrail checks the actual database before the answer ships. If no such card exists, the invented answer dies in the wrapper. The user never sees the invention. Ordinary code hands them a fallback instead, an honest “I can’t find that gift card.”

Guardrails come in a few standard shapes:

• Validate claims against real data. Does that gift card exist?
• Constrain the format so the reply is valid, parseable output
• Filter unsafe or off-brand content
• Refuse out-of-scope requests entirely

This is Module 5’s oldest instinct, never trust input, extended one step further. Now never trust output either, and verify what can be verified.

What is human review?

And for stakes no rate can cover, human reviewConcept · lights on your maphuman reviewKeeping a person between the model and any consequence that money, health, or law cannot take back. The model drafts while a person approves, and the right question asks where people still belong once error rates are measured, not whether software has earned their removal. keeps a person in the loop. The AI drafts the refund email, and a human clicks send.

The pattern shows up wherever errors are costly or irreversible: refunds, medical, legal, publishing. And it is Module 10’s tradeoff discipline matched to measured error rates, not a failure of nerve. The design question is never “is the AI good enough to remove the human?” but “which decisions deserve a human, given the rates we measured?”

The mental model to remember

A hallucination is confident, fluent, false, the normal machinery having a normal moment. It is managed, never fully eliminated.

Model behavior is probabilistic, and the same input yields a distribution of outputs. The bedrock promise of every earlier module (same input, same output) bends here.

So testing became evals: hundreds of scored cases, quality as rates against thresholds, run on every change like CI.

Guardrails wrap deterministic checks around probabilistic output. Never trust output either, and verify what can be verified.

Human review matches autonomy to measured stakes, weighing error rate against consequence.

You should now be able to hear “the evals say 96%” and ask the two questions that matter: 96% of what, and what does the other 4% cost?