What Are AI Hallucinations? The AI Isn't Lying. It's Predicting.
When a hallucination gets reported, the instinct is to file it against the model. The fix is almost never in the model.
The instinct after the first hallucination report.
The AI feature ships. A user reports a wrong answer. Someone files a bug, pings the ML team, and waits for the next model version.
That reflex makes sense. It's also usually solving the wrong thing.
LLMs are not retrieval systems. They're prediction systems. When a model generates an answer, it's producing the most statistically likely sequence of tokens based on training data, not looking anything up. Most of the time, the most likely sequence happens to be correct. When it isn't, the model has no internal signal that something went wrong. It outputs the wrong answer with the same confidence it uses for right ones.
That's the base behavior of the system. The product question is what you build around it.
The spec question I ask before building any AI feature.
Is this feature asking the model to reason, or is it asking the model to recall?
Summarisation, tone rewriting, general Q&A on broad topics: the model is drawing on patterns from training, and a slightly wrong answer is usually not a trust-destroying event. The failure mode is low-stakes.
Policy answers, pricing information, and account-specific data are different, because a wrong answer there has a real user consequence. And the difference isn't which model you're using. It's whether the model is being asked to surface specific facts it cannot reliably know.
Reasoning over provided context is reliable. Recall from training data isn't. That distinction shapes the entire architecture decision.
RAG is not a default. It's a tradeoff.
Retrieval-augmented generation is the go-to response to hallucinations in high-stakes domains. Before the model generates a response, the system retrieves the relevant documents, live account data, or current policy text and passes them as context. The model reasons over known facts rather than recalling possibly-invented ones.
It's the right investment when the failure mode of a wrong answer is severe, the source of truth is structured and retrievable, and the latency and cost overhead fit the use case.
It's not always the right investment. RAG adds retrieval latency, infrastructure complexity, and an indexing problem you now own. For a use case where hallucination rate is low and the consequence of a wrong answer is minor, that investment may be a worse tradeoff than confidence thresholds, output validation, or prompting the model to say it doesn't know.
I try to document why I chose one approach over the other in the spec. If the model changes or the use case expands, that reasoning is what makes the decision legible to whoever picks it up next.
How I'd instrument hallucination rate.
"Hallucination rate" sounds like a clear metric. In practice it's one of the harder things to measure at scale.
For closed-domain tasks, where there's a ground truth you can check against, automated evaluation is tractable. Sample outputs, run them against the source document or database, flag divergences. That gives a directional signal and lets you track regression across model versions.
For open-domain tasks, you're largely working from user signals: explicit feedback like thumbs down or report buttons, implicit signals like a user immediately re-asking the same question, and manual review sampling. The sample size needed for statistical confidence at low hallucination rates is large. I'd build the logging infrastructure before you need the data, not after.
The guardrail I watch is downstream trust: does engagement with the AI feature decline over time? A hallucination rate that looks acceptable in evaluation can still be quietly eroding trust in ways that don't surface until you're looking at 90-day retention on the feature.
The question that changes how I spec AI work.
Not "how do we reduce hallucinations" but "how do we design the system so the model never has to guess."
Those are different problems. The first puts the burden on the model. The second puts it on the architecture. The architecture is the part I actually control.
Once that framing clicks, the spec questions shift. Not "what prompt reduces wrong answers" but "what does the model need in context to answer this correctly" and "what should the system do when that context isn't available." Those answers belong in the PRD, not in a future model upgrade ticket.