Why Language Models Hallucinate — The Mechanics and What Actually Helps

Confident fabrication isn't a bug — it falls straight out of next-token prediction. This piece takes the mechanism apart layer by layer: no truth term in the loss, lossy compression that fails at the edges, sampling that draws from the tail, and scoring rules that pay models to guess. Then it ranks the countermeasures that actually work: grounding, mechanical citation checking, constrained output, and uncertainty estimation.

Survey of Hallucination in Natural Language Generation

An exam you can't leave blank

Picture yourself taking a fill-in-the-blank exam with one unusual rule: you are not allowed to hand in a blank. Every question has to be filled with something plausible before you can move on. Working from fragments of memory and a feel for how the answer should sound, you write down "I think it was a name like this."

That is exactly the position a large language model is in. A hallucination is output that is not true but delivered in fluent prose that looks exactly like the truth. What makes it dangerous is that the false part doesn't degrade. A nonexistent paper title and a nonexistent API method arrive with precisely the same fluency as a correct one. The reader has nothing to go on.

And it isn't a bug. It follows directly from how the model was trained: predict the next word. Below I take the mechanism apart in four layers, then separate the countermeasures that work from the ones people assume work.

Two kinds, and don't mix them

Before designing anything, split the problem in two.

• Factuality hallucination: conflicts with the world. "The Eiffel Tower is in London."
• Faithfulness hallucination: conflicts with the input you provided. A summary picks up a number that wasn't in the source; a translation grows a sentence the original never had.

You detect these in completely different ways. The first requires checking against an external knowledge source; the second only requires checking against the input text. Most incidents in summarization, translation, and meeting-note pipelines are the second kind — and bolting on a web search as a "fact check" catches none of them. Decide which one you're fighting before you write a line of code.

The intuition: the objective has no slot for truth

Strip pretraining down and one thing is happening. Pour in an enormous amount of text, make the model assign a probability to the next token at every position, and penalize it in proportion to how wrong that probability was.

Here $x_t$ is the $t$-th token (a word fragment), $x_{<t}$ is everything that came before it, and $p_\theta$ is the probability the model — with parameters $\theta$ — assigns. All the formula says is: assign high probability to the token that actually came next and the loss is small; assign low probability and get punished hard. Where this loss comes from is covered in Entropy and Cross-Entropy.

Now notice what isn't there: no term anywhere measures whether the sentence is true. The model is maximizing plausibility, not truth. In the training data, the phrase "as shown in the paper" was almost always followed by a real paper title. What the model learns is the pattern "a string shaped like a paper title goes in this position" — not the constraint "only write papers that exist."

Which is why the dangerous spots are always the same ones: proper nouns, numbers, dates, URLs, citations, part numbers. Every one of them has a highly regular form while its content is low-frequency information that has to be memorized case by case. When the form is learned and the content isn't, the model fills in the form.

Layer 1: compressed knowledge fails at the edges

Another way to see it is compression. The parameter count is finite, and the training data is squeezed into it lossily. Facts that appear constantly survive as thick pathways; a fact that appeared once gets absorbed into a nearby pattern and disappears.

This explains why the lies are usually near misses. A real author name paired with a real paper title in a combination that never existed. A date off by exactly one year. A function name blended with one from a different library. Outright nonsense is rare compared with substitution from the neighborhood, because that's what lossy compression does. From a verification standpoint this is nasty: near misses pass every plausibility check you can think of. Only actual lookup catches them.

Layer 2: sampling draws from the tail

Even after training, every generation adds another layer of risk. The model doesn't emit an answer — it emits a probability distribution over the whole vocabulary, and one token is sampled from it. Raise the temperature $T$ and the distribution flattens, making low-probability candidates easier to draw.

The key point is that the draw happens independently for every token. Even at a 0.5% per-token error rate, a 300-token answer makes "something went wrong somewhere" the normal outcome rather than the exception. Longer answers contain more errors not because the model tires out, but because it ran more lotteries.

And the crucial corollary: setting the temperature to zero does not remove hallucination. It only means the highest-probability token is always chosen — so if the model puts peak probability on a paper title that doesn't exist, that title now comes out deterministically, every single time. Temperature zero removes variance, not error. Any "safe mode" built on the assumption that temperature=0 means safe rests on that confusion.

Layer 3: the scoring rule pays for guessing

The third layer sits in neither training nor generation. It's in evaluation.

Most benchmarks award one point for a correct answer, zero for a wrong one, and zero for no answer at all. Under that scoring rule, the expected-value-maximizing strategy when you don't know is obvious: guess. If abstaining scores the same as being wrong, guessing strictly dominates, because guessing sometimes lands. It's the same arithmetic as filling in every bubble on a multiple-choice test with no penalty for wrong answers.

The same pressure shows up in post-training on human preferences. Put a confident answer next to "I don't know" and ask a rater to pick, and the confident one tends to win. Across the whole pipeline, then, the environment consistently rewards the model that guesses. Not saying "I don't know" is something we trained in.

Read in reverse, this also points at a fix. Build a scoring rule for your task where abstaining doesn't score the same as being wrong — give partial credit for "I don't know" — and both your model selection and your tuning start pulling in a different direction.

Models partly know what they don't know

There's some good news. Information about whether the model can answer correctly survives, partially, inside it. Ask the same question several times at a higher temperature: for things it actually knows, the answers converge; for things it doesn't, you get a different name every time. Using that variance as a detector is the idea behind self-consistency checking (the SelfCheckGPT family).

The implementation is almost embarrassingly plain.

def flag_hallucination(ask, question, n=5, thresh=0.6):
    answers = [ask(question, temperature=1.0) for _ in range(n)]
    base = answers[0]
    # do the other samples entail base's claims? (use an NLI model)
    agree = sum(entails(a, base) for a in answers[1:]) / (n - 1)
    return agree < thresh, base, agree

The cost scales linearly with n, but it needs no external knowledge base at all. The practical pattern is to apply it selectively — to the dollar amounts, the dosages, the statute numbers — wherever being wrong is expensive.

Fix 1: stop asking it to recall, make it read

The single highest-leverage change is to take recall out of the loop. Retrieve relevant documents, put them in the prompt, then ask the question. That's retrieval-augmented generation, and what it really does is turn a memory problem into a reading-comprehension problem. Comprehension you can check, because the evidence is right there.

But RAG is not magic, and a badly built one makes things worse.

• A retrieval miss amplifies the error. Handed an irrelevant document, the model will contort it into something that reads as supporting evidence. The hallucination now has a source attached, which makes it harder to spot than the ungrounded version.
• Being in the context doesn't mean being used. Information placed in the middle of a long context is reportedly more likely to be ignored than information at either end.
• When documents contradict each other, the model silently picks one and asserts it. It will not report the conflict.

The build details are in RAG Fundamentals and Design Patterns, and how to tell whether your version is actually helping is in Evaluating RAG in Practice.

Fix 2: require citations, then check them mechanically

This is the best return on effort in production.

1. Every answer must carry the document ID and the quoted span from the material you supplied.
2. After generation, verify by plain string matching that the quoted text actually exists in that document.
3. Flag or drop any sentence that fails.

The whole point of step 3 is that it uses no model. String matching is deterministic, fast, and incapable of hallucinating on its own. The worst possible version of this is asking for URLs: the model will happily assemble URLs that don't exist, with convincing domains and convincing paths. Restrict the output to IDs of documents you handed it and nothing else.

Fix 3: give it less room

Free prose is the format most prone to fabrication. Where you can, fix the shape of the answer in advance: a JSON schema, an enumerated set of choices, a fixed label set that includes "none of the above." An answer that isn't in the enum structurally cannot be produced.

For the same reason, don't cram several questions into one prompt. Split them and demand evidence for each — you get a finer grain to verify at.

How this shows up on the job

Who, and when Application engineers wiring up internal document search or support responses hit this the moment model output goes straight to a user. In medicine, law, and finance, where a wrong answer causes real harm, this stops being a quality concern and becomes the primary release criterion.

Parameters and tools you'll actually touch On the generation side: temperature, top_p, and logprobs (per-token log probabilities) — proper nouns that squeaked through at low probability are exactly the spans worth flagging. On the retrieval side: top_k, chunk size and overlap, and the reranker. On the evaluation side: FActScore (split long text into atomic claims and verify them one at a time), TruthfulQA (how much of the folklore that fools humans does the model reproduce), and an NLI model for entailment checks. Most APIs expose a seed for determinism — that's a reproducibility tool, not a correctness tool.

Pitfalls that turn into incidents

• Don't call temperature=0 a "safe mode." It fixes the output; the error stays fixed too, and ships every time.
• Don't stop evaluating the moment RAG is in. A retrieval system with poor precision produces more confident wrong answers than the bare model.
• Don't measure faithfulness failures with a fact checker. Whether a summary drifted from its source is answerable only by comparison against the source, never by external knowledge.
• Don't leave factuality scoring entirely to an LLM-as-a-judge. The judge draws its misconceptions from the same distribution.
• Don't count "please include references" as a mitigation. With no evidence supplied, the references get generated along with everything else.
• Don't leave output length unbounded. Same model, longer answer, more claims — and the error opportunities scale with the claim count.

How it gets asked in interviews and design reviews "Does setting temperature to 0 stop hallucination?" — No, and the follow-through is what matters: separate sampling variance from pretraining-induced error as two different layers. "Doesn't RAG solve it?" — Only while retrieval is hitting; the real question is whether your design specifies what happens when it misses (abstain? return a confidence score?).

Takeaways

• Hallucination isn't a bug; it's what next-token prediction optimizes for. The loss function has no truth term.
• The risk concentrates on proper nouns, numbers, dates, and citations — regular in form, low-frequency in content.
• Lowering the temperature doesn't remove it. It removes variance.
• In rough order of effectiveness: supply the evidence (RAG) → verify citations mechanically → constrain the output shape → measure uncertainty and let it abstain.
• Build a scoring rule for your own task that gives credit for "I don't know," and both selection and iteration start pointing somewhere better.