Guide · Reliability, Security & Trust

How do you stop an LLM from hallucinating?

Published 1 August 2026

You cannot stop a language model from hallucinating. You can make hallucinations rare, cheap to detect and impossible to ship unnoticed. Every control that works is one of three kinds: remove the need to guess, make guessing visible, or make the consequence of a guess harmless. Anything that is only a plea in the prompt — "do not make things up" — belongs to none of them and buys almost nothing.

First, classify what you are seeing

The controls differ per type, so triage before you fix:

  • Missing-knowledge fabrication. The answer was never in the context and the model produced a plausible one. Fixed by retrieval and by permitting refusal.
  • Context-contradicting. The right fact was in the context and the answer contradicts it. Usually an attention problem: overlong context, buried detail, or competing instructions. Fixed by trimming and by citation.
  • Format hallucination. Invented field names, invented enum values, invented tool arguments. Fixed by schema validation, not by prompting.
  • Confident arithmetic and dates. Numbers that look right and are not. Fixed by moving the computation into code.

Treating all four as one problem is why "we improved the prompt" rarely moves the error rate.

Remove the need to guess

  • Put the facts in the window. Retrieval is the single largest lever. A model asked about a document it can read is doing extraction; the same model asked from memory is doing recall, and recall is where fabrication lives.
  • Make refusal a legitimate, specified output. Not "say you don't know if unsure" — an explicit branch: {"answer": null, "reason": "not_in_context"}. Models fabricate partly because every path in your schema demands an answer.
  • Narrow the question. "Summarise this contract" invites invention. "List each termination clause with its section number, or return an empty list" does not.
  • Cut the context you do not need. Contradicting-the-context errors climb with irrelevant volume. Ten relevant chunks beat fifty padded ones.

Make guessing visible

  • Require verifiable citations. Demand a quoted span plus a locator (section, line, chunk id) for each claim, then check the span exists in the source with string matching. This is the highest-value control in the whole list because the check is code, not judgement.
  • Validate every structured output against a schema. Reject unknown fields and out-of-set enum values rather than coercing them. An invented field silently accepted is an invented field in your database.
  • Check entities against a real list. Product names, SKUs, table names, user ids — if you have the authoritative set, intersect against it and treat non-members as failures.
  • Run the answer twice on high-stakes paths. Where two independent samples disagree, escalate. Disagreement is a surprisingly reliable proxy for fabrication. It is only cheap under two conditions: the shared prompt prefix is cached, so the second call is a cache read rather than a second full prefill; and sampling is not deterministic, since two greedy samples of the same input agree by construction and tell you nothing.

Make the consequence harmless

  • Never let the model be the last step before a side effect. A write, a payment, an email — put a deterministic check or a human between the generation and the effect.
  • Move computation out. Totals, dates, percentages: have the model produce the inputs and let your code do the arithmetic. Then a hallucinated number cannot exist.
  • Surface provenance in the UI. An answer shown next to its source is one a user can falsify in two seconds. An answer shown alone is one they will trust.
  • Fail closed. When validation fails, show "couldn't verify this" rather than the unverified answer. The unverified answer is the expensive outcome.

A worked example

A support assistant answering from a policy handbook fabricated a 30-day refund window; the real policy is 14 days. Prompt-side attempts — "be accurate", "only use the handbook" — reduced nothing measurably.

What fixed it, in order of impact: retrieval scoped to the refunds section instead of the whole handbook; an output schema of { answer, quote, section, confident }; a code check that quote appears verbatim in the retrieved text; and a UI that renders the quote under the answer. The fabrication did not vanish from the model's output — it now fails the quote check and never reaches the user. The residual error rate became measurable, because every rejection is logged.

What does not work

  • Asking the model how confident it is, and trusting the number. Self-reported confidence is generated text, not a calibrated probability.
  • Asking the model to check its own last answer in the same conversation. It will defend it more often than it corrects it.
  • Threatening or emphatic instructions. Measurable effect close to zero once retrieval and validation are in place.
  • Turning temperature to zero. It makes output repeatable, not true — you get the same fabrication every time.

Measure it, or you are guessing too

Keep a fixed set of cases with known answers, including cases whose answer is "not in the source", and score them on every prompt or model change. Two numbers matter: the fabrication rate on answerable cases, and the false-refusal rate on the unanswerable ones. Every control above trades one against the other, and without both numbers you cannot tell whether a change helped.

Where this fits

This is a reliability problem, so it sits in the Reliability, Security & Trust part of the track alongside evals and guardrails. The judgement being tested is whether a proposed control is enforceable in code or merely stated in a prompt — and how the system behaves when the control fires.

Keep going