How to Reduce LLM Hallucinations: A Playbook

You ask the model a question. It answers in a clear, confident voice — names, dates, a citation — and it’s completely wrong. Not garbled: plausible, and false. That’s a hallucination, and it’s the single biggest thing standing between a demo that impresses and a product you’d trust in front of a user.

The good news and the bad news are the same fact: you can’t switch hallucination off, because it’s not a bug bolted onto the model — it’s how the model works. But you can stack a set of defences that take it from “unshippable” to “reliable enough, and honest about the rest.”

Here’s the playbook I actually use, and teach.

In one breath: You can’t eliminate hallucination — an LLM predicts plausible text, not true text. So you stack layers: ground it in real sources, constrain the prompt, let it abstain when it doesn’t know, tune the knobs, and verify before trusting — then measure what’s left. No single layer is enough; the stack is.

Why LLMs hallucinate, in one idea

Strip away the mystique and a language model is a next-token predictor. It was trained to continue text with the most statistically likely next word, over and over. It was never trained to check whether the result is true — truth was never in the loop.

That works astonishingly well when the answer is well-represented in its training. It fails silently when it isn’t: faced with a question it has thin or no knowledge of, the model doesn’t stop and say “I don’t know.” It does the only thing it can — it generates the most plausible-sounding continuation. A fabricated citation looks, statistically, exactly like a real one.

So “just tell it not to lie” can’t work as a fix. The model isn’t lying; it has no concept of truth to violate. Every real technique below does the same underlying thing: it changes the setup so the plausible answer and the true answer are forced closer together.

Pause & recall. Why is a made-up citation so convincing — and why does that mean scolding the model in the prompt won’t fix it? One sentence.

Reveal the answer

Because the model generates what’s statistically plausible, and a fake citation is shaped exactly like a real one — it has no truth signal to violate, so telling it “don’t make things up” changes nothing about the mechanism.

The playbook: five layers of defence

No single move removes hallucination. The reliable pattern is layered — each layer catches a different failure, and you rarely need all five at full strength. Reach for the ones your stakes demand.

Layer 1 — Ground it in real sources

The highest-leverage move: stop asking the model to answer from memory, and put the facts in front of it at answer time. Retrieve the relevant documents, hand them to the model, and tell it to answer from those — with citations, so the answer is checkable.

This is retrieval-augmented generation, and it works because it moves knowledge out of the model’s blurry weights and into text on the desk. Grounding responses in retrieved sources is widely reported to cut hallucinations by roughly 60–80% — the biggest single dial you have.

This is exactly what the Writing Bed does: it answers only from my blog posts and shows you which post each answer came from. If you want the mechanism from scratch — embeddings, retrieval, the lot — that post builds it end to end, and choosing grounding over baking knowledge into the weights is its own decision.

The honest limit: grounding is necessary but not sufficient. If retrieval fetches the wrong passage, or the model wanders off the passage it was given, you’re back to a confident wrong answer — now wearing a citation. Grounding raises the floor dramatically; it doesn’t close the door. That’s why there are four more layers.

Layer 2 — Constrain the prompt

The prompt is your cheapest, fastest line of defence, and the first place teams under-invest. Three instructions earn their keep on almost every task:

“Answer only from the provided context.” Ties the model to the sources instead of its memory.
“If the answer isn’t in the context, say you don’t know.” Gives it explicit permission to stop — the seed of Layer 3.
“Cite the source for each claim.” Makes drift visible to you and the reader.

Two more that help: put the important rules at the start of the prompt (that’s where they bite hardest), and ask the model to reason step by step before answering — working through intermediate steps catches errors a straight-to-answer leap would sail past. Where the output feeds other code, demand a structured format so a malformed or padded answer fails loudly instead of quietly.

None of this is glamorous, and all of it is compounding. A prompt that says “be accurate” is decoration; a prompt that names the sources, the abstain rule, and the citation format is a specification.

Layer 3 — Let it abstain (refusal is a feature)

This is the layer almost everyone skips, and the one I’d fight hardest to keep. A confident wrong answer is worse than “I don’t know.” The wrong answer costs a user their trust — and sometimes far more. The refusal costs them one extra click.

So design the “I don’t know” path deliberately, as a first-class outcome rather than a failure. Give the model an explicit escape hatch, and make refusing graceful — point the user somewhere useful instead of just shrugging.

Sprout is built around exactly this: ask it something outside what it actually knows about me and it declines, rather than inventing a plausible me. That refusal isn’t a rough edge — it’s the whole reason you can trust the answers it does give. A tool that answers everything is a tool that answers confidently when it shouldn’t.

Layer 4 — Tune the knobs

The cheap wins, often overlooked:

Lower the temperature for factual tasks. Temperature controls randomness; near the bottom of its range the model takes fewer creative liberties and sticks closer to the likeliest — and usually safer — continuation. Save the high settings for brainstorming, not facts.
Keep prompts focused. Models genuinely lose track of things buried in the middle of an enormous dump of context. Feeding the few relevant passages beats pasting everything — sharper and less prone to drift.
Pick a model that fits the job. A stronger model on a hard, high-stakes task hallucinates less than a weaker one you’ve over-asked. Match the tool to the difficulty.

None of these rewrite your architecture. They’re settings — and they move the needle before you write a line of new code.

Layer 5 — Verify before you trust

For anything high-stakes, don’t let the first draft be the final word. Add a checking step:

A second pass. Have the model (or a second one) check the answer against the sources — “is every claim here supported by this context?” Catching a contradiction is easier than avoiding one.
Surface the evidence. Return citations the user can click. You won’t catch everything; a checkable answer lets the reader catch the rest. Citations aren’t decoration — they’re how an honest system ships something that can still occasionally be wrong.
Human-in-the-loop where it counts. For medical, legal, financial or safety-critical output, a person reviews before it acts. Not every feature needs this; the ones where a wrong answer is expensive absolutely do.

You can’t fix what you don’t measure

Everything above is guesswork until you put a number on it. “It seems to hallucinate less” is not progress you can defend or repeat.

So build a small hallucination eval: a fixed set of questions — including ones you know fall outside the system’s knowledge — and score two things on every change: is each answer actually supported by its sources (faithfulness), and does it refuse the out-of-scope ones instead of improvising? Now a prompt tweak that helps is a fact, and one that quietly makes things worse is caught before it ships. I wrote a whole guide on building that eval from scratch — it’s the difference between tuning and hoping.

Match the defence to the stakes

You don’t need all five layers everywhere. Over-engineering a low-stakes feature wastes effort; under-engineering a high-stakes one is how you make the news.

Low stakes (an internal brainstorming helper): a good prompt and a sensible temperature may be plenty.
Medium stakes (a customer-facing assistant): ground it, constrain the prompt, and let it abstain.
High stakes (anything a wrong answer could harm): all of the above, plus verification and a human in the loop — and measure relentlessly.

The skill isn’t knowing the techniques. It’s reading the stakes and spending your defence budget where a wrong answer actually costs something.

What I watch teams get wrong

“Just prompt it not to hallucinate.” The model has no truth signal to obey. Change the setup, not the scolding.
Grounding once and declaring victory. Retrieval can fetch the wrong thing; the model can still wander. Grounding is a floor, not a ceiling.
No abstain path. If the system can’t say “I don’t know,” it will make something up. That’s not optional.
Never measuring. Without an eval you’re managing a number you can’t see.
Chasing zero. You can’t reach it. Aim for “reliable, and honest about the edges” — and know where those edges are.

Where to start

Pick your highest-stakes feature and do three things this week: ground it in real sources with citations, give it an explicit way to refuse, and write twenty test questions — some answerable, some not — to measure it. That trio removes most of the damage. Add temperature tuning and a verification pass as the stakes climb.

Hallucination isn’t a flaw to be ashamed of; it’s a property to be managed. The engineers whose systems people trust aren’t the ones who eliminated it — they’re the ones who built around it honestly, and can tell you exactly where the edges are.

Check yourself.

Why can’t a prompt instruction like “don’t make things up” fix hallucination on its own?

Grounding is powerful but “necessary, not sufficient” — name one way a grounded system still hallucinates.

Why is a graceful “I don’t know” worth designing as a first-class feature?

What two things should a hallucination eval measure?

If any answer won’t come, that’s your reread map — the layers above hold all four.

Want to see two of these layers running? The Writing Bed grounds and cites; Sprout refuses when it should. Poke both in the Greenhouse and watch for the moment each one declines — that’s the playbook working. 🌱

When the model makes things up