Three ways to give a model your knowledge
A model out of the box knows a lot about the world and nothing about yours. Your product docs, your policies, your writing, last week’s numbers — none of it is in there.
There are exactly three ways to fix that, and teams argue about them constantly: retrieval-augmented generation (RAG), fine-tuning, and long context. Most “RAG vs fine-tuning” posts leave the third one out, then quietly recommend whatever the author sells.
Here’s the honest version, from someone who’s shipped all three and teaches the trade-off to engineering teams every week.
In one breath: Reach for RAG when the answer depends on knowledge that’s private or keeps changing and you want citations; fine-tuning when you need the model to behave differently — a voice, a format, a domain’s reasoning; and long context when the whole corpus is small, fixed, and you just want to move fast. Most production systems end up combining them.
Let’s earn that summary.
What each one actually changes
The confusion comes from treating these as three flavours of the same thing. They’re not — each changes a different part of the system.
RAG changes what the model knows right now
At query time you find the few passages relevant to the question and paste only those into the prompt. The model’s weights never move; you’ve just handed it the right page of the open book. Add a document and it’s usable instantly; every answer can point back to its source. I pulled this apart from first principles in the Writing Bed post — that’s RAG end to end.
Fine-tuning changes how the model behaves
You train on your own examples and the weights actually shift. This is the right tool when you need a behaviour the base model doesn’t have — a house style, a strict output format, the reasoning patterns of a specialised domain. What it’s not good at is being a fact store: knowledge baked into weights goes stale the day you publish something new, and it can’t tell you which document an answer came from.
Long context changes what fits in one call
Modern models take enormous prompts — hundreds of thousands to a million-plus tokens. So skip the machinery and paste everything in. No infrastructure, no index, nothing to maintain. The catch is that you pay for those tokens on every single request, latency climbs with prompt size, and models genuinely lose track of things buried in the middle of a giant dump.
Pause & recall. Before reading on: which of the three would let you answer “what changed in our returns policy this morning” — and which one physically can’t, no matter how well you built it? Say why in one sentence.
Reveal the answer
RAG and long context can, because the fresh text is in front of the model at query time. Fine-tuning can’t — this morning’s change isn’t in the weights until you retrain.
Side by side
| RAG | Fine-tuning | Long context | |
|---|---|---|---|
| Changes | What it knows now | How it behaves | What fits per call |
| Best for | Fresh / private facts | Voice, format, domain reasoning | Small, fixed corpora |
| Knowledge freshness | Instant (add a file) | Stale until retrained | Instant (paste it) |
| Can cite sources | Yes | No | Yes, if you ask |
| Setup effort | Medium (index + retrieval) | High (data + training) | Low |
| Per-query cost | Low (a few chunks) | Low (no retrieval) | High (whole corpus, every call) |
| Latency | Low | Low | Grows with prompt |
| Maintenance | Re-index on change | Retrain on change | None |
Read down the freshness and behaviour rows and the split falls out on its own: no single column wins everything, which is exactly why the answer is “it depends” — and why the next section is a framework, not a verdict.
How to choose: walk these questions in order
Don’t start from the technique. Start from what your problem actually needs, in this order:
- Does the answer depend on fresh or private knowledge? If yes, the knowledge must be in front of the model at answer time — that’s RAG or long context, never fine-tuning alone.
- Is that knowledge large or always changing? Large or changing → RAG, because you index once and update cheaply. Small and fixed → long context, because an index isn’t worth building for a page of facts.
- Do you need the model to behave differently — a tone, a strict format, a specialist’s reasoning? That’s a weights problem → fine-tuning. It’s a separate axis from knowledge, which is why it can stack on top of RAG.
- Need both fresh facts and special behaviour? Then it’s a hybrid — more on that below.
- Whatever you lean toward, measure it on your own data first. The cheapest option that passes your eval wins. “It felt better” is not an eval — here’s how to build one.
Notice steps 1–2 and step 3 are asking about different things — knowledge versus behaviour. Half of all bad architecture decisions come from mixing those two up.
The part the vendor blogs skip: cost over time
“RAG is cheaper” is true on day one and not the whole story. What matters is cost as your data and traffic grow.
| Upfront | Per query | As data grows | Maintenance cadence | |
|---|---|---|---|---|
| RAG | Build the index | Cheap — a few chunks | Flat — retrieval stays small | Re-index on change (cheap) |
| Fine-tuning | Expensive (data + training) | Cheapest — no retrieval | Flat at inference | Retrain to update (expensive, periodic) |
| Long context | ~None | Expensive — whole corpus every call | Grows linearly, forever | None |
The trap is long context. It’s free to build, so it feels cheapest — but you re-pay for the entire corpus on every request, and that bill scales with both your data size and your traffic. It’s a brilliant prototype and an expensive production default. Fine-tuning is the mirror image: painful upfront, then the cheapest per call, as long as your knowledge doesn’t need to be current.
Scale the tool to the problem, not to the tutorial. A page of facts doesn’t need a vector database; a corpus that grows every week shouldn’t live in a prompt you pay for on every call.
When the answer is “combine them”
The real world rarely fits one column. A support assistant needs your brand voice and format (fine-tuning) and today’s product facts with citations (RAG). That pairing is common enough to have a name — RAFT, retrieval-augmented fine-tuning: fine-tune the model for the behaviour, then wrap it in retrieval for the fresh knowledge.
The rule of thumb: reach for the hybrid only once a single approach has demonstrably failed your eval — not on day one. Every layer you add is another thing to maintain.
What I watch teams get wrong
Same handful of mistakes, again and again:
- Fine-tuning to “add knowledge.” The most expensive way to get a stale, un-citable fact store. Nine times out of ten they wanted RAG.
- Building RAG for a page of facts. If the whole corpus fits comfortably in a prompt and never changes, an index is ceremony. Just paste it.
- Treating long context as free because it “just works” in the demo. It works right up until the token bill and the latency arrive together.
- Choosing on vibes. No eval set, no numbers — just a hunch that one “seems smarter.” Build twenty real questions with known answers before you pick.
- Treating the decision as permanent. It isn’t. Prototype on long context, graduate to RAG when the corpus grows, add fine-tuning only when behaviour demands it. Revisit as you scale.
A worked example: how I chose for the Writing Bed
The Writing Bed answers questions from my blog posts. Three options, one decision:
- Long context? No. My writing grows every time I publish, so I’d pay to stuff every post into every request, and that bill only ever climbs. Fine for a weekend prototype; wrong as a default.
- Fine-tuning? No. I’d have to retrain on every new post, the model still couldn’t cite which post an answer came from, and it would confidently paraphrase things I never wrote.
- RAG? Yes. It updates the instant I add a file, it answers with sources so you can check it hasn’t drifted, and it scales the same whether I have ten posts or ten thousand.
The corpus is small enough that the vector store is just an in-memory list — no Pinecone, no pgvector. That’s the same principle running the other way: RAG was right, but the heavy version of RAG would have been its own over-engineering. Match the tool to the size of the problem in front of you.
Check yourself.
- Which axis does fine-tuning change — what the model knows, or how it behaves?
- Why is long context cheap to build but expensive to run at scale?
- A teammate wants to fine-tune to teach the model this quarter’s pricing. What do you suggest instead, and why?
- When is reaching for the RAG-plus-fine-tuning hybrid premature?
If any answer won’t come, that’s your map of what to reread — the tables above hold all four.
Where to start
If you take one thing from this: prototype with long context, measure, then earn every layer of complexity. Paste it all in to see if the model can do the task at all. If knowledge is the gap and it’s growing, move to RAG. If behaviour is the gap, fine-tune. Add a hybrid only when a single approach has failed a real eval — never before.
The technique is the easy, portable part. The judgement — knowing which gap you actually have — is the valuable part, and it’s the same judgement whether you’re wiring up a weekend tool or an enterprise assistant.
Want to see the RAG end of this built from nothing? That’s the Writing Bed. Want to watch a model decide and act rather than just answer? Meet Scout — then go poke all three in the Greenhouse. 🌱
