Chunking Strategies for RAG: What Works

Your retrieval is returning junk.

The obvious suspect is the embedding model. So you swap it for a better one. Almost nothing changes.

The real culprit is usually a decision made much earlier, with far less thought: where you cut the text.

Chunking strategies for RAG get treated as plumbing — a line of setup code copied from a tutorial and never revisited. But the cut decides what your system can find. And a better embedding model won’t rescue a bad one.

The good news: it’s one idea, not ten.

In one breath: Chunking strategies for RAG come down to one trade-off, not a menu of ten. Small chunks match a question precisely but arrive without enough context to answer it. Large chunks carry the context but blur the match and cost more to run. The reliable default is recursive splitting at roughly 512 tokens — it cuts on paragraph and sentence boundaries, so it respects the structure the author already put there, and costs nothing extra to compute. Attach metadata to every chunk — source, section, date — so answers can be filtered and cited. Then measure before you get clever: peer-reviewed evaluations keep finding that expensive semantic chunking fails to justify its cost, and one 2026 benchmark put recursive splitting at 69% end-to-end accuracy against semantic chunking’s 54%. Blind fixed-length cutting is the one genuinely bad option.

What a chunk actually is

A chunk isn’t “a piece of text.” It’s two things at once.

First, it’s the unit of retrieval — the smallest thing your system can find. Nothing smaller ever comes back. Nothing larger comes back without dragging its neighbours along.

Second, and this is the half that gets missed, it’s the unit of citation. When your app says “according to this source,” the chunk is what it points at.

Split a claim across two chunks and there’s nothing clean to cite.

This is also why you can’t skip chunking by embedding whole documents. An embedding is a single point in meaning-space. A 3,000-word document covering ten topics collapses into one blurry average of all ten — close to nothing in particular.

Chunk it, and each idea becomes its own findable point.

I built how retrieval works end to end — embeddings, vectors, similarity — from scratch in an earlier post. This one is about the decision sitting a layer beneath it. (Still weighing whether to retrieve at all? Start there instead.)

Pause & recall. Why does one embedding for a long, multi-topic document retrieve badly — even with an excellent embedding model?

Reveal the answer

Because the embedding is a single point averaging every topic in the document. The average sits near none of them, so a question about any one topic matches it only weakly. The model isn’t the problem; the unit is.

The one dial: precision versus context

Small chunks retrieve precisely and answer poorly. The vector is sharp because the chunk is about one thing. But what arrives may be a fragment — the answer, stripped of the setup that makes it usable.

Large chunks answer well and retrieve poorly. The context is definitely in there. But the vector averages everything inside, so it matches nothing sharply, and you pay for the irrelevant majority.

Watch it happen on one passage.

The text we’re cutting

"Recursive splitting cuts on paragraph breaks
 first. If a chunk is still too big, it falls
 back to sentences, then to words. The Writing
 Bed uses it at ~1,500 characters."

The question we’ll ask it: “What does recursive splitting fall back to?”

Cut small — around 128 tokens

chunk → "back to sentences, then to words."

A perfect keyword match. But falls back from what? The chunk can’t say. Precise, and useless.

Cut medium — around 512 tokens

chunk → the whole passage above

Matches the question, and contains the answer. This is the usable middle.

Cut large — around 2,000 tokens

chunk → the passage + 12 unrelated paragraphs

The answer is in there. But the vector now averages thirteen topics, so it matches the question weakly — and you paid for twelve paragraphs nobody asked for.

Every chunking strategy is an attempt to cheat this trade-off — precision of small chunks, context of large ones.

Hold that, and you can judge any strategy without anyone explaining it: which end of the dial is it buying back, and what does it charge you?

The strategies, boring to clever

Strategy	How it cuts	Cost	When it earns its place
Fixed-size	Every N characters, structure be damned	Trivial	Uniform text; a baseline to beat
Recursive	Paragraphs, then sentences, then words — until chunks fit	Trivial	The default. Respects structure, costs nothing
Sentence	On sentence boundaries, grouped to a target size	Trivial	Dense prose; strong cost-to-quality ratio
Document-aware	On the document’s own structure — headings, code blocks	Low	Markdown, HTML, code
Semantic	Embeds sentences, cuts where meaning shifts	High — embeds everything twice	Unstructured text where topics drift mid-page
LLM-based	A model reads the document and picks boundaries	Very high	Small, high-value corpora
Late chunking	Embeds the whole document first, then pools per chunk	Moderate; needs long-context embedding	Text thick with pronouns and back-references

You rarely implement these by hand. Recursive splitting is RecursiveCharacterTextSplitter in LangChain, or the equivalent node parsers in LlamaIndex. Messy PDFs need a parser like Unstructured first, so there’s real structure to cut on.

The strategy is your decision. The library call is one line of config.

What “recursive” actually does

The name sounds clever. The mechanism is a ladder of fallbacks.

The ladder

Target: 512 tokens. Try in order —

  1. Split on "\n\n"   (paragraphs)
  2. Split on "\n"     (lines)
  3. Split on ". "     (sentences)
  4. Split on " "      (words)

It stops at the first level where the chunks fit.

Why that matters

It always cuts at the biggest natural boundary available. So chunks land on paragraph breaks rather than mid-sentence — and it costs nothing, because the structure was already in the text. Recursive splitting simply doesn’t destroy it.

Late chunking, properly explained

The newest strategy and the most misunderstood. Worth doing slowly.

Step 1 — the document

"The Writing Bed indexes every post I publish.
 It cut answer time by 40%."

Two sentences. The second depends entirely on the first.

Step 2 — cut it the normal way

chunk 1 → "The Writing Bed indexes every post I publish."
chunk 2 → "It cut answer time by 40%."

Chunk 2 is orphaned. On its own, “It” means nothing.

Step 3 — watch the answer disappear

Search: "Writing Bed performance"

  chunk 1 → strong match
            (names it, says nothing about speed)
  chunk 2 → near zero
            (holds the answer, names no subject)

The only chunk containing the answer is invisible to the search. That’s the failure normal chunking can’t fix.

Step 4 — what late chunking changes

Embed the WHOLE document first
  → "It" is encoded while "The Writing Bed"
    is still visible to the model

Then slice into chunks
  → chunk 2 keeps that knowledge

Search: "Writing Bed performance"
  chunk 2 → matches

Same text. Same boundaries. Same number of chunks. The only change is when you embed — before the cut instead of after.

The catch

It needs a long-context embedding model, and there’s little independent benchmarking behind it yet. Worth watching. Not yet worth defaulting to.

What the benchmarks actually say

The literature is more interesting than the guides suggest. The answer isn’t simply “simple wins.”

Study	Setup	Finding
Qu, Tu & Bao, Oct 2024 (arXiv 2410.13070)	Semantic vs fixed-size across document retrieval, evidence retrieval and answer generation	Fixed-size consistently outperformed semantic chunking on realistic documents; “the computational costs associated with semantic chunking are not justified by consistent performance gains”
Bennani & Moslonka, Jan 2026 (arXiv 2601.14123)	Natural Questions, SPLADE retrieval, Mistral-8B generator	Sentence chunking is the most cost-effective method, matching semantic up to ~5k tokens; overlap gave no measurable benefit; a “context cliff” degrades quality beyond ~2.5k tokens
Shaukat, Adnan & Kuhn, Mar 2026 (arXiv 2603.06976)	36 segmentation methods × 5 embedding models	Content-aware chunking significantly beat naive fixed-length splitting (nDCG@5 ≈ 0.459 vs below 0.244). Bigger embedding models scored higher but stayed “sensitive to suboptimal segmentation”

Together they say something sharper than “keep it simple.”

Respecting structure matters. Paying a model to find the structure usually doesn’t.

Cutting blindly every N characters is genuinely bad — the March 2026 study nearly doubled retrieval quality just by moving off it. But the expensive end keeps failing to justify itself. The 2024 study found fixed-size beating semantic outright. The 2026 one found plain sentence chunking matching it, far more cheaply.

Recursive splitting sits exactly in that gap. Content-aware, but free.

A better embedding model won’t save bad chunking. Larger models scored higher overall, yet stayed sensitive to poor segmentation. A team testing three embedding models while never testing a chunk size is tuning the wrong dial.

Most teams I meet are doing precisely that.

Third-party benchmarks agree. A February 2026 FloTorch run over 50 academic papers put recursive splitting at 512 tokens on 69% end-to-end accuracy against semantic chunking’s 54%. It’s reported through Prem AI’s round-up rather than a paper I can link directly, so weigh it accordingly.

The most telling number: the semantic chunker produced fragments averaging 43 tokens.

That’s the mechanism in miniature. Cut precisely where meaning shifts and you get chunks that are correct and far too small to answer from. Precision won. Context lost. Answers worse.

The overlap myth

Nearly every guide — mine included, until I checked — tells you to overlap adjacent chunks by 10–20%, so an idea straddling a boundary isn’t sliced in half.

Without overlap

chunk 1: "...cosine similarity measures the angle"
chunk 2: "between two vectors, ignoring length."

Ask “what is cosine similarity?” and neither chunk answers. The definition was cut down the middle.

With 20% overlap

chunk 1: "...cosine similarity measures the angle
          between two vectors, ignoring length."
chunk 2: "between two vectors, ignoring length.
          Nearest neighbours are the smallest angles."

Chunk 1 now answers it. That rescue is what you’re paying for.

What the evidence says

The January 2026 analysis varied overlap systematically and found it gave no measurable benefit while increasing indexing cost. Every overlapped token is one you embed, store and pay for twice.

Scope that honestly — one study isn’t a law. It used a sparse retriever (SPLADE), which matches on terms rather than dense semantic similarity, on a single dataset. Teams running dense retrieval still report overlap helping.

So overlap is a hedge, not a law. The rescue above is real. It may just not happen often enough to pay for itself.

Test it on your own corpus. If your documents are well-structured, the paragraph boundaries recursive splitting already respects may be doing the job you’re paying overlap for.

The half everyone forgets: metadata

Chunking isn’t only about where you cut. It’s about what you attach — and this is where the cheapest wins hide.

A bare chunk

"Start at 512 tokens and measure."

Can’t filter by date. Can’t prefer the canonical source. Can’t check permissions. Can’t cite.

The same chunk, enriched

text:    "Start at 512 tokens and measure."
source:  "Where you cut the text"
section: "Where to start"
date:    2026-08-27

Four things the second one can do that the first can’t:

Filter before searching. “Only 2026 policies” becomes a cheap metadata filter — instead of hoping the embedding encodes recency. It doesn’t.
Rank on more than similarity. Prefer newer. Prefer canonical over a stale duplicate.
Respect permissions. Retrieval that ignores who’s asking is a data leak with good UX.
Cite. The Writing Bed can name the post an answer came from purely because every chunk carries its source.

That last one connects chunking to trust. Citations are how an honest system ships something that can still occasionally be wrong — and the chunk is what a citation points at.

One free trick: prepend the section heading to the chunk’s text before embedding it. Costs nothing, gives the vector real context.

Do that before you go anywhere near a semantic chunker.

What I chose for the Writing Bed, and why

Roughly 1,500-character chunks — about 375 tokens — recursive, with overlap, each carrying the post it came from.

The reasoning is simple. These essays are structured prose where one idea usually runs two to four paragraphs. Recursive splitting cuts on paragraph breaks, so chunks land on real boundaries. At ~375 tokens a chunk is a complete argument: big enough to answer from, small enough that its vector is about one thing.

And the corpus is tiny. Fancier methods would cost more in complexity than they could return.

Now the honest part: the overlap is an unexamined default.

I set it because it sounded prudent. Then I read a peer-reviewed paper suggesting it may be buying me nothing but index size. My corpus is dense, structured essays — roughly the case where paragraph-aware splitting should already handle boundary problems.

What would change my mind is evidence, not argument: twenty golden questions, run with overlap and without, comparing whether the right chunk comes back.

Until I run that, I’m reporting a choice, not defending one.

How to actually pick

Your documents	Start with	Because
Anything, no strong opinion	Recursive, ~512 tokens	The default the benchmarks keep validating
Essays, articles, prose	Recursive or sentence, 400–800	Ideas run in paragraphs; cut on them
Markdown, HTML, code	Document-aware	The structure is already there
Short factual Q&A	~256 tokens	Factoid queries reward precision
Legal, financial, technical	1,024, document-aware	A clause is meaningless without its surroundings
Transcripts, chat logs	Sentence or semantic	No structure to lean on; topics drift
Small, high-value corpus	Consider LLM-based	The only place the cost can be justified

Two rules outrank the table.

Stay under the cliff. Quality degrades past roughly 2.5k tokens of retrieved context. Retrieving eight 1,000-token chunks is likely making answers worse, not better informed.

Change one variable at a time. Otherwise you’ll never know which change helped.

You can’t tune what you don’t measure

Chunk size is a hyperparameter. Treat it like one: baseline, change one thing, measure, keep what wins.

The trap is measuring the wrong layer.

If you only score the final answer, you can’t tell a retrieval failure from a generation failure. Score retrieval on its own first. For questions where you know which chunk should come back — does it?

Fix that before touching the prompt.

Twenty golden questions is enough to start. I wrote a full guide to building that eval from scratch.

Plainly: if you’re not going to measure, stop reading comparisons of chunking strategies. You won’t be able to tell whether any of them helped.

What I watch teams get wrong

Blaming the embedding model. It’s usually the chunking. Cheaper to test, too.
Cutting by character count alone. Slices tables, code and clauses in half.
Reaching for semantic chunking first. The expensive option that keeps losing. Earn your way there.
Bare chunks. No source, no section, no date — so no filtering, no ranking, no citations.
Retrieving more chunks to be safe. Past the cliff, that’s noise, not knowledge.
Re-chunking without re-embedding. New chunks need new vectors. Mixed generations break similarity silently.
Tuning by vibes. “That feels better” isn’t a result you can defend.

Where to start

Split recursively at 512 tokens. Attach source and section to every chunk, and prepend that heading to the text you embed. Write twenty questions you know the answers to. Check whether the right chunk comes back.

Only then change one variable — size first, then overlap, then strategy.

That’s a day’s work, and it beats most pipelines I’m shown.

The advanced strategies aren’t wrong. They’re answers to a question most teams haven’t yet earned the right to ask.

Check yourself.

What are the two things a chunk is the unit of — and why does the second matter for trust?

State the core trade-off in one sentence. Which end does a 128-token chunk win, and which does it lose?

Semantic chunking is theoretically better and empirically worse. What does the 43-token fragment finding suggest about why?

In the late chunking example, why can’t chunk 2 be found — and what fixes it?

Anything that won’t come is your reread map.

Want to see one of these decisions running? The Writing Bed is ~1,500-character recursive chunks with the source attached, live in the Greenhouse.

Ask it something specific and watch which post it cites. That’s a chunk, doing exactly the job it was cut for. 📚

Where you cut the text