Chunking: Breaking Down Long Documents

Part of the free Generative AI course on LogicWiz, module: Nova Reads the Archives.

Episode 16: Death by a Thousand Tickets

"Nova can find the right document. The problem is the document is forty pages long — and she only needs one paragraph of it."

The Forty-Page Runbook

Arjun feeds LogicWizNews's engineering runbook into the archive. It's one document — forty pages covering logins, billing, email, outages, everything. He embeds it as a single vector, stores it, and asks Nova a login question.

Retrieval returns... the whole runbook. All forty pages, because the one vector represents all of it. Nova now has to read a novel to answer one question, most of it irrelevant, and her answer comes back vague and slow.

Here's the failure in one line: you cannot embed a big document as one vector. Cram forty pages of mixed topics into 1536 numbers and the meaning averages out into mush — the vector is a little bit about everything and precisely about nothing. Retrieval can't tell that page 3 is about logins.

Make it concrete. Say the runbook has just five short sections, each resolving a different kind of ticket:

(Five is just to keep it readable. A real runbook has hundreds of sections, and the whole archive holds millions — but the failure is identical at any scale.)

Embed all five sections together as one vector, and that single vector now has to stand for logins and billing and email and outages and API keys at the same time. It settles on the blurry average of all five — sharply about none of them. So when a reader asks "I can't log in after resetting my password," retrieval compares that question to the one everything-vector, where the login meaning is buried under four unrelated topics. Nova gets handed the whole runbook and has to wade through billing, email, and outages to find the single login line. Slow, noisy, vague — exactly the problem we started with.

Now chunk the runbook — one vector per section, five vectors instead of one. The login question now matches the Logins chunk directly and ignores the other four. Retrieval hands back just those two sentences about session tokens. Same document, same embedding model — the only difference is that each section kept its own identity instead of being averaged into the pile.

{{visual:chunk-vs-whole}}

The fix is chunking: slice big documents into small, coherent pieces before embedding, so each vector represents one focused idea. Get chunking right and Nova retrieves the exact paragraph. Get it wrong and the best embeddings in the world can't save you.

⚠️ Warning: Chunking happens before embedding, which happens before the LLM. It's a pre-processing step in the offline clock. Bad chunks poison everything downstream — vague retrieval, wasted tokens, worse answers.

flowchart LR
    A["Big document"] --> B["Chunking"]
    B --> C["Embedding"]
    C --> D[("Vector store")]
    D --> E["Retrieval → LLM"]

So the question becomes: how do you cut? Turns out there are five common strategies, from dead-simple to LLM-powered. Let's meet them.

Strategy 1: Fixed-Size Chunking

The most intuitive method: chop the text into equal-sized pieces — every N characters, words, or tokens. Hit the limit, start a new chunk. Done.

The one nuance is overlap. If you cut cleanly at 500 characters, a sentence that straddles the boundary gets split in half, and both chunks lose the thread. So you let chunks overlap — chunk 2 starts a little before chunk 1 ended, carrying some shared text to preserve continuity.

{{visual:fixed-overlap}}

Fixed-size is fast, dumb, and surprisingly fine when your data is uniform. Here's the real library doing it — notice the chunk_size and chunk_overlap knobs:

{{visual:l16-fixed-walkthrough}}

{{cell:l16-fixed}}

Its weakness is obvious: it has no idea what the text means. It'll happily slice through the middle of a sentence or glue two unrelated topics into one chunk. For messy, varied documents, we need something that respects meaning.

Strategy 2: Semantic Chunking

What if the content itself decided where to cut? Semantic chunking splits at topic shifts, not at a fixed length.

The mechanism reuses the exact tool from last episode — cosine similarity:

1. Break the document into sentences.
2. Start a chunk with the first sentence. Keep adding sentences.
3. After each sentence, check the cosine similarity between the growing chunk and the next sentence.
4. When that similarity drops sharply, the topic just changed — finalise the chunk, and start a fresh one with the sentence that caused the drop.

That similarity cliff is the signal that "authentication" just became "billing." Watch the boundary get detected live:

{{visual:semantic-boundary}}

The result is beautiful: each chunk is one coherent topic. A document about three separate issues becomes three clean, self-contained chunks. Here's the boundary logic on a handful of sentences:

{{visual:l16-semantic-walkthrough}}

{{cell:l16-semantic}}

💡 Tip: Semantic chunking costs more — you embed every sentence just to decide the boundaries, before you even embed the final chunks. Worth it when topic-purity matters; overkill when your data is already uniform.

Strategy 3: Recursive Chunking

Semantic chunking is smart but pricey. Recursive chunking is a clever middle ground that respects a document's natural seams for free.

The idea: try to split on the biggest natural separator first — paragraphs (\n\n). If a resulting piece is still too big, split that piece on the next separator down — sentences. Still too big? Split on words. It recurses, going finer only where it has to.

flowchart TD
    A["Take a segment"] --> B{"Bigger than<br/>the size limit?"}
    B -- "No" --> C["Keep as a chunk ✓"]
    B -- "Yes" --> D["Split on the next<br/>separator down"]
    D --> A

The payoff: chunks respect paragraph and sentence boundaries whenever possible, only cutting mid-sentence as a last resort. Watch it split a structured document on \n\n first, dropping to finer separators only where a piece is still too big:

{{visual:l16-recursive-walkthrough}}

{{cell:l16-recursive}}

It's the default choice for most RAG systems — a great balance of quality and cost, and no embedding calls needed to decide boundaries.

Strategy 4: Document-Structure Chunking

Some documents come pre-chunked by their author — you just have to respect the structure they already wrote. Document-structure chunking uses the document's own headings and sections as boundaries.

A Markdown doc has # headings. An HTML page has <h1>, <section>. A well-formatted runbook has Title → Introduction → Section 1 → Section 2 → Conclusion. Each becomes its own chunk.

The advantage is that you preserve exactly the meaning the author intended — they already decided what belongs together. If page 3 is the "Billing" section, it becomes the "Billing" chunk, cleanly. (In practice you often combine it with recursive chunking: split by heading, then recursively split any section that's still too long.)

Strategy 5: LLM-Based Chunking

The first four are heuristics — mechanical rules. The fifth hands the whole job to an LLM: "here's a document, split it into semantically coherent chunks."

Because the model actually understands the text, it can make judgement calls the rules can't — grouping related ideas even across awkward formatting. It's the smartest option.

It's also the riskiest:

• Hallucination — the model is supposed to only split the text, but it might subtly reword, drop, or invent content while doing it. In a task that's meant to be pure extraction, that's a real hazard.
• Cost & latency — you're paying for a full LLM pass over every document.
• Hard to debug — when a rule-based chunker misbehaves you can read the rule. When an LLM draws a weird boundary, good luck explaining why.

flowchart LR
    D["Document"] --> LLM["LLM (understands meaning)"]
    LLM --> C["Coherent chunks"]
    LLM -.->|risk| H["hallucination · cost · opacity"]

📌 Summary: Fixed-size — fast, uniform data. Semantic — topic-pure, costs embeddings. Recursive — respects natural seams, great default. Structure-based — honours the author's headings. LLM-based — smartest and riskiest. Five knives; the trick is picking the right one.

How Big Should a Chunk Be?

Every strategy has one dial: chunk size. There's no universal right answer — it's an experiment — but a few forces pull on it.

• The model's token limit is the ceiling. A big-context model can take big chunks; a small model forces small ones. You literally cannot exceed it.
• Bigger chunks cost more. More memory to process, more tokens per retrieval, slower generation.
• The precision/context trade-off. Small chunks are precise but can lose surrounding context (the sentence that explains the one you retrieved). Big chunks keep context but dilute relevance and drag in noise.

{{visual:chunk-size-tradeoff}}

So how do you actually pick? Not by guessing:

1. Start from the benchmark the model's documentation recommends.
2. Nudge by a plus-minus delta — try a bit bigger, a bit smaller.
3. Test against your real data and measure retrieval quality.
4. Only then ship it to production.

💡 Tip: Choose the strategy by your data, and the size by testing. Simple, uniform data → fixed-size is fine. Prose with clear topics → semantic or recursive. Structured docs → structure-based. And when you combine strategies (a hybrid), remember you inherit the pros and the cons of each.

What Nova Learns Next

Nova can now slice documents into clean, embeddable chunks, and she can find the closest ones by meaning. But at ten million chunks, "compare the query against every single vector" — the brute-force loop we wrote in Episode 15 — grinds to a halt.

Next episode: indexing and retrieval — how vector databases organise millions of chunks so search is instant instead of a full scan, how to route a query to the right pile of data, and the final polish step, re-ranking, that puts the single best chunk on top before Nova ever sees it. That's the last piece of the RAG engine.