Chunking is the step in a retrieval-augmented generation (RAG) pipeline where long documents are split into smaller pieces—chunks—before they are embedded and indexed. When a user asks a question, the system retrieves the most relevant chunks and passes them to the language model as context. If the chunks are badly cut, the right information never reaches the model, no matter how good the model is.

Why it matters now: as companies connect AI assistants and agents to manuals, contracts, tickets, and codebases, retrieval quality has become the main bottleneck. Research from Chroma found that the choice of chunking strategy alone can shift retrieval recall by up to 9 percentage points (Chroma: Evaluating Chunking Strategies for Retrieval), and Anthropic showed that adding context to each chunk can cut failed retrievals by 49% (Anthropic: Contextual Retrieval). Chunking is cheap to change and has an outsized effect on answer quality.

Why RAG needs chunking at all

Three constraints force you to split documents:

• Embedding limits: embedding models accept a limited number of tokens per input, and long inputs get blurry representations. If you need a refresher, see what embeddings are and why RAG depends on them.

• Precision: a 40-page PDF embedded as one vector matches almost every question vaguely and none precisely. Smaller units let search find the exact paragraph.

• Cost and context: sending only the relevant passages to the model keeps prompts short, cheaper, and less distracting.

The trade-off is that every cut can separate a fact from the context that explains it. A chunk that says “revenue grew 3% over the previous quarter” is useless if it no longer says which company or which quarter.

The main chunking strategies

Fixed-size chunking

Split text every N tokens or characters, optionally with overlap between neighboring chunks. It is fast and predictable, but it happily cuts sentences, tables, and code in half.

Recursive splitting

Try to split at the largest natural boundary first (paragraphs), and only fall back to smaller units (lines, sentences, words) when a piece is still too long. LangChain’s RecursiveCharacterTextSplitter is the best-known implementation, and in Chroma’s benchmark a 200-token recursive splitter with no overlap was consistently strong across metrics.

Structure-aware splitting

Use the document’s own structure: Markdown headings, HTML sections, slide boundaries, or code functions and classes. Each chunk maps to a real unit of meaning, and you can prepend the heading path (for example, “Refund policy > International orders”) so the chunk carries its location with it (Chroma docs: chunking strategies).

Semantic chunking

Embed sentences or paragraphs, measure how similar neighbors are, and cut where similarity drops—a sign the topic changed. It produces more coherent chunks for unstructured text, at a higher compute cost.

LLM-based and contextual chunking

Ask a language model to decide boundaries, or to write a short note that situates each chunk within the whole document. Anthropic’s contextual retrieval prepends a 50–100 token explanation to every chunk before embedding and keyword indexing; combined with reranking, it reduced retrieval failures by 67% in their tests. Because each call includes the full document, prompt caching is what keeps this approach affordable.

How to choose chunk size and overlap

• Start small to medium: roughly 200–500 tokens is a sensible starting range for prose. Smaller chunks are more precise; larger ones carry more context.

• Use overlap sparingly: overlap protects facts that straddle a boundary, but it duplicates content, raises storage cost, and can return near-identical results.

• Match the content type: FAQs work well one question per chunk; contracts by clause; code by function; tables as whole tables with their headers.

• Keep metadata: store source, title, section, date, and permissions with each chunk so you can filter and cite.

• Test, don’t guess: the best settings depend on your documents and your questions.

How chunking fits with the rest of the RAG pipeline

Chunking is the first lever, but it works together with the others. Hybrid search catches exact terms that pure vector search misses, and reranking reorders the candidates so the best chunks reach the model first. To know whether a chunking change actually helped, measure it with RAG evaluation metrics such as recall@k on a set of real questions.

Common chunking mistakes

• Splitting tables and lists mid-row: numbers lose their column labels and become meaningless.

• Dropping headings: chunks without their section title lose most of their context.

• One size for every source: a policy manual, a Slack export, and a codebase need different rules.

• Ignoring PDFs’ layout: headers, footers, and page numbers end up glued into the middle of sentences.

• Never re-chunking: when documents or embedding models change, revisit the strategy and re-index.

FAQ

What is the best chunk size for RAG?

There is no universal number. Many teams start around 200–500 tokens with little or no overlap, then tune based on retrieval metrics from their own questions and documents.

Does a bigger context window make chunking unnecessary?

Not really. Even with long context windows, sending entire document collections is slow and expensive, and models can miss details buried in very long prompts. Retrieval over well-made chunks is still more efficient.

Should chunks overlap?

Sometimes. A small overlap can help with facts split across boundaries, but benchmarks show overlap is not automatically better and it increases index size.

Who should learn chunking strategies?

AI engineers building assistants and agents, and data professionals preparing company knowledge for search. It is one of the most practical skills for making RAG systems accurate.

Ready to build RAG systems that find the right answer and the data pipelines behind them? Explore these Coderhouse courses:

• AI Engineering Course — design retrieval pipelines, agents, and production LLM applications.

• Data Analytics Course — organize, clean, and measure the data your AI systems rely on.