Chunking: Splitting Documents Well

Why Chunk Boundaries Matter

Search works on chunks, not whole documents. A chunk that cuts a sentence in half is hard to retrieve and confusing to answer from; a chunk that mixes three topics embeds as a blur.

Simple Character Chunking

def chunk_text(text, size=500, overlap=50):
    chunks = []
    step = size - overlap
    for i in range(0, len(text), step):
        chunks.append(text[i:i + size])
    return chunks

chunks = chunk_text(long_document)
print(len(chunks))

Splitting by Structure

Split on paragraphs, headings or code blocks instead of fixed sizes. A markdown-aware splitter keeps sections intact and labels them.

sections = []
current = []
for line in markdown_document.splitlines():
    if line.startswith("#"):
        if current:
            sections.append("
".join(current))
        current = [line]
    else:
        current.append(line)
sections.append("
".join(current))

Choosing Chunk Size

• Small chunks (200-400 chars) - Precise retrieval, more calls.
• Medium chunks (400-800 chars) - A balanced default.
• Large chunks (1000+ chars) - Fewer calls, broader context, more noise.

Key Points

• Keep sentences and headings intact.
• Overlap prevents topics from splitting at boundaries.
• Split by structure for better labels and retrieval.
• Map each chunk back to its source for citations.