Scaling and Sharding in Production

Chunking Strategy

Chunk size and overlap shape retrieval quality. Use paragraphs of a few hundred characters with a small overlap so ideas are not split across boundaries. Store the source title and section with every chunk for citations.

Sharding

Split a large collection across several nodes by a shard key so each node stays lean. Searches fan out across shards and merge results, adding machines horizontally as data grows.

Replication and Availability

Keep replica copies per shard so a node loss does not take the collection down. Write against the primary replica and read from any of them, like a standard database topology.

Monitor Daily

• Latency percentiles - P95 and P99, not just averages.
• Recall drift - Re-run your eval set weekly.
• Disk and memory - Indexes can outgrow RAM fast.
• Write backlogs - Embedding jobs falling behind.

Key Points

• Shard by key and replicate for availability.
• Watch P95 latency and eval recall, not just uptime.
• Plan memory; large indexes need RAM.
• Keep embedding jobs, indexes and sources in sync.