RAG That Survives Production: Chunking, Reranking, and Eval

Why do RAG prototypes fall over in production?

Because the prototype was measured on generation and production is limited by retrieval. In a demo, every test question has its answer sitting in one clean paragraph. Real users ask questions whose answer spans three documents, uses none of the same words, or is not in the corpus at all. Swapping in a better model does not fix any of those. Better retrieval does.

Across the RAG systems we have taken from prototype to production, the ranking of what actually moved answer quality has been consistent:

1. Chunking strategy
2. Reranking
3. Query rewriting
4. Prompt construction
5. Base model choice

Model choice is last. It is also the first thing every team wants to change.

Chunking is a content problem, not a parameter

Fixed-size chunking is the default and it is wrong for most technical corpora, because it cuts through the middle of the structures that carry the meaning, a table, a config block, a numbered procedure. A chunk containing step 4 of 7 with no heading is retrievable and useless.

What has worked for us:

• Split on structure first. Headings, list boundaries, code fences. Only fall back to token counts inside a section that is too long.
• Carry the heading path into every chunk. Prefixing Security > Key rotation > Automating rotation to the chunk text costs a few tokens and measurably improves both embedding quality and the model's ability to interpret what it received.
• Overlap around 15%. Enough that a sentence cut at a boundary appears whole in one of the two chunks.
• Keep tables intact even when they blow the size budget. A half table is worse than no table.

Reranking is the highest-leverage 30 lines you will write

Embedding search is fast and approximate. It optimises for "this passage is about the same subject", which is not the same as "this passage answers the question". A cross-encoder reranker reads the query and each candidate together and scores actual relevance.

The pattern is: retrieve wide, rerank narrow.

# Retrieve 50 candidates by vector similarity, rerank, keep 5.
candidates = vector_store.search(query_embedding, top_k=50)

scored = reranker.score(
    query=user_query,
    documents=[c.text for c in candidates],
)

context = [
    c for c, s in sorted(zip(candidates, scored), key=lambda p: -p[1])
    if s > RELEVANCE_FLOOR
][:5]

Two details that matter more than the model you pick. First, retrieve far wider than you intend to use, the reranker can only fix ordering, not recall. Second, keep the RELEVANCE_FLOOR. Passing five weak passages because you promised the prompt five passages is how a system learns to answer questions it should have declined.

Hybrid search, because embeddings cannot spell

Vector search fails on exact tokens: error codes, version numbers, flag names, part numbers. ERR_CONN_5521 embeds near every other error string. Run BM25 keyword search alongside the vector search and fuse the results. Reciprocal Rank Fusion is about ten lines and needs no tuning. On corpora full of identifiers this is routinely a larger win than any embedding model upgrade.

Evaluation, or you are guessing

You cannot improve retrieval you are not measuring, and "it looked better in the playground" is not measurement. The minimum viable evaluation setup:

• A golden set. 100–200 real questions with the passages that should be retrieved, labelled by someone who knows the domain. This is a week of unglamorous work and it is the whole foundation.
• Retrieval metrics, separately from answer metrics. Recall@k and MRR tell you whether the right passage was found. Track them on their own, because a good answer from lucky retrieval is a bug that will surface later.
• Answer grading. An LLM judge scoring faithfulness, does every claim trace to a retrieved passage, plus periodic human spot-checks to keep the judge honest.
• Regression gating in CI. Every change to chunking, embeddings, or prompts runs the golden set. A drop over threshold fails the build.

What we would skip next time

Building a bespoke vector store. Postgres with pgvector carried every production workload we have run under roughly 5M chunks, kept retrieval in the same transactional store as the metadata we needed to filter on, and removed a whole system from the operational surface. Reach for a dedicated vector database when you have measured that you need one.

References

• Reciprocal Rank Fusion (Cormack et al., 2009), the original hybrid-fusion paper.
• pgvector, Postgres vector similarity search.
• Anthropic. Contextual Retrieval, on prefixing chunks with their surrounding context.
• BEIR benchmark, for comparing retrieval approaches across domains.