Retrieval math for RAG: embeddings, chunking, hybrid search and reranking

In a RAG system, the model can only use what retrieval hands it. When the answer is wrong, the cause is usually upstream of the model: the right passage was never retrieved, or it was retrieved but buried. Here’s the math of the retrieval half, which is where most of the fixable problems are.

Similarity: cosine, and why to normalise

An embedding model maps each chunk of text to a vector. Retrieval finds the chunks whose vectors are closest to the query’s. Cosine similarity compares direction only:

cos(q, d) = (q · d) / (‖q‖ · ‖d‖)

Normalise every vector to unit length once, at indexing time, and cosine becomes a plain dot product. That’s cheaper, and it’s what fast vector indexes are built for. Most embedding models are trained with cosine similarity, so normalising costs nothing in quality.

Memory: what a million chunks cost

At 1,024 dimensions, one million chunks take:

Precision	Memory
float32	4.1 GB
float16	2.0 GB
int8	1.0 GB
binary (1 bit per dimension)	128 MB

Quantising vectors to int8 typically loses little recall. Binary vectors lose more, but they’re 32× smaller and very fast to compare. A common pattern is to search the binary index first, then rescore the top candidates with full-precision vectors. That’s how on-device retrieval stays fast, where memory is the constraint.

Approximate search: HNSW

Comparing the query against every vector is exact but linear in collection size. HNSW (Malkov and Yashunin, 2018) builds a layered proximity graph. A search enters at the sparse top layer and greedily walks toward the query, descending layer by layer. Its cost grows roughly logarithmically instead of linearly.

The knobs trade recall for speed: how many neighbours each node keeps, and how wide the candidate list is during search. Measure recall@k, the fraction of the true top-k neighbours the index returns, against exact search on a sample of your own queries. “Fast” with 80% recall means one relevant chunk in five silently disappears.

Chunking

Chunk size is a real trade-off:

• Small chunks embed one idea each and retrieve precisely, but they lose surrounding context, so the model gets a sentence without the paragraph it depends on.
• Large chunks keep context, but their embedding averages several ideas and matches none of them sharply.

Chunk on structure where it exists: headings, functions, sections. Add a little overlap at the boundaries so ideas split across a boundary appear whole in one chunk. Store the parent section’s title or path with each chunk, so the model knows where a passage came from.

Hybrid search wins

Dense embeddings capture meaning but are weak on exact tokens: error codes, function names, product SKUs. BM25, the classic term-matching ranker, is the opposite. Run both and merge with reciprocal rank fusion (Cormack et al., 2009):

RRF(d) = Σ over rankers  1 / (k + rank_r(d))        k = 60

RRF needs no score calibration between the two systems, only ranks. Documents that both rankers place high rise to the top. Hybrid retrieval is usually the cheapest quality improvement available in a RAG system.

Reranking

The first stage returns, say, 50 candidates quickly. A cross-encoder reranker then reads the query and each candidate together and scores them jointly. That’s far more accurate than comparing two independently computed vectors, and far more expensive. Reranking 50 candidates of about 512 tokens each means processing about 25,600 tokens per query.

The two-stage shape, cheap recall and then expensive precision, is the standard because neither stage alone is good enough.

Then think about order

Retrieved chunks are context, and context has positions. Liu et al. (2023) showed that models use information at the start and end of a long context better than in the middle. Put the strongest evidence first, and don’t pad the context with twenty weak chunks when three strong ones will do.

Rules I use

1. Measure retrieval separately from generation: recall@k on a labelled sample of real queries, before blaming the model.
2. Normalise vectors and quantise them if memory matters. Check recall after quantising.
3. Use hybrid retrieval with RRF as the default, and add a reranker when precision matters.
4. Chunk on structure, with metadata, and keep the retrieved set small and strong.

Sources: Malkov and Yashunin, HNSW (2018). Robertson and Zaragoza, The Probabilistic Relevance Framework: BM25 and Beyond (2009). Cormack, Clarke and Büttcher, Reciprocal Rank Fusion (SIGIR 2009). Liu et al., Lost in the Middle (2023). Memory figures are computed for one million 1,024-dimensional vectors.