Going deeper

Chunking comes first

If the answer is split across two chunks, no retriever can bring it back. Start with chunks that follow the document’s structure (headings, sections), add a small overlap, and attach the title and section to each chunk. Sizes are something to measure on your data, not to copy.

Two retrievers that fail differently

Vector search compares meanings, usually through cosine similarity. It finds “reimbursement” for “refund”, but it’s weak on exact strings like ERR-4512. BM25 scores term matches, weighting rare terms and correcting for document length. It finds ERR-4512 and misses synonyms. Combine them, since each one covers the other’s gaps.

Their scores aren’t comparable, so Reciprocal Rank Fusion (Cormack et al., 2009) uses only positions:

RRF(d) = Σ 1 / (k + rank_i(d))     k = 60

A reranker can then read the query with each candidate, using a cross-encoder or a model. That’s slower and more precise, so it only sees the top few dozen. The usual order:

const visible = await acl.filter(user);
const a = await bm25(query, visible, 50);
const b = await vectors(query, visible, 50);
const top = rerank(query, rrf([a, b]).slice(0, 30)).slice(0, 5);
if (top[0].score < threshold) return abstain();

The agentic part

• Rewrite: turn “and for the blue one?” into a standalone query, or split a compound question.
• Route: choose between the policy base, the orders database and the web.
• Judge and retry: if results are weak, reformulate. Cap the retries.
• Abstain: saying “I can’t find this” beats a confident invention. It needs a threshold you’ve tested.

HyDE (Gao et al., 2022) is a useful trick for vocabulary gaps: generate a hypothetical answer, then search with that text rather than the question.

Measure the two stages separately

When an answer is wrong, the first question is whether the right passage was retrieved. Recall@k tells you if it was among the top k. MRR tells you how high it came. If recall is low, no prompt will help, so fix retrieval. If recall is high and answers are wrong, look at generation.

Permissions belong before the search

Filtering after ranking loses results, and it can reveal what hidden documents contain, because they still influence term statistics and rankings. Filter on the user’s rights first. And remember that retrieved text is untrusted input: a document can contain instructions, which is the next module’s subject.