Module 7 of 10, lesson 1 of 5

Agentic RAG

Know what an agent adds to RAG, and when it only weighs the system down.

Classic RAG follows a fixed route. The question is turned into a vector, nearby passages are retrieved, then added to the prompt. The LLM is called once, to generate the answer.

Agentic RAG also uses the model to decide: which source to query, how to rephrase, whether the result is good enough.

The chain, from document to answer

The whole chain

A RAG has two halves. The preparation runs once, then again when documents change. The answer runs for every question. Everything the agent can find was decided on the left.

Most RAG failures start on the left, long before the model writes a word.

1. Chunking

A model cannot read a whole library for each question, so documents are cut into passages. Too long, and one passage mixes several subjects. Too short, and it loses its meaning.

Cut where the document cuts. A fixed size could have split “within 30 days” from the rule it belongs to.

2. Embedding

An embedding model turns each passage into a vector, a list of numbers that captures its meaning. Texts that say the same thing get close vectors, even with different words.

The model places sentences by meaning, not by spelling. These values are illustrative, and real vectors hold hundreds or thousands of numbers.

3. The vector store

The store keeps each vector with its text, its source and its access rights. It answers one question quickly: which passages are nearest to this vector? Keep access rights here, so you can filter before ranking.

The three nearest passages go on to the next step, and the warranty is left out. This 2D map is a sketch of a space with far more dimensions, and the scores are illustrative: closeness is usually measured by cosine similarity.

4. The types of search

A vector store answers one kind of question. Others need exact words, a database query, a graph of relations or a look inside the files. For an agent, each of these is a separate tool, which is why choosing the source matters.

Cyan searches documents, violet searches structured data, green reaches a source directly. Pick from the question, not from the tool you already have.

5. Combining searches and reranking

The hybrid search above is the usual default for documents. Run both searches, merge their rankings, then let a slower model reread only the best candidates.

Each search ranks a different document first. Article 12 wins because both found it, and BM25 catches exact codes that vectors miss.

6. The agent

So far the path never changes. The agent turns search into a tool it can use, skip or try again. That is the only difference between a classic RAG and an agentic one.

Without an agent the path is fixed. With one, each decision adds a model call, so cap the tries.

Four common roles

Routing: pick the right source, or none if the question is out of scope.
Query decomposition: split a complex question, hand out the parts, merge the answers.
ReAct: adapt the next search to the previous result.
Plan and execute: draw up a full plan before searching.

What matters most

Fixed-size chunking is a starting point, not a solution. Hybrid search (keywords and meaning) and reranking often improve quality more than an agent does.
Measure retrieval on its own first: if the right passage doesn’t arrive, the best agent in the world will answer badly.
Everything you retrieve is untrusted input. A document can contain instructions aimed at the agent.
With permissioned data, filter before ranking. A filter applied afterward can let forbidden documents influence results.

The trade-off

Each model decision adds a call, so cost and latency. For a simple question on one source, classic RAG stays faster and cheaper. And neither one fully removes hallucinations.

Fix retrieval before adding an agent.

Worked example: Merging two rankings

Keyword search finds the exact article number and vector search finds the meaning. Reciprocal rank fusion merges them using ranks only, so the two kinds of score never need comparing.

function fuse(rankings, k = 60) {
  const scores = new Map();
  for (const ranking of rankings) {
    ranking.forEach((id, rank) => {
      scores.set(id, (scores.get(id) ?? 0) + 1 / (k + rank + 1));
    });
  }
  return [...scores].sort((a, b) => b[1] - a[1]).map(([id]) => id);
}

const keywords = ['refund-art-12', 'shipping-faq', 'returns-guide'];
const vectors = ['returns-guide', 'refund-art-12', 'warranty'];

fuse([keywords, vectors]);

['refund-art-12', 'returns-guide', 'shipping-faq', 'warranty']

Article 12 comes first because both searches rank it well, even though neither is sure of it alone.

Try it in the editor

1function rrf(rankings, { k = 60, limit = 10 } = {}) {2  // TODO3  return [];4}56module.exports = { rrf };7

Give the agent a skillMerge rankings with RRF

Open in the editor

Legal notice and privacy