Agentic RAG or plain RAG?

Retrieval-augmented generation (RAG) gives a language model the documents it needs at the moment it answers. In its plain form the shape is fixed: take the question, search the index once, put the best passages in front of the model, write the answer. Agentic RAG puts a loop around that shape. The system decides whether to search at all, what to search for, which tool to call, and whether what it found is enough to answer.

Neither one is the better design. They suit different questions, and the question on the desk decides which one the work needs.

What one pass does well

Most questions an expert looks up are single lookups. What does the current policy say about the notice period? Which clause in this contract covers early termination? What is the tolerance for this inspection step? One search, one passage, one answer.

For questions like these, plain RAG is hard to beat. Cost is one search and one model call. Latency is predictable. The audit trail is short: the question, the passages retrieved, the answer, and the source each passage came from. Testing is direct, because you can check whether the right passage came back and whether the answer followed from it.

The original RAG work paired a generator with a retriever over an external index. That pairing is still the right default. Changing knowledge lives in the index, where it can be updated without touching the model.

Four signs one pass is not enough

1. The second lookup depends on the first. The question “is the guarantor on this facility still in good standing?” needs two facts: who the guarantor is, from the contract, and the guarantor's current status, from another source. Trivedi and colleagues observed that one-step retrieve-and-read is insufficient for multi-step questions, because what to retrieve next depends on what has already been found.
2. The question's words are not the document's words. An operator asks about “the late-payment rule”; the policy says “overdue receivables”. The search misses even though the answer is in the index. Rewriting the query before retrieving, as in the Rewrite-Retrieve-Read approach, closes that gap between the input text and the knowledge retrieval needs.
3. The answer lives in a system, not a document. A current balance, an open order, a machine's last calibration date. No passage holds it. The right step is a tool call: a read-only SQL query or an API request, with the result recorded like a retrieved passage.
4. The evidence may not support the answer. Self-RAG trains a model to decide when to retrieve and to assess both the retrieved passages and its own output. A simpler version, often enough, is a separate check that each claim in the answer points to a passage that supports it, and that stops when one does not.

If none of the four shows up in the work, plain RAG is the design.

What the loop costs

Every hop is another search, another model call, or another tool call. Cost grows with the number of hops, and latency becomes variable: most questions may finish quickly while a few take many steps.

The audit trail grows too. A reviewer now needs every query the system wrote, every passage and tool result it read, the order it read them in, and the reason it stopped. Without that record, a wrong answer from a five-hop loop is close to impossible to explain.

Loops also fail in their own ways. They keep searching when the answer is not in the index. They drift from the original question. They rewrite a query into something the operator never asked. Each needs a limit set before the loop runs: a maximum number of hops, a stop when evidence is missing, and a path to a named person when the limit is reached.

How to test which one a task needs

1. Collect real questions from the desk, and have the institution's expert write or grade the correct answer and its source before anything is built.
2. Run plain RAG first on that set. Record two things per question: did the right passage come back, and did the expert accept the answer.
3. Sort the failures. Missed because of wording points to query rewriting. Missed because a second fact was needed points to multiple hops. Missed because the fact sits in a system points to a tool call. Answered without support points to an evidence check.
4. Add only the step that fixes the largest group of failures, then rerun on questions held back from development. Keep the step if it shows a gain there; drop it if it does not.
5. Route by question where both kinds exist. Simple lookups take the single pass; the questions that need hops take the loop. The router is itself tested on the same set.

The result is often a plain pipeline with one added step, not a general agent.

What retrieval never changes

Retrieval changes the evidence the model reads. It does not change the model's weights, and it does not change who decides.

The sources a loop may search and the tools it may call are named in the agent's written harness, read-only by default. The loop cannot add a source, a tool, or a permission because a question seems to need one. Policy and authority stay outside the model and outside the index. When the evidence is not enough, the system says so, and the case goes to a named person with what was found and where the search stopped.

Count the hops in the real questions first. The design follows from the count.