Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 47 of 57

Agentic RAG

0 of 57 complete

Why it matters

A fixed retrieve-then-generate pipeline cannot adapt when the first search misses, the question needs two sources, or the answer requires a calculation. An agent can reformulate, switch tools and iterate, which handles long-tail questions a pipeline fails. The cost is that the control flow is generated at runtime, so budget, tool design and trajectory logging become the engineering work. Effort shifts from pipeline construction to constraining a planner.

Mental model

Agentic RAG is a research assistant with library access: given a question, they choose which catalogue, which shelf and when to stop looking. The metaphor breaks because the assistant's judgement is a sampled model output, so the same question can produce a different search path each time.

How it works

Mechanism, step by stepInteractive
  1. 1Define a small tool set with a typed input and a bounded result size: keyword search, vector search, structured lookup and document fetch.
  2. 2Write tool descriptions that state when to use each one and what a result contains, since selection depends on them.
  3. 3Give the agent a step budget and a token budget, and make exhaustion return the best current answer rather than an error.
  4. 4Require the agent to record which tool call supported each claim, so citations survive the trajectory.
  5. 5Constrain iteration: stop when a new search returns no new document ids, not when the model feels finished.
  6. 6Log the full trajectory of queries, tool calls, results and decisions, and evaluate it, not only the final text.

Step 1 of 6

Define a small tool set with a typed input and a bounded result size: keyword search, vector search, structured lookup and document fetch.

Step 1/6 · step 1

Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.

What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.

Text version

The page's steps: 1. Define a small tool set with a typed input and a bounded result size: keyword search, vec… 2. Write tool descriptions that state when to use each one and what a result contains, since… 3. Give the agent a step budget and a token budget, and make exhaustion return the best curr… 4. Require the agent to record which tool call supported each claim, so citations survive th… 5. Constrain iteration: stop when a new search returns no new document ids, not when the mod… 6. Log the full trajectory of queries, tool calls, results and decisions, and evaluate it, n…

  1. Define a small tool set with a typed input and a bounded result size: keyword search, vector search, structured lookup and document fetch.
  2. Write tool descriptions that state when to use each one and what a result contains, since selection depends on them.
  3. Give the agent a step budget and a token budget, and make exhaustion return the best current answer rather than an error.
  4. Require the agent to record which tool call supported each claim, so citations survive the trajectory.
  5. Constrain iteration: stop when a new search returns no new document ids, not when the model feels finished.
  6. Log the full trajectory of queries, tool calls, results and decisions, and evaluate it, not only the final text.

Trade-offs

ChoiceGood forCostFails when
Fixed pipelinePredictable latency and costCannot adapt to a failed first searchQueries vary widely in difficulty
Agent with many toolsHandles diverse tasksTool confusion and longer trajectoriesTool descriptions overlap
Agent with two toolsReliable selectionLimited coverageThe task needs structured data as well as text
Unbounded reasoningHighest chance of a complete answerCost and tail latencyThe product has an interactive budget
Trade-off mapInteractive

Good for

Predictable latency and cost

Cost

Cannot adapt to a failed first search

Fails when

Queries vary widely in difficulty

  1. Queries vary widely in difficulty
  2. Tool descriptions overlap
  3. The task needs structured data as well as text
  4. The product has an interactive budget

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Fixed pipeline is good for Predictable latency and cost, costs Cannot adapt to a failed first search, and fails when Queries vary widely in difficulty. Agent with many tools is good for Handles diverse tasks, costs Tool confusion and longer trajectories, and fails when Tool descriptions overlap. Agent with two tools is good for Reliable selection, costs Limited coverage, and fails when The task needs structured data as well as text. Unbounded reasoning is good for Highest chance of a complete answer, costs Cost and tail latency, and fails when The product has an interactive budget

Failure modes

  • Tool thrashing. The agent alternates between two searches without new results and burns the budget. Debug: count distinct result ids per step and stop when a step adds none.
  • Silent tool failure. A search error is returned as an empty result and the agent answers from memory. Debug: distinguish empty from failed in the tool response and assert the agent sees the difference.
  • Trajectory without provenance. The final answer cannot be traced to a tool call, so it cannot be cited or audited. Debug: require a call id per claim and fail the response when one is missing.

In production

Track steps per request, tool error rate and the share of trajectories that hit the budget. Cap steps and result sizes, and sample trajectories weekly for tool misuse.

Common misconceptions

People think: An agent is always better than a pipeline.

Actually: It buys adaptivity with latency and variance; a fixed pipeline wins when queries are homogeneous.

People think: More tools give the agent more capability.

Actually: Overlapping tools create selection errors and longer trajectories more often than they add coverage.

Interview angle

Justify the agent by query heterogeneity, then constrain it with a small tool set, budgets and stopping rules. Evaluate trajectories as well as answers. An autonomous loop with no cost model and no way to explain a bad answer is the answer that fails the follow-up.

Sources