Agentic RAG vs Traditional RAG in .NET in 2026 — When Each Wins, Real Code, Production Metrics

Traditional RAG is what every "ChatGPT for your docs" tutorial builds: embed the question, fetch top-k chunks, stuff them into a prompt, return the answer. It works beautifully for ~75% of the questions you'd ask a customer-support assistant.

Then a partner asks Mattrx Insights "My conversion rate dropped 18% last week. Look at my webhook logs, check the dashboard error rate, find related docs, and tell me what's wrong" — and traditional RAG falls over. The question can't be answered by one round of retrieval. It needs: log inspection, SQL query, doc lookup, recent-events check, hypothesis generation, validation. That's an agent.

Agentic RAG is RAG where the LLM acts — picks tools, plans steps, retrieves multiple times, self-critiques, and synthesizes. It can answer questions traditional RAG can't. It also costs ~10× more per query, takes ~4× longer, and is harder to debug. If you use it for everything, you'll burn $40,000/month on questions a $4,000/month traditional RAG would have answered just fine.

The right architecture in 2026 isn't "pick one." It's a router + both — traditional RAG for the 75% of queries it handles in 2 seconds for $0.004, agentic RAG for the 25% that need an agent. This guide is the complete .NET implementation of both, with Semantic Kernel wiring, real Mattrx code, and the production metrics from running both side-by-side: Mattrx Help (traditional, ~12k queries/day, $0.004/query) and Mattrx Insights (agentic, ~1,800 queries/day, $0.038/query). Plus the router that decides which a query goes to and saves us ~$3,200/month vs "agentic everything."

---

TL;DR

The decision matrix:

Dimension	Traditional RAG	Agentic RAG
Steps per query	1 (retrieve → generate)	3–8 (plan → tool calls → critique → synth)
Tool calls	0	2–6 on average
Cost / query	$0.004	$0.038 (~10×)
End-to-end latency p95	2.1 s	8.2 s (~4×)
Best for	FAQ, doc lookup, "where is X"	Multi-step analysis, debugging, "why is X"
Accuracy on simple questions	78%	71% (slightly lower — extra steps add noise)
Accuracy on complex questions	32% (often hallucinates)	84%
Debuggability	Easy (one prompt)	Hard (multi-step trace)
Failure modes	Hallucination, missed context	Tool-loop, premature stop, infinite plan
Right model	gpt-4o-mini	gpt-4o (mini struggles with planning)

The 2026 rule of thumb: Use a router. Traditional RAG by default. Agentic RAG when the question requires multiple tools, multiple knowledge sources, or iteration.

Mattrx production wins from running both with a router (4-week build):

• Total daily queries served by AI: ~13,800
• Routed to traditional RAG: 78% (~10,800/day)
• Routed to agentic RAG: 22% (~3,000/day)
• Avg cost per query (blended): $0.012 — vs $0.038 if everything were agentic
• Monthly savings from routing: ~$3,200
• Accuracy uplift on complex queries: 32% (traditional) → 84% (agentic) on the routed-to-agent slice
• User-rated "this was helpful": 84% (Mattrx Help) and 89% (Mattrx Insights) — both higher than human first-response (61%)
• Tool calls/query on the agentic path (avg): 3.4
• Agent loop max iterations hit: 2% of queries (we kill at 6)
• Hallucination rate on complex queries: 18% (traditional) → 6% (agentic)

Neither is "better." They're tools for different kinds of questions. Use a router.

---

1. The mental model — what makes a RAG "agentic"

Traditional RAG = retrieve(question) → generate(prompt). Agentic RAG = agent(question), where the agent decides what to retrieve, in what order, and stops only when it's confident.

Side-by-side

TRADITIONAL RAG                          AGENTIC RAG
───────────────                          ───────────

   User question                            User question
        │                                        │
        ▼                                        ▼
   Embed query                            ┌──────────────┐
        │                                  │  PLANNER     │  "What do I need?"
        ▼                                  │  (LLM)       │
   Vector search top-k                     └──────┬───────┘
        │                                         │
        ▼                                         ▼
   Build prompt                            Pick tool 1
   (system + chunks                              │
    + question)                                   ▼
        │                                  Execute tool 1
        ▼                                         │
   LLM generates                                  ▼
        │                                  ┌──────────────┐
        ▼                                  │  CRITIC      │  "Enough? Or do
   Stream tokens                          │  (LLM)       │   I need more?"
                                          └──────┬───────┘
                                                  │
                                       ┌──────────┴──────────┐
                                       │                      │
                                       ▼                      ▼
                                   Yes — synth         No — pick tool 2
                                   final answer             │
                                                            ▼
                                                  Loop until confident
                                                  or max iterations (6)
                                                            │
                                                            ▼
                                                  Synthesize final answer
                                                  with all gathered context

The four things only agents can do

1. Decompose — split "compare Q1 to last year and recommend optimizations" into sub-questions.
2. Iterate — if the first retrieval returned junk, reformulate and try again.
3. Choose tools — pick searchDocs for definitions, runSqlQuery for numbers, getRecentLogs for debugging.
4. Self-critique — judge whether the answer is grounded before returning it.

If your use case doesn't need any of those four, traditional RAG is the right choice. Don't pay 10× for capability you don't use.

The four costs of going agentic

1. Money — each LLM call costs $0.005–$0.030. Agentic queries often make 4–8 LLM calls.
2. Latency — sequential tool calls add up. p95 climbs from 2s to 8s+.
3. Debuggability — when a traditional RAG is wrong, you can read 1 prompt and 1 response. When an agent is wrong, you read 6 prompts, 6 tool results, and a plan tree.
4. Failure modes — agents can loop forever, stop too early, or use the wrong tool. None of these can happen with traditional RAG.

---

2. Mattrx context — two products, two architectures

We run both patterns in production:

Product	What it does	Query shape	Architecture
Mattrx Help	In-product docs assistant	"How do I X?", "What does error 4012 mean?"	Traditional RAG (covered in the Semantic Kernel + Azure AI Search guide)
Mattrx Insights	Analytical assistant	"Why did conversions drop?", "Debug my integration"	Agentic RAG

Mattrx Insights has these tools available to the agent:

docs.search(query)           → vector search over docs
analytics.runQuery(spec)     → typed analytics queries (read-only)
events.getRecent(filter)     → last N webhook/customer events for this tenant
logs.search(query, window)   → search application logs (last 24h)
config.getStatus()           → current integration config + health
metrics.compare(a, b)        → period-over-period analytics summary

Six tools. The agent picks 2–4 of them per query. The user never knows which ones — they just see the answer with citations.

---

3. Traditional RAG — the recap (so you can see the contrast)

// Mattrx.Application/Help/TraditionalRag.cs
public sealed class TraditionalRagService(
    Kernel kernel,
    HybridRetriever retriever,
    PromptBuilder promptBuilder,
    ICurrentUser user)
{
    public async IAsyncEnumerable<RagEvent> AskAsync(
        string question, [EnumeratorCancellation] CancellationToken ct)
    {
        // 1. ONE retrieval round
        var chunks = await retriever.RetrieveAsync(question, user.TenantId, topK: 6, ct);
        yield return new RagEvent("sources", chunks.Select(c => new { c.Id, c.Title, c.SourceUrl }));

        // 2. ONE prompt build
        var chatHistory = promptBuilder.Build(question, chunks, []);

        // 3. ONE LLM call (streamed)
        var chat = kernel.GetRequiredService<IChatCompletionService>();
        await foreach (var part in chat.GetStreamingChatMessageContentsAsync(
            chatHistory,
            new OpenAIPromptExecutionSettings { Temperature = 0.2, MaxTokens = 600 },
            kernel, ct))
        {
            if (!string.IsNullOrEmpty(part.Content))
                yield return new RagEvent("token", part.Content);
        }

        yield return new RagEvent("done", null);
    }
}

Two function calls. One LLM call. ~$0.004. 2 seconds. Done.

This is the right answer for 75% of queries. Don't abandon it.

---

4. Agentic RAG — the building blocks

A production agentic RAG in .NET needs five pieces:

1. Tools — typed functions the agent can call (with side-effect awareness).
2. System prompt — instructions, role, "when to stop" rules.
3. Loop — the orchestration that keeps calling the LLM until done.
4. Critic — a check that says "you have enough info" or "go look more."
5. Budget guards — max iterations, max cost, max tool calls.

We use Semantic Kernel's auto function-calling for the loop, with custom budget enforcement on top.

4.1 Defining tools as Semantic Kernel plugins

// Mattrx.Application/Insights/Tools/DocsSearchTool.cs
public sealed class DocsSearchTool(HybridRetriever retriever, ICurrentUser user)
{
    [KernelFunction("search_docs")]
    [Description("Search Mattrx documentation, API reference, and runbooks by natural-language query. " +
                 "Use this for 'how to', 'what is', 'when does' style questions. " +
                 "Returns up to 6 relevant doc snippets with citations.")]
    public async Task<IReadOnlyList<DocSnippet>> SearchAsync(
        [Description("The natural-language question to search for.")] string query,
        CancellationToken ct)
    {
        var chunks = await retriever.RetrieveAsync(query, user.TenantId, topK: 6, ct);
        return chunks.Select(c => new DocSnippet
        {
            Id = c.Id, Title = c.Title, Excerpt = c.Content, Url = c.SourceUrl,
        }).ToList();
    }
}

// Mattrx.Application/Insights/Tools/AnalyticsTool.cs
public sealed class AnalyticsTool(IAnalyticsRepo analytics, ICurrentUser user)
{
    [KernelFunction("run_analytics_query")]
    [Description("Run a typed analytics query against the tenant's data. " +
                 "Supported metrics: revenue, conversion_rate, click_through_rate, event_count. " +
                 "Use this when the user asks for numbers, comparisons, or trends.")]
    public async Task<AnalyticsResult> RunAsync(
        [Description("Metric name: revenue, conversion_rate, click_through_rate, or event_count.")] string metric,
        [Description("ISO date range start, inclusive, e.g. 2026-04-01.")] string from,
        [Description("ISO date range end, inclusive, e.g. 2026-04-30.")] string to,
        [Description("Optional campaign filter — empty for all.")] string campaign,
        CancellationToken ct)
    {
        // Server-side identity — the agent CANNOT pass a different tenant id
        return await analytics.QueryAsync(new AnalyticsSpec
        {
            TenantId = user.TenantId,
            Metric = metric,
            From = DateOnly.Parse(from),
            To = DateOnly.Parse(to),
            Campaign = string.IsNullOrEmpty(campaign) ? null : campaign,
        }, ct);
    }
}

// Mattrx.Application/Insights/Tools/LogsTool.cs
public sealed class LogsTool(ILogSearch logs, ICurrentUser user)
{
    [KernelFunction("search_recent_logs")]
    [Description("Search recent application logs (last 24h) for the tenant. " +
                 "Use this when the user reports an error, slowness, or unexpected behavior.")]
    public async Task<IReadOnlyList<LogEntry>> SearchAsync(
        [Description("Search query — error code, keyword, or partial trace.")] string query,
        [Description("Optional log level: error, warn, info — empty for all.")] string level,
        CancellationToken ct)
    {
        return await logs.SearchAsync(user.TenantId, query, level, hours: 24, ct);
    }
}

Three rules every tool must follow:

1. Server-side identity — user.TenantId is from the JWT, not from the agent. The agent cannot access another tenant's data.
2. Read-only by default — agent tools query data; they do not mutate. Mutations need explicit user confirmation, not agent autonomy.
3. Rich Description attributes — the LLM reads these to choose tools. Bad descriptions = bad tool choices.

4.2 The orchestrating service (Semantic Kernel auto-function-calling loop)

// Mattrx.Application/Insights/AgenticRagService.cs
public sealed class AgenticRagService(
    Kernel kernel,
    DocsSearchTool docsTool,
    AnalyticsTool analyticsTool,
    LogsTool logsTool,
    EventsTool eventsTool,
    ICurrentUser user,
    ILogger<AgenticRagService> log)
{
    private const int MaxIterations = 6;
    private const decimal MaxCostUsd = 0.50m;          // hard budget per query

    public async IAsyncEnumerable<AgentEvent> AskAsync(
        string question, [EnumeratorCancellation] CancellationToken ct)
    {
        // 1. Clone the kernel and add the tools as plugins
        var scopedKernel = kernel.Clone();
        scopedKernel.Plugins.AddFromObject(docsTool, "Docs");
        scopedKernel.Plugins.AddFromObject(analyticsTool, "Analytics");
        scopedKernel.Plugins.AddFromObject(logsTool, "Logs");
        scopedKernel.Plugins.AddFromObject(eventsTool, "Events");

        // 2. System prompt — tells the agent how to behave
        var history = new ChatHistory(SystemPrompt);
        history.AddUserMessage(question);

        // 3. The settings turn on auto function calling — SK handles the loop for tools,
        //    but we wrap our own iteration cap on top
        var settings = new OpenAIPromptExecutionSettings
        {
            Temperature = 0.1,
            MaxTokens = 1200,
            ToolCallBehavior = ToolCallBehavior.AutoInvokeKernelFunctions,
            FunctionChoiceBehavior = FunctionChoiceBehavior.Auto(),
        };

        var chat = scopedKernel.GetRequiredService<IChatCompletionService>();

        int iter = 0;
        decimal cost = 0m;
        var sb = new StringBuilder();

        while (iter < MaxIterations)
        {
            iter++;
            yield return new AgentEvent("iter_start", new { iteration = iter });

            ChatMessageContent response;
            try
            {
                response = await chat.GetChatMessageContentAsync(history, settings, scopedKernel, ct);
            }
            catch (Exception ex)
            {
                log.LogError(ex, "Agent call failed at iter {Iter}", iter);
                yield return new AgentEvent("error", new { message = "agent_call_failed" });
                yield break;
            }

            cost += EstimateCostFromResponse(response);
            yield return new AgentEvent("step_done", new { iteration = iter, costUsd = cost });

            // Budget guard
            if (cost >= MaxCostUsd)
            {
                yield return new AgentEvent("error", new { message = "budget_exceeded" });
                yield break;
            }

            history.Add(response);

            // If no tool calls were issued AND we have content, we have our answer
            if (response.Items.OfType<FunctionCallContent>().Any() == false &&
                !string.IsNullOrWhiteSpace(response.Content))
            {
                sb.Append(response.Content);
                yield return new AgentEvent("final_answer", response.Content);
                break;
            }

            // SK auto-invokes the function calls — the loop continues with the results
        }

        if (iter >= MaxIterations)
        {
            yield return new AgentEvent("error", new { message = "max_iterations_exceeded" });
        }

        yield return new AgentEvent("done", new { iterations = iter, costUsd = cost });
    }

    private const string SystemPrompt = """
        You are Mattrx Insights, an analytical assistant for Mattrx marketers.

        TOOL USE RULES:
        - You have these tools: search_docs, run_analytics_query, search_recent_logs, get_recent_events.
        - Decompose the user's question. Use multiple tools as needed.
        - If a tool returns insufficient data, REFORMULATE and try again — once.
        - Stop calling tools once you can answer with grounded evidence.
        - Maximum 5 tool calls per question.

        ANSWER RULES:
        - Cite every claim with the tool result that supported it ([doc:title], [analytics:metric], [log:line]).
        - If you cannot answer with the tools provided, say so plainly. Do NOT speculate.
        - Be concise: under 300 words for the answer body.
        - End with a "Recommended next steps" section if relevant.

        ETHICS RULES:
        - Never recommend actions that affect other tenants' data.
        - Never include partner-private data in the answer if the user is not that partner.
        """;
}

4.3 What this code does step-by-step

User: "My conversion rate dropped 18% last week. Why?"

  ITER 1 — Agent plans:
    "I need (a) recent conversion data (b) recent errors/incidents (c) any related docs."
    Tool call → run_analytics_query("conversion_rate", "2026-05-20", "2026-05-27", "")
    Tool call → run_analytics_query("conversion_rate", "2026-05-13", "2026-05-20", "")
    Result: 4.2% (last week) vs 5.1% (week prior) — confirmed 18% drop.

  ITER 2 — Agent thinks:
    "Was there an error?"
    Tool call → search_recent_logs("conversion", "error")
    Result: 1,240 entries of "Webhook delivery failed: 502" between 2026-05-22 and 2026-05-26.

  ITER 3 — Agent thinks:
    "Look at recent webhook events to confirm scope."
    Tool call → events.getRecent({kind: "webhook", status: "failed", since: "2026-05-22"})
    Result: 1,184 failed deliveries to partner XYZ between 2026-05-22 and 2026-05-26.

  ITER 4 — Agent thinks:
    "Find docs about this error class."
    Tool call → search_docs("webhook 502 retry policy")
    Result: doc snippet about retry config + idempotency key handling.

  ITER 5 — Agent has enough; synthesizes:

    "Your conversion rate dropped 18% (5.1% → 4.2%) between May 20 and May 27 [analytics:conversion_rate].
     The likely cause: 1,184 failed webhook deliveries to partner XYZ between May 22–26, all returning
     502 Bad Gateway [log:webhook-delivery] [events:failed]. Each failed delivery costs you a tracked
     conversion event.

     Recommended next steps:
     1. Check partner XYZ's webhook endpoint health.
     2. Confirm your retry policy is at default 3 retries with exponential backoff [doc:retry-policy].
     3. Backfill the missing conversions via /api/events/replay (run after partner XYZ confirms uptime)."

  → No more tool calls. Loop exits. Final answer streamed back to user.
  Total: 5 LLM calls + 4 tool calls. Cost ~$0.042. Latency ~9 seconds.

A traditional RAG with the same question would have searched docs once, retrieved generic "improve conversion" content, and produced a vague answer that wouldn't have mentioned partner XYZ or the webhook failures. It can't, because it only has one tool.

---

5. The router — deciding which RAG handles each query

The router is the single most cost-effective piece of the architecture. It classifies each incoming query as "needs an agent" or "doesn't" and dispatches accordingly.

5.1 The classifier

// Mattrx.Application/Rag/QueryRouter.cs
public sealed class QueryRouter(Kernel kernel)
{
    public async Task<RouteDecision> ClassifyAsync(string question, CancellationToken ct)
    {
        var chat = kernel.GetRequiredService<IChatCompletionService>();

        var history = new ChatHistory(ClassifierPrompt);
        history.AddUserMessage(question);

        var response = await chat.GetChatMessageContentAsync(
            history,
            new OpenAIPromptExecutionSettings
            {
                Temperature = 0,
                MaxTokens = 50,
                ResponseFormat = "json_object",
            },
            kernel, ct);

        var json = JsonSerializer.Deserialize<RouteDecision>(response.Content!)
                   ?? new RouteDecision("traditional", "fallback");
        return json;
    }

    private const string ClassifierPrompt = """
        You are a query router. Classify each user question as either:

        "traditional" — answerable by retrieving 1 set of docs and responding. Examples:
          - "How do I configure X?"
          - "What does error 4012 mean?"
          - "Where is the API key in the dashboard?"

        "agentic" — requires multi-step reasoning, multiple data sources, debugging, or
                    comparison. Examples:
          - "Why did my conversion rate drop?"
          - "Compare my Q1 to Q2 and recommend changes."
          - "Debug my webhook integration."
          - "Is something wrong with my account?"

        Respond ONLY with JSON: {"route":"traditional"|"agentic","reason":"<short why>"}
        """;
}

public sealed record RouteDecision(string Route, string Reason);

5.2 The orchestrator

// Mattrx.Api/Rag/RagEndpoints.cs
app.MapPost("/api/rag/ask",
    async (RagAskRequest req, HttpContext ctx,
           QueryRouter router, TraditionalRagService traditional, AgenticRagService agentic,
           CancellationToken ct) =>
    {
        ctx.Response.Headers.ContentType = "text/event-stream";

        var decision = await router.ClassifyAsync(req.Question, ct);

        await WriteSseAsync(ctx.Response, "route",
            new { route = decision.Route, reason = decision.Reason }, ct);

        if (decision.Route == "agentic")
        {
            await foreach (var evt in agentic.AskAsync(req.Question, ct))
                await WriteSseAsync(ctx.Response, evt.Type, evt.Payload, ct);
        }
        else
        {
            await foreach (var evt in traditional.AskAsync(req.Question, ct))
                await WriteSseAsync(ctx.Response, evt.Type, evt.Payload, ct);
        }
    })
.RequireAuthorization();

Why the router uses gpt-4o-mini at temperature 0: it's a classification task. The smaller, cheaper model handles it cleanly with the right prompt. Total router overhead: ~$0.0002 per query and ~80ms p95.

5.3 The savings

Without router (everything goes agentic):
  13,800 queries/day × $0.038 = $524.40/day = ~$15,732/month

With router (78% traditional, 22% agentic):
  10,800 × $0.004 + 3,000 × $0.038 + 13,800 × $0.0002 (router) = $43.20 + $114 + $2.76 = $159.96/day = ~$4,800/month

Monthly savings: ~$10,900 (yes, the bigger Mattrx Help volume tips this even further).

That's the single highest-ROI engineering decision in the AI stack. Build the router first.

---

6. The full architecture (one picture)

┌──────────────────────────────────────────────────────────────────────┐
│ User question                                                         │
└────────────────────────────┬─────────────────────────────────────────┘
                             │
                             ▼
                ┌────────────────────────┐
                │ QueryRouter (gpt-4o-mini) │
                │ "traditional" or "agentic"? │
                └─────────┬────────┬─────┘
                          │        │
        ┌─────────────────┘        └─────────────────┐
        │                                              │
        ▼ (78% of queries)                            ▼ (22%)
┌─────────────────────────┐                ┌──────────────────────────┐
│ TraditionalRagService   │                │ AgenticRagService          │
│  - 1 retrieval          │                │  - Plan                    │
│  - 1 LLM call           │                │  - Tool 1                  │
│  - Stream               │                │  - Tool 2                  │
│                         │                │  - Critic / iterate        │
│  Tools used: 1          │                │  - Tool N                  │
│  Avg cost: $0.004       │                │  - Synthesize              │
│  Avg latency: 2.1s      │                │                            │
└────────────┬────────────┘                │  Tools available: 6        │
             │                              │  Avg tools/query: 3.4       │
             │                              │  Max iterations: 6           │
             │                              │  Avg cost: $0.038            │
             │                              │  Avg latency: 8.2s           │
             │                              └─────────────┬────────────────┘
             │                                            │
             └─────────────────┬──────────────────────────┘
                               ▼
                ┌──────────────────────────┐
                │ Stream SSE to client      │
                └──────────────────────────┘

  Telemetry: every step traced via OpenTelemetry -> Application Insights
  Budget: per-tenant daily cap, hard 429 above
  Eval: nightly golden set runs both paths on representative questions

---

7. Real production metrics — Traditional vs Agentic vs Routed

After 4 weeks of running both with the router:

Metric	Traditional only (Mattrx Help)	Agentic only (Mattrx Insights)	Routed (90% of new queries)
Daily queries	12,000	1,800	13,800 (combined)
Avg cost / query	$0.004	$0.038	$0.012 (blended)
Avg latency p95	2.1 s	8.2 s	2.1s (traditional) / 8.2s (agentic)
Tool calls / query (avg)	0	3.4	n/a (routed)
Accuracy on simple questions	78%	71%	78% (router sends them to traditional)
Accuracy on complex questions	32%	84%	84% (router sends them to agentic)
Hallucination rate	4% (simple) / 18% (complex)	6%	overall 5%
Max iterations hit	n/a	2% of queries	0.4%
Monthly OpenAI bill	~$1,440	~$2,050	~$4,800 total (vs $15,700 agentic-only)
User-rated "this was helpful"	84%	89%	86% overall
Support tickets deflected / month	520	180 (complex)	700 combined

The routed mode is strictly better than either alone:

• Better accuracy than traditional (because complex queries get the agent).
• Cheaper than agentic-only (because simple queries skip the agent).
• Acceptable latency (most queries finish in 2 seconds; only the 22% that need agentic pay 8s).

---

8. The mental checklist — before adding agentic RAG to your stack

• Do at least 15% of your queries genuinely need multiple data sources, multi-step reasoning, or tool use? (If not, stay traditional.)
• Can you afford ~10× per-query cost on the agentic share?
• Have you built a router (with eval coverage) so simple queries don't hit the agent?
• Are your tools server-side identity-scoped? (Agent must not be able to access other tenants.)
• Do tools enforce read-only by default? (Mutations need user confirmation.)
• Are tool descriptions clear enough that the LLM picks the right tool?
• Is there a hard cap: max iterations + max cost + per-tenant daily limit?
• Is every iteration traced via OpenTelemetry so you can debug failures?
• Does the eval set include both simple and complex queries, and does it measure routing accuracy?
• Is the agent's final answer required to cite tool results (not hallucinate)?

If any answer is "I'm not sure" — fix it before users see it.

---

9. Honest stuff

• Most apps don't need agentic RAG. If your queries are FAQ-shaped, stay traditional. The 10× cost and 4× latency aren't worth it.
• The router is the highest-leverage piece. Built before the agent, it saves real money from day one.
• Tool descriptions are 50% of the agent's quality. Spend time on them. They're effectively the agent's "manual."
• Tools must be server-side identity-scoped. The single most dangerous bug is the agent accessing another tenant. Don't trust the prompt for any security-relevant input.
• Hard caps prevent runaway cost. Max 6 iterations, max $0.50/query, max 5,000 queries/tenant/day. We've hit each of these in production.
• gpt-4o-mini struggles with multi-step planning. Use gpt-4o for the agentic path — the per-call cost difference is dwarfed by the smaller number of calls (better plans -> fewer iterations).
• The biggest debugging wins come from tracing every iteration. Without OpenTelemetry, an agent that loops forever is a black box.
• Self-critique helps a lot. A system prompt rule "before answering, check if you've cited each claim with a tool result" cut Mattrx Insights' hallucinations from 9% to 6%.
• The eval set matters more than the prompt. Without it, prompt tuning is a coin flip.

---

10. The right mental model

In one line: Traditional RAG is retrieve -> generate. Agentic RAG is plan -> loop(tool -> critique) -> synthesize. A router decides which to use.

Three habits that prevent 90% of the pain in this guide:

1. Build the router first. Before you build the agent. It's cheaper, it saves money on day one, and you'll need it forever.
2. Treat tools as a public API. Server-side identity. Read-only by default. Rich descriptions. Multi-tenant tested.
3. Hard caps on iterations + cost + daily volume. Agents will try to loop. Will try to spend $5 on a $0.04 question. The guards are not optional.

Apply that, and the next "should we go agentic?" question takes a 10-minute classification of your query distribution, not a quarter of architecture debate.

---

Further reading

• Semantic Kernel — Auto function calling — the loop SK provides out of the box.
• Semantic Kernel — Agents — higher-level agent abstractions for multi-agent scenarios.
• Microsoft — Agentic RAG patterns — Microsoft's reference architecture.
• Anthropic — Building effective agents — when (and when not) to go agentic. The canonical essay.
• LangChain — Agents — Python ecosystem analog; the concepts transfer cleanly.
• PrepStack — Build a RAG Chatbot in C# with Semantic Kernel + Azure AI Search — the traditional RAG deep-dive this guide builds on.
• PrepStack — React + AI: Building a ChatGPT-Like Application — the front-end + Node sibling on the same RAG concepts.

---

Deciding whether to go agentic for a specific feature? Email randhir.jassal@gmail.com with the query distribution (how many simple vs complex queries you actually have) — happy to help draft the routing thresholds and tool list that match your workload.