Polyglot Architecture for Enterprise AI — .NET API Gateway + Python RAG/Agent Service (Real Code, Real Diagram)

Your enterprise app is .NET. Your existing UI is ExtJS. Your users live inside those screens. Now product wants RAG, agents, evals — and the entire AI ecosystem in 2026 ships in Python first. You don''t want to rewrite anything, and you don''t want to bolt a Frankenstein on top.

A polyglot architecture is the answer: keep .NET as your trusted system-of-record and security boundary, run the AI workload as a separate Python service, talk to it over a clean contract, and observe both sides end-to-end. This guide is a real recipe — diagram, code, deployment, and operational playbook — for shipping that pattern in production.

---

The architecture at a glance

ExtJS web UI ─────► .NET API Gateway ─REST/gRPC─► Python RAG/Agent service
                    (ASP.NET Core)                 (FastAPI + LangChain + LangGraph)
                    auth · rate limit              ingestion · retrieval · orchestration
                    contract · security
                                                       │              │
                                                       ▼              ▼
                                                 Qdrant vector DB   LLM provider
                                                 (Docker)           (Azure OpenAI / OpenAI)

Observability spans every arrow:
  Langfuse          → LLM traces, prompts, evals, token cost
  OpenTelemetry     → end-to-end traces across .NET + Python + Qdrant + LLM

Five components, two languages, one observability fabric. We''ll build it bottom-up, then show the request flow, then show the code each layer actually runs.

---

1. Why polyglot, and why now

1.1 The honest problem statement

In 2026, three forces are pulling against each other inside every enterprise stack:

1. Your business logic is in .NET. Years of domain models, validation rules, RBAC, audit trails, integrations. None of that needs to move.
2. Your UI is ExtJS / WinForms / WPF / Razor. It works. Users know it. Rewriting the UI is a multi-quarter project that adds zero AI value.
3. The AI ecosystem ships Python first. LangChain, LangGraph, llama-index, vLLM, transformers, sentence-transformers, Ragas, DSPy — Python releases land months before .NET ports (if they exist at all).

You have three architectural options:

Option	Outcome
Rewrite the AI stack in .NET (Semantic Kernel only)	Slow to adopt new techniques; smaller library surface; locked behind .NET release cycles.
Rewrite the business app in Python	Throws away years of trusted code; introduces ~100% new bugs; takes 12+ months.
Polyglot — keep .NET, add Python alongside	Best tool per layer; clean security boundary; both stacks evolve independently.

The polyglot route is the only one that respects existing investment and lets you adopt the AI ecosystem at full speed.

1.2 What "polyglot" actually means here

It does not mean "we use 4 languages because it''s cool." It means:

• Single product, single front door. ExtJS users see one URL, one auth flow, one error model.
• .NET owns the contract. Every external caller goes through ASP.NET Core. AuthN, AuthZ, rate-limiting, contract versioning, audit logging — .NET.
• Python owns the AI workload. RAG, embeddings, retrieval, prompt assembly, multi-agent orchestration — FastAPI service, never directly exposed to the internet.
• The boundary is a typed contract — REST with OpenAPI, or gRPC with .proto. No language detail leaks.

This is a textbook bounded context split. The .NET service knows nothing about LangChain. The Python service knows nothing about your customer SSO provider. They meet at a single, narrow, well-tested wire format.

---

2. What each layer is for (and why this exact split)

2.1 ExtJS web UI — keep it, don''t touch it

You don''t need to migrate to React, Angular, or Blazor to ship AI features. ExtJS is just an HTTP client at the end of the day. Add a panel, a button, a chat drawer — whatever your UX calls for — and have it call the existing .NET API. The fact that the AI runs in Python is invisible to the browser.

2.2 .NET API Gateway — the security boundary

ASP.NET Core 9+ is excellent at being a typed gateway. It owns:

• Authentication (NextAuth/Cookie/JWT/Entra ID/Okta — your existing scheme).
• Authorization (RBAC, claims, row-level checks).
• Rate limiting (Microsoft.AspNetCore.RateLimiting).
• Contract versioning (/v1/, /v2/).
• Audit logging (who asked what, when, with what claims).
• Outbound HTTP/gRPC to Python (resilience policies via Polly).
• PII redaction before the prompt ever leaves your network.
• Quotas + billing accounting (token cost attribution to tenant/user).

The Python service is never exposed to the internet. It listens on an internal subnet / Docker network only. This is non-negotiable.

2.3 Python RAG / Agent service — the AI workhorse

A FastAPI service that does everything LangChain and LangGraph make easy:

• Ingestion — chunking, embedding, upserting to Qdrant.
• Retrieval — semantic search + optional BM25 hybrid + re-ranking.
• Prompt construction — system + retrieved chunks + user query + tool schemas.
• LLM calls — Azure OpenAI / OpenAI / Anthropic / Bedrock, pluggable.
• Multi-agent orchestration — LangGraph state machine (planner → tool-user → critic → summariser).
• Streaming — SSE / WebSocket for token-by-token UX.
• Evals — running test suites of (query, expected) pairs nightly.

2.4 Qdrant — the vector database

Qdrant runs in a Docker container next to your Python service. It stores embeddings (cosine / dot-product), supports filters, payload indexing, and sub-100 ms ANN searches up to ~10M vectors on a single node. It''s open source, no per-call fee, runs identically in dev and prod.

Alternatives: pgvector (cheapest if you already have Postgres), Weaviate, Pinecone (managed). Qdrant is the sweet spot for self-hosted enterprises.

2.5 LLM provider — pluggable, never hard-coded

Azure OpenAI for compliance-bound enterprises (data residency, customer-managed keys, no training on inputs). OpenAI direct for fastest model access. Anthropic / Bedrock for Claude. The Python service abstracts this behind a single interface so you can swap providers without touching .NET.

2.6 Observability — the part most teams skip and regret

• Langfuse — LLM-native tracing. Every prompt, every retrieved chunk, every token count, every latency, every eval score. Critical for debugging "why did the model answer wrong?"
• OpenTelemetry — end-to-end distributed tracing across .NET → Python → Qdrant → LLM. So when a request is slow, you can see exactly which span is the culprit.

Both run on every request. Tracing is the only way you''ll keep your sanity once this is in production.

---

3. The request flow, end to end

A user types "Show me last quarter''s compliance exceptions for the Mumbai branch" into the ExtJS UI:

1. Browser → POST /api/v1/assistant/query with bearer token.
2. .NET Gateway authenticates the JWT, checks claims (tenant_id, roles), records the audit event, runs rate-limit check, redacts any PII patterns, then proxies (gRPC) to Python.
3. Python RAG service receives the typed request, embeds the query with text-embedding-3-small, searches Qdrant filtered by tenant_id, retrieves the top 8 chunks, builds the prompt, calls Azure OpenAI with streaming enabled, optionally invokes LangGraph agents (e.g. an SQL-tool agent for live data), and streams tokens back.
4. Langfuse records: the prompt, the retrieved chunks (with scores), the LLM call (model, latency, tokens in/out, cost), the eval score (if a critic agent ran).
5. OpenTelemetry records a single trace with spans for: gateway-auth → gateway-rate-limit → grpc-call → embed → qdrant-search → llm-call → response-stream. Each span carries the same trace_id.
6. The browser renders streaming markdown in the chat drawer. Total round trip: 1.4 s to first token, 4.1 s to complete.

You can stand this up in two weeks. Let''s see the code.

---

4. The .NET API Gateway — production-grade code

4.1 Program.cs — minimal but real

using Grpc.Net.ClientFactory;
using Microsoft.AspNetCore.Authentication.JwtBearer;
using Microsoft.AspNetCore.RateLimiting;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;
using Polly;
using Polly.Extensions.Http;
using System.Threading.RateLimiting;

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddAuthentication(JwtBearerDefaults.AuthenticationScheme)
    .AddJwtBearer(o =>
    {
        o.Authority = builder.Configuration["Auth:Authority"];
        o.Audience  = builder.Configuration["Auth:Audience"];
    });

builder.Services.AddAuthorization();

builder.Services.AddRateLimiter(o =>
{
    o.GlobalLimiter = PartitionedRateLimiter.Create<HttpContext, string>(ctx =>
    {
        var tenant = ctx.User.FindFirst("tenant_id")?.Value ?? "anonymous";
        return RateLimitPartition.GetTokenBucketLimiter(tenant, _ => new()
        {
            TokenLimit       = 100,
            TokensPerPeriod  = 20,
            ReplenishmentPeriod = TimeSpan.FromSeconds(10),
            QueueLimit       = 0,
        });
    });
});

// Typed gRPC client to the Python service, with retries + circuit breaker.
builder.Services.AddGrpcClient<RagService.RagServiceClient>(o =>
{
    o.Address = new Uri(builder.Configuration["Rag:GrpcUrl"]!);
})
.AddPolicyHandler(HttpPolicyExtensions
    .HandleTransientHttpError()
    .WaitAndRetryAsync(3, n => TimeSpan.FromMilliseconds(200 * n)))
.AddPolicyHandler(HttpPolicyExtensions
    .HandleTransientHttpError()
    .CircuitBreakerAsync(5, TimeSpan.FromSeconds(30)));

// OpenTelemetry — traces propagate to Python via W3C headers automatically.
builder.Services.AddOpenTelemetry().WithTracing(t => t
    .SetResourceBuilder(ResourceBuilder.CreateDefault().AddService("api-gateway"))
    .AddAspNetCoreInstrumentation()
    .AddHttpClientInstrumentation()
    .AddGrpcClientInstrumentation()
    .AddOtlpExporter()); // any OTLP collector (Tempo, Jaeger, Honeycomb, Datadog)

var app = builder.Build();

app.UseAuthentication();
app.UseAuthorization();
app.UseRateLimiter();

app.MapPost("/api/v1/assistant/query", async (
    AssistantQueryRequest req,
    HttpContext ctx,
    RagService.RagServiceClient rag,
    ILogger<Program> log) =>
{
    var tenantId = ctx.User.FindFirst("tenant_id")?.Value;
    var userId   = ctx.User.FindFirst("sub")?.Value;
    if (tenantId is null || userId is null) return Results.Unauthorized();

    log.LogInformation("AssistantQuery tenant={Tenant} user={User} chars={Chars}",
        tenantId, userId, req.Query.Length);

    var sanitized = PiiRedactor.Redact(req.Query); // your existing PII rules

    var grpcReq = new QueryRequest
    {
        Query    = sanitized,
        TenantId = tenantId,
        UserId   = userId,
        TopK     = req.TopK ?? 8,
    };

    var resp = await rag.QueryAsync(grpcReq, deadline: DateTime.UtcNow.AddSeconds(30));
    return Results.Ok(new AssistantQueryResponse(
        Answer:    resp.Answer,
        Citations: resp.Citations.Select(c => new Citation(c.SourceId, c.Score)).ToList(),
        TraceId:   resp.TraceId));
})
.RequireAuthorization();

app.Run();

public record AssistantQueryRequest(string Query, int? TopK);
public record AssistantQueryResponse(string Answer, IReadOnlyList<Citation> Citations, string TraceId);
public record Citation(string SourceId, float Score);

What''s worth noticing:

• Tenant-aware rate limiting — the token bucket is keyed by tenant_id, not IP.
• Trace propagation is automatic — AddGrpcClientInstrumentation injects W3C traceparent headers; FastAPI''s OTel middleware picks them up. One trace ID from browser to LLM.
• Polly resilience — 3 retries with exponential backoff, then a circuit breaker. The Python service can hiccup without dragging down the gateway.
• PII redaction before the wire — your gateway is the last place under your control where redaction can happen.

4.2 The gRPC contract (rag.proto)

syntax = "proto3";
package rag.v1;

service RagService {
  rpc Query (QueryRequest)  returns (QueryResponse);
  rpc Ingest (IngestRequest) returns (IngestResponse);
  rpc StreamQuery (QueryRequest) returns (stream StreamToken);
}

message QueryRequest {
  string query     = 1;
  string tenant_id = 2;
  string user_id   = 3;
  int32  top_k     = 4;
}

message QueryResponse {
  string  answer    = 1;
  repeated Citation citations = 2;
  string  trace_id  = 3;
}

message Citation {
  string source_id = 1;
  float  score     = 2;
}

message StreamToken {
  string text = 1;
  bool   done = 2;
}

message IngestRequest {
  string tenant_id        = 1;
  repeated Document docs  = 2;
}

message Document {
  string source_id = 1;
  string text      = 2;
  map<string, string> metadata = 3;
}

message IngestResponse {
  int32 chunks_indexed = 1;
}

Both sides generate code from this file. No type drift, no JSON-string-to-object guesswork.

---

5. The Python RAG / Agent service — the actual AI

5.1 FastAPI service with LangChain RAG + LangGraph agent

# app/main.py
import os, asyncio, uuid
from contextlib import asynccontextmanager

from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
from opentelemetry import trace
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor

from langchain_openai import AzureChatOpenAI, AzureOpenAIEmbeddings
from langchain_qdrant import Qdrant
from langchain.prompts import ChatPromptTemplate
from langchain.schema.runnable import RunnablePassthrough
from langchain.schema.output_parser import StrOutputParser
from qdrant_client import QdrantClient
from langfuse.callback import CallbackHandler as LangfuseCallback

from app.agent_graph import build_agent_graph

tracer = trace.get_tracer("rag-service")

# ---- Bootstrapped once at startup ----
qdrant = QdrantClient(url=os.environ["QDRANT_URL"])

embeddings = AzureOpenAIEmbeddings(
    azure_deployment="text-embedding-3-small",
    api_version="2024-10-21",
)

llm = AzureChatOpenAI(
    azure_deployment="gpt-4o",
    api_version="2024-10-21",
    temperature=0,
    streaming=True,
)

vectorstore = Qdrant(client=qdrant, collection_name="docs", embeddings=embeddings)

system_prompt = """You are a careful enterprise assistant.
Use ONLY the context below to answer. If the answer is not in the context,
say "I don''t have that information."

Context:
{context}

Cite sources by their [source_id] in square brackets after each claim.
"""

prompt = ChatPromptTemplate.from_messages([
    ("system", system_prompt),
    ("human",  "{query}"),
])

agent_graph = build_agent_graph(llm=llm, vectorstore=vectorstore)


# ---- FastAPI ----
@asynccontextmanager
async def lifespan(app: FastAPI):
    if not qdrant.collection_exists("docs"):
        qdrant.create_collection(
            "docs",
            vectors_config={"size": 1536, "distance": "Cosine"},
        )
    yield

app = FastAPI(lifespan=lifespan)
FastAPIInstrumentor.instrument_app(app)  # OTel auto-instrumentation


class QueryRequest(BaseModel):
    query: str
    tenant_id: str
    user_id: str
    top_k: int = 8


class Citation(BaseModel):
    source_id: str
    score: float


class QueryResponse(BaseModel):
    answer: str
    citations: list[Citation]
    trace_id: str


@app.post("/v1/query", response_model=QueryResponse)
async def query(req: QueryRequest) -> QueryResponse:
    if not req.query.strip():
        raise HTTPException(status_code=400, detail="empty query")

    trace_id = uuid.uuid4().hex
    lf = LangfuseCallback(
        session_id=req.user_id,
        user_id=req.user_id,
        metadata={"tenant_id": req.tenant_id, "trace_id": trace_id},
    )

    with tracer.start_as_current_span("rag.query") as span:
        span.set_attribute("tenant_id", req.tenant_id)
        span.set_attribute("user_id", req.user_id)

        # Retrieval — Qdrant filtered by tenant for hard isolation
        with tracer.start_as_current_span("rag.retrieve"):
            docs = await asyncio.to_thread(
                vectorstore.similarity_search_with_score,
                query=req.query,
                k=req.top_k,
                filter={"must": [{"key": "tenant_id", "match": {"value": req.tenant_id}}]},
            )
        citations = [
            Citation(source_id=d.metadata["source_id"], score=float(s))
            for d, s in docs
        ]
        context = "\n\n".join(
            f"[{d.metadata[''source_id'']}]\n{d.page_content}" for d, _ in docs
        )

        # Generation
        with tracer.start_as_current_span("rag.generate"):
            chain = (
                {"context": lambda _: context, "query": RunnablePassthrough()}
                | prompt
                | llm
                | StrOutputParser()
            )
            answer = await chain.ainvoke(req.query, config={"callbacks": [lf]})

    return QueryResponse(answer=answer, citations=citations, trace_id=trace_id)


@app.post("/v1/agent")
async def agent(req: QueryRequest):
    """Multi-step agentic query — planner -> tool -> critic -> summarise."""
    result = await agent_graph.ainvoke(
        {"query": req.query, "tenant_id": req.tenant_id, "user_id": req.user_id},
        config={"callbacks": [LangfuseCallback(user_id=req.user_id)]},
    )
    return result

5.2 The LangGraph agent — planner / tool-user / critic

# app/agent_graph.py
from typing import TypedDict
from langgraph.graph import StateGraph, END
from langchain.tools import tool


class AgentState(TypedDict):
    query: str
    tenant_id: str
    user_id: str
    plan: str
    tool_calls: list[dict]
    observations: list[str]
    draft: str
    critique: str
    final: str


@tool
def sql_query(question: str) -> str:
    """Run a read-only SQL query against the data warehouse. Use for
    structured questions like ''how many'', ''sum'', ''last 7 days''."""
    return f"(stub result for: {question})"


@tool
def vector_search(question: str) -> str:
    """Search the document index for unstructured questions."""
    return f"(top 3 docs about: {question})"


tools = {"sql_query": sql_query, "vector_search": vector_search}


def build_agent_graph(llm, vectorstore):
    g = StateGraph(AgentState)

    def plan(state: AgentState) -> AgentState:
        prompt = f"""Given this user query, output a JSON list of tool calls.
Available tools: sql_query (for numbers), vector_search (for documents).
Query: {state[''query'']}
"""
        plan = llm.invoke(prompt).content
        return {**state, "plan": plan, "tool_calls": _parse_calls(plan)}

    def execute(state: AgentState) -> AgentState:
        observations = []
        for call in state["tool_calls"]:
            tool_fn = tools[call["name"]]
            observations.append(tool_fn.invoke(call["args"]))
        return {**state, "observations": observations}

    def draft(state: AgentState) -> AgentState:
        prompt = f"""Synthesise an answer using these observations.
Cite each fact with its tool name. Query: {state[''query'']}
Observations: {state[''observations'']}
"""
        return {**state, "draft": llm.invoke(prompt).content}

    def critique(state: AgentState) -> AgentState:
        prompt = f"""Critique this answer for accuracy, completeness, and
hallucination. If acceptable, return ''OK''. Otherwise list issues.
Draft: {state[''draft'']}
"""
        return {**state, "critique": llm.invoke(prompt).content}

    def finalise(state: AgentState) -> AgentState:
        return {**state, "final": state["draft"]}

    g.add_node("plan", plan)
    g.add_node("execute", execute)
    g.add_node("draft", draft)
    g.add_node("critique", critique)
    g.add_node("finalise", finalise)

    g.set_entry_point("plan")
    g.add_edge("plan",     "execute")
    g.add_edge("execute",  "draft")
    g.add_edge("draft",    "critique")
    g.add_conditional_edges(
        "critique",
        lambda s: "finalise" if "OK" in s["critique"] else "draft",
        {"finalise": "finalise", "draft": "draft"},
    )
    g.add_edge("finalise", END)
    return g.compile()


def _parse_calls(plan_text: str) -> list[dict]:
    import json, re
    m = re.search(r"\[.*\]", plan_text, re.DOTALL)
    return json.loads(m.group(0)) if m else []

What''s worth noticing:

• Tenant filter in retrieval is mandatory. Without it, your RAG service can leak documents across tenants.
• The plan → execute → draft → critique → finalise loop is a classic agentic pattern. The conditional edge from critique lets it iterate until the answer passes self-review (with a max iteration guard in real code).
• Langfuse callback is attached at every entry point — every LLM call is traced with its prompt, completion, and cost.

5.3 Ingestion endpoint — adding documents to the index

# app/ingest.py
from langchain.text_splitter import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=800,
    chunk_overlap=100,
    separators=["\n\n", "\n", ". ", " "],
)

@app.post("/v1/ingest")
async def ingest(req: IngestRequest):
    chunks = []
    for d in req.docs:
        for i, chunk in enumerate(splitter.split_text(d.text)):
            chunks.append({
                "page_content": chunk,
                "metadata": {
                    **d.metadata,
                    "source_id": d.source_id,
                    "chunk_index": i,
                    "tenant_id": req.tenant_id,
                },
            })
    await asyncio.to_thread(
        vectorstore.add_texts,
        texts=[c["page_content"] for c in chunks],
        metadatas=[c["metadata"] for c in chunks],
    )
    return {"chunks_indexed": len(chunks)}

Ingestion is just chunk → embed → upsert. The same tenant_id lives on every chunk so retrieval filters can enforce isolation.

---

6. Wiring it all together with Docker Compose

# docker-compose.yml
services:
  gateway:
    build: ./gateway
    ports: ["8080:8080"]
    environment:
      Rag__GrpcUrl: "http://rag:50051"
      Auth__Authority: "https://login.example.com/"
      Auth__Audience:  "https://api.example.com"
      OTEL_EXPORTER_OTLP_ENDPOINT: "http://otel-collector:4317"
    depends_on: [rag, otel-collector]

  rag:
    build: ./rag
    expose: ["50051", "8000"]   # NOT published to host — only on the internal network
    environment:
      QDRANT_URL:                 "http://qdrant:6333"
      AZURE_OPENAI_API_KEY:       "${AZURE_OPENAI_API_KEY}"
      AZURE_OPENAI_ENDPOINT:      "${AZURE_OPENAI_ENDPOINT}"
      LANGFUSE_PUBLIC_KEY:        "${LANGFUSE_PUBLIC_KEY}"
      LANGFUSE_SECRET_KEY:        "${LANGFUSE_SECRET_KEY}"
      LANGFUSE_HOST:              "https://cloud.langfuse.com"
      OTEL_EXPORTER_OTLP_ENDPOINT: "http://otel-collector:4317"
      OTEL_SERVICE_NAME:           "rag-service"
    depends_on: [qdrant, otel-collector]

  qdrant:
    image: qdrant/qdrant:latest
    expose: ["6333"]
    volumes: ["qdrant-data:/qdrant/storage"]

  otel-collector:
    image: otel/opentelemetry-collector-contrib:latest
    command: ["--config=/etc/otel-config.yaml"]
    volumes: ["./otel-config.yaml:/etc/otel-config.yaml"]
    expose: ["4317"]

volumes:
  qdrant-data:

The shape that matters:

• Only the gateway is reachable from outside. Qdrant, the RAG service, and the collector all live on the internal Docker network with expose: (not ports:).
• The gateway → RAG hop is HTTP/2 + gRPC over the internal network. No TLS hand-shake cost; runs at near-line-speed.
• Secrets are environment variables sourced from a secret manager (Azure Key Vault, AWS Secrets Manager, Vault). Never committed.

In Kubernetes, the same shape becomes: one LoadBalancer Service for the gateway, three ClusterIP Services for the rest, NetworkPolicies that block all egress from the RAG namespace except to the LLM provider and Langfuse.

---

7. Observability — Langfuse + OpenTelemetry side-by-side

These two are not redundant. They answer different questions.

7.1 OpenTelemetry — "where is the latency?"

A trace for one user query looks like:

api-gateway: POST /api/v1/assistant/query           [1420 ms]
├── auth.validate-jwt                                  [3 ms]
├── auth.check-claims                                  [1 ms]
├── rate-limit.acquire                                 [0.4 ms]
├── grpc.call rag.RagService/Query                  [1410 ms]
│   ├── rag-service: rag.query                      [1395 ms]
│   │   ├── rag.retrieve                              [180 ms]
│   │   │   └── qdrant.search                         [165 ms]
│   │   └── rag.generate                            [1190 ms]
│   │       └── azure-openai.chat.completions       [1170 ms]

You see immediately: the LLM took 1190 ms (~85% of the wall-clock). Qdrant added 165 ms (~12%). The gateway overhead is single-digit milliseconds. If a query is slow, this view tells you why.

7.2 Langfuse — "why did the model answer wrong?"

For the same trace, Langfuse shows you:

• The exact prompt sent to the model, including the retrieved chunks.
• The completion, with token-by-token streaming if it was streamed.
• Tokens in / out, cost in USD, model version.
• The citations the user got back, and whether they were ranked high enough.
• If a critic agent ran, its eval score (faithfulness, answer relevance).

When a user complains "the bot said our return policy is 60 days, it''s 30," you open Langfuse, find that trace, see that chunk #4 (an outdated PDF) was retrieved with score 0.78, ranked above chunk #2 (the current policy, score 0.74), and you have a concrete reproducible bug — fix the source document, re-ingest, ship.

OTel tells you when something is slow. Langfuse tells you when something is wrong.

7.3 Evals — the closed loop

# tests/eval_rag.py
import pandas as pd
from langfuse import Langfuse
from ragas.metrics import faithfulness, answer_relevancy, context_precision

lf = Langfuse()

dataset = pd.read_csv("tests/golden_queries.csv")  # 200 (query, expected_answer) pairs

results = []
for _, row in dataset.iterrows():
    response = client.query(row["query"], tenant_id="test-tenant")
    results.append({
        "query":      row["query"],
        "expected":   row["expected_answer"],
        "answer":     response.answer,
        "contexts":   [c.source_id for c in response.citations],
    })

scores = {
    "faithfulness":      faithfulness.score(results),
    "answer_relevancy":  answer_relevancy.score(results),
    "context_precision": context_precision.score(results),
}

for k, v in scores.items():
    lf.score(name=k, value=v, comment=f"nightly eval {pd.Timestamp.utcnow()}")

if scores["faithfulness"] < 0.85:
    raise SystemExit("Faithfulness regression — block deploy")

Run this nightly in CI. When you change a prompt, swap a model, or tweak chunking, you catch quality regressions before they ship. The same evals show up in Langfuse so PMs and SMEs can review them.

---

8. Benefits — what this architecture actually buys you

Concern	How polyglot helps
Existing .NET investment	Zero rewrite; the gateway is plain ASP.NET Core
Existing ExtJS UI	Untouched; AI features arrive as new endpoints
Velocity on new AI features	Python side ships at the speed of LangChain releases
Security & compliance	One auth boundary, one audit log, PII redaction at the edge
Multi-tenancy	Tenant filter enforced in Qdrant query AND in .NET claims
LLM provider switch	Single Python class to swap; .NET never knew
Cost attribution	Langfuse tags tokens by tenant_id from the gateway
Debugging	OTel for latency, Langfuse for quality, both linked by trace_id
Local dev	docker compose up brings up the whole stack
Scaling	Each service scales independently (RAG is the bottleneck; scale that pod)

---

9. Problems this architecture solves (concretely)

1. "We can''t ship RAG because LangChain isn''t in .NET." → It doesn''t need to be. Python service does the AI; .NET stays in charge of the contract.
2. "Security said no Python service exposed to the internet." → It isn''t. Only the .NET gateway is.
3. "We need to swap from OpenAI to Azure OpenAI for compliance." → One env var on the RAG service. .NET doesn''t change.
4. "Our RAG hallucinates and we don''t know why." → Open Langfuse, read the prompt, fix the chunk source, re-ingest.
5. "P95 latency went from 1.2s to 4.8s after the model update." → OTel waterfall shows which span got slower; usually the LLM call.
6. "Tenant A is seeing Tenant B''s documents." → Qdrant filter enforces tenant_id on every search. Add an integration test.
7. "We want to A/B test a new prompt." → Langfuse supports prompt versioning + experiments. Roll out to 10% of traffic; compare scores.

---

10. When NOT to do this

• Greenfield AI-only product — go pure Python. The polyglot tax (two languages, two deploy pipelines, contract maintenance) is pure overhead if you have no .NET legacy.
• Tiny app, single instance — overkill. Use Semantic Kernel in-process and ship.
• No observability budget — Langfuse + OTel are not optional in this design. If you can''t fund them, you can''t operate this architecture safely.
• Hard real-time (<200 ms total) — the gRPC hop + LLM call combine to ~1 s minimum. RAG is not for sub-second SLAs.

---

11. Integrating this into your existing project — a 2-week plan

Week 1 — boundary

• Day 1–2: Add ASP.NET Core gateway endpoints (/api/v1/assistant/query, /api/v1/assistant/ingest) returning stub data.
• Day 3: Wire ExtJS to call the new endpoint; show a chat drawer with mocked responses.
• Day 4–5: Author the rag.proto contract; generate both .NET and Python clients/servers from it.

Week 2 — AI workload

• Day 6–7: Stand up FastAPI service + Qdrant in Docker Compose; ingest one source corpus.
• Day 8: Wire LangChain RAG chain with Azure OpenAI; first end-to-end call from ExtJS works.
• Day 9: Add Langfuse + OTel — both sides instrumented; trace IDs propagate.
• Day 10: Write 20 eval queries; run nightly in CI.

After two weeks you have a real, observable, production-shaped polyglot AI stack. The rest is content, prompts, and evals — which is exactly where you want to be spending time.

---

12. Closing — the right mental model

Think of this architecture as two services with one trust boundary, not "we now have a Python problem." The .NET side is your enterprise. The Python side is your AI lab. They meet at a contract — one that''s narrow enough to maintain, typed enough to test, observable enough to debug.

The teams that succeed with this pattern share three habits:

1. They keep the contract narrow. Every new field on QueryRequest is reviewed like a wire-format change. No language detail leaks across.
2. They treat observability as a feature. Langfuse, OTel, evals — not "nice to have," but on the critical path.
3. They version everything. Prompt versions, model versions, gRPC versions. When something regresses, they can roll back the exact thing that changed.

Adopt those habits and the polyglot architecture stops being a maintenance burden and starts being a competitive advantage — you ship AI at Python speed without spending a single quarter on a rewrite.

---

Further reading

• LangChain docs — the RAG chain you saw is one of dozens of patterns.
• LangGraph docs — for multi-agent and tool-using flows.
• Qdrant docs — vector indexes, filters, payload schemas.
• Langfuse docs — LLM tracing and evals.
• OpenTelemetry for .NET and for Python.
• ASP.NET Core gRPC — the typed wire format we used.

---

Building something like this? Email randhir.jassal@gmail.com with your architecture diagram and the question you''re stuck on — happy to compare notes.