MCP Deep Dive, Part 3: Build a Production-Grade MCP Server From Scratch

You can stand up an MCP server that returns data in an afternoon. Standing up one that an agent hammers 85,000 times a day, across tenants, without leaking stack traces, OOM-ing on a big query, or dropping calls on deploy — that's a service, not a demo. This part builds the real thing.

This is Part 3 of a 15-part deep dive on Model Context Protocol (MCP). Part 1 made the case (N×M glue → N+M), and Part 2 drew the architecture (host, client, server; tools, resources, prompts; the initialize handshake). Now we build one of the three Mattrx servers — mattrx-analytics — end to end, and cover the production concerns that separate a toy from a service. Auth (Part 6/7), streaming (Part 9), and the deep Azure hosting story (Part 13) get their own parts; here we build the server itself.

TL;DR

Concern	Demo server (before)	Production server (after)
Bootstrap	hand-rolled JSON-RPC	MCP SDK + DI + transport
Tools	stringly-typed blob args	described, typed params → real schema
Errors	exceptions leak / 500	tool errors vs protocol errors
Data	everything is a "tool"	read data as resources (by URI)
Results	return everything	paginated, capped, cancellable
Ops	no health / telemetry	/healthz, /readyz, OTel per call

• Build on the MCP SDK + DI; wire transport, tools, resources, and prompts declaratively.
• Typed, described tool params → the SDK emits a JSON Schema the model calls correctly.
• Distinguish tool errors (an isError result the agent can read) from protocol errors; never leak stack traces.
• Model read data as resources (URI), not everything as a tool.
• Cap + paginate every result (page ≤ 200, opaque cursor); honor CancellationToken.
• /healthz + /readyz → Azure Container Apps does zero-downtime deploys without dropping calls.
• One OTel span per tool call (tenant, tool, outcome, ms) → per-tenant p95 and error rate.
• The server is stateless behind the gateway → scale to N replicas.
• mattrx-analytics serves ~85k tool calls/day at read p95 120 ms.
• Typed tool errors are part of why the agent tool-call error rate is 0.8%.

The one mental shift: an MCP server is a service, not a script. Everything you'd demand of a production API — schemas, typed errors, pagination, cancellation, health, telemetry — an MCP server needs too, because an agent is a more demanding, less forgiving client than a human.

The running example: mattrx-analytics

Mattrx runs .NET 9 / ASP.NET Core with the official MCP C# SDK. mattrx-analytics is the read server: it exposes get_campaign_kpis and query_events as tools, campaign records as resources (mattrx://analytics/campaigns/{id}), and a campaign_health_review prompt. It runs as an Azure Container App behind our gateway, over Streamable HTTP + SSE in production and stdio in local dev. Let's build it, concern by concern, with the demo version that breaks and the production version that holds.

The server, in one picture

                 mattrx-analytics  (Azure Container App)
   +-------------------------------------------------------------+
   |  HTTP endpoints:                                            |
   |    /mcp      (Streamable HTTP + SSE)  <- agents / clients   |
   |    /healthz  (liveness)   /readyz  (readiness)             |
   +-------------------------------------------------------------+
                 |
                 v
   +-------------------------------------------------------------+
   |  MCP server (SDK)                                           |
   |    Tools:     get_campaign_kpis, query_events              |
   |    Resources: mattrx://analytics/campaigns/{id}            |
   |    Prompts:   campaign_health_review                       |
   +-------------------------------------------------------------+
                 |  DI: the same domain services the rest of Mattrx uses
                 v
   +-------------------------------------------------------------+
   |  Domain: ICampaignQueries, IEventQueries -> Azure SQL      |
   |  Cross-cutting: AiPrincipal, OTel spans, health checks     |
   +-------------------------------------------------------------+

1. Bootstrap: use the SDK, not hand-rolled JSON-RPC

Before

The first instinct is to expose a couple of HTTP endpoints and parse JSON-RPC by hand. You end up reimplementing framing, initialize, capability negotiation, and schema generation — badly.

After

The MCP SDK gives you the server; you register capabilities and dependencies through DI. The server's tool handlers use the same domain services as the rest of the app — no duplication.

// Program.cs — the mattrx-analytics MCP server.
var builder = WebApplication.CreateBuilder(args);

builder.Services
    .AddMcpServer(o => o.ServerInfo = new() { Name = "mattrx-analytics", Version = "2.4.0" })
    .WithHttpTransport(o => o.Stateless = false)   // Streamable HTTP + SSE; keep session for streams
    .WithTools<AnalyticsTools>()
    .WithResources<CampaignResources>()
    .WithPrompts<AnalyticsPrompts>();

// Tool handlers depend on the same query services as the rest of Mattrx.
builder.Services.AddScoped<ICampaignQueries, CampaignQueries>();
builder.Services.AddScoped(sp => sp.GetRequiredService<IPrincipalAccessor>().Current); // AiPrincipal

builder.Services.AddOpenTelemetry().WithTracing(t => t.AddSource("Mattrx.Mcp"));
builder.Services.AddHealthChecks().AddCheck<AzureSqlHealthCheck>("sql", tags: ["ready"]);

var app = builder.Build();
app.MapMcp("/mcp");                                            // the MCP endpoint
app.MapHealthChecks("/healthz");                               // liveness
app.MapHealthChecks("/readyz", new() { Predicate = c => c.Tags.Contains("ready") });
app.Run();

Diagnostic: the SDK owns the protocol (framing, initialize, tools/list, schema emission) so you own only your capabilities. Hand-rolling JSON-RPC is effort spent re-creating a solved problem, with new bugs.

Mattrx metric: because handlers reuse the existing ICampaignQueries, exposing the analytics domain over MCP added almost no new query code — the server is a thin, governed surface over services we already had.

2. Tools done right — typed params, real schemas

Before

A vague tool with a stringly-typed blob. The model has to guess your argument format, and gets it wrong.

// BEFORE: what is "args"? The model has to invent the format.
[McpServerTool(Name = "kpis")]
public async Task<string> Kpis(string args, CancellationToken ct)
{
    var parts = args.Split(',');    // hand-parsing a format nobody documented
    var data = await _campaigns.GetKpisAsync(parts[0], parts[1], ct);
    return JsonSerializer.Serialize(data);
}

After

A precise name, described typed parameters (the SDK turns these into a JSON Schema the model reads), and a structured return type.

[McpServerToolType]
public sealed class AnalyticsTools(ICampaignQueries campaigns, AiPrincipal principal)
{
    [McpServerTool(Name = "get_campaign_kpis")]
    [Description("Return the KPI time-series for one campaign in the caller's tenant.")]
    public async Task<CampaignKpis> GetCampaignKpis(
        [Description("Campaign id (GUID) within the caller's tenant.")] string campaignId,
        [Description("ISO-8601 date range, e.g. 2026-06-01/2026-06-30.")] string range,
        CancellationToken ct)
    {
        var window = DateRange.Parse(range);
        return await campaigns.GetKpisAsync(principal.TenantId, campaignId, window, ct);
    }
}

Diagnostic: the model calls a tool from its schema. A described, typed signature is the schema — the model fills it correctly. A string args blob makes the model guess, and every guess is a failed call.

Mattrx metric: precise tool schemas are a direct contributor to the agent tool-call error rate dropping to 0.8% — the model stopped mis-formatting arguments because the schema told it exactly what to send.

3. Error handling — tool errors vs protocol errors

Before

Exceptions bubble out of the handler and surface as an opaque protocol error or a 500 with a stack trace — leaking internals and giving the agent nothing to act on.

After

Draw the line that most MCP tutorials skip. A tool error is a result (isError: true) the agent can read and recover from ("not_found" → try a different id). A protocol error is for a malformed request. Unexpected exceptions are logged server-side and returned as a safe, generic tool error — never a stack trace.

public async Task<CallToolResult> GetCampaignKpis(GetKpisArgs args, CancellationToken ct)
{
    if (!Guid.TryParse(args.CampaignId, out var id))
        return ToolResults.Error("invalid_campaign_id", "campaignId must be a GUID.");

    var campaign = await campaigns.FindAsync(principal.TenantId, id, ct);
    if (campaign is null)
        return ToolResults.Error("not_found", $"No campaign {id} in this tenant.");

    try
    {
        var kpis = await campaigns.GetKpisAsync(principal.TenantId, id, args.Range, ct);
        return ToolResults.Structured(kpis);                 // success: structured content
    }
    catch (Exception ex)
    {
        logger.ToolFailed(ex, "get_campaign_kpis");          // full detail stays server-side
        return ToolResults.Error("internal", "The tool failed; try again shortly."); // safe surface
    }
}

Diagnostic: a tool failure must not become a protocol crash. If the agent gets a dead connection instead of a readable not_found, it can't adapt — and if it gets your SQL exception text, you've leaked your schema to whoever's driving the model.

Mattrx metric: returning typed tool errors instead of throwing is another reason the tool-call error rate sits at 0.8% — the agent receives an actionable error and retries intelligently, rather than the whole call dying.

4. Resources — not everything is a tool

Before

Every read is a tool, so fetching a record means the model reasons about, chooses, and calls get_campaign — burning a round trip and context to retrieve data the host already knows it needs.

After

Expose read data as a resource with a URI template and a content type, and handle not-found cleanly. The host attaches it by URI (see Part 2); the model reads it.

[McpServerResourceType]
public sealed class CampaignResources(ICampaignQueries campaigns, AiPrincipal principal)
{
    [McpServerResource(
        UriTemplate = "mattrx://analytics/campaigns/{campaignId}",
        MimeType = "application/json")]
    [Description("A campaign record (name, status, budget, audience) in the caller's tenant.")]
    public async Task<ReadResourceResult> GetCampaign(string campaignId, CancellationToken ct)
    {
        if (!Guid.TryParse(campaignId, out var id))
            return ResourceResults.NotFound(campaignId);

        var c = await campaigns.FindAsync(principal.TenantId, id, ct);
        return c is null
            ? ResourceResults.NotFound(campaignId)
            : ResourceResults.Json($"mattrx://analytics/campaigns/{campaignId}", c);
    }
}

Diagnostic: the rule from Part 2 holds in the code — model-controlled actions are tools; application-controlled data is a resource. Overusing tools bloats the model's decision space and its token bill.

Mattrx metric: serving "the record the user is viewing" as a resource instead of a tool call is part of how we hold context tokens at 3.5k (down from 14k) — the host attaches it directly instead of the model fishing for it.

5. Pagination, caps, and cancellation

Before

query_events runs SELECT * ... WHERE campaign = @id and returns every matching row — against an Events table with ~180M rows. One call OOMs the server or floods the model's context.

After

Cap the page size, return an opaque cursor (never OFFSET on a huge table), and honor cancellation so an abandoned agent call stops working.

[McpServerTool(Name = "query_events")]
[Description("Query a campaign's events, newest first. Returns one page; pass `cursor` to continue.")]
public async Task<EventPage> QueryEvents(
    [Description("Campaign id (GUID).")] string campaignId,
    [Description("Opaque cursor from a previous page, or null for the first page.")] string? cursor,
    [Description("Page size, 1-200 (default 50).")] int pageSize = 50,
    CancellationToken ct = default)
{
    pageSize = Math.Clamp(pageSize, 1, 200);        // the model does NOT get to ask for 180M rows
    var page = await events.QueryAsync(principal.TenantId, campaignId, cursor, pageSize, ct);
    return new EventPage(page.Items, page.NextCursor);  // cursor-based, not OFFSET
}

The same discipline applies to tools/list and resources/list — both are paginated by the protocol; return a nextCursor when your catalog is large.

Diagnostic: an unbounded tool result is a double foot-gun — memory on the server, and cost + confusion in the model's context. Assume every tool could one day match a billion rows, because at Mattrx's scale one of them does.

Mattrx metric: capping pages at 200 and using cursors keeps query_events at read p95 120 ms even against the 180M-row Events table, and keeps a single tool call from ever ballooning the agent's context.

6. Health, readiness, and telemetry

Before

No health checks. A dropped SQL connection makes every tool call 500 silently, and a deploy kills in-flight calls because nothing tells the platform when the server is ready.

After

Expose /healthz (liveness) and /readyz (readiness gated on dependencies), and emit one OpenTelemetry span per tool call. Readiness lets Azure Container Apps roll deploys without dropping traffic.

// One activity per tool call: tenant, tool, outcome, duration -> App Insights.
using var activity = ActivitySource.StartActivity("mcp.tool_call");
activity?.SetTag("mcp.tool", toolName);
activity?.SetTag("mattrx.tenant", principal.TenantId);

var result = await next(ct);

activity?.SetTag("mcp.outcome", result.IsError ? "error" : "ok");

Diagnostic: an agent won't tell you it's getting errors — it'll just quietly perform worse. Per-call spans turn "the assistant feels off" into a chart of p95 and error rate per tool and per tenant. (Deep observability — tracing a full agent run across servers — is Part 10; this is the server-level baseline.)

Mattrx metric: readiness probes give zero-downtime deploys on Container Apps, and per-call OTel spans are what make read p95 120 ms and the 0.8% error rate measurable in the first place — you can't improve what you don't emit.

A tool call, inside the server

  tools/call get_campaign_kpis { campaignId, range }
        |
        v
  1. Deserialize + schema-validate args   -- bad JSON => JSON-RPC PROTOCOL error
        |
        v
  2. Auth + scope check  (Parts 6/7)      -- tenant bound from the token
        |
        v
  3. Invoke handler
        |-- invalid input -> TOOL error { isError, "invalid_campaign_id" }
        |-- not found     -> TOOL error { isError, "not_found" }
        |-- unexpected    -> log detail; TOOL error { isError, "internal" }  (no stack trace)
        +-- success       -> structured result { CampaignKpis }
        |
        v
  4. OTel span (tenant, tool, outcome, ms) + audit
        |
        v
  serialize -> client

The numbers, in one place

Metric	Demo server (before)	Production server (after)
Tool arg errors	model guesses format	rare — typed schema
Tool-call error rate	high (opaque failures)	0.8%
Error surface	stack traces leak	safe typed tool errors
Max result size	unbounded (OOM risk)	≤ 200 / page, cursored
Read-tool p95	varies / stalls	120 ms
Deploys	drop in-flight calls	zero-downtime (readiness)
Observability	none	OTel span per call
Calls/day served	—	~85,000

Production-server checklist

• Use the MCP SDK; register tools/resources/prompts via DI, reuse existing domain services.
• Give tools precise names + described, typed params — that's the schema the model uses.
• Return structured results, not hand-serialized strings.
• Separate tool errors (isError result) from protocol errors; log detail, return safe messages.
• Expose read data as resources (URI + MIME), with clean not-found handling.
• Cap and paginate every result; use opaque cursors; honor CancellationToken.
• Paginate tools/list / resources/list when the catalog is large.
• Ship /healthz + /readyz; gate readiness on real dependencies.
• Emit one telemetry span per tool call (tenant, tool, outcome, duration).
• Keep the server stateless so it scales horizontally.

The honest stuff: when the full build is overkill

1. A local stdio dev tool. A few tools over stdio needs none of the HTTP, readiness, or scaling machinery. Don't build a service for a helper script.
2. Everything-as-a-tool. Reads that just fetch a record are resources. Overusing tools bloats the model's choices and cost.
3. Skipping result caps. The single most common production MCP mistake. Cap and paginate from line one, not after the first OOM.
4. Leaking internals in errors. Stack traces and SQL messages in tool errors are an info leak. Log server-side; return safe strings.
5. Hand-rolling the protocol. The SDK does framing, negotiation, and schema generation. Reimplementing it is wasted, buggy effort.
6. Ignoring cancellation. An abandoned agent call that keeps querying is wasted capacity and cost. Thread and honor the token.
7. In-memory session state. Stash per-session state in process memory and you can't scale out. Keep the server stateless; let the transport/gateway own session.

The model to carry forward

An MCP server is a service, not a script. An agent is a relentless, literal, unforgiving client: it calls from your schemas, it retries on your errors, it will happily ask for a billion rows. So build the server the way you'd build any production API you expected a demanding client to abuse — typed contracts, safe errors, bounded results, health, and telemetry — and it will hold at 85,000 calls a day.

Three habits that keep a server production-grade:

1. Type your tools. Described, typed parameters are the schema the model depends on — vague tools produce failed calls.
2. Make errors data, not exceptions. Return a tool error the agent can reason about; log the detail, never leak it.
3. Cap and paginate everything. Treat every tool as if it could match a billion rows, because eventually one will.

In Part 4 we build the other side: an MCP client that connects to any server, discovers tools, and drives them from an agent loop — the consumer of everything we just built.

Continue the series — MCP Deep Dive

1. Why Model Context Protocol Kills Integration Glue Code for Good
2. Inside the MCP Architecture: Hosts, Clients, and Servers
3. Build a Production-Grade MCP Server From Scratch (you are here)
4. Build an MCP Client That Connects to Any AI Tool
5. Custom MCP Tools Your AI Agents Can Actually Trust
6. MCP Authentication With OAuth and Entra ID, Done Right
7. Scoped Authorization for MCP Tools (Least Privilege for Agents)
8. Securing MCP Against Prompt Injection and Tool Abuse
9. Streaming and Long-Running Tools Over MCP
10. Debugging and Observability for MCP in Production
11. Rolling MCP Out Across the Enterprise
12. Building MCP Servers in C# and .NET 9
13. Hosting MCP on Azure at Real Scale
14. Wiring MCP Into OpenAI and Agent Frameworks
15. Running MCP in Production — Lessons From Mattrx

Further reading

• MCP Deep Dive, Part 4: Build an MCP Client That Connects to Any AI Tool
• MCP Deep Dive, Part 12: Building MCP Servers in C# and .NET 9
• AI-Native Architecture: The 9-Layer Blueprint Every Enterprise Will Adopt by 2027

---

Building an MCP server and want a second pair of eyes on your tool contracts or error handling? I'm always happy to compare notes — reach me at randhir.jassal@gmail.com.