System Design
How We Deliver 15 Million Webhooks a Day Without Losing a Single Event
Delivering 15M webhooks a day to endpoints you don't control is deceptively hard. Here's the outbox + queue + retry design that never drops an event.
20 min read7
Posts and interview questions tagged Distributed Systems.
Delivering 15M webhooks a day to endpoints you don't control is deceptively hard. Here's the outbox + queue + retry design that never drops an event.
Chunk-and-embed RAG hits a wall at scale. Open Knowledge Format (OKF) feeds a context engine structured, governed knowledge — the Mattrx rebuild with code.
Most enterprises bolt AI onto a backend built for CRUD. We rebuilt Mattrx around nine AI-native layers in production. Here is the blueprint, with code.
Part 6: AI & data governance as a control plane — classification at ingestion, entitlement-aware retrieval, consent & purpose limits, data lineage, and policy-as-code in C# + Python.
Part 5: multi-tenant context engineering — tenant-scoped retrieval, per-tenant prompts and caches, noisy-neighbor fairness, model routing, and per-tenant cost attribution in C# + Python.
Part 4 : the enterprise design that makes GenAI production-grade — eval gates, injection + PII defense, cost control, and tracing.
Part 3: multi-agent architecture that survives production — a supervisor + specialized agents (C# orchestration, Python agent graph) with bounded loops.
Part 2: giving enterprise AI a memory that works — working, short-term, and long-term tiers in C# + Python, tenant-isolated, summarized, and governed.
Part 1 of a context-engineering series: why naive RAG hallucinates and the C#+Python context layer that fixes it — rewriting, re-ranking, budgeting.
How Mattrx swapped synchronous REST calls for Kafka — decoupling the ingestion pipeline, killing cascading failures, and cutting failures by 90%.
Mattrx's Azure bill spiked overnight. The real causes — runaway logging, autoscale, orphaned resources, egress — and the fixes that cut it 60%.
.NET 11 vs .NET 10 benchmarked on Mattrx in production — the real throughput, latency, and memory wins, the new features, and whether to upgrade.