Design a Notification System — Push, SMS & Email at Scale

"Design a notification system" looks like an integration task — call Twilio, call SendGrid, done. The interview lives in everything around those calls: how do you fan one event out to a million recipients across three channels, make sure a marketing blast never delays someone's login OTP, avoid double-sending when your queue redelivers a message, and survive Twilio having a bad afternoon? It's a queues-and-reliability problem wearing an integration costume.

A notification system delivers messages to users across multiple channels — push (APNs/FCM), SMS (Twilio), email (SES) — triggered by events elsewhere in your product. It has to absorb huge bursts (a campaign to every user at once), guarantee that important messages get through, and stay polite (no duplicate spam, respect preferences). The hard parts are fan-out, priority, deduplication, and retries — all of which point at the same answer: put a queue in the middle. (New to the method? Start with the framework.)

TL;DR — the design at a glance

Concern	Decision
Core shape	Enqueue → per-channel workers → providers (decouple ingestion from delivery)
Fan-out	Expand recipients → one queued message per (user, channel)
Priority	Separate queues — transactional never waits behind bulk
Delivery guarantee	At-least-once + dedup (exactly-once across third parties is a myth)
Failures	Retry with backoff + jitter, then dead-letter queue
Politeness	User preferences, quiet hours, per-user rate limiting

• It's a queue problem. The queue absorbs bursts, decouples ingestion from third-party providers, and enables retries.
• Fan-out turns one event into many messages — expand the recipient list and enqueue per (user, channel), in batches for huge audiences.
• Separate transactional from bulk. An OTP must never sit behind a million-message marketing blast — use priority queues.
• Delivery is at-least-once, so dedup. Redelivery is a fact of queues; a dedup key (eventId + userId + channel) stops double-sends.
• Providers fail — plan for it. Exponential backoff + jitter, circuit breakers per provider, a dead-letter queue, and ideally a fallback provider.
• Respect the user. Preferences, opt-outs, quiet hours, and frequency caps aren't nice-to-haves — they're legal (TCPA/GDPR) and they prevent notification fatigue.
• "Sent" ≠ "delivered" ≠ "read." Track the real states via provider delivery receipts.

The one mental shift: a notification system is a buffer between your events and unreliable third parties. You don't call providers inline from your app — you enqueue, and workers deliver on their own schedule, retrying and prioritizing. That one decoupling is what makes it scale, survive provider outages, and keep the OTP moving while the newsletter waits.

Step 1 — Requirements & scope

Functional:

• Send notifications on push, SMS, and email, triggered by product events.
• Support templates (localized, with variables) and user preferences (channel opt-ins, quiet hours).
• Handle both transactional (OTP, password reset — urgent, per-user) and promotional (campaigns — huge fan-out, best-effort).

Parked: in-app notification center UI, rich analytics dashboards, WhatsApp/other channels (same pattern, more adapters).

Non-functional:

• High throughput — millions/day, with campaign spikes.
• Reliable — at-least-once delivery; important messages must not be silently dropped.
• Low latency for transactional — an OTP has seconds to be useful.
• Resilient to provider failures — Twilio/SES going down shouldn't lose messages.
• Non-spammy — dedup + rate limiting per user.

Step 2 — Back-of-envelope estimates

Say 100 million notifications/day:

Average:  100,000,000 / 86,400 s  ≈  ~1,160 /sec
Campaign spike: a blast to 50M users in minutes  ≈  tens of thousands /sec

The average is trivial; the spikes are the design driver. A campaign fans one API call out to tens of millions of messages in a burst — which is exactly why a queue sits in the middle to absorb it, so you never hand that spike straight to a rate-limited provider.

Step 3 — API & data model

POST /notifications
  { recipients, type, templateId, data, channels?, priority }
  -> { batchId }

Core entities:

notification   id, userId, type, channel, templateId, payload,
               status (queued|sent|delivered|failed), attempts, createdAt
preferences    userId, channel opt-ins, quietHours, deviceTokens
template       templateId, channel, subject/body with {{placeholders}}

status and attempts are what make delivery observable and retryable; preferences is what keeps you compliant and polite.

Step 4 — High-level architecture

The whole system is an enqueue-then-deliver pipeline:

  Event sources (services, campaigns)
        │
        ▼
  [ Notification Service ]  — validate · apply prefs · dedup · render template
        │
        ▼
  [ Message Queue ]                    (priority: transactional vs bulk)
     │              │              │
     ▼              ▼              ▼
 [Push worker]  [SMS worker]  [Email worker]     — scale independently
     │              │              │
     ▼              ▼              ▼
   APNs / FCM     Twilio          SES            — third-party providers
     │              │              │
     └──── delivery receipts (webhooks) ────▶ update status  ·  DLQ on repeated failure

The queue decouples the fast, spiky ingestion from the slow, rate-limited providers. Workers pull at a rate the providers can handle, retry on failure, and scale per channel independently (email volume ≠ SMS volume).

Step 5 — Deep-dive: the four hard parts

Fan-out

One event ("your team shipped a release") can target millions of users. The service expands the recipient list and enqueues one message per (user, channel) — for a huge audience, in batches so a single request doesn't block producing millions of messages. Fan-out is where a tiny API call becomes a tidal wave; the queue is the seawall.

Priority — the OTP must not wait

A marketing blast enqueues tens of millions of messages. If a login OTP lands behind them in the same FIFO queue, it arrives ten minutes late and is useless. Fix: separate queues by priority — a high-priority transactional queue with its own workers, and a bulk queue for campaigns. Transactional traffic is small and urgent; bulk is huge and patient. Never let them share a lane.

Deduplication — don't double-send

Queues are at-least-once: a worker crash after sending but before acking causes redelivery, and the user gets the notification twice. Guard every send with an idempotency key — eventId + userId + channel — checked against a fast store (Redis) before dispatch. If the key's already marked sent, skip. (Same discipline as an idempotent consumer — see the rate limiter and outbox patterns.)

Retries + dead-letter queue

Providers fail — timeouts, 500s, throttling. A failed send is retried with exponential backoff + jitter (so retries don't thundering-herd the provider), up to a max. After the last attempt, the message goes to a dead-letter queue for inspection instead of being lost or retried forever. Wrap each provider in a circuit breaker so a sustained outage stops hammering it, and ideally fail over to a backup provider for the channel.

Step 6 — Delivery tracking & preferences

• Delivery tracking: "we handed it to Twilio" is not "the phone buzzed." Providers send delivery receipts via webhooks; consume them to move status from sent → delivered (or bounced/failed), and handle bounces (a hard email bounce should suppress future sends to that address).
• Preferences & rate limiting: apply user opt-ins and quiet hours before enqueueing, and cap frequency per user (batch low-priority notifications into a digest rather than firing 30 in an hour). This is both compliance and anti-fatigue.

Step 7 — Bottlenecks & scaling

• The queue absorbs the spike — the single most important scaling property; campaigns drain at provider-safe rates.
• Workers scale horizontally per channel; add consumers to drain a backlog.
• Provider limits — throttle to each provider's rate limit; circuit-break and fail over on outages.
• Template rendering — precompile and cache templates; render once per unique (template, locale, variables).
• Idempotency/dedup store — Redis, keyed by the dedup key, with a TTL long enough to cover retries.
• Separate pipelines for transactional vs bulk so one never starves the other.

Step 8 — Trade-offs & wrap-up

• At-least-once + dedup vs exactly-once: true exactly-once across third parties is impossible (you can't un-send an SMS); at-least-once with idempotent dedup is the real target.
• Priority queues vs one queue: a little more infrastructure for the guarantee that urgent beats bulk. Always worth it.
• Sync vs queued: even transactional sends go through the queue (for retries/observability) — just on a fast lane, not inline in the request.
• Channel trade-offs: push is free-ish but droppable; SMS is reliable but costs money and has strict rate limits; email is cheap but spam-filtered. Pick per message importance.

The design checklist

• Enqueue → per-channel workers → providers (decouple ingestion from delivery).
• Fan-out expands recipients and enqueues per (user, channel), batched.
• Priority queues — transactional separated from bulk.
• Dedup with an idempotency key (eventId+userId+channel).
• Retry (backoff + jitter) → DLQ; circuit breaker per provider.
• Delivery receipts update status; handle bounces.
• Preferences, quiet hours, per-user rate limiting.
• Chose channels deliberately (cost vs reliability).

In production at Mattrx

Mattrx fans out roughly 15M events a day — budget.threshold.crossed, campaign.completed, and conversion.tracked — to customer-configured webhook endpoints and to users over email and push. The first version dispatched inline in the request that produced the event: a deploy that recycled the app mid-flight dropped every in-memory delivery, and one slow customer endpoint dragged API latency up with it. Moving to the outbox-plus-Service-Bus design above decoupled the write path from delivery entirely.

Metric	Before	After
Events dropped per deploy	Thousands	0 (outbox + queue)
Delivery model	Inline in the request path (best-effort)	Outbox → Azure Service Bus → worker pool
First-attempt delivery success	n/a (no queue, no retry)	~96%
Eventual delivery success	Lost on first endpoint failure	~99.98% (backoff + retry)
API write-path p95	Coupled to slow customer endpoints	120ms (delivery decoupled)
Failing-endpoint handling	Retried inline, blocked workers	Auto-disabled after 20 consecutive failures

The queue is the shock absorber: the write path is now indifferent to how fast — or whether — a customer's endpoint responds, and a bad deploy can no longer lose a single event.

The architecture at Mattrx

  Domain event (budget.threshold.crossed, ...)
        │  outbox row in the SAME DB txn
        ▼
  [ Dispatcher ] ──▶ [ Azure Service Bus · partitioned by tenantId ]
                            │
                            ▼
                     [ .NET 9 workers ]  dedup by eventId (Redis) · retry+backoff · circuit breaker (20 fails ─▶ disable)
                       │           │            │
                       ▼           ▼            ▼
                    webhook      email         push       (delivery receipts ─▶ status)

Production implementation (Mattrx)

Here is the core of Mattrx's delivery path: an outbox relay enqueues each event onto the webhook-deliveries Service Bus queue, and a processor pool dedups against Redis, dispatches over HTTP, then backs off or dead-letters on failure.

using Azure.Messaging.ServiceBus;
using Microsoft.Extensions.Logging;
using StackExchange.Redis;

namespace Mattrx.Webhooks.Delivery;

/// <summary>Immutable unit of work carried on the `webhook-deliveries` queue.</summary>
public sealed record WebhookDelivery(
    Guid EventId,           // stable outbox id; also the customer-side dedup key
    string TenantId,
    string EventType,       // budget.threshold.crossed | campaign.completed | conversion.tracked
    Uri EndpointUrl,
    string Body,            // serialized JSON payload, HMAC-signed downstream
    DateTimeOffset OccurredAt);

// ---------- 1. Producer: outbox relay -> Service Bus ----------

public sealed class WebhookDeliveryProducer(
    ServiceBusSender sender,
    ILogger<WebhookDeliveryProducer> logger)
{
    public async Task EnqueueAsync(WebhookDelivery delivery, CancellationToken ct)
    {
        var message = new ServiceBusMessage(BinaryData.FromObjectAsJson(delivery))
        {
            MessageId    = delivery.EventId.ToString(), // broker-side dupe suppression
            Subject      = delivery.EventType,
            ContentType  = "application/json",
            PartitionKey = delivery.TenantId,           // per-tenant fairness + locality
            ApplicationProperties = { ["attempt"] = 1, ["tenantId"] = delivery.TenantId },
        };

        await sender.SendMessageAsync(message, ct);
        logger.LogInformation("Enqueued {EventType} {EventId} for tenant {TenantId}",
            delivery.EventType, delivery.EventId, delivery.TenantId);
    }
}

// ---------- 2. Worker: dedup -> dispatch -> backoff / dead-letter ----------

public sealed class WebhookDeliveryWorker(
    ServiceBusProcessor processor,
    ServiceBusSender sender,               // used to schedule delayed retries
    IConnectionMultiplexer redis,
    IHttpClientFactory httpFactory,
    EndpointCircuitBreaker breaker,
    ILogger<WebhookDeliveryWorker> logger) : IAsyncDisposable
{
    private const int MaxAttempts = 8;
    private static readonly TimeSpan DedupTtl = TimeSpan.FromHours(24);

    public async Task StartAsync(CancellationToken ct)
    {
        processor.ProcessMessageAsync += OnMessageAsync;
        processor.ProcessErrorAsync   += OnErrorAsync;
        await processor.StartProcessingAsync(ct);
    }

    private async Task OnMessageAsync(ProcessMessageEventArgs args)
    {
        var ct = args.CancellationToken;
        var delivery = args.Message.Body.ToObjectFromJson<WebhookDelivery>();
        var db = redis.GetDatabase();
        var dedupKey = $"wh:sent:{delivery.EventId}";

        // Idempotency: at-least-once delivery means we may see a delivered event again.
        if (await db.KeyExistsAsync(dedupKey))
        {
            await args.CompleteMessageAsync(args.Message, ct);
            return;
        }

        // Circuit breaker: endpoint auto-disabled after 20 consecutive failures.
        if (await breaker.IsOpenAsync(delivery.EndpointUrl))
        {
            await args.DeadLetterMessageAsync(args.Message, "endpoint_disabled",
                "Circuit open: endpoint auto-disabled after 20 consecutive failures.", ct);
            return;
        }

        try
        {
            await DispatchAsync(delivery, ct);

            // Mark delivered so duplicates and stragglers are dropped for 24h.
            await db.StringSetAsync(dedupKey, "1", DedupTtl);
            await breaker.RecordSuccessAsync(delivery.EndpointUrl);
            await args.CompleteMessageAsync(args.Message, ct);
        }
        catch (NonRetryableDeliveryException ex)
        {
            // 4xx (except 408/429): the endpoint rejected us for good. Dead-letter, never retry.
            await args.DeadLetterMessageAsync(args.Message, "non_retryable", ex.Message, ct);
        }
        // Catch everything except a genuine shutdown cancellation. HttpClient timeouts surface as
        // TaskCanceledException too, so we only bail out when our own token is actually cancelled.
        catch (Exception ex) when (ex is not OperationCanceledException || !ct.IsCancellationRequested)
        {
            var attempt = args.Message.ApplicationProperties.TryGetValue("attempt", out var a)
                ? Convert.ToInt32(a) : 1;
            var tripped = await breaker.RecordFailureAsync(delivery.EndpointUrl);

            if (attempt >= MaxAttempts || tripped)
            {
                await args.DeadLetterMessageAsync(args.Message,
                    tripped ? "circuit_tripped" : "max_attempts", ex.Message, ct);
                return;
            }

            // Exponential backoff + full jitter, capped at 5 min. Reschedule a fresh copy
            // and complete the original so we own the retry clock, not the broker.
            var delay = Backoff(attempt);
            await ScheduleRetryAsync(delivery, attempt + 1, delay, ct);
            await args.CompleteMessageAsync(args.Message, ct);
            logger.LogWarning(ex, "Webhook {EventId} attempt {Attempt} failed; retry in {Delay}",
                delivery.EventId, attempt, delay);
        }
    }

    private async Task DispatchAsync(WebhookDelivery delivery, CancellationToken ct)
    {
        var client = httpFactory.CreateClient("webhook");
        using var request = new HttpRequestMessage(HttpMethod.Post, delivery.EndpointUrl)
        {
            Content = new StringContent(delivery.Body, System.Text.Encoding.UTF8, "application/json"),
        };
        request.Headers.Add("X-Mattrx-Event-Id", delivery.EventId.ToString());
        request.Headers.Add("X-Mattrx-Event-Type", delivery.EventType);

        using var response = await client.SendAsync(request, ct);

        // 4xx (except 408/429) is the customer's problem — dead-letter fast, do not retry.
        if (!response.IsSuccessStatusCode && IsNonRetryable(response.StatusCode))
            throw new NonRetryableDeliveryException(response.StatusCode);

        response.EnsureSuccessStatusCode();
    }

    private async Task ScheduleRetryAsync(
        WebhookDelivery delivery, int nextAttempt, TimeSpan delay, CancellationToken ct)
    {
        var retry = new ServiceBusMessage(BinaryData.FromObjectAsJson(delivery))
        {
            MessageId    = $"{delivery.EventId}:{nextAttempt}",
            Subject      = delivery.EventType,
            ContentType  = "application/json",
            PartitionKey = delivery.TenantId,
            ApplicationProperties = { ["attempt"] = nextAttempt, ["tenantId"] = delivery.TenantId },
        };
        await sender.ScheduleMessageAsync(retry, DateTimeOffset.UtcNow.Add(delay), ct);
    }

    private static TimeSpan Backoff(int attempt)
    {
        // Base-2 growth in seconds with full jitter, hard-capped at 5 minutes.
        var ceiling = Math.Min(Math.Pow(2, attempt), TimeSpan.FromMinutes(5).TotalSeconds);
        return TimeSpan.FromSeconds(Random.Shared.NextDouble() * ceiling); // full jitter
    }

    private static bool IsNonRetryable(System.Net.HttpStatusCode code) =>
        (int)code is >= 400 and < 500
        && code is not (System.Net.HttpStatusCode.RequestTimeout
                        or System.Net.HttpStatusCode.TooManyRequests);

    private Task OnErrorAsync(ProcessErrorEventArgs args)
    {
        logger.LogError(args.Exception, "Service Bus processor error in {Source}", args.ErrorSource);
        return Task.CompletedTask;
    }

    public async ValueTask DisposeAsync()
    {
        await processor.StopProcessingAsync();
        await processor.DisposeAsync();
    }
}

// ---------- 3. Circuit breaker: 20 consecutive failures -> endpoint disabled ----------

public sealed class EndpointCircuitBreaker(IConnectionMultiplexer redis)
{
    private const int Threshold = 20;
    private static readonly TimeSpan OpenFor = TimeSpan.FromMinutes(30);

    public Task<bool> IsOpenAsync(Uri endpoint) =>
        redis.GetDatabase().KeyExistsAsync($"wh:cb:open:{endpoint}");

    public Task RecordSuccessAsync(Uri endpoint) =>          // one success clears the streak
        redis.GetDatabase().KeyDeleteAsync($"wh:cb:fails:{endpoint}");

    public async Task<bool> RecordFailureAsync(Uri endpoint) // true when this failure trips it
    {
        var db = redis.GetDatabase();
        var fails = await db.StringIncrementAsync($"wh:cb:fails:{endpoint}");
        if (fails < Threshold) return false;
        await db.StringSetAsync($"wh:cb:open:{endpoint}", "1", OpenFor);
        return true;
    }
}

public sealed class NonRetryableDeliveryException(System.Net.HttpStatusCode statusCode)
    : Exception($"Endpoint returned non-retryable status {(int)statusCode}.")
{
    public System.Net.HttpStatusCode StatusCode { get; } = statusCode;
}

The producer never blocks on a customer's endpoint — it only writes to the queue — and the worker owns the retry clock via scheduled messages, so a slow or dead endpoint can never back-pressure the API. The Redis SET-based dedup key makes at-least-once delivery safe to retry, while the breaker sheds load from any endpoint that has failed 20 times in a row.

The honest stuff: caveats and when it's overkill

1. A small app should not build channel integrations. Use Twilio/SendGrid/OneSignal directly; build this pipeline only when volume, reliability, and multi-channel actually demand it.
2. Exactly-once delivery is a myth here. You cannot un-send an SMS or recall an email. Aim for at-least-once + dedup, and be honest that a rare duplicate is possible.
3. Never mix transactional and promotional. A campaign must not be able to delay an OTP — separate queues, separate workers, full stop.
4. Respect preferences and the law. Opt-outs, quiet hours, and frequency caps are TCPA/GDPR/CAN-SPAM obligations, not features — and ignoring them tanks engagement via notification fatigue.
5. Providers will fail. Design for Twilio being down: retries, circuit breakers, DLQ, and a fallback provider. Don't assume the happy path.
6. Fan-out can self-DDoS. One event → tens of millions of messages can overwhelm your own workers and the providers. Batch the expansion and rate-limit to provider limits.
7. "Sent" is not "read." Track delivery states honestly; a notification handed to a provider can still bounce, be filtered, or land on a device with notifications off.

The model to carry forward

A notification system is a queue that buffers your events from unreliable third-party channels. Everything hard about it — fan-out, priority, dedup, retries — is solved by not calling providers inline: you enqueue, and workers deliver at a safe rate with retries and a dead-letter safety net. Add priority lanes so urgent beats bulk, idempotent dedup so at-least-once doesn't mean double-send, and preference/rate-limit checks so you stay polite and legal. That's the whole system.

Three habits this problem teaches:

1. Put a queue in the middle. It's the answer to bursts, provider outages, and retries all at once.
2. Separate urgent from bulk. The OTP-behind-the-newsletter failure is the one interviewers probe for.
3. Assume redelivery, so dedup. At-least-once is the reality; an idempotency key is the one-line fix.

Further reading

• How to Crack Any System Design Interview: A Repeatable Framework
• Design a Distributed Cache — Consistent Hashing, Eviction & Replication
• Outbox Pattern — A Complete Guide with Order Processing Example

---

Prepping for a system design round and want the notification-system follow-ups drilled — or the next "Design X" walked through the same way? I'm always happy to help; reach me at randhir.jassal@gmail.com.