Design a Payment System — Idempotency, Ledgers, and Exactly-Once at Scale

"Design a payment system" is the one where scale is not the point. Nobody cares that it does ten thousand transactions a second; they care that it never does one transaction twice, never loses one, and can prove — line by line, months later — exactly where every cent went. The interview lives in four words: idempotency, ledger, and reconciliation. Retries must not double-charge, the books must always balance, and when your record and the bank's record disagree, you must catch it. Get those right and the rest is plumbing.

A payment system charges customers, records the movement of money, and stays correct in the face of retries, timeouts, and an external provider you don't control. Its defining property isn't throughput — it's correctness under uncertainty: a request times out and you genuinely don't know if the charge went through. Everything in the design exists to make that situation safe: an idempotency key so a retry can't double-charge, a double-entry ledger so the books are auditable and always balance, and reconciliation so a lost message can't silently corrupt the truth. (New to the method? Start with the framework.)

TL;DR — the design at a glance

Concern	Decision
Double-charge safety	Idempotency key (unique) — a retry returns the first result, never re-charges
Money records	Double-entry ledger — every movement is balanced entries; balances are derived, not stored
Provider events	At-least-once + dedup by event id (exactly-once is a myth across an external provider)
Consistency	Ledger writes are atomic (DB transaction); the external charge is bridged by a state machine
Safety net	Reconciliation — compare your ledger to the provider daily; flag every discrepancy
Card data	Never store it — tokenize via the provider (PCI)

• Payments is a correctness problem, not a scale problem — optimize for "never wrong," not "very fast."
• The idempotency key is the single most important idea: the same key must produce the same result, so a client retry after a timeout is safe.
• Double-entry ledger: every transaction posts balanced debit/credit entries that net to zero; never mutate a single "balance" — derive it from the immutable entries.
• You cannot wrap an external charge and your DB write in one transaction — use a state machine (pending → succeeded/failed) confirmed by the provider's webhook.
• The webhook is at-least-once, so dedup by event id or you'll apply the same payment twice.
• Reconciliation is the backstop: a lost webhook means your ledger is missing a real charge — a daily compare against the provider catches it.
• Never trust the client for amounts, and never store raw card data — tokenize through the provider.

The one mental shift: a payment system is designed around the moment you don't know what happened — the request timed out. Every core piece (idempotency key, state machine, webhook confirmation, reconciliation) is an answer to "did that charge actually go through?" Design backward from that uncertainty and the pieces are inevitable.

Step 1 — Requirements & scope

Functional:

• Charge a customer through a payment provider (Stripe/Adyen/etc.).
• Record every money movement in a ledger; support refunds.
• Consume provider webhooks for authoritative settlement (async).
• Reconcile internal records against the provider.

Parked: fraud/risk scoring, multi-currency FX, payouts/marketplace splits, chargeback dispute flows.

Non-functional:

• Correctness above all — never lose or double-count money.
• Idempotent — retries must be safe.
• Strongly consistent ledger — the books always balance.
• Auditable — every entry traceable, immutable.
• Available — but correctness wins ties.

Step 2 — Back-of-envelope estimates

This is the design where you say the quiet part out loud: it's not a scale problem.

Even a large processor: a few million transactions/day  ≈  ~tens–hundreds/sec

That fits comfortably on a well-indexed relational database. So we spend the budget not on sharding and caching but on invariants: uniqueness constraints for idempotency, a balanced ledger, and a reconciliation job. Correctness is the scarce resource here, not QPS.

Step 3 — API

POST /payments
  Idempotency-Key: <client-generated uuid>
  { customerId, amount, currency, sourceToken }
  -> { paymentId, status }                     201 (or the prior result on retry)

POST /webhooks/provider     (payment_intent.succeeded, .failed, charge.refunded)
GET  /payments/{paymentId}

Two rules encoded here: the Idempotency-Key is client-generated and makes the charge safe to retry, and the amount is server-validated against what the customer actually owes — never taken on trust.

Step 4 — Data model

payments        paymentId, idempotencyKey (UNIQUE), customerId, amount, currency,
                status (pending|succeeded|failed|refunded), providerRef, createdAt
ledger_entries  entryId, transactionId, accountId, amount (SIGNED), createdAt   (immutable, append-only)
accounts        accountId, type (cash|receivable|revenue|...)
webhook_events  eventId (UNIQUE), processedAt                                   (dedup log)

• The UNIQUE(idempotencyKey) constraint is the idempotency mechanism — the database enforces "one charge per key."
• ledger_entries is append-only and immutable; a balance is SUM(amount) over an account, never a mutable field.
• webhook_events with UNIQUE(eventId) makes at-least-once webhooks safe.

Step 5 — High-level architecture

  Client ──▶ [ Payment API ] ── idempotency check ──▶ [ Provider (Stripe) ]
                   │  (pending)                              │
                   ▼                                         │ webhook (at-least-once)
             [ Ledger (double-entry, atomic) ] ◀── dedup ───┘  payment_intent.succeeded
                   │
             [ Reconciliation job ] ── daily compare vs provider ──▶ flag discrepancies

The API records intent and calls the provider; the webhook delivers the authoritative outcome; the ledger records the balanced entries atomically; the reconciliation job guarantees that even a dropped webhook can't leave the books wrong.

Step 6 — Deep-dive: the hard parts

Idempotency — the whole ballgame

A client charges, the request times out, the client retries. Did the first attempt succeed? You can't know from the client's side — so the server must make the retry safe. The client sends an idempotency key; the server stores it under a unique constraint before doing anything expensive. On a retry, the key already exists, so you return the stored result instead of charging again. Pass the same key to the provider too (Stripe's Idempotency-Key header), so even the external call dedups. One key, one charge, forever.

You can't make the charge and the ledger one transaction

The provider is external — you cannot enroll "call Stripe" and "write my ledger" in a single ACID transaction. So you use a state machine: record the payment as pending (committed), call the provider, and let the webhook drive the authoritative transition to succeeded/failed, at which point you post the ledger — all in one local transaction. The internal ledger is strongly consistent; the external settlement is eventually consistent; the webhook is the bridge.

Double-entry ledger — the books must balance

Never store a single mutable balance. Instead, every money movement posts balanced entries — debits equal credits, netting to zero — into an immutable, append-only ledger. A successful charge might debit Cash and credit the customer's Receivable. Balances are derived (SUM over an account), so the ledger is auditable, replayable, and self-checking: if a transaction's entries don't sum to zero, you reject it. This is the same event-sourced discipline banks have used for centuries.

Webhooks are at-least-once — dedup them

The provider will occasionally deliver the same payment_intent.succeeded twice. Applying it twice double-credits the ledger. Guard every handler with the UNIQUE(eventId) dedup log: record the event id first; if it's already there, the second delivery is a no-op.

Reconciliation — the backstop that makes it trustworthy

What if a webhook is lost? The charge succeeded at the provider but your ledger never recorded it — a silent, invisible gap. Reconciliation closes it: a scheduled job pulls the provider's transaction list and compares it to your ledger, flagging anything that exists on one side but not the other. Reconciliation is what lets you sleep, because it turns "hope the webhook arrived" into "prove the books match."

Step 7 — Bottlenecks, failure & scaling

• Provider timeout (the classic): you called Stripe and got no response. Don't blindly retry the charge — retry with the same idempotency key (safe), or query the provider for the intent's status. Reconciliation catches whatever slips through.
• Ledger scale: append-only writes scale well; partition by account/time; balances via periodic snapshots + entries since.
• Outbox for downstream events: publish "payment.succeeded" to other services via the outbox pattern so a crash can't drop the event.
• Refunds/reversals are just more balanced entries — never delete or mutate existing ones.

Step 8 — Trade-offs & wrap-up

• Strong (ledger) vs eventual (settlement): the internal ledger is strongly consistent; the external world is eventual; reconciliation bridges the two.
• Idempotency key: client- vs server-generated: client-generated keys make client retries safe end-to-end — the right default for payments.
• Double-entry vs single balance: double-entry costs more writes and buys auditability, self-checking, and the trust that a payments system lives or dies on.
• Sync vs async: real systems are async — authorize, then let the webhook confirm capture — because the provider is, itself, eventually consistent.
• PCI: never let raw card numbers touch your servers; tokenize through the provider and store only the token.

The design checklist

• Idempotency key with a UNIQUE constraint; same key → same result; pass it to the provider too.
• State machine (pending → succeeded/failed/refunded) confirmed by the provider webhook.
• Double-entry ledger, append-only, balances derived — reject unbalanced transactions.
• Dedup webhooks by event id (UNIQUE).
• Reconciliation job comparing ledger vs provider daily.
• Provider-timeout handling via same-key retry / status query, not a blind re-charge.
• Outbox for reliable downstream events.
• Never store card data; validate amounts server-side.

The honest stuff: caveats and when it's overkill

1. A hobby project can use Stripe Checkout and stop. If the provider hosts the flow and you just store the resulting token, you don't need a ledger and reconciliation engine. Build this when you are the system of record for money.
2. Exactly-once is a myth — say so. Across an external provider and a flaky network, you get at-least-once. Idempotency keys and event-id dedup are how you make it behave as once.
3. A single mutable balance column is a time bomb. Concurrent updates, partial failures, and no audit trail. Double-entry isn't academic — it's the only design that survives an auditor.
4. The timeout is the real test. Most bugs live in "I called the provider and don't know what happened." If your design has no answer there (same-key retry, status query, reconciliation), it's not a payment system.
5. Lost webhooks happen. Never treat the webhook as guaranteed. Reconciliation isn't optional garnish — it's the only thing that catches a silently dropped settlement.
6. Don't reconcile by mutating history. Fixing a discrepancy means posting a new correcting transaction, never editing or deleting past entries. The ledger is immutable.
7. PCI scope is expensive — stay out of it. The moment a raw PAN touches your server you inherit a compliance burden. Tokenize and never see the card.

In production at Mattrx

Mattrx bills ~thousands of tenant workspaces on monthly subscriptions plus usage overages, charged through Stripe. The first billing service was dangerously naive: it charged inline in the request, and a timeout on Stripe's response left it not knowing whether the charge landed — so a retry occasionally double-charged a customer, and support cleaned it up by hand. Worse, "paid" was a boolean on the subscription row, so there was no auditable trail and monthly reconciliation against Stripe was a spreadsheet. We rebuilt it as exactly the design above: a client idempotency key on every charge, a double-entry ledger as the system of record, event-id-deduped Stripe webhooks, and a nightly reconciliation job.

Metric	Before	After
Double-charges on retry	A handful a month (manual refunds)	0 (idempotency key + Stripe Idempotency-Key)
Money record	paid boolean on the subscription	Immutable double-entry ledger (balances always reconcile)
Webhook double-application	Occasional double-credit	0 (UNIQUE(eventId) dedup)
Reconciliation vs Stripe	Manual spreadsheet, ~hours/month	Automated nightly, discrepancies auto-flagged
Billing disputes from double-charges	Non-zero	0 in the last two quarters

The billing write path stopped being able to charge twice, the books became provably correct, and a dropped Stripe webhook is now caught by reconciliation the next morning instead of surfacing as an angry support ticket.

The architecture at Mattrx

  React billing UI ──▶ [ Azure App Service · .NET 9 billing API ]
                              │  reserve idempotency key (pending)
                              ▼
                        [ Stripe ]  (same Idempotency-Key)
                              │  payment_intent.succeeded  (at-least-once)
                              ▼
                 [ Webhook handler ] ── dedup by eventId ──▶ [ Ledger (Azure SQL, double-entry) ]
                              ▲                                        │
        nightly reconcile ────┴──── [ Reconciliation job ] ◀──────────┘  compare vs Stripe, flag gaps

Production implementation (Mattrx)

Here is the core of Mattrx's billing path on ASP.NET Core / .NET 9: an idempotent charge command that reserves the key before touching Stripe, and an event-id-deduped webhook handler that posts a balanced double-entry ledger transaction — the whole thing enforcing "one key, one charge" and "the books always net to zero."

using Microsoft.Extensions.Logging;

namespace Mattrx.Billing;

public sealed record ChargeCommand(
    string IdempotencyKey, Guid TenantId, long AmountMinor, string Currency, string SourceToken);

public sealed class BillingService(
    IPaymentRepository payments,
    ILedger ledger,
    IStripeClient stripe,
    IUnitOfWork uow,
    ILogger<BillingService> logger)
{
    // Charge a tenant. Safe to retry: the same IdempotencyKey never charges twice.
    public async Task<PaymentView> ChargeAsync(ChargeCommand cmd, CancellationToken ct)
    {
        // 1) A retry with the same key returns the original result — no second charge.
        var existing = await payments.FindByIdempotencyKeyAsync(cmd.IdempotencyKey, ct);
        if (existing is not null)
            return existing.ToView();

        // 2) Reserve the key by inserting a pending row. The UNIQUE(idempotency_key)
        //    constraint makes two concurrent attempts collapse into one.
        var payment = Payment.Pending(cmd.IdempotencyKey, cmd.TenantId, cmd.AmountMinor, cmd.Currency);
        try
        {
            await payments.InsertAsync(payment, ct);
            await uow.SaveChangesAsync(ct);
        }
        catch (DuplicateKeyException)
        {
            // Lost the race; the winner's row is authoritative.
            return (await payments.FindByIdempotencyKeyAsync(cmd.IdempotencyKey, ct))!.ToView();
        }

        // 3) Call Stripe with the SAME key so Stripe also dedups on its side.
        var intent = await stripe.CreatePaymentIntentAsync(new PaymentIntentRequest(
            cmd.AmountMinor, cmd.Currency, cmd.SourceToken), idempotencyKey: cmd.IdempotencyKey, ct);

        await payments.AttachProviderRefAsync(payment.Id, intent.Id, ct);
        await uow.SaveChangesAsync(ct);

        // Authoritative success arrives via webhook; the caller sees "pending".
        return payment.ToView();
    }
}

// Stripe webhook: at-least-once delivery. Dedup, then post the ledger atomically.
public sealed class StripeWebhookHandler(
    IWebhookEventLog eventLog,
    IPaymentRepository payments,
    ILedger ledger,
    IUnitOfWork uow)
{
    public async Task HandleAsync(StripeEvent evt, CancellationToken ct)
    {
        await using var tx = await uow.BeginTransactionAsync(ct);

        // Dedup: UNIQUE(event_id). A redelivered event is a no-op.
        if (!await eventLog.TryRecordAsync(evt.Id, evt.Type, ct))
        {
            await tx.CommitAsync(ct);
            return;
        }

        if (evt.Type == "payment_intent.succeeded")
        {
            var payment = await payments.GetByProviderRefAsync(evt.PaymentIntentId, ct);
            payment.MarkSucceeded(evt.OccurredAt);

            // Double-entry: debit Cash, credit the tenant's Receivable. Must net to zero.
            await ledger.PostAsync(LedgerTransaction.Create(
                reference: payment.Id,
                LedgerEntry.Debit(Accounts.Cash, payment.AmountMinor, payment.Currency),
                LedgerEntry.Credit(Accounts.Receivable(payment.TenantId), payment.AmountMinor, payment.Currency)),
                ct);

            await payments.UpdateAsync(payment, ct);
        }

        await uow.SaveChangesAsync(ct);
        await tx.CommitAsync(ct);
    }
}

public sealed class Ledger(ILedgerEntryStore entries) : ILedger
{
    // Invariant enforced on every post: a transaction's signed entries net to zero,
    // so the books can never be written into an unbalanced state.
    public async Task PostAsync(LedgerTransaction txn, CancellationToken ct)
    {
        if (txn.Entries.Sum(e => e.SignedAmountMinor) != 0)
            throw new UnbalancedLedgerException(txn.Reference);

        foreach (var entry in txn.Entries)
            await entries.AppendAsync(entry, ct);   // append-only; never update or delete
    }
}

The charge path reserves the idempotency key before calling Stripe and passes the same key onward, so a client retry and a provider retry both collapse to one charge; the webhook handler dedups on event_id and posts a balanced ledger transaction inside one local DB transaction; and Ledger.PostAsync refuses to write anything that doesn't net to zero — so at Mattrx the books are correct by construction, and the nightly reconciliation job exists to catch the one thing code can't: a settlement Stripe recorded but never told us about.

The model to carry forward

A payment system is a machine for staying correct when you don't know what happened. The timeout is the enemy, and every part of the design is a defense against it: an idempotency key so retries are safe, a state machine so an external charge and an internal record don't have to be one transaction, a double-entry ledger so the truth is auditable and self-checking, deduped webhooks so at-least-once behaves like once, and reconciliation so a lost message can't quietly corrupt the books. Optimize for never wrong, not very fast, and the whole design falls into place.

Three habits this problem teaches:

1. Lead with idempotency. The first thing out of your mouth should be "the client sends an idempotency key" — it frames the entire correctness story.
2. Make money a ledger, not a number. Immutable balanced entries with derived balances is the only design an auditor (or a bug) can't break.
3. Assume the webhook can be lost. Reconciliation is what turns hope into proof, and it's the answer interviewers wait for.

Further reading

• How to Crack Any System Design Interview: A Repeatable Framework
• Outbox Pattern — A Complete Guide with Order Processing Example
• Saga Pattern for Microservices — A Complete Guide

---

Prepping for a system design round and want the payment-system follow-ups drilled — or the next "Design X" walked through the same way? I'm always happy to help; reach me at randhir.jassal@gmail.com.