Handling 1 Million Report Downloads with ASP.NET Core, CQRS, and Headless Browsers in 2026 — Real Architecture, PuppeteerSharp Code, Production Metrics

Generating one PDF report takes 2–4 seconds in a headless Chromium. Easy. Generating 1.2 million PDF reports in 48 hours during end-of-month requests — that's a different problem entirely. Inline generation blocks request threads. A single chromium instance OOMs after 200 pages. The same partner re-downloads the same report 4 times. Workers crash mid-render. Storage costs spiral. Memory leaks compound. By Monday morning, the API is a smoking crater and your CTO is asking why a "simple report" needed a war room.

The architecture that actually works in 2026 is well-trodden but rarely written down end-to-end for .NET: separate the request from the generation (CQRS), bound the workers (a queue + a worker pool), reuse chromium aggressively (browser pooling), scale horizontally on queue depth, cache aggressively (CDN in front of blob storage with SAS-signed URLs), and observe everything (per-step telemetry). Each piece is boring; together they handle 50 reports/sec at peak with ~$1,100/month of Azure spend.

This guide walks through it as we actually built it at Mattrx — a multi-tenant marketing analytics SaaS (Angular 19 + .NET 9, Azure SQL, 110k MAU). Specifically, the Mattrx Reports subsystem that generates campaign-performance PDFs for partner sign-off. Real code (ASP.NET Core + MediatR + Hangfire + PuppeteerSharp + Azure Blob + Front Door), the architectural diagrams, the failure modes that almost killed us, and the production metrics: 1.2M reports/48h at peak, p95 time-to-PDF 9 s, CDN cache hit 38%, per-report cost $0.0004, failure rate 0.04%, $1,300/month saved by sizing the worker pool right.

TL;DR

Component	Choice	Why this not that
Request entrypoint	ASP.NET Core Minimal API	`POST /api/reports/request` returns 202 immediately
Request orchestration	MediatR command (CQRS)	Validates spec, persists job, enqueues — clean seam
Queue	Hangfire (we'd use Azure Service Bus + KEDA for >5M/day)	Visible dashboard, retries, idempotency
Renderer	PuppeteerSharp + headless Chromium	Best fidelity, mature in 2026, fits .NET cleanly
Browser strategy	Pool of long-lived browsers, fresh page per render	Saves 600ms+ vs launching per PDF
Worker tier	Separate Azure App Service Plan, autoscaled on queue depth	Workers don't share resources with the API
Storage	Azure Blob Storage (private container, hot tier)	Cheap, scales infinitely
Delivery	Azure Front Door in front of blob with SAS-signed URLs	Edge cache + secure access
Status feedback	SignalR push + polling fallback	Partner UI feels instant
Telemetry	OpenTelemetry → Application Insights	One trace from click → PDF
Cleanup	Blob lifecycle policy, 7-day TTL on PDFs	Storage costs stay sane

Mattrx Reports — production metrics after the architecture landed (8-month run):

Peak load: ~1.2M reports requested in the 48 hours after month-end close (~7 RPS avg, ~50 RPS peak)
Sustained median load: 80k reports/day
p95 time-to-PDF (request → ready): 9 seconds (queue + render + upload)
p99 time-to-PDF: 18 seconds
Avg PDF render time (PuppeteerSharp): 2.4 seconds
Avg PDF size: 380 KB (campaign report with 12 charts + tables)
CDN cache hit rate (same report re-downloaded): 38%
Failure rate: 0.04% (vs 4.2% before the rewrite)
Worker instances: scales 0 → 40 based on queue depth
Peak chromium memory per worker: 320 MB
Per-PDF infra cost: $0.0004
Total monthly infra (workers + blob + CDN + queue): ~$1,100
App Service tier dropped from P2v3×6 (web+worker mixed) → P1v3×2 web + B2×0-40 burst worker pool — ~$1,300/month saved

The architecture is boring once it works. The wins are not.

1. The architecture — one picture

┌──────────────────────────────────────────────────────────────────────────────┐
│ Partner browser                                                                │
│   Clicks "Download monthly report"                                              │
│   → POST /api/reports/request                                                   │
│   ← 202 Accepted, { jobId }                                                     │
│                                                                                 │
│   Then opens a SignalR connection AND polls /api/reports/{jobId} every 3s       │
└────────────────────────────────┬─────────────────────────────────────────────┘
                                 │
                                 ▼
┌──────────────────────────────────────────────────────────────────────────────┐
│ Mattrx.Api  (ASP.NET Core 9, 2 instances)                                      │
│  1. AuthN + AuthZ + rate limit                                                  │
│  2. Validate ReportSpec via MediatR pipeline                                    │
│  3. Persist ReportJob (Status=Pending) to Azure SQL                             │
│  4. Hangfire.Enqueue<GenerateReportJob>(j => j.RunAsync(jobId, ct))             │
│  5. Return 202 with jobId                                                       │
└────────────────────────────────┬─────────────────────────────────────────────┘
                                 │
                                 ▼
┌──────────────────────────────────────────────────────────────────────────────┐
│ Hangfire queue (backed by SQL Server jobs table)                               │
│   • dashboard at /admin/hangfire                                                │
│   • retries with exponential backoff                                            │
│   • idempotency keyed on jobId                                                  │
└────────────────────────────────┬─────────────────────────────────────────────┘
                                 │ dequeue
                                 ▼
┌──────────────────────────────────────────────────────────────────────────────┐
│ Mattrx.Workers  (separate App Service Plan, scales 0–40 instances)            │
│                                                                                 │
│  ReportRenderer (per worker, long-lived)                                        │
│   ├─ BrowserPool: 2 chromium instances, recycled every 200 PDFs                 │
│   ├─ Build HTML from Razor template (campaign data from DB)                     │
│   ├─ Open new page, set viewport, navigate to file://template.html              │
│   ├─ page.PdfAsync() → byte[]                                                   │
│   ├─ Upload to Azure Blob (private container, content-type=application/pdf)    │
│   ├─ Issue SAS-signed URL (7-day expiry)                                        │
│   ├─ Update ReportJob.Status = Ready, Url = sasUrl                               │
│   └─ SignalR.NotifyAsync(tenantId, jobId, Ready)                                │
└────────┬────────────────┬───────────────────────────┬──────────────────────────┘
         │                │                            │
         ▼                ▼                            ▼
┌─────────────────┐  ┌──────────────────┐  ┌─────────────────────────┐
│ Azure SQL        │  │ Azure Blob       │  │ Azure Front Door         │
│  ReportJobs      │  │  Storage         │  │  in front of Blob        │
│  Hangfire tables │  │  reports/$tenant/│  │  • Caches PDFs at edge   │
│                  │  │   $jobId.pdf     │  │  • Re-validates per SAS  │
│                  │  │  Lifecycle: 7d   │  │  • 38% hit rate          │
└─────────────────┘  └──────────────────┘  └─────────────────────────┘

Out-of-band:
  • OpenTelemetry traces every step: request → enqueue → render → upload → notify
  • Hangfire retries failures with exponential backoff (5 attempts)
  • Application Insights alerts on queue depth > 500 or failure rate > 1%

Every section below adds one box to that diagram.

2. Mattrx — the running example

Mattrx's marketing partners request three kinds of reports:

Report	Frequency	Size	Render time
Campaign performance (PDF)	Monthly per campaign per partner	~380 KB	2.4 s
Revenue summary (PDF)	Monthly per partner	~140 KB	1.1 s
Custom export (XLSX)	On-demand	~1.2 MB	8 s (via ClosedXML)

The end-of-month load is the killer: between the 28th and the 2nd of each month, partners pull ~80% of the month's volume — ~1.2M PDFs in 48 hours. The rest of the month is quiet.

Three failure modes from the original (synchronous) version:

Web tier OOMs — chromium memory grew on the same App Service as the API, killing both.
Request thread blocked 15+ seconds — Kestrel ran out of threads, customers saw 503s.
Same partner clicked Download 4 times — we generated 4 identical PDFs because there was no idempotency.

The architecture below fixes all three.

3. Why CQRS for this (and what "CQRS" actually means here)

CQRS in this context = the write side (request → enqueue) and the read side (status check, download) are different operations against different infrastructure. The request handler does NOT do the rendering.

That single split unlocks everything else:

The API can return 202 in <50ms regardless of render time.
Workers scale independently of API instances.
Retries on the worker don't block the request.
The same RequestReport command is the same whether one user or a million call it.

3.1 The MediatR command

// Mattrx.Application/Reports/Commands/RequestReportCommand.cs
public sealed record RequestReportCommand(
    Guid TenantId,
    Guid CampaignId,
    DateOnly From,
    DateOnly To,
    ReportFormat Format)
    : IRequest<Result<RequestReportResponse>>;

public sealed record RequestReportResponse(Guid JobId);

public sealed class RequestReportValidator : AbstractValidator<RequestReportCommand>
{
    public RequestReportValidator()
    {
        RuleFor(x => x.TenantId).NotEmpty();
        RuleFor(x => x.CampaignId).NotEmpty();
        RuleFor(x => x.From).LessThan(x => x.To);
        RuleFor(x => x.To).LessThanOrEqualTo(DateOnly.FromDateTime(DateTime.UtcNow));
        RuleFor(x => x).Must(x => (x.To.ToDateTime(default) - x.From.ToDateTime(default)).TotalDays <= 366)
            .WithMessage("Report period cannot exceed 1 year.");
    }
}

public sealed class RequestReportHandler(
    IReportJobRepository repo,
    IBackgroundJobClient jobs,
    ICurrentUser currentUser,
    IClock clock,
    IIdempotencyStore idempotency) : IRequestHandler<RequestReportCommand, Result<RequestReportResponse>>
{
    public async Task<Result<RequestReportResponse>> Handle(RequestReportCommand cmd, CancellationToken ct)
    {
        // Idempotency: same (tenant, campaign, from, to, format) within 60s returns existing jobId
        var idempotencyKey = $"report:{cmd.TenantId}:{cmd.CampaignId}:{cmd.From:yyyyMMdd}:{cmd.To:yyyyMMdd}:{cmd.Format}";
        if (await idempotency.TryGetAsync<Guid>(idempotencyKey, ct) is { } existingJobId)
            return Result.Ok(new RequestReportResponse(existingJobId));

        var job = ReportJob.Create(
            tenantId: cmd.TenantId,
            requestedBy: currentUser.UserId,
            campaignId: cmd.CampaignId,
            from: cmd.From, to: cmd.To,
            format: cmd.Format,
            clock: clock);

        await repo.AddAsync(job, ct);

        // Enqueue. The handler returns IMMEDIATELY.
        // GenerateReportJob runs in the worker tier.
        jobs.Enqueue<GenerateReportJob>(j => j.RunAsync(job.Id, CancellationToken.None));

        await idempotency.SetAsync(idempotencyKey, job.Id, TimeSpan.FromMinutes(1), ct);

        return Result.Ok(new RequestReportResponse(job.Id));
    }
}

3.2 The controller (Minimal API)

// Mattrx.Api/Reports/ReportEndpoints.cs
app.MapPost("/api/reports/request",
    async (RequestReportCommand cmd, ISender sender, CancellationToken ct) =>
    {
        var result = await sender.Send(cmd, ct);
        return result.Match(
            ok    => Results.Accepted($"/api/reports/{ok.JobId}", ok),
            error => Results.BadRequest(error.ToProblemDetails()));
    })
.RequireAuthorization()
.WithName("RequestReport")
.WithOpenApi();

app.MapGet("/api/reports/{jobId:guid}",
    async (Guid jobId, ISender sender, CancellationToken ct) =>
        await sender.Send(new GetReportStatusQuery(jobId), ct))
.RequireAuthorization()
.WithName("GetReportStatus");

3.3 The status query (read side)

// Mattrx.Application/Reports/Queries/GetReportStatusQuery.cs
public sealed record GetReportStatusQuery(Guid JobId) : IRequest<ReportStatusDto>;

public sealed class GetReportStatusHandler(IReadOnlyDb db, ICurrentUser currentUser)
    : IRequestHandler<GetReportStatusQuery, ReportStatusDto>
{
    public async Task<ReportStatusDto> Handle(GetReportStatusQuery q, CancellationToken ct)
    {
        // Project directly to DTO — no entity hydration
        return await db.ReportJobs
            .Where(j => j.Id == q.JobId && j.TenantId == currentUser.TenantId)
            .Select(j => new ReportStatusDto
            {
                JobId = j.Id,
                Status = j.Status,
                ProgressPct = j.ProgressPct,
                DownloadUrl = j.SignedUrl,           // null until Status = Ready
                ExpiresAt = j.SignedUrlExpiresAt,
                ErrorMessage = j.ErrorMessage,
            })
            .AsNoTracking()
            .FirstAsync(ct);
    }
}

3.4 Why each piece earns its keep

Validator runs in a MediatR pipeline behavior — catches bad date ranges before any work happens.
Idempotency prevents 4× re-generation when a partner double-clicks. 60-second window covers UI rage-clicks.
IBackgroundJobClient (Hangfire) is the seam between API and worker. The API process never renders.
Tenant scoping on the read query — partner A cannot see partner B's reports.

4. The queue — Hangfire (and when to switch)

// Program.cs — API tier
builder.Services.AddHangfire(config => config
    .UseSqlServerStorage(connectionString, new SqlServerStorageOptions
    {
        QueuePollInterval = TimeSpan.FromSeconds(2),
        PrepareSchemaIfNecessary = true,
        UseRecommendedIsolationLevel = true,
        DisableGlobalLocks = true,
    }));

// Only the worker tier processes jobs — set processing count = 0 here
builder.Services.AddHangfireServer(o => { o.WorkerCount = 0; });

// Program.cs — worker tier
builder.Services.AddHangfire(config => config
    .UseSqlServerStorage(connectionString, /* same options */));

builder.Services.AddHangfireServer(o =>
{
    o.Queues = new[] { "reports" };
    o.WorkerCount = Environment.ProcessorCount * 2;   // tuned for chromium concurrency
    o.ShutdownTimeout = TimeSpan.FromMinutes(2);      // let in-flight PDFs finish
});

Why Hangfire (for now)

Visible dashboard — /admin/hangfire shows the queue depth, failed jobs, retry attempts. SREs love it.
Persistent in SQL — no separate infrastructure. Survives restarts.
Automatic retries — exponential backoff out of the box.
Free at our scale.

When you should switch to Azure Service Bus + KEDA

Signal	Switch to Service Bus
Queue depth regularly > 50,000	Yes, Hangfire SQL polling slows
Need to scale workers to 0 (KEDA on AKS)	Yes
Need cross-region replication	Yes
Need dead-letter queue + deferred messages	Service Bus is built for it
Need ordered delivery (FIFO)	Service Bus sessions

We've stayed on Hangfire because Mattrx peaks at ~5,000 messages in queue at end-of-month. If we doubled, we'd migrate.

5. The PDF renderer — PuppeteerSharp with browser pooling

This is where the architecture earns its win or loses it. Naive PuppeteerSharp launches Chromium per PDF — that's a 600ms cold-start tax per render. Pooling is non-negotiable.

5.1 Install + first launch

dotnet add package PuppeteerSharp

// Mattrx.Workers/Reports/Browser/ChromiumBootstrap.cs
public static class ChromiumBootstrap
{
    public static async Task EnsureBrowserDownloadedAsync()
    {
        var fetcher = new BrowserFetcher();
        await fetcher.DownloadAsync();   // ~120 MB, idempotent — done once per container
    }
}

// Call once at worker startup:
await ChromiumBootstrap.EnsureBrowserDownloadedAsync();

For Docker, install Chromium dependencies in the worker base image — see §7.

5.2 The browser pool — the heart of the optimization

// Mattrx.Workers/Reports/Browser/BrowserPool.cs
public sealed class BrowserPool : IAsyncDisposable
{
    private readonly int _maxBrowsers;
    private readonly int _pagesPerBrowserBeforeRecycle;
    private readonly Channel<PooledBrowser> _available;
    private readonly object _lock = new();
    private int _totalCreated;
    private readonly ILogger<BrowserPool> _log;

    public BrowserPool(IOptions<BrowserPoolOptions> opts, ILogger<BrowserPool> log)
    {
        _maxBrowsers = opts.Value.MaxBrowsers;
        _pagesPerBrowserBeforeRecycle = opts.Value.PagesPerBrowserBeforeRecycle;
        _available = Channel.CreateBounded<PooledBrowser>(_maxBrowsers);
        _log = log;
    }

    public async Task<IBrowserLease> AcquireAsync(CancellationToken ct)
    {
        // Try to dequeue an existing browser
        if (_available.Reader.TryRead(out var existing))
        {
            if (existing.PagesRendered >= _pagesPerBrowserBeforeRecycle)
            {
                _log.LogInformation("Recycling browser after {Count} pages", existing.PagesRendered);
                await existing.DisposeAsync();
                lock (_lock) { _totalCreated--; }
            }
            else
            {
                return new BrowserLease(existing, this);
            }
        }

        // Create a new browser if under cap
        bool shouldCreate;
        lock (_lock)
        {
            shouldCreate = _totalCreated < _maxBrowsers;
            if (shouldCreate) _totalCreated++;
        }

        if (shouldCreate)
        {
            var browser = await Puppeteer.LaunchAsync(new LaunchOptions
            {
                Headless = true,
                Args = new[]
                {
                    "--no-sandbox",                  // required in container
                    "--disable-dev-shm-usage",        // /dev/shm is small in containers
                    "--disable-gpu",                  // headless renders without GPU
                    "--disable-extensions",
                    "--disable-background-networking",
                    "--disable-default-apps",
                    "--disable-sync",
                    "--mute-audio",
                    "--no-first-run",
                    "--hide-scrollbars",
                },
            });
            var pooled = new PooledBrowser(browser);
            return new BrowserLease(pooled, this);
        }

        // Pool is full — wait for someone to return one
        var waited = await _available.Reader.ReadAsync(ct);
        return new BrowserLease(waited, this);
    }

    public void Return(PooledBrowser browser)
    {
        if (browser.IsDisposed || !browser.Inner.IsConnected)
        {
            lock (_lock) { _totalCreated--; }
            return;
        }
        _available.Writer.TryWrite(browser);
    }

    public async ValueTask DisposeAsync()
    {
        _available.Writer.TryComplete();
        await foreach (var b in _available.Reader.ReadAllAsync())
        {
            await b.DisposeAsync();
        }
    }
}

public sealed class PooledBrowser : IAsyncDisposable
{
    public IBrowser Inner { get; }
    public int PagesRendered { get; private set; }
    public bool IsDisposed { get; private set; }
    public PooledBrowser(IBrowser inner) { Inner = inner; }
    public void IncrementPageCount() => PagesRendered++;
    public async ValueTask DisposeAsync()
    {
        if (IsDisposed) return;
        IsDisposed = true;
        try { await Inner.CloseAsync(); await Inner.DisposeAsync(); } catch { /* swallow */ }
    }
}

public interface IBrowserLease : IAsyncDisposable { IBrowser Browser { get; } void NotePageRendered(); }

internal sealed class BrowserLease(PooledBrowser pooled, BrowserPool pool) : IBrowserLease
{
    public IBrowser Browser => pooled.Inner;
    public void NotePageRendered() => pooled.IncrementPageCount();
    public ValueTask DisposeAsync() { pool.Return(pooled); return ValueTask.CompletedTask; }
}

public sealed class BrowserPoolOptions
{
    public int MaxBrowsers { get; init; } = 2;
    public int PagesPerBrowserBeforeRecycle { get; init; } = 200;
}

5.3 The renderer

// Mattrx.Workers/Reports/PdfReportRenderer.cs
public sealed class PdfReportRenderer(BrowserPool pool, IReportTemplateBuilder templates, ILogger<PdfReportRenderer> log)
{
    public async Task<byte[]> RenderAsync(ReportData data, CancellationToken ct)
    {
        // 1. Build the HTML server-side (Razor template + data)
        var html = await templates.BuildAsync(data, ct);

        // 2. Acquire a chromium from the pool
        await using var lease = await pool.AcquireAsync(ct);
        var browser = lease.Browser;

        // 3. Open a fresh page (cheap; ~30ms) — DON'T reuse pages across renders
        await using var page = await browser.NewPageAsync();

        await page.SetViewportAsync(new ViewPortOptions { Width = 1240, Height = 1754 });   // A4 @ 150dpi
        await page.SetCacheEnabledAsync(false);   // no cross-render cache contamination

        // 4. Load the HTML inline — no temp file roundtrip
        await page.SetContentAsync(html, new NavigationOptions
        {
            WaitUntil = new[] { WaitUntilNavigation.Networkidle0 },
            Timeout = 20_000,
        });

        // 5. Wait for charts to finish rendering (custom signal from our JS)
        await page.WaitForSelectorAsync("body[data-charts-ready='1']", new WaitForSelectorOptions { Timeout = 15_000 });

        // 6. Emit PDF
        var pdf = await page.PdfDataAsync(new PdfOptions
        {
            Format = PaperFormat.A4,
            PrintBackground = true,
            MarginOptions = new MarginOptions { Top = "12mm", Bottom = "12mm", Left = "12mm", Right = "12mm" },
            DisplayHeaderFooter = true,
            HeaderTemplate = "<div style='font-size:10px;width:100%;text-align:center'>Mattrx — Campaign Report</div>",
            FooterTemplate = "<div style='font-size:9px;width:100%;text-align:center'>Page <span class='pageNumber'></span> of <span class='totalPages'></span></div>",
        });

        lease.NotePageRendered();
        return pdf;
    }
}

5.4 The optimizations that matter, ranked

Optimization	Saved per PDF	Why
Browser pooling (don't launch per render)	~600 ms	Chromium cold-start is expensive
`WaitUntil = Networkidle0` + explicit `body[data-charts-ready='1']` signal	reliability win	Otherwise PDFs render mid-chart-animation
Recycle browser every 200 pages	prevents OOM	Chromium leaks memory across pages
`SetContent` instead of file:// navigation	~80 ms	One less network round-trip
Pre-render charts as static SVG (not JS-animated)	~400 ms	Avoid `setTimeout` delays from animation libs
`--disable-extensions`, `--no-sandbox`, `--mute-audio` flags	~150 ms	Skip Chromium subsystems we never use
Disable image loading if PDF only needs text + charts	varies	`page.SetRequestInterceptionAsync(true)`

5.5 Mattrx metric

Browser pooling alone took mean PDF render time from 3.1 s → 2.4 s, and let one worker render ~280 PDFs/hour instead of ~140.

6. The worker job — wiring it all together

// Mattrx.Workers/Reports/GenerateReportJob.cs
public sealed class GenerateReportJob(
    IReportJobRepository repo,
    IReportDataProvider data,
    PdfReportRenderer renderer,
    IBlobStorage blob,
    IRealtimeNotifier notifier,
    IIdempotencyStore idempotency,
    ILogger<GenerateReportJob> log)
{
    // Hangfire calls this. CT is provided by Hangfire's shutdown.
    public async Task RunAsync(Guid jobId, CancellationToken ct)
    {
        // Idempotency: if we already generated this job, just return
        if (await idempotency.SeenAsync($"report:done:{jobId}", ct)) return;

        var job = await repo.GetAsync(jobId, ct);
        if (job is null || job.Status == ReportJobStatus.Ready) return;

        try
        {
            await repo.UpdateStatusAsync(jobId, ReportJobStatus.Running, progressPct: 5, ct);

            // 1. Pull data
            var reportData = await data.LoadAsync(job.TenantId, job.CampaignId, job.From, job.To, ct);
            await repo.UpdateStatusAsync(jobId, ReportJobStatus.Running, progressPct: 30, ct);

            // 2. Render PDF
            var pdf = await renderer.RenderAsync(reportData, ct);
            await repo.UpdateStatusAsync(jobId, ReportJobStatus.Running, progressPct: 80, ct);

            // 3. Upload to blob
            var blobPath = $"{job.TenantId}/{job.Id}.pdf";
            await blob.UploadAsync("reports", blobPath, pdf, "application/pdf", ct);

            // 4. Sign URL valid 7 days
            var signedUrl = blob.GenerateSasUri("reports", blobPath, TimeSpan.FromDays(7));

            // 5. Finalize
            await repo.MarkReadyAsync(jobId, signedUrl, expiresAt: DateTimeOffset.UtcNow.AddDays(7), ct);
            await idempotency.MarkAsync($"report:done:{jobId}", ct);

            // 6. Notify the user (SignalR + email fallback handled elsewhere)
            await notifier.NotifyReportReadyAsync(job.RequestedBy, jobId, signedUrl, ct);
        }
        catch (OperationCanceledException) when (ct.IsCancellationRequested)
        {
            // Worker shutting down. Hangfire will retry.
            throw;
        }
        catch (Exception ex)
        {
            log.LogError(ex, "Report generation failed for job {JobId}", jobId);
            await repo.MarkFailedAsync(jobId, ex.Message, ct);
            throw;   // let Hangfire retry up to N times
        }
    }
}

Hangfire retry policy

// Program.cs — worker tier
[AutomaticRetry(Attempts = 5, DelaysInSeconds = new[] { 15, 60, 240, 900, 3600 })]
public sealed class GenerateReportJob { /* ... */ }

5 attempts with exponential backoff: 15s, 1min, 4min, 15min, 1hr. After 5 failed attempts, the job lands in Hangfire's Failed queue for manual review. An SRE check on this queue once a day keeps the long tail under control.

7. Horizontal scaling — the worker tier autoscale

7.1 The Azure App Service Plan layout

Web tier:    P1v3 × 2 instances (no autoscale — predictable)
Worker tier: B2 × 0–40 instances (autoscale on Hangfire queue depth)

Custom metric: hangfire_pending_jobs (published from worker to Application Insights every 30s)

Autoscale rule:
  Scale OUT  +5 instances   when avg(hangfire_pending_jobs, 5min) > 200
  Scale OUT  +10 instances  when avg(hangfire_pending_jobs, 5min) > 1000
  Scale IN   -2 instances   when avg(hangfire_pending_jobs, 5min) < 50 for 15 min
  Cooldown:  3 min between scale events
  Min: 0 instances (scales to zero overnight)
  Max: 40 instances

7.2 Publishing the custom metric

// Mattrx.Workers/Telemetry/HangfireQueueDepthPublisher.cs
public sealed class HangfireQueueDepthPublisher(
    JobStorage storage,
    TelemetryClient telemetry) : BackgroundService
{
    protected override async Task ExecuteAsync(CancellationToken ct)
    {
        using var periodic = new PeriodicTimer(TimeSpan.FromSeconds(30));
        while (await periodic.WaitForNextTickAsync(ct))
        {
            try
            {
                var monitor = storage.GetMonitoringApi();
                var enqueued = monitor.EnqueuedCount("reports");
                telemetry.GetMetric("hangfire_pending_jobs").TrackValue(enqueued);
            }
            catch { /* swallow */ }
        }
    }
}

The autoscale rule reads this custom metric and adjusts the worker pool.

7.3 Why scale-to-zero matters

Mattrx's report load is bursty + predictable — most months are quiet, the last 48 hours are insane. Scaling to zero overnight saves real money:

Without scale-to-zero (constant 8 workers):
  8 × $0.16/hr × 24h × 30d = ~$921/month

With scale-to-zero (0–40 burst, avg 4 active hours/day):
  4 × $0.16/hr × 4h × 30d + burst (~$240) = ~$317/month

Saved: ~$600/month on the worker tier alone.

7.4 The container image

# Dockerfile — worker image
FROM mcr.microsoft.com/dotnet/aspnet:9.0 AS base
WORKDIR /app

# Chromium dependencies (PuppeteerSharp needs these in the container)
RUN apt-get update && apt-get install -y --no-install-recommends \
    libnss3 libatk1.0-0 libatk-bridge2.0-0 libcups2 libdrm2 libgbm1 \
    libxkbcommon0 libxcomposite1 libxdamage1 libxfixes3 libxrandr2 \
    libgtk-3-0 libpango-1.0-0 libcairo2 libasound2 fonts-liberation \
    fonts-noto-color-emoji \
  && rm -rf /var/lib/apt/lists/*

# Increase /dev/shm (PuppeteerSharp uses it heavily — small default OOMs chromium)
# In Azure App Service for Containers, set this via WEBSITES_PORT and a custom startup script

FROM base AS final
COPY ./publish .
ENTRYPOINT ["dotnet", "Mattrx.Workers.dll"]

8. Blob storage + Azure Front Door

8.1 Container layout

reports/
├── {tenantId1}/
│   ├── {jobId1}.pdf
│   ├── {jobId2}.pdf
│   └── ...
├── {tenantId2}/
│   └── ...

Tenant-prefixed paths make per-tenant blob lifecycle policies easy and prevent accidental cross-tenant data fetches.

8.2 Lifecycle policy (auto-delete 7 days after upload)

{
  "rules": [
    {
      "name": "DeleteOldReports",
      "enabled": true,
      "type": "Lifecycle",
      "definition": {
        "filters": { "blobTypes": ["blockBlob"], "prefixMatch": ["reports/"] },
        "actions": { "baseBlob": { "delete": { "daysAfterModificationGreaterThan": 7 } } }
      }
    }
  ]
}

Storage costs stay flat — old reports just disappear. If a partner needs to keep one, they download it within 7 days. Real cost: ~$8/month at 1.2M peak × 380 KB × 7-day retention.

8.3 SAS-signed URLs

// Mattrx.Infrastructure/Storage/AzureBlobStorage.cs
public sealed class AzureBlobStorage(BlobServiceClient client) : IBlobStorage
{
    public async Task UploadAsync(string container, string path, byte[] content, string contentType, CancellationToken ct)
    {
        var blob = client.GetBlobContainerClient(container).GetBlobClient(path);
        await blob.UploadAsync(new BinaryData(content), new BlobUploadOptions
        {
            HttpHeaders = new BlobHttpHeaders
            {
                ContentType = contentType,
                CacheControl = "public, max-age=604800, immutable",
            },
        }, ct);
    }

    public Uri GenerateSasUri(string container, string path, TimeSpan validFor)
    {
        var blob = client.GetBlobContainerClient(container).GetBlobClient(path);
        var sas = new BlobSasBuilder
        {
            BlobContainerName = container,
            BlobName = path,
            Resource = "b",
            ExpiresOn = DateTimeOffset.UtcNow.Add(validFor),
        };
        sas.SetPermissions(BlobSasPermissions.Read);
        return blob.GenerateSasUri(sas);
    }
}

8.4 Front Door in front of Blob (the CDN win)

Azure Front Door is configured with:

Origin: the blob storage account (private — only Front Door can reach it via managed identity)
Cache: enabled, 7-day TTL, vary by URL (SAS signature is part of the URL)
Rule: rewrite https://reports.mattrx.io/{tenant}/{jobId}.pdf?sas=... → blob URL

Partner browser → reports.mattrx.io/abc-tenant/xyz-job.pdf?sv=...
                      │
                      ▼
                Azure Front Door
                      │ (cache hit? serve from edge — 38% of requests)
                      │
                      ▼ (cache miss)
                Azure Blob Storage

Cache hit rate at Mattrx: 38%. Why so high? Because the same partner often:

Re-downloads the same PDF to forward to a colleague
Opens the link from a different browser/device
Downloads on month-end recap, then again at quarter-close

Each cached download is zero blob read + zero egress through the storage account. Front Door is ~$0.07/GB out vs blob storage's $0.087/GB — the cache saves both.

9. Status feedback — SignalR + polling fallback

Partners hate watching a spinner without progress. Two paths:

9.1 SignalR push (the happy path)

// Mattrx.Api/Reports/ReportsHub.cs
[Authorize]
public sealed class ReportsHub(IUserContext user) : Hub
{
    public override async Task OnConnectedAsync()
    {
        await Groups.AddToGroupAsync(Context.ConnectionId, $"tenant:{user.TenantId}");
        await base.OnConnectedAsync();
    }
}

// Worker publishes the event via the API's Hub (or a Service Bus topic the API subscribes to)
public sealed class ReportReadyNotifier(IHubContext<ReportsHub> hub) : IRealtimeNotifier
{
    public Task NotifyReportReadyAsync(Guid userId, Guid jobId, Uri url, CancellationToken ct) =>
        hub.Clients.Group($"user:{userId}")
                   .SendAsync("report-ready", new { jobId, url = url.ToString() }, ct);
}

9.2 Polling (the always-works path)

The Angular side polls /api/reports/{jobId} every 3 seconds. SignalR is the fast path; polling is the safety net (some corporate networks block WebSocket upgrades).

// Angular signal-based polling
this.statusInterval = interval(3000).pipe(
  switchMap(() => this.api.getReportStatus(this.jobId)),
  takeWhile(s => s.status !== 'Ready' && s.status !== 'Failed', true),
);

10. Failure modes we hit (and the fixes)

10.1 Chromium memory leak under load

Symptom: Worker RSS climbs from 320 MB → 1.4 GB over 4 hours, then OOM-restart by App Service.

Fix: Recycle browser every 200 pages (PagesPerBrowserBeforeRecycle = 200). After: RSS steady at ~330 MB.

10.2 Same job rendered twice

Symptom: Network blip caused Hangfire to re-enqueue a job that was actually finishing. Two PDFs uploaded; second one had the same content but a different URL.

Fix: IIdempotencyStore.SeenAsync($"report:done:{jobId}") check at the top of GenerateReportJob.RunAsync. The second invocation no-ops.

10.3 The CDN cached an expired SAS

Symptom: Partner clicked a 7-day-old link, Front Door served the cached PDF, but the blob was already deleted by lifecycle policy. PDF rendered fine; analytics dashboard for "report opens" was inflated.

Fix: Set Front Door cache-max-age to match SAS expiry (7 days). When SAS expires, the cache also expires. No more zombie cached PDFs.

10.4 Chart rendering library used `setTimeout` for animations

Symptom: PDFs missing the last chart, or rendered mid-animation. WaitUntil = Networkidle0 wasn't enough.

Fix: Charts publish document.body.dataset.chartsReady = '1' after they finish. page.WaitForSelectorAsync("body[data-charts-ready='1']") blocks until the explicit signal.

10.5 Long-running job exceeded Hangfire timeout

Symptom: A 50-page report took 45 seconds — Hangfire's default 30-minute timeout was fine, but Application Insights traces showed the job as "Failed" because the SignalR send timed out.

Fix: SignalR notification wrapped in fire-and-forget with its own short timeout + logged failures. Notification is best-effort; the report is still ready and the polling endpoint will return it.

11. The full Mattrx production metrics

After 8 months in production:

Metric	Value
Reports generated (total)	~5.2M
Peak load (48-hour month-end)	1.2M
Peak RPS	~50
Avg PDF render time	2.4 s
p95 time-to-PDF (request → ready)	9 s
p99 time-to-PDF	18 s
Avg PDF size	380 KB
Storage at peak (7-day retention)	~456 GB
CDN cache hit rate	38%
Worker instances (autoscale range)	0 → 40
Worker memory (avg per instance)	330 MB
Worker memory (peak per instance)	480 MB
Hangfire queue depth (peak)	~4,800 jobs
Hangfire queue depth (steady)	< 10
Failure rate	0.04%
Manual reprocess (per month)	< 5 jobs
Per-PDF infra cost	$0.0004
Total monthly infra (reports subsystem)	~$1,100
Monthly savings vs old architecture	~$1,300

The architecture pays for itself many times over in cost-of-incidents avoided.

12. The detection cookbook — what to watch in production

DAILY (automated):
  • Hangfire failed-jobs count                     → SRE inbox at > 10
  • Application Insights "Avg time to PDF" metric  → alert at p95 > 30s
  • Worker memory p95                              → alert at > 500 MB
  • Front Door cache hit rate                      → alert at < 25% (something changed)

WEEKLY (manual):
  • Reprocess any Hangfire dead-letter jobs that look transient
  • Check blob storage size against lifecycle policy
  • Review autoscale events — did we leave instances running too long?

MONTHLY (planning):
  • Cost report: per-PDF cost trending
  • Capacity: peak worker count vs cap (40) — bump if hit 80%
  • Browser pool tuning: PagesPerBrowserBeforeRecycle vs RSS growth pattern

13. The mental checklist — before shipping a report-generation system

If any answer is "I'm not sure" — fix it before peak traffic.

14. Honest stuff

Don't render PDFs inline from the API. Period. Even if your load is "small," one slow render blocks a thread; ten do real damage.
PuppeteerSharp is the right choice in 2026 for .NET PDF generation. It's actively maintained, matches the rendering quality of the browser, and the pool pattern is well-known. Alternatives: QuestPDF (faster, no chromium needed, but limited HTML/CSS), iText (commercial license), Aspose (expensive).
Browser pooling is non-negotiable. Launching chromium per PDF is the single most common mistake.
Memory leaks. Recycle. Chromium has known per-page leaks that accumulate. Recycle every 100–300 pages depending on your template complexity.
The CDN is the cheapest performance win. A 38% cache hit rate at zero engineering effort is rare. Use it.
Scale-to-zero works for bursty workloads. Mattrx peaks for 48 hours/month. Paying for idle workers the other 720 hours is waste.
The .NET worker side is the easy part. The HTML template + chart-rendering coordination is what eats time. Budget for it.
Hangfire is good enough for most apps. Switch to Azure Service Bus when queue depth regularly exceeds 50k or you need KEDA/AKS scale-to-zero on a non-App-Service platform.
Watch the dead-letter queue. A failed report is a partner who can't sign off on their month. SRE-pageable.

15. The right mental model

In one line: The pattern is accept fast → enqueue → render in a pool → store → notify → cache at the edge. Every box scales independently. Failures retry without blocking the next request. The user never waits on a thread for the render.

Three habits that prevent 90% of the pain in this guide:

The API never renders. Hardline rule. Render happens on a worker, on a different process, ideally on a different App Service Plan.
Browser pooling on day one. Don't ship "let's launch chromium per call" code, ever. The migration later is painful.
Storage + CDN, not DB. PDFs are big binary blobs. Blob storage + Front Door is the right tier for them. SQL is not.

Apply that, and the next time someone says "we need to generate reports at scale" you'll have a 48-hour architecture instead of a 6-week project.

Handling 1 Million Report Downloads with ASP.NET Core, CQRS, and Headless Browsers in 2026 — Real Architecture, PuppeteerSharp Code, Production Metrics

Generating one PDF report takes 2–4 seconds in a headless Chromium. Easy. Generating 1.2 million PDF reports in 48 hours during end-of-month requests — that's a different problem entirely. Inline generation blocks request threads. A single chromium instance OOMs after 200 pages. The same partner re-downloads the same report 4 times. Workers crash mid-render. Storage costs spiral. Memory leaks compound. By Monday morning, the API is a smoking crater and your CTO is asking why a "simple report" needed a war room.

The architecture that actually works in 2026 is well-trodden but rarely written down end-to-end for .NET: separate the request from the generation (CQRS), bound the workers (a queue + a worker pool), reuse chromium aggressively (browser pooling), scale horizontally on queue depth, cache aggressively (CDN in front of blob storage with SAS-signed URLs), and observe everything (per-step telemetry). Each piece is boring; together they handle 50 reports/sec at peak with ~$1,100/month of Azure spend.

This guide walks through it as we actually built it at Mattrx — a multi-tenant marketing analytics SaaS (Angular 19 + .NET 9, Azure SQL, 110k MAU). Specifically, the Mattrx Reports subsystem that generates campaign-performance PDFs for partner sign-off. Real code (ASP.NET Core + MediatR + Hangfire + PuppeteerSharp + Azure Blob + Front Door), the architectural diagrams, the failure modes that almost killed us, and the production metrics: 1.2M reports/48h at peak, p95 time-to-PDF 9 s, CDN cache hit 38%, per-report cost $0.0004, failure rate 0.04%, $1,300/month saved by sizing the worker pool right.

TL;DR

Component	Choice	Why this not that
Request entrypoint	ASP.NET Core Minimal API	`POST /api/reports/request` returns 202 immediately
Request orchestration	MediatR command (CQRS)	Validates spec, persists job, enqueues — clean seam
Queue	Hangfire (we'd use Azure Service Bus + KEDA for >5M/day)	Visible dashboard, retries, idempotency
Renderer	PuppeteerSharp + headless Chromium	Best fidelity, mature in 2026, fits .NET cleanly
Browser strategy	Pool of long-lived browsers, fresh page per render	Saves 600ms+ vs launching per PDF
Worker tier	Separate Azure App Service Plan, autoscaled on queue depth	Workers don't share resources with the API
Storage	Azure Blob Storage (private container, hot tier)	Cheap, scales infinitely
Delivery	Azure Front Door in front of blob with SAS-signed URLs	Edge cache + secure access
Status feedback	SignalR push + polling fallback	Partner UI feels instant
Telemetry	OpenTelemetry → Application Insights	One trace from click → PDF
Cleanup	Blob lifecycle policy, 7-day TTL on PDFs	Storage costs stay sane

Mattrx Reports — production metrics after the architecture landed (8-month run):

Peak load: ~1.2M reports requested in the 48 hours after month-end close (~7 RPS avg, ~50 RPS peak)
Sustained median load: 80k reports/day
p95 time-to-PDF (request → ready): 9 seconds (queue + render + upload)
p99 time-to-PDF: 18 seconds
Avg PDF render time (PuppeteerSharp): 2.4 seconds
Avg PDF size: 380 KB (campaign report with 12 charts + tables)
CDN cache hit rate (same report re-downloaded): 38%
Failure rate: 0.04% (vs 4.2% before the rewrite)
Worker instances: scales 0 → 40 based on queue depth
Peak chromium memory per worker: 320 MB
Per-PDF infra cost: $0.0004
Total monthly infra (workers + blob + CDN + queue): ~$1,100
App Service tier dropped from P2v3×6 (web+worker mixed) → P1v3×2 web + B2×0-40 burst worker pool — ~$1,300/month saved

The architecture is boring once it works. The wins are not.

1. The architecture — one picture

┌──────────────────────────────────────────────────────────────────────────────┐
│ Partner browser                                                                │
│   Clicks "Download monthly report"                                              │
│   → POST /api/reports/request                                                   │
│   ← 202 Accepted, { jobId }                                                     │
│                                                                                 │
│   Then opens a SignalR connection AND polls /api/reports/{jobId} every 3s       │
└────────────────────────────────┬─────────────────────────────────────────────┘
                                 │
                                 ▼
┌──────────────────────────────────────────────────────────────────────────────┐
│ Mattrx.Api  (ASP.NET Core 9, 2 instances)                                      │
│  1. AuthN + AuthZ + rate limit                                                  │
│  2. Validate ReportSpec via MediatR pipeline                                    │
│  3. Persist ReportJob (Status=Pending) to Azure SQL                             │
│  4. Hangfire.Enqueue<GenerateReportJob>(j => j.RunAsync(jobId, ct))             │
│  5. Return 202 with jobId                                                       │
└────────────────────────────────┬─────────────────────────────────────────────┘
                                 │
                                 ▼
┌──────────────────────────────────────────────────────────────────────────────┐
│ Hangfire queue (backed by SQL Server jobs table)                               │
│   • dashboard at /admin/hangfire                                                │
│   • retries with exponential backoff                                            │
│   • idempotency keyed on jobId                                                  │
└────────────────────────────────┬─────────────────────────────────────────────┘
                                 │ dequeue
                                 ▼
┌──────────────────────────────────────────────────────────────────────────────┐
│ Mattrx.Workers  (separate App Service Plan, scales 0–40 instances)            │
│                                                                                 │
│  ReportRenderer (per worker, long-lived)                                        │
│   ├─ BrowserPool: 2 chromium instances, recycled every 200 PDFs                 │
│   ├─ Build HTML from Razor template (campaign data from DB)                     │
│   ├─ Open new page, set viewport, navigate to file://template.html              │
│   ├─ page.PdfAsync() → byte[]                                                   │
│   ├─ Upload to Azure Blob (private container, content-type=application/pdf)    │
│   ├─ Issue SAS-signed URL (7-day expiry)                                        │
│   ├─ Update ReportJob.Status = Ready, Url = sasUrl                               │
│   └─ SignalR.NotifyAsync(tenantId, jobId, Ready)                                │
└────────┬────────────────┬───────────────────────────┬──────────────────────────┘
         │                │                            │
         ▼                ▼                            ▼
┌─────────────────┐  ┌──────────────────┐  ┌─────────────────────────┐
│ Azure SQL        │  │ Azure Blob       │  │ Azure Front Door         │
│  ReportJobs      │  │  Storage         │  │  in front of Blob        │
│  Hangfire tables │  │  reports/$tenant/│  │  • Caches PDFs at edge   │
│                  │  │   $jobId.pdf     │  │  • Re-validates per SAS  │
│                  │  │  Lifecycle: 7d   │  │  • 38% hit rate          │
└─────────────────┘  └──────────────────┘  └─────────────────────────┘

Out-of-band:
  • OpenTelemetry traces every step: request → enqueue → render → upload → notify
  • Hangfire retries failures with exponential backoff (5 attempts)
  • Application Insights alerts on queue depth > 500 or failure rate > 1%

Every section below adds one box to that diagram.

2. Mattrx — the running example

Mattrx's marketing partners request three kinds of reports:

Report	Frequency	Size	Render time
Campaign performance (PDF)	Monthly per campaign per partner	~380 KB	2.4 s
Revenue summary (PDF)	Monthly per partner	~140 KB	1.1 s
Custom export (XLSX)	On-demand	~1.2 MB	8 s (via ClosedXML)

The end-of-month load is the killer: between the 28th and the 2nd of each month, partners pull ~80% of the month's volume — ~1.2M PDFs in 48 hours. The rest of the month is quiet.

Three failure modes from the original (synchronous) version:

Web tier OOMs — chromium memory grew on the same App Service as the API, killing both.
Request thread blocked 15+ seconds — Kestrel ran out of threads, customers saw 503s.
Same partner clicked Download 4 times — we generated 4 identical PDFs because there was no idempotency.

The architecture below fixes all three.

3. Why CQRS for this (and what "CQRS" actually means here)

CQRS in this context = the write side (request → enqueue) and the read side (status check, download) are different operations against different infrastructure. The request handler does NOT do the rendering.

That single split unlocks everything else:

The API can return 202 in <50ms regardless of render time.
Workers scale independently of API instances.
Retries on the worker don't block the request.
The same RequestReport command is the same whether one user or a million call it.

3.1 The MediatR command

// Mattrx.Application/Reports/Commands/RequestReportCommand.cs
public sealed record RequestReportCommand(
    Guid TenantId,
    Guid CampaignId,
    DateOnly From,
    DateOnly To,
    ReportFormat Format)
    : IRequest<Result<RequestReportResponse>>;

public sealed record RequestReportResponse(Guid JobId);

public sealed class RequestReportValidator : AbstractValidator<RequestReportCommand>
{
    public RequestReportValidator()
    {
        RuleFor(x => x.TenantId).NotEmpty();
        RuleFor(x => x.CampaignId).NotEmpty();
        RuleFor(x => x.From).LessThan(x => x.To);
        RuleFor(x => x.To).LessThanOrEqualTo(DateOnly.FromDateTime(DateTime.UtcNow));
        RuleFor(x => x).Must(x => (x.To.ToDateTime(default) - x.From.ToDateTime(default)).TotalDays <= 366)
            .WithMessage("Report period cannot exceed 1 year.");
    }
}

public sealed class RequestReportHandler(
    IReportJobRepository repo,
    IBackgroundJobClient jobs,
    ICurrentUser currentUser,
    IClock clock,
    IIdempotencyStore idempotency) : IRequestHandler<RequestReportCommand, Result<RequestReportResponse>>
{
    public async Task<Result<RequestReportResponse>> Handle(RequestReportCommand cmd, CancellationToken ct)
    {
        // Idempotency: same (tenant, campaign, from, to, format) within 60s returns existing jobId
        var idempotencyKey = $"report:{cmd.TenantId}:{cmd.CampaignId}:{cmd.From:yyyyMMdd}:{cmd.To:yyyyMMdd}:{cmd.Format}";
        if (await idempotency.TryGetAsync<Guid>(idempotencyKey, ct) is { } existingJobId)
            return Result.Ok(new RequestReportResponse(existingJobId));

        var job = ReportJob.Create(
            tenantId: cmd.TenantId,
            requestedBy: currentUser.UserId,
            campaignId: cmd.CampaignId,
            from: cmd.From, to: cmd.To,
            format: cmd.Format,
            clock: clock);

        await repo.AddAsync(job, ct);

        // Enqueue. The handler returns IMMEDIATELY.
        // GenerateReportJob runs in the worker tier.
        jobs.Enqueue<GenerateReportJob>(j => j.RunAsync(job.Id, CancellationToken.None));

        await idempotency.SetAsync(idempotencyKey, job.Id, TimeSpan.FromMinutes(1), ct);

        return Result.Ok(new RequestReportResponse(job.Id));
    }
}

3.2 The controller (Minimal API)

// Mattrx.Api/Reports/ReportEndpoints.cs
app.MapPost("/api/reports/request",
    async (RequestReportCommand cmd, ISender sender, CancellationToken ct) =>
    {
        var result = await sender.Send(cmd, ct);
        return result.Match(
            ok    => Results.Accepted($"/api/reports/{ok.JobId}", ok),
            error => Results.BadRequest(error.ToProblemDetails()));
    })
.RequireAuthorization()
.WithName("RequestReport")
.WithOpenApi();

app.MapGet("/api/reports/{jobId:guid}",
    async (Guid jobId, ISender sender, CancellationToken ct) =>
        await sender.Send(new GetReportStatusQuery(jobId), ct))
.RequireAuthorization()
.WithName("GetReportStatus");

3.3 The status query (read side)

// Mattrx.Application/Reports/Queries/GetReportStatusQuery.cs
public sealed record GetReportStatusQuery(Guid JobId) : IRequest<ReportStatusDto>;

public sealed class GetReportStatusHandler(IReadOnlyDb db, ICurrentUser currentUser)
    : IRequestHandler<GetReportStatusQuery, ReportStatusDto>
{
    public async Task<ReportStatusDto> Handle(GetReportStatusQuery q, CancellationToken ct)
    {
        // Project directly to DTO — no entity hydration
        return await db.ReportJobs
            .Where(j => j.Id == q.JobId && j.TenantId == currentUser.TenantId)
            .Select(j => new ReportStatusDto
            {
                JobId = j.Id,
                Status = j.Status,
                ProgressPct = j.ProgressPct,
                DownloadUrl = j.SignedUrl,           // null until Status = Ready
                ExpiresAt = j.SignedUrlExpiresAt,
                ErrorMessage = j.ErrorMessage,
            })
            .AsNoTracking()
            .FirstAsync(ct);
    }
}

3.4 Why each piece earns its keep

Validator runs in a MediatR pipeline behavior — catches bad date ranges before any work happens.
Idempotency prevents 4× re-generation when a partner double-clicks. 60-second window covers UI rage-clicks.
IBackgroundJobClient (Hangfire) is the seam between API and worker. The API process never renders.
Tenant scoping on the read query — partner A cannot see partner B's reports.

4. The queue — Hangfire (and when to switch)

// Program.cs — API tier
builder.Services.AddHangfire(config => config
    .UseSqlServerStorage(connectionString, new SqlServerStorageOptions
    {
        QueuePollInterval = TimeSpan.FromSeconds(2),
        PrepareSchemaIfNecessary = true,
        UseRecommendedIsolationLevel = true,
        DisableGlobalLocks = true,
    }));

// Only the worker tier processes jobs — set processing count = 0 here
builder.Services.AddHangfireServer(o => { o.WorkerCount = 0; });

// Program.cs — worker tier
builder.Services.AddHangfire(config => config
    .UseSqlServerStorage(connectionString, /* same options */));

builder.Services.AddHangfireServer(o =>
{
    o.Queues = new[] { "reports" };
    o.WorkerCount = Environment.ProcessorCount * 2;   // tuned for chromium concurrency
    o.ShutdownTimeout = TimeSpan.FromMinutes(2);      // let in-flight PDFs finish
});

Why Hangfire (for now)

Visible dashboard — /admin/hangfire shows the queue depth, failed jobs, retry attempts. SREs love it.
Persistent in SQL — no separate infrastructure. Survives restarts.
Automatic retries — exponential backoff out of the box.
Free at our scale.

When you should switch to Azure Service Bus + KEDA

Signal	Switch to Service Bus
Queue depth regularly > 50,000	Yes, Hangfire SQL polling slows
Need to scale workers to 0 (KEDA on AKS)	Yes
Need cross-region replication	Yes
Need dead-letter queue + deferred messages	Service Bus is built for it
Need ordered delivery (FIFO)	Service Bus sessions

We've stayed on Hangfire because Mattrx peaks at ~5,000 messages in queue at end-of-month. If we doubled, we'd migrate.

5. The PDF renderer — PuppeteerSharp with browser pooling

This is where the architecture earns its win or loses it. Naive PuppeteerSharp launches Chromium per PDF — that's a 600ms cold-start tax per render. Pooling is non-negotiable.

5.1 Install + first launch

dotnet add package PuppeteerSharp

// Mattrx.Workers/Reports/Browser/ChromiumBootstrap.cs
public static class ChromiumBootstrap
{
    public static async Task EnsureBrowserDownloadedAsync()
    {
        var fetcher = new BrowserFetcher();
        await fetcher.DownloadAsync();   // ~120 MB, idempotent — done once per container
    }
}

// Call once at worker startup:
await ChromiumBootstrap.EnsureBrowserDownloadedAsync();

For Docker, install Chromium dependencies in the worker base image — see §7.

5.2 The browser pool — the heart of the optimization

// Mattrx.Workers/Reports/Browser/BrowserPool.cs
public sealed class BrowserPool : IAsyncDisposable
{
    private readonly int _maxBrowsers;
    private readonly int _pagesPerBrowserBeforeRecycle;
    private readonly Channel<PooledBrowser> _available;
    private readonly object _lock = new();
    private int _totalCreated;
    private readonly ILogger<BrowserPool> _log;

    public BrowserPool(IOptions<BrowserPoolOptions> opts, ILogger<BrowserPool> log)
    {
        _maxBrowsers = opts.Value.MaxBrowsers;
        _pagesPerBrowserBeforeRecycle = opts.Value.PagesPerBrowserBeforeRecycle;
        _available = Channel.CreateBounded<PooledBrowser>(_maxBrowsers);
        _log = log;
    }

    public async Task<IBrowserLease> AcquireAsync(CancellationToken ct)
    {
        // Try to dequeue an existing browser
        if (_available.Reader.TryRead(out var existing))
        {
            if (existing.PagesRendered >= _pagesPerBrowserBeforeRecycle)
            {
                _log.LogInformation("Recycling browser after {Count} pages", existing.PagesRendered);
                await existing.DisposeAsync();
                lock (_lock) { _totalCreated--; }
            }
            else
            {
                return new BrowserLease(existing, this);
            }
        }

        // Create a new browser if under cap
        bool shouldCreate;
        lock (_lock)
        {
            shouldCreate = _totalCreated < _maxBrowsers;
            if (shouldCreate) _totalCreated++;
        }

        if (shouldCreate)
        {
            var browser = await Puppeteer.LaunchAsync(new LaunchOptions
            {
                Headless = true,
                Args = new[]
                {
                    "--no-sandbox",                  // required in container
                    "--disable-dev-shm-usage",        // /dev/shm is small in containers
                    "--disable-gpu",                  // headless renders without GPU
                    "--disable-extensions",
                    "--disable-background-networking",
                    "--disable-default-apps",
                    "--disable-sync",
                    "--mute-audio",
                    "--no-first-run",
                    "--hide-scrollbars",
                },
            });
            var pooled = new PooledBrowser(browser);
            return new BrowserLease(pooled, this);
        }

        // Pool is full — wait for someone to return one
        var waited = await _available.Reader.ReadAsync(ct);
        return new BrowserLease(waited, this);
    }

    public void Return(PooledBrowser browser)
    {
        if (browser.IsDisposed || !browser.Inner.IsConnected)
        {
            lock (_lock) { _totalCreated--; }
            return;
        }
        _available.Writer.TryWrite(browser);
    }

    public async ValueTask DisposeAsync()
    {
        _available.Writer.TryComplete();
        await foreach (var b in _available.Reader.ReadAllAsync())
        {
            await b.DisposeAsync();
        }
    }
}

public sealed class PooledBrowser : IAsyncDisposable
{
    public IBrowser Inner { get; }
    public int PagesRendered { get; private set; }
    public bool IsDisposed { get; private set; }
    public PooledBrowser(IBrowser inner) { Inner = inner; }
    public void IncrementPageCount() => PagesRendered++;
    public async ValueTask DisposeAsync()
    {
        if (IsDisposed) return;
        IsDisposed = true;
        try { await Inner.CloseAsync(); await Inner.DisposeAsync(); } catch { /* swallow */ }
    }
}

public interface IBrowserLease : IAsyncDisposable { IBrowser Browser { get; } void NotePageRendered(); }

internal sealed class BrowserLease(PooledBrowser pooled, BrowserPool pool) : IBrowserLease
{
    public IBrowser Browser => pooled.Inner;
    public void NotePageRendered() => pooled.IncrementPageCount();
    public ValueTask DisposeAsync() { pool.Return(pooled); return ValueTask.CompletedTask; }
}

public sealed class BrowserPoolOptions
{
    public int MaxBrowsers { get; init; } = 2;
    public int PagesPerBrowserBeforeRecycle { get; init; } = 200;
}

5.3 The renderer

// Mattrx.Workers/Reports/PdfReportRenderer.cs
public sealed class PdfReportRenderer(BrowserPool pool, IReportTemplateBuilder templates, ILogger<PdfReportRenderer> log)
{
    public async Task<byte[]> RenderAsync(ReportData data, CancellationToken ct)
    {
        // 1. Build the HTML server-side (Razor template + data)
        var html = await templates.BuildAsync(data, ct);

        // 2. Acquire a chromium from the pool
        await using var lease = await pool.AcquireAsync(ct);
        var browser = lease.Browser;

        // 3. Open a fresh page (cheap; ~30ms) — DON'T reuse pages across renders
        await using var page = await browser.NewPageAsync();

        await page.SetViewportAsync(new ViewPortOptions { Width = 1240, Height = 1754 });   // A4 @ 150dpi
        await page.SetCacheEnabledAsync(false);   // no cross-render cache contamination

        // 4. Load the HTML inline — no temp file roundtrip
        await page.SetContentAsync(html, new NavigationOptions
        {
            WaitUntil = new[] { WaitUntilNavigation.Networkidle0 },
            Timeout = 20_000,
        });

        // 5. Wait for charts to finish rendering (custom signal from our JS)
        await page.WaitForSelectorAsync("body[data-charts-ready='1']", new WaitForSelectorOptions { Timeout = 15_000 });

        // 6. Emit PDF
        var pdf = await page.PdfDataAsync(new PdfOptions
        {
            Format = PaperFormat.A4,
            PrintBackground = true,
            MarginOptions = new MarginOptions { Top = "12mm", Bottom = "12mm", Left = "12mm", Right = "12mm" },
            DisplayHeaderFooter = true,
            HeaderTemplate = "<div style='font-size:10px;width:100%;text-align:center'>Mattrx — Campaign Report</div>",
            FooterTemplate = "<div style='font-size:9px;width:100%;text-align:center'>Page <span class='pageNumber'></span> of <span class='totalPages'></span></div>",
        });

        lease.NotePageRendered();
        return pdf;
    }
}

5.4 The optimizations that matter, ranked

Optimization	Saved per PDF	Why
Browser pooling (don't launch per render)	~600 ms	Chromium cold-start is expensive
`WaitUntil = Networkidle0` + explicit `body[data-charts-ready='1']` signal	reliability win	Otherwise PDFs render mid-chart-animation
Recycle browser every 200 pages	prevents OOM	Chromium leaks memory across pages
`SetContent` instead of file:// navigation	~80 ms	One less network round-trip
Pre-render charts as static SVG (not JS-animated)	~400 ms	Avoid `setTimeout` delays from animation libs
`--disable-extensions`, `--no-sandbox`, `--mute-audio` flags	~150 ms	Skip Chromium subsystems we never use
Disable image loading if PDF only needs text + charts	varies	`page.SetRequestInterceptionAsync(true)`

5.5 Mattrx metric

Browser pooling alone took mean PDF render time from 3.1 s → 2.4 s, and let one worker render ~280 PDFs/hour instead of ~140.

6. The worker job — wiring it all together

// Mattrx.Workers/Reports/GenerateReportJob.cs
public sealed class GenerateReportJob(
    IReportJobRepository repo,
    IReportDataProvider data,
    PdfReportRenderer renderer,
    IBlobStorage blob,
    IRealtimeNotifier notifier,
    IIdempotencyStore idempotency,
    ILogger<GenerateReportJob> log)
{
    // Hangfire calls this. CT is provided by Hangfire's shutdown.
    public async Task RunAsync(Guid jobId, CancellationToken ct)
    {
        // Idempotency: if we already generated this job, just return
        if (await idempotency.SeenAsync($"report:done:{jobId}", ct)) return;

        var job = await repo.GetAsync(jobId, ct);
        if (job is null || job.Status == ReportJobStatus.Ready) return;

        try
        {
            await repo.UpdateStatusAsync(jobId, ReportJobStatus.Running, progressPct: 5, ct);

            // 1. Pull data
            var reportData = await data.LoadAsync(job.TenantId, job.CampaignId, job.From, job.To, ct);
            await repo.UpdateStatusAsync(jobId, ReportJobStatus.Running, progressPct: 30, ct);

            // 2. Render PDF
            var pdf = await renderer.RenderAsync(reportData, ct);
            await repo.UpdateStatusAsync(jobId, ReportJobStatus.Running, progressPct: 80, ct);

            // 3. Upload to blob
            var blobPath = $"{job.TenantId}/{job.Id}.pdf";
            await blob.UploadAsync("reports", blobPath, pdf, "application/pdf", ct);

            // 4. Sign URL valid 7 days
            var signedUrl = blob.GenerateSasUri("reports", blobPath, TimeSpan.FromDays(7));

            // 5. Finalize
            await repo.MarkReadyAsync(jobId, signedUrl, expiresAt: DateTimeOffset.UtcNow.AddDays(7), ct);
            await idempotency.MarkAsync($"report:done:{jobId}", ct);

            // 6. Notify the user (SignalR + email fallback handled elsewhere)
            await notifier.NotifyReportReadyAsync(job.RequestedBy, jobId, signedUrl, ct);
        }
        catch (OperationCanceledException) when (ct.IsCancellationRequested)
        {
            // Worker shutting down. Hangfire will retry.
            throw;
        }
        catch (Exception ex)
        {
            log.LogError(ex, "Report generation failed for job {JobId}", jobId);
            await repo.MarkFailedAsync(jobId, ex.Message, ct);
            throw;   // let Hangfire retry up to N times
        }
    }
}

Hangfire retry policy

// Program.cs — worker tier
[AutomaticRetry(Attempts = 5, DelaysInSeconds = new[] { 15, 60, 240, 900, 3600 })]
public sealed class GenerateReportJob { /* ... */ }

7. Horizontal scaling — the worker tier autoscale

7.1 The Azure App Service Plan layout

Web tier:    P1v3 × 2 instances (no autoscale — predictable)
Worker tier: B2 × 0–40 instances (autoscale on Hangfire queue depth)

Custom metric: hangfire_pending_jobs (published from worker to Application Insights every 30s)

Autoscale rule:
  Scale OUT  +5 instances   when avg(hangfire_pending_jobs, 5min) > 200
  Scale OUT  +10 instances  when avg(hangfire_pending_jobs, 5min) > 1000
  Scale IN   -2 instances   when avg(hangfire_pending_jobs, 5min) < 50 for 15 min
  Cooldown:  3 min between scale events
  Min: 0 instances (scales to zero overnight)
  Max: 40 instances

7.2 Publishing the custom metric

// Mattrx.Workers/Telemetry/HangfireQueueDepthPublisher.cs
public sealed class HangfireQueueDepthPublisher(
    JobStorage storage,
    TelemetryClient telemetry) : BackgroundService
{
    protected override async Task ExecuteAsync(CancellationToken ct)
    {
        using var periodic = new PeriodicTimer(TimeSpan.FromSeconds(30));
        while (await periodic.WaitForNextTickAsync(ct))
        {
            try
            {
                var monitor = storage.GetMonitoringApi();
                var enqueued = monitor.EnqueuedCount("reports");
                telemetry.GetMetric("hangfire_pending_jobs").TrackValue(enqueued);
            }
            catch { /* swallow */ }
        }
    }
}

The autoscale rule reads this custom metric and adjusts the worker pool.

7.3 Why scale-to-zero matters

Mattrx's report load is bursty + predictable — most months are quiet, the last 48 hours are insane. Scaling to zero overnight saves real money:

Without scale-to-zero (constant 8 workers):
  8 × $0.16/hr × 24h × 30d = ~$921/month

With scale-to-zero (0–40 burst, avg 4 active hours/day):
  4 × $0.16/hr × 4h × 30d + burst (~$240) = ~$317/month

Saved: ~$600/month on the worker tier alone.

7.4 The container image

# Dockerfile — worker image
FROM mcr.microsoft.com/dotnet/aspnet:9.0 AS base
WORKDIR /app

# Chromium dependencies (PuppeteerSharp needs these in the container)
RUN apt-get update && apt-get install -y --no-install-recommends \
    libnss3 libatk1.0-0 libatk-bridge2.0-0 libcups2 libdrm2 libgbm1 \
    libxkbcommon0 libxcomposite1 libxdamage1 libxfixes3 libxrandr2 \
    libgtk-3-0 libpango-1.0-0 libcairo2 libasound2 fonts-liberation \
    fonts-noto-color-emoji \
  && rm -rf /var/lib/apt/lists/*

# Increase /dev/shm (PuppeteerSharp uses it heavily — small default OOMs chromium)
# In Azure App Service for Containers, set this via WEBSITES_PORT and a custom startup script

FROM base AS final
COPY ./publish .
ENTRYPOINT ["dotnet", "Mattrx.Workers.dll"]

8. Blob storage + Azure Front Door

8.1 Container layout

reports/
├── {tenantId1}/
│   ├── {jobId1}.pdf
│   ├── {jobId2}.pdf
│   └── ...
├── {tenantId2}/
│   └── ...

Tenant-prefixed paths make per-tenant blob lifecycle policies easy and prevent accidental cross-tenant data fetches.

8.2 Lifecycle policy (auto-delete 7 days after upload)

{
  "rules": [
    {
      "name": "DeleteOldReports",
      "enabled": true,
      "type": "Lifecycle",
      "definition": {
        "filters": { "blobTypes": ["blockBlob"], "prefixMatch": ["reports/"] },
        "actions": { "baseBlob": { "delete": { "daysAfterModificationGreaterThan": 7 } } }
      }
    }
  ]
}

Storage costs stay flat — old reports just disappear. If a partner needs to keep one, they download it within 7 days. Real cost: ~$8/month at 1.2M peak × 380 KB × 7-day retention.

8.3 SAS-signed URLs

// Mattrx.Infrastructure/Storage/AzureBlobStorage.cs
public sealed class AzureBlobStorage(BlobServiceClient client) : IBlobStorage
{
    public async Task UploadAsync(string container, string path, byte[] content, string contentType, CancellationToken ct)
    {
        var blob = client.GetBlobContainerClient(container).GetBlobClient(path);
        await blob.UploadAsync(new BinaryData(content), new BlobUploadOptions
        {
            HttpHeaders = new BlobHttpHeaders
            {
                ContentType = contentType,
                CacheControl = "public, max-age=604800, immutable",
            },
        }, ct);
    }

    public Uri GenerateSasUri(string container, string path, TimeSpan validFor)
    {
        var blob = client.GetBlobContainerClient(container).GetBlobClient(path);
        var sas = new BlobSasBuilder
        {
            BlobContainerName = container,
            BlobName = path,
            Resource = "b",
            ExpiresOn = DateTimeOffset.UtcNow.Add(validFor),
        };
        sas.SetPermissions(BlobSasPermissions.Read);
        return blob.GenerateSasUri(sas);
    }
}

8.4 Front Door in front of Blob (the CDN win)

Azure Front Door is configured with:

Origin: the blob storage account (private — only Front Door can reach it via managed identity)
Cache: enabled, 7-day TTL, vary by URL (SAS signature is part of the URL)
Rule: rewrite https://reports.mattrx.io/{tenant}/{jobId}.pdf?sas=... → blob URL

Partner browser → reports.mattrx.io/abc-tenant/xyz-job.pdf?sv=...
                      │
                      ▼
                Azure Front Door
                      │ (cache hit? serve from edge — 38% of requests)
                      │
                      ▼ (cache miss)
                Azure Blob Storage

Cache hit rate at Mattrx: 38%. Why so high? Because the same partner often:

Re-downloads the same PDF to forward to a colleague
Opens the link from a different browser/device
Downloads on month-end recap, then again at quarter-close

Each cached download is zero blob read + zero egress through the storage account. Front Door is ~$0.07/GB out vs blob storage's $0.087/GB — the cache saves both.

9. Status feedback — SignalR + polling fallback

Partners hate watching a spinner without progress. Two paths:

9.1 SignalR push (the happy path)

// Mattrx.Api/Reports/ReportsHub.cs
[Authorize]
public sealed class ReportsHub(IUserContext user) : Hub
{
    public override async Task OnConnectedAsync()
    {
        await Groups.AddToGroupAsync(Context.ConnectionId, $"tenant:{user.TenantId}");
        await base.OnConnectedAsync();
    }
}

// Worker publishes the event via the API's Hub (or a Service Bus topic the API subscribes to)
public sealed class ReportReadyNotifier(IHubContext<ReportsHub> hub) : IRealtimeNotifier
{
    public Task NotifyReportReadyAsync(Guid userId, Guid jobId, Uri url, CancellationToken ct) =>
        hub.Clients.Group($"user:{userId}")
                   .SendAsync("report-ready", new { jobId, url = url.ToString() }, ct);
}

9.2 Polling (the always-works path)

The Angular side polls /api/reports/{jobId} every 3 seconds. SignalR is the fast path; polling is the safety net (some corporate networks block WebSocket upgrades).

// Angular signal-based polling
this.statusInterval = interval(3000).pipe(
  switchMap(() => this.api.getReportStatus(this.jobId)),
  takeWhile(s => s.status !== 'Ready' && s.status !== 'Failed', true),
);

Metric	Value
Reports generated (total)	~5.2M
Peak load (48-hour month-end)	1.2M
Peak RPS	~50
Avg PDF render time	2.4 s
p95 time-to-PDF (request → ready)	9 s
p99 time-to-PDF	18 s
Avg PDF size	380 KB
Storage at peak (7-day retention)	~456 GB
CDN cache hit rate	38%
Worker instances (autoscale range)	0 → 40
Worker memory (avg per instance)	330 MB
Worker memory (peak per instance)	480 MB
Hangfire queue depth (peak)	~4,800 jobs
Hangfire queue depth (steady)	< 10
Failure rate	0.04%
Manual reprocess (per month)	< 5 jobs
Per-PDF infra cost	$0.0004
Total monthly infra (reports subsystem)	~$1,100
Monthly savings vs old architecture	~$1,300

The architecture pays for itself many times over in cost-of-incidents avoided.

12. The detection cookbook — what to watch in production

DAILY (automated):
  • Hangfire failed-jobs count                     → SRE inbox at > 10
  • Application Insights "Avg time to PDF" metric  → alert at p95 > 30s
  • Worker memory p95                              → alert at > 500 MB
  • Front Door cache hit rate                      → alert at < 25% (something changed)

WEEKLY (manual):
  • Reprocess any Hangfire dead-letter jobs that look transient
  • Check blob storage size against lifecycle policy
  • Review autoscale events — did we leave instances running too long?

MONTHLY (planning):
  • Cost report: per-PDF cost trending
  • Capacity: peak worker count vs cap (40) — bump if hit 80%
  • Browser pool tuning: PagesPerBrowserBeforeRecycle vs RSS growth pattern

13. The mental checklist — before shipping a report-generation system

If any answer is "I'm not sure" — fix it before peak traffic.

14. Honest stuff

Don't render PDFs inline from the API. Period. Even if your load is "small," one slow render blocks a thread; ten do real damage.
PuppeteerSharp is the right choice in 2026 for .NET PDF generation. It's actively maintained, matches the rendering quality of the browser, and the pool pattern is well-known. Alternatives: QuestPDF (faster, no chromium needed, but limited HTML/CSS), iText (commercial license), Aspose (expensive).
Browser pooling is non-negotiable. Launching chromium per PDF is the single most common mistake.
Memory leaks. Recycle. Chromium has known per-page leaks that accumulate. Recycle every 100–300 pages depending on your template complexity.
The CDN is the cheapest performance win. A 38% cache hit rate at zero engineering effort is rare. Use it.
Scale-to-zero works for bursty workloads. Mattrx peaks for 48 hours/month. Paying for idle workers the other 720 hours is waste.
The .NET worker side is the easy part. The HTML template + chart-rendering coordination is what eats time. Budget for it.
Hangfire is good enough for most apps. Switch to Azure Service Bus when queue depth regularly exceeds 50k or you need KEDA/AKS scale-to-zero on a non-App-Service platform.
Watch the dead-letter queue. A failed report is a partner who can't sign off on their month. SRE-pageable.

15. The right mental model

Three habits that prevent 90% of the pain in this guide:

The API never renders. Hardline rule. Render happens on a worker, on a different process, ideally on a different App Service Plan.
Browser pooling on day one. Don't ship "let's launch chromium per call" code, ever. The migration later is painful.
Storage + CDN, not DB. PDFs are big binary blobs. Blob storage + Front Door is the right tier for them. SQL is not.

Apply that, and the next time someone says "we need to generate reports at scale" you'll have a 48-hour architecture instead of a 6-week project.

Get the next issue

Keep reading

Get the next issue

Keep reading