Microservices Interview Questions for Senior Developers — Boundaries, Saga & Consistency, Resilience & Observability (Deep Answers)
30 senior microservices interview questions with deep answers and real code: service boundaries, saga & eventual consistency, resilience, and observability.
- Author
- Randhir Jassal
- Published
- Reading time
- 21 min read
- Views
- 5 views
Junior microservices interviews ask "what is a microservice." Senior interviews ask why your services share a database (so they're really one service that can't deploy independently), what happens when the payment call times out halfway through an order, and how you debug a request that touched six services at 3am. Microservices trade in-process simplicity for distributed-systems hard problems — and seniority is knowing which problems you just bought and how to pay for them. This guide is 30 questions across boundaries & decomposition, data & consistency, resilience & communication, and observability & production.
Each answer is a short answer you could say in one breath, a deep dive into the mechanism (with code), and the senior signal the interviewer is listening for. Every example is from Mattrx — a real multi-tenant marketing-analytics platform on .NET 9 whose services (API, a Kafka ingestion service, a Service-Bus report service, a webhook-delivery service, and a Python AI service) talk over Kafka, Azure Service Bus, and Event Grid. No Foo and Bar.
How to use this guide
Read the deep dive even when the short answer is obvious — seniority is in the why, the failure mode, and the trade-off. The blocks build on each other. If you can explain each code comment out loud, you're ready.
Block A — service boundaries & decomposition
1. When should you NOT use microservices?
Short answer: Almost always at the start. Microservices solve organizational and scaling problems you don't have yet; for a new product or a small team, a well-structured modular monolith ships faster and is far easier to change. Split only when a module has a real, independent scaling or deployment need.
Deep dive: The biggest failure mode is going distributed too early and getting a distributed monolith — all the operational pain of microservices with none of the independence.
Right order: modular monolith -> extract a service when a module has a
DISTINCT scaling / deploy / team-ownership need (Mattrx extracted the
Kafka ingestion service first because it scales on event volume, not API traffic).
Senior signal: You default to a modular monolith and can name the specific trigger (independent scaling/deploy/ownership) that justifies extracting a service.
2. How do you decide service boundaries?
Short answer: By business capability / DDD bounded context, not by technical layer. A service owns a cohesive slice of the business (billing, ingestion, reporting) end-to-end — its data, logic, and API — so most changes stay inside one service.
Deep dive: Boundaries drawn by layer ("a data service, a logic service, a UI service") force every feature to cross all of them — the opposite of independent. Mattrx's services map to capabilities: Ingestion, Campaigns, Reports, Notifications.
BAD (by layer): [ UI svc ] -> [ business svc ] -> [ data svc ] (every change spans all 3)
GOOD (by domain): [ Campaigns ] [ Ingestion ] [ Reports ] (a change stays in one)
Senior signal: You cite DDD bounded contexts / business capabilities and reject layer-based splits because they maximize cross-service coupling.
3. Database-per-service — why, and what's the anti-pattern?
Short answer: Each service owns its own database and no other service touches it directly. The classic anti-pattern is a shared database — the moment two services read/write the same tables, they're coupled at the schema and can't deploy or scale independently.
Deep dive: A shared DB turns a schema change into a cross-team coordination event and re-creates the monolith's coupling invisibly.
ANTI-PATTERN: [ svc A ] --\
> [ ONE shared DB ] (schema change breaks both; no isolation)
[ svc B ] --/
CORRECT: [ svc A -> DB-A ] [ svc B -> DB-B ] (talk via APIs/events only)
Senior signal: You treat "another service reads my tables" as a boundary violation, not a shortcut, and enforce database-per-service.
4. If each service owns its data, how does a service get data it doesn't own?
Short answer: Three options: API composition (call the owning service synchronously), a local read replica kept up to date via events (the owner publishes changes; you store a copy of just what you need), or a CQRS read model that joins across services asynchronously. Never reach into another service's DB.
Deep dive: Mattrx's Reports service keeps a small local copy of campaign names (updated from campaign.updated events) so it doesn't call the Campaigns service on every render.
// Reports service consumes Campaigns' events and maintains a local read copy.
public async Task Consume(ConsumeContext<CampaignRenamed> ctx)
{
await _localCampaigns.UpsertNameAsync(ctx.Message.CampaignId, ctx.Message.Name, ctx.CancellationToken);
}
Senior signal: You reach for event-driven data replication for hot paths and API composition for cold ones — and never a cross-service DB read.
5. What is a distributed monolith, and how do you avoid it?
Short answer: Services that must be deployed together, call each other synchronously in long chains, and often share a DB — you paid the network/ops cost of microservices but kept the coupling of a monolith. Avoid it with clear boundaries, async communication, and independent deployability.
Deep dive: The tell is "we have to release these three services in a specific order." If you can't deploy one service without the others, they're one service wearing three uniforms.
Senior signal: You use "can I deploy this service alone?" as the litmus test, and flag synchronous call-chains + shared DBs as distributed-monolith smells.
6. Synchronous vs asynchronous communication — when each?
Short answer: Synchronous (HTTP/gRPC) when the caller needs an immediate answer to proceed (a query, a validation). Asynchronous (events/messages) for everything else — it decouples services, absorbs spikes, and survives the callee being down. Prefer async; it's what makes services independent.
Deep dive: Every synchronous hop adds its latency and its failure probability to yours. Mattrx ingests events async (Kafka) and only calls synchronously where a user is waiting on the answer.
Senior signal: You default to async for decoupling and reserve sync for "the caller literally can't continue without the response," counting each sync hop as added latency + failure surface.
7. How do you query data that spans multiple services?
Short answer: API composition (a gateway/BFF calls each service and stitches the result) for simple cases, or a CQRS read model — a service that subscribes to events from several services and maintains a pre-joined, query-optimized view — for complex or high-traffic reads.
Deep dive: Mattrx's dashboard reads a pre-aggregated read model (fed by ingestion events) instead of fanning out to every service per request.
API composition: gateway -> [svc A] + [svc B] + [svc C] -> merge (simple, N calls/request)
CQRS read model: services --events--> [ read model DB ] <- one fast query (complex/hot reads)
Senior signal: You match the pattern to read complexity/traffic and know a cross-service JOIN is a smell that usually means a missing read model.
8. How do you evolve an API/event contract without breaking consumers?
Short answer: Only make backward-compatible changes (add optional fields, never remove/rename in place), version when you must break, and use consumer-driven contract tests so a producer change that breaks a consumer fails CI, not production.
Deep dive: Because services deploy independently, at any moment old and new versions run together — every change must be compatible across that window (see expand/contract, Q27).
Senior signal: You treat every contract change as "old and new run simultaneously," add-only by default, and back it with contract tests.
Block B — data consistency & messaging
9. Why can't you use a distributed transaction (2PC) across services?
Short answer: Two-phase commit requires every participant to hold locks and stay available through both phases — it doesn't scale, couples availability (one slow service blocks all), and most modern stores don't support it across boundaries. Distributed systems trade ACID-across-services for eventual consistency.
Deep dive: This is the CAP reality: across a network partition you can't have both strong consistency and availability, so you choose availability + eventual consistency and reconcile with sagas.
Senior signal: You reject 2PC on availability/coupling grounds and pivot to sagas + eventual consistency rather than wishing for cross-service ACID.
10. Saga pattern — orchestration vs choreography?
Short answer: A saga is a sequence of local transactions across services, each with a compensating action to undo it on failure (no global rollback). Orchestration = one coordinator drives the steps; choreography = services react to each other's events with no central brain.
Deep dive: Orchestration is easier to reason about and debug (the flow is in one place); choreography is more decoupled but the flow is emergent and harder to follow. Mattrx uses orchestration for its multi-step report flow.
// Orchestration saga: one coordinator runs the steps and compensates on failure.
public async Task Handle(GenerateReport cmd, CancellationToken ct)
{
var hold = await _quota.ReserveAsync(cmd.TenantId, ct); // step 1
try
{
var pdf = await _render.RenderAsync(cmd, ct); // step 2
await _delivery.SendAsync(cmd.TenantId, pdf, ct); // step 3
}
catch
{
await _quota.ReleaseAsync(hold, ct); // COMPENSATE step 1
throw;
}
}
Senior signal: You define a saga by its compensations (not a rollback), and pick orchestration vs choreography on the "debuggability vs decoupling" trade-off.
11. What does eventual consistency actually mean for the product?
Short answer: After a change, different services (and the UI) may briefly disagree until events propagate — usually milliseconds to seconds. You must design the UX and business rules to tolerate that window (optimistic UI, "processing…" states) rather than assume instant global truth.
Deep dive: A user who publishes a campaign might not see it in analytics for a second — fine, if the UI is honest about it. The danger is a business rule that assumes immediate consistency (e.g., "block if over budget" checked against stale data).
Senior signal: You translate eventual consistency into concrete UX/business handling and flag rules that silently assume instant consistency.
12. What is the dual-write problem, and how does the outbox pattern fix it?
Short answer: The dual-write problem: you update your DB and publish an event as two separate operations — a crash in between leaves them inconsistent (state changed, event lost, or vice-versa). The outbox pattern writes the event into an outbox table in the same DB transaction as the state change; a relay publishes it afterward.
Deep dive: One atomic commit guarantees "state changed ⇔ event recorded." Mattrx's webhook/eventing runs on this.
// BUG (dual write): a crash between these two loses the event.
await _db.SaveChangesAsync(ct);
await _bus.PublishAsync(new CampaignPublished(id), ct); // if this fails, event is lost forever
// FIX (outbox): event + state commit atomically; a relay publishes from the outbox later.
using var tx = await _db.BeginTransactionAsync(ct);
_db.Campaigns.Update(campaign);
_db.Outbox.Add(new OutboxMessage(Guid.NewGuid(), "campaign.published", Serialize(evt)));
await _db.SaveChangesAsync(ct);
await tx.CommitAsync(ct); // atomic: state change AND event are persisted together
Senior signal: You name the dual-write race and reach for the outbox (or transactional-log/CDC) so publishing can't diverge from the state change.
13. Why must every event consumer be idempotent?
Short answer: Message delivery is at-least-once — the same message will be redelivered (redeploys, rebalances, ack loss). If applying it twice changes the outcome, you corrupt data. So consumers must dedup by a stable event id (or make the operation naturally idempotent).
Deep dive: Mattrx's KPI consumer checks a dedup store before applying, so a redelivered conversion isn't counted twice.
// At-least-once: the same event can arrive again. Dedup by a stable id before applying.
public async Task Consume(ConsumeContext<ConversionTracked> ctx)
{
if (!await _dedup.TryMarkProcessedAsync(ctx.Message.EventId, ctx.CancellationToken))
return; // already applied -> no-op
await _kpis.ApplyAsync(ctx.Message, ctx.CancellationToken);
}
Senior signal: You treat idempotency as mandatory (not optional), and know at-least-once redelivery is a certainty, not an edge case.
14. Exactly-once delivery — why is it a myth?
Short answer: Across a network, a sender can't know if a lost ack means "not delivered" or "delivered, ack lost," so it must retry -> at-least-once. True exactly-once delivery is impossible; you achieve exactly-once effect with at-least-once delivery + idempotent consumers.
Deep dive: "Exactly-once" features (e.g., Kafka's) are really at-least-once + transactional dedup within one system — they don't extend across an external HTTP call or a third party.
Senior signal: You correct "exactly-once" to "at-least-once + idempotency = exactly-once effect," and know its boundaries.
15. Message queue vs event streaming (Service Bus/RabbitMQ vs Kafka)?
Short answer: A queue (Service Bus/RabbitMQ) is for commands/work — a message is consumed once by one worker, then gone; great for task distribution and competing consumers. A log/stream (Kafka) is for events — a durable, replayable, ordered log many consumers read independently at their own offset.
Deep dive: Mattrx uses Kafka for high-volume ingestion (replayable, multiple consumers) and Service Bus for the report command queue (do-this-once work).
Queue (Service Bus): producer -> [ queue ] -> one worker consumes & removes (work distribution)
Log (Kafka): producer -> [ partitioned log ] -> many consumers, own offsets, replayable
Senior signal: You split "command -> queue" from "event -> log," and cite replay + multiple independent consumers as Kafka's differentiator.
16. How do you handle out-of-order or duplicate events?
Short answer: Duplicates: idempotent consumers (Q13). Ordering: partition/key by the entity (Kafka guarantees order within a partition), carry a version/sequence and ignore stale updates, and design handlers to be order-tolerant where possible.
Deep dive: Ordering is only guaranteed within a partition, so Mattrx keys events by campaignId to keep a campaign's events ordered on one partition.
Senior signal: You separate the dedup problem from the ordering problem and use partition keys + version checks rather than assuming global order.
Block C — resilience & communication
17. What problem does a circuit breaker solve?
Short answer: It stops a service from hammering a failing dependency. After a failure threshold it opens (fails fast without calling), waits, then half-opens to test recovery. This prevents wasted work, frees threads, and stops one sick service from dragging down its callers.
Deep dive: Without a breaker, callers pile up waiting on a dead dependency, exhaust their thread/connection pools, and fail too — a cascade.
// .NET resilience: retry (jittered) + circuit breaker + timeout on an outbound call.
services.AddHttpClient<PaymentClient>()
.AddResilienceHandler("payment", b => b
.AddRetry(new() { MaxRetryAttempts = 3, BackoffType = DelayBackoffType.Exponential, UseJitter = true })
.AddCircuitBreaker(new() { FailureRatio = 0.5, MinimumThroughput = 20, BreakDuration = TimeSpan.FromSeconds(30) })
.AddTimeout(TimeSpan.FromSeconds(3)));
Senior signal: You describe the closed->open->half-open states and tie the breaker to preventing thread-pool exhaustion and cascades.
18. Retries — how do you do them safely?
Short answer: Retry only transient, idempotent operations, with exponential backoff + jitter, a bounded attempt count, and a budget. Naive fixed-interval retries synchronize across callers and cause a retry storm that DDoSes the recovering service.
Deep dive: Jitter de-synchronizes retries; the budget stops retries from amplifying load during an outage.
// Exponential backoff + FULL jitter, bounded — never a tight fixed-interval retry loop.
.AddRetry(new() { MaxRetryAttempts = 4, Delay = TimeSpan.FromMilliseconds(200),
BackoffType = DelayBackoffType.Exponential, UseJitter = true });
Senior signal: You require jitter + a budget and only retry idempotent/transient calls, naming the retry storm as the failure to avoid.
19. Timeouts & bulkheads — why does every call need them?
Short answer: Every network call needs a timeout or a hung dependency ties up your threads/connections forever. Bulkheads isolate resources (separate connection pools/thread limits per dependency) so one slow dependency can't consume all of them and sink the whole service.
Deep dive: The default "no timeout" is the root of most cascading failures — a slow callee becomes a slow caller becomes a dead caller.
Senior signal: You say "no call without a timeout" and use bulkheads to contain a slow dependency's blast radius.
20. How do cascading failures happen, and how do you stop them?
Short answer: One slow/failing service causes its callers to block (no timeout), exhaust their pools, and fail — which fails their callers, rippling upstream. Stop it with timeouts, circuit breakers, bulkheads, retries with budgets, and load shedding.
Deep dive: The cascade is a resource-exhaustion chain reaction; each resilience pattern caps one link so failure stays local.
[A] -> [B] -> [C fails/slow]
| | ^ B's threads block on C, B exhausts pool, B fails, A fails...
+-- timeout + circuit breaker at each hop stops the ripple -> failure stays at C
Senior signal: You explain the cascade as pool exhaustion propagating upstream and list the specific patterns that cap each hop.
21. What do service discovery and the API gateway do?
Short answer: Service discovery lets services find each other's (dynamic) network locations without hardcoded addresses (DNS/registry, or the platform's built-in service DNS in Kubernetes/Container Apps). An API gateway is the single entry point for clients — routing, auth, rate limiting, and aggregation at the edge.
Deep dive: Mattrx runs on Azure Container Apps (built-in service discovery + ingress) with a gateway handling auth, per-tenant rate limiting, and TLS.
Senior signal: You separate east-west discovery (service-to-service) from the north-south gateway (client entry) and know the platform often provides discovery.
22. What are backpressure and load shedding?
Short answer: Backpressure is a consumer signaling "slow down" so a fast producer doesn't overwhelm it (bounded queues/channels, consumer-paced pulling). Load shedding is deliberately rejecting excess work (429s, dropping low-priority messages) to protect the service instead of collapsing under it.
Deep dive: Mattrx's ingestion consumer uses a bounded channel + worker pool so a Kafka spike buffers and paces instead of OOM-ing the service.
Senior signal: You'd rather shed load gracefully (429 / drop) than let a service fall over, and use bounded buffers for backpressure.
23. Why do you need a dead-letter queue?
Short answer: A DLQ captures messages that repeatedly fail processing (after max retries) instead of blocking the queue or being lost. It turns "poison message stuck in a retry loop" into "parked for inspection," keeping the pipeline flowing.
Deep dive: Without a DLQ, one un-processable message either blocks the partition/queue forever or vanishes. Mattrx dead-letters webhook deliveries after 8 attempts.
Senior signal: You add a DLQ + alerting for poison messages and know a stuck message can otherwise halt the whole consumer.
Block D — observability, deployment & production
24. Why is distributed tracing non-negotiable?
Short answer: A single user request fans out across many services; without a trace/correlation id propagated through every hop, a failure is un-debuggable — you have logs from six services and no way to connect them. Distributed tracing (OpenTelemetry, W3C traceparent) stitches one request into one timeline.
Deep dive: Mattrx propagates a trace id across HTTP and the bus; one trace shows the whole request path and where the latency/error was.
// One trace spans every hop; OTel propagates it via the W3C traceparent header automatically.
using var activity = ActivitySource.StartActivity("GenerateReport");
activity?.SetTag("tenant.id", tenantId);
// downstream HTTP + Service Bus calls carry traceparent -> one correlated trace across services.
Senior signal: You call tracing table-stakes for microservices and name correlation-id propagation (OTel/traceparent) as how you debug across services.
25. What does good logging/metrics look like across services?
Short answer: Structured (JSON) logs, centralized (shipped to one place), trace-correlated (every log line carries the trace id), plus RED metrics (Rate, Errors, Duration) per service. Free-text logs scattered across services are useless at scale.
Deep dive: With trace id on every log, you jump from a slow trace to exactly the log lines for that request across all services.
Senior signal: You insist on structured + centralized + trace-correlated logs and per-service RED metrics, not grep-across-N-boxes.
26. Liveness vs readiness health checks, and graceful shutdown?
Short answer: Liveness = "restart me if I'm dead"; readiness = "route traffic to me only when I can serve" (deps up, warm). On shutdown, stop accepting new work, finish in-flight requests/messages, then exit — so a deploy doesn't drop or duplicate work.
Deep dive: Failing readiness pulls a pod from the load balancer without killing it; graceful shutdown (honor the stop signal, commit offsets) prevents mid-message loss on every deploy.
Senior signal: You separate liveness from readiness and design graceful shutdown around the stop signal so rolling deploys are lossless.
27. How do you deploy services independently with schema changes?
Short answer: Expand/contract (a.k.a. parallel change): never make a breaking change in one deploy. Expand (add the new column/field, both old and new work) -> migrate (backfill, dual-write) -> contract (remove the old once nothing uses it). This lets producers and consumers deploy in any order.
Deep dive: Because old and new run simultaneously, a rename must be add-new -> migrate -> drop-old across separate deploys.
Expand: add new nullable column / optional event field (old + new both work)
Migrate: backfill data + dual-write; ship consumers reading the new field
Contract: once no one reads the old field, drop it (still no breaking deploy)
Senior signal: You never ship a breaking migration in one release and describe expand/contract for backward-compatible, order-independent deploys.
28. How do you test microservices?
Short answer: A test pyramid skewed to fast tests: lots of unit tests, contract tests to guarantee service-to-service compatibility, some integration tests per service (with test containers), and few end-to-end tests (slow, flaky — use sparingly). Consumer-driven contracts catch breaking API changes in CI.
Deep dive: Full E2E across all services is brittle; contract tests give most of the cross-service confidence at a fraction of the cost.
Senior signal: You lean on contract tests over broad E2E and can explain why E2E-heavy suites become the flaky bottleneck.
29. How does Conway's law shape microservices?
Short answer: Systems mirror the communication structure of the org that builds them (Conway's law). So you design teams around services (one team owns a service end-to-end) — otherwise a service split across teams becomes a coordination nightmare, and shared ownership becomes no ownership.
Deep dive: Independent deployability requires independent teams; if a change needs three teams, the service boundary is wrong (or the team boundary is).
Senior signal: You tie service boundaries to team ownership (Conway's law / team topologies) and treat "who deploys this" as a design input.
30. What are the top ways microservices go wrong?
Short answer: Over-decomposition (nano-services), a shared database, chatty synchronous call-chains (a distributed monolith), no observability (un-debuggable), missing resilience (cascades), and ignoring eventual consistency in business rules. Most "microservices failed us" stories are one of these.
Deep dive: The through-line: teams adopt the topology of microservices without the disciplines (boundaries, async, idempotency, observability) that make it work.
Senior signal: You name concrete failure modes and frame microservices as "distributed-systems discipline," not "small services," showing you've felt the pain.
Bonus — what makes an answer "senior" in microservices?
Short answer: A junior lists patterns; a senior reasons about what you traded for them. "Should we use microservices?" gets an enthusiastic "yes, they scale!" from a junior and "only when a module needs independent scaling/deploy/ownership — otherwise a modular monolith, because you're buying network failure, eventual consistency, and distributed debugging" from a senior.
Deep dive: The recurring senior lens across all 30: independence has a cost (network, consistency, ops), so every service, every sync call, and every shared resource must earn it. Name the failure mode, the compensation, and the trade-off.
Senior signal: You volunteer the cost/failure mode before the benefit, and you can say "here's when a monolith is the right answer."
The model to carry forward
Microservices aren't "small services" — they're a bet that independent deployability is worth distributed-systems pain. Everything follows: own your data (database-per-service), talk async where you can, accept eventual consistency and reconcile with sagas + the outbox, make every consumer idempotent, wrap every call in timeouts/breakers/retries-with-jitter, and make the whole thing observable with distributed tracing. Answer each question with the trade you're making and the failure it prevents, and you'll sound like someone who has operated this — which is the signal.
Further reading
- ASP.NET Core Interview Questions for Senior Developers — Middleware, DI, Kestrel & Auth (Deep Answers)
- Saga Pattern for Microservices — A Complete Guide
- Outbox Pattern — A Complete Guide with Order Processing Example
Prepping for a senior backend/architecture interview and want these drilled live, or a mock focused on sagas and resilience? I'm always happy to help — reach me at randhir.jassal@gmail.com.
Get the next issue
A short, curated email with the newest posts and questions.