End-to-end system design playbook

20.1 End-to-end: the journey of one request Added

A customer taps "Pay" in the mobile app. Here's every hop, and the concept that applies at each one.

app.MapPost("/payments", async (PaymentRequest req, [FromHeader(Name = "Idempotency-Key")] Guid key,
                                PaymentsDb db, CancellationToken ct) =>
{
    if (await db.Payments.AnyAsync(p => p.IdempotencyKey == key, ct))
        return Results.Ok();                                    // retry: same result, no double charge

    var payment = Payment.Create(req, key);
    db.Payments.Add(payment);
    db.Outbox.Add(OutboxMessage.From(new PaymentCompleted(payment.Id)));  // same transaction
    await db.SaveChangesAsync(ct);                              // unique index backs up the check
    return Results.Created($"/payments/{payment.Id}", payment.Id);
})
.RequireAuthorization("payments.write")
.RequireRateLimiting("per-user");

20.2 System design interview framework Added

A repeatable 7-step structure for a 45–60 minute design discussion. Structure beats brilliance.

20.3 How to answer any "X vs Y" question Added

One formula covers Sync vs Async, SQL vs NoSQL, Queue vs Event, ETL vs ELT…

20.4 Trade-off cheat sheet Added

Common either/or decisions, with a rule of thumb for when to choose each side.

20.5 Worked example: real-time payment notification platform Added

"Design a service that notifies customers (push, SMS, email) within 5 seconds of any card or account transaction."

6. Deep dives

• Duplicates: dedupe key = transactionId + channel in Redis (TTL 24 h). Consumers are idempotent.
• Ordering: partition by accountId so a customer's events are processed in order.
• Spikes: queues level the load. Workers auto-scale on queue depth.
• Provider outage: circuit breaker per provider, failover SMS provider, DLQ + replay.
• Security: mask PII in payloads (last 4 digits only), managed identities, encryption at rest and in transit.
• Observability: trace from transactionId → delivered. SLO = % delivered within 5 s.

7. Trade-offs

Near-real-time events instead of synchronous calls from core banking (decoupled, but eventually consistent). At-least-once delivery + dedupe instead of costly exactly-once. NoSQL for history (scale) instead of SQL (ad-hoc queries go to the lakehouse instead).

processor.ProcessMessageAsync += async args =>
{
    var evt = args.Message.Body.ToObjectFromJson<TransactionPosted>()!;
    foreach (var channel in await prefs.GetChannelsAsync(evt.CustomerId, args.CancellationToken))
    {
        // Claim (transaction, channel) once for 24 h; a redelivery finds the key and skips
        bool first = await redis.StringSetAsync($"notif:{evt.TransactionId}:{channel}", 1,
                                                TimeSpan.FromHours(24), When.NotExists);
        if (first) await senders[channel].SendAsync(evt, args.CancellationToken);
    }
    await args.CompleteMessageAsync(args.Message);
};

20.6 Design review checklist Added

Questions to run through before you present or approve a design.