Section 22

Top 10 system design use cases in .NET

Ten real problems, solved the .NET 10 way. Each one follows the same path: the problem statement (with numbers), the solution as an animated flow plus the key trade-offs, the steps to build it with the exact APIs and NuGet packages, and the test scenarios: positive, negative and edge cases, with the .NET tool for each.

These are implementation-focused. The classic architecture case studies (URL shortener, rate limiter, chat, notifications, feed, ledger) live in 21 Design Case Studies.

22.1 Flash-sale order & inventory without overselling Added

Thousands of people press "Buy" in the same second for a few thousand items. Take the stock in one atomic step, answer fast, and do the slow work (payment, order rows) in the background.

Problem statement

A retailer (or a bank's rewards store) sells 5,000 limited items at 12:00 sharp. About 200,000 shoppers arrive in the first minute. A normal "read stock, then write stock" flow sells the same last item to many people, and the database melts under the spike.

Functional

  • Show remaining stock (can be a few seconds old)
  • Reserve and place an order; pay within 10 minutes
  • Limit: 2 items per customer
  • Release stock if payment fails or times out
  • A double-click or retry must not buy twice

Non-functional

  • Peak: 50,000 requests/s on "Buy" for about 2 minutes
  • Latency: reserve answer p99 < 200 ms
  • Correctness: 0 oversold items, ever
  • Availability: 99.95% during the sale window
  • Durability: no accepted order is lost

Constraints & assumptions: one Azure region; SQL Server holds the real stock and orders; Redis is available; the payment provider is external and slow (1–3 s); the sale time is known in advance, so you can pre-scale.

Solution

Key decisionTrade-off
Gate in Redis, truth in SQL. A Lua script checks and decrements stock in one atomic step in memory.Very fast, but Redis and SQL can drift, so SQL keeps a final WHERE Stock >= @qty guard and a job reconciles.
Reserve now, write later. The API answers 202 and a queue absorbs the spike.The shopper sees "pending" for a moment, and you need a timeout to release unpaid stock.
Idempotency key per checkout. Retries return the first result.One extra Redis key per order (kept 24 h).
Admission control. Rate limit or a waiting room in front of "Buy".Some shoppers wait or get 429, but the system stays up.
Pre-scale before 12:00. Set minimum replicas ahead of time.You pay for idle capacity for a few minutes; autoscale reacts too slowly for a 2-minute spike.

Steps to implement in .NET 10

  1. Set up: dotnet new webapi (minimal APIs) plus an Aspire 13 AppHost with Aspire.Hosting.Redis, Aspire.Hosting.SqlServer and Aspire.Hosting.Azure.ServiceBus, so Redis, SQL and the Service Bus emulator start locally with one command.
  2. Model: EF Core 10 (Microsoft.EntityFrameworkCore.SqlServer) with Product (Id, Stock, [Timestamp] byte[] RowVersion), Order (PK = orderId) and an outbox table. RowVersion makes admin edits (price, stock top-ups) fail with DbUpdateConcurrencyException instead of silently overwriting.
  3. Load stock into Redis before the sale: StringSetAsync("stock:{sku}", 5000) with StackExchange.Redis.
  4. Reserve endpoint: run the Lua script with IDatabase.ScriptEvaluateAsync (core trick 1). It also adds the order to a sorted set scored by its expiry time, so the sweeper can find unpaid reservations.
  5. Idempotency: require an Idempotency-Key header (an IETF draft name, widely used). Check it inside the same Lua script, so "seen before?" and "take stock" can't race.
  6. Publish OrderReserved with Azure.Messaging.ServiceBus. Set MessageId = orderId and turn on duplicate detection on the queue.
  7. Order worker: a BackgroundService with ServiceBusProcessor writes the order using the SQL guard (core trick 2), in one transaction with the outbox row.
  8. Admission control: built-in Microsoft.AspNetCore.RateLimiting (AddRateLimiter, token bucket per client, 429 + Retry-After), plus Azure Front Door WAF rate rules at the edge.
  9. Release expired reservations: a BackgroundService using new PeriodicTimer(TimeSpan.FromSeconds(15), timeProvider) reads expired entries (SortedSetRangeByScoreAsync up to "now"), returns their stock and removes them. Inject TimeProvider so tests can fake time.
  10. Deploy & observe: Azure Container Apps with min replicas raised before the sale, Azure Managed Redis, Azure.Monitor.OpenTelemetry.AspNetCore for traces and metrics (reserved, sold out, released). Run the load test a day before.

Core trick 1: atomic reserve + idempotency in one Lua script

const string ReserveLua = """
    -- KEYS: 1 stock, 2 customer count, 3 idempotency key, 4 reservations (sorted set)
    if redis.call('EXISTS', KEYS[3]) == 1 then return -2 end            -- seen before
    local qty, limit = tonumber(ARGV[1]), tonumber(ARGV[2])
    local stock = tonumber(redis.call('GET', KEYS[1]) or '0')
    if stock < qty then return -1 end                                     -- sold out
    if tonumber(redis.call('GET', KEYS[2]) or '0') + qty > limit then return -3 end
    redis.call('DECRBY', KEYS[1], qty)
    redis.call('INCRBY', KEYS[2], qty)
    redis.call('SET', KEYS[3], ARGV[3], 'EX', 86400)                      -- remember orderId 24 h
    redis.call('ZADD', KEYS[4], ARGV[4], ARGV[3])                        -- score = expiry time
    return stock - qty
    """;

var tag = "{sku-123}";            // hash tag: all keys land on the same cluster slot
var left = (long)await db.ScriptEvaluateAsync(ReserveLua,
    [$"stock:{tag}", $"cust:{tag}:{customerId}", $"idem:{tag}:{idemKey}", $"resv:{tag}"],
    [qty, 2, orderId.ToString(), clock.GetUtcNow().AddMinutes(10).ToUnixTimeSeconds()]);
return left switch
{
    -1 => Results.Conflict("Sold out"),
    -2 => Results.Accepted($"/orders/{await db.StringGetAsync($"idem:{tag}:{idemKey}")}"),
    -3 => Results.UnprocessableEntity("Limit is 2 per customer"),
    _  => Results.Accepted($"/orders/{orderId}")
};

Core trick 2: the SQL guard in the worker (EF Core ExecuteUpdateAsync)

await using var tx = await db.Database.BeginTransactionAsync(ct);
if (await db.Orders.AnyAsync(o => o.Id == msg.OrderId, ct)) return;       // redelivered message

var rows = await db.Products
    .Where(p => p.Id == msg.ProductId && p.Stock >= msg.Qty)              // final guard
    .ExecuteUpdateAsync(s => s.SetProperty(p => p.Stock, p => p.Stock - msg.Qty), ct);

if (rows == 0)                                                           // Redis and SQL disagreed
{
    await tx.RollbackAsync(ct);
    await ReleaseInRedisAsync(msg, ct);
    return;
}
db.Orders.Add(Order.AwaitingPayment(msg.OrderId, msg.CustomerId, msg.ProductId, msg.Qty));
await db.SaveChangesAsync(ct);                                           // order + outbox row together
await tx.CommitAsync(ct);

Testing scenarios

TypeScenarioExpected resultHow to test (.NET)
✅ PositiveBuy 1 when stock is 10202 + orderId; Redis stock 9; order row "AwaitingPayment"xUnit v3 + WebApplicationFactory; Testcontainers Redis + MsSql
✅ PositiveSame Idempotency-Key sent twiceSame orderId both times; stock goes down onceTwo calls with one key; assert stock and order count
✅ PositivePayment succeeds within 10 minutesOrder "Paid"; reservation removed; stock not returnedService Bus emulator (Testcontainers.ServiceBus) or a fake payment handler
✅ PositiveCustomer buys exactly 2 (the limit)Both acceptedIntegration test with one customerId
❌ NegativeStock is 0409 Sold out; no order, no messageSeed stock 0; assert queue is empty
❌ NegativeSame customer tries a 3rd item422; stock unchangedIntegration test
❌ NegativeMissing Idempotency-Key400 with a ProblemDetails bodyHttpClient without the header; Verify snapshot of the body
❌ NegativeRedis is down503 (fail closed: never sell blind)Stop the Redis container mid-test, or a Polly chaos fault on the Redis call
❌ NegativePayment failsOrder "Cancelled"; stock back in RedisFake payment returns a decline
⚠️ Edge500 parallel buys for the last 100 itemsExactly 100 accepted, 400 sold out, stock never below 0Task.WhenAll against Testcontainers Redis (sample below)
⚠️ EdgeReservation reaches 10:00 exactlyStock returned once; a late payment is refundedFakeTimeProvider.Advance(TimeSpan.FromMinutes(10))
⚠️ EdgeService Bus delivers the same message twiceOne order row onlyCall the worker handler twice with the same message
⚠️ EdgeRedis says 1 left, SQL says 0SQL guard rejects; reservation released; shopper told "sold out"Seed the two stores differently
⚠️ Edge50,000 req/s for 2 minutesp99 < 200 ms; error rate < 0.1%; 0 oversoldk6 (free) or NBomber (commercial licence for organisations)

Sample test: no overselling under concurrency (the factory starts new RedisBuilder("redis:7.4").Build() and points the app at it)

public sealed class FlashSaleTests(SaleApiFactory factory) : IClassFixture<SaleApiFactory>
{
    [Fact]
    public async Task Never_sells_more_than_the_stock()
    {
        var ct = TestContext.Current.CancellationToken;
        await factory.SeedStockAsync("sku-123", 100);

        var calls = Enumerable.Range(0, 500).Select(async i =>
        {
            using var client = factory.CreateClient();
            var req = new HttpRequestMessage(HttpMethod.Post, "/orders")
            { Content = JsonContent.Create(new { sku = "sku-123", qty = 1, customerId = $"c{i}" }) };
            req.Headers.Add("Idempotency-Key", Guid.NewGuid().ToString());
            return (await client.SendAsync(req, ct)).StatusCode;
        });
        var results = await Task.WhenAll(calls);

        Assert.Equal(100, results.Count(s => s == HttpStatusCode.Accepted));
        Assert.Equal(400, results.Count(s => s == HttpStatusCode.Conflict));
        Assert.Equal(0, await factory.GetRedisStockAsync("sku-123"));
    }
}
The .NET stack for this use case
ComponentLibrary / APINotes
Atomic gateStackExchange.Redis ScriptEvaluateAsyncLua runs as one step; use hash tags on Redis Cluster
Final guardEF Core 10 ExecuteUpdateAsync + [Timestamp] rowversionConditional UPDATE; optimistic concurrency for edits
Async ordersAzure.Messaging.ServiceBusDuplicate detection on MessageId
Admissionbuilt-in Microsoft.AspNetCore.RateLimiting429 + Retry-After
Expiry sweeperbuilt-in BackgroundService, PeriodicTimer, TimeProviderFake time in tests
Testsxunit.v3, Microsoft.AspNetCore.Mvc.Testing, Testcontainers.Redis; NBomber commercial licence or k6NBomber is free only for personal use

Never "read stock, then write stock" in two calls. Whatever store you use, the check and the decrement must be one atomic operation.

  • Check-and-decrement in one atomic step: a Redis Lua script or UPDATE ... WHERE Stock >= @qty.
  • Answer 202 fast; a queue and a worker do the slow part.
  • Idempotency key + reservation expiry turn retries and abandoned carts into non-events.
  • Prove it with a Task.WhenAll race test, not with hope.

5,000 items. 200,000 shoppers. 12:00 sharp. How do you sell exactly 5,000? 🛒

⚛️ Check and take stock in ONE atomic step (Redis Lua or a conditional SQL UPDATE)

📨 Answer 202 in milliseconds, let a queue write the orders

🔑 Idempotency keys: a double-click never buys twice

⏳ Unpaid after 10 minutes? The stock goes back automatically

🧪 Test it: 500 parallel requests for 100 items must give exactly 100 orders

Overselling is a design bug, not bad luck.

#SystemDesign #DotNet #Redis #Ecommerce #Testing

22.2 Seat / appointment booking with temporary holds Added

Let a user hold a seat or time slot for a few minutes while they pay, block everyone else, and release it automatically if they walk away.

Problem statement

Concert tickets, flight seats and branch appointments all have the same shape: a limited set of named slots. Users need a short, exclusive hold while they fill in details or pay. Without holds, two people pay for the same seat; with holds that never expire, seats look "sold out" when they are not.

Functional

  • See free, held and booked seats/slots
  • Hold 1–6 seats for 10 minutes, all or nothing
  • Confirm (after payment) or cancel a hold
  • Holds release by themselves when time runs out
  • Show a countdown; other screens update live

Non-functional

  • Peak: 2,000 hold requests/s at a big on-sale (appointments: ~50/s)
  • Latency: hold p99 < 300 ms; seat map p95 < 100 ms
  • Correctness: zero double bookings
  • Availability: 99.9%
  • Freshness: seat map may be 2–3 s stale; hold/confirm never

Constraints & assumptions: SQL Server is the source of truth; up to 60,000 seats per event; app servers are NTP-synced (seconds of skew don't matter for a 10-minute hold); all times are stored in UTC (DateTimeOffset).

Solution

Key decisionTrade-off
Hold columns on the seat row + a conditional UPDATE. WHERE BookedBy IS NULL AND (HoldExpiresAt IS NULL OR HoldExpiresAt <= @now).One source of truth and transactional with booking; SQL row locks become the bottleneck at very big on-sales.
Lazy expiry. An expired hold counts as free in every query; no exact-time job is needed.Expired values stay in the row until a sweeper or the next hold overwrites them.
All-or-nothing multi-seat holds in one transaction.One taken seat fails the whole request; the UI must suggest alternatives.
Redis SET NX PX in front for huge on-sales (plus a waiting room).Much faster, but now two stores must agree; SQL still does the final confirm.
10-minute hold, one extension.Longer holds mean fewer failed payments but more "fake sold out" seats.

Steps to implement in .NET 10

  1. Set up: minimal API + Aspire AppHost (Aspire.Hosting.SqlServer); EF Core 10 with Microsoft.EntityFrameworkCore.SqlServer.
  2. Model: Seat (Id, EventId, HeldBy, HoldExpiresAt as DateTimeOffset?, BookedBy, [Timestamp] RowVersion), index on (EventId, Id). A Booking row per confirmed payment with a unique PaymentId.
  3. Time: register builder.Services.AddSingleton(TimeProvider.System) and never call DateTime.UtcNow directly. Tests swap in FakeTimeProvider.
  4. Hold endpoint: one ExecuteUpdateAsync inside a transaction; if fewer rows changed than seats asked for, roll back (core trick 1). Reject duplicate seat IDs and more than 6 seats up front (400).
  5. Confirm endpoint: UPDATE ... SET BookedBy = @me WHERE HeldBy = @me AND HoldExpiresAt > @now, idempotent on PaymentId. If the hold expired but the seat is still free, book it; if someone else has it, trigger an automatic refund.
  6. Seat map reads: HybridCache (Microsoft.Extensions.Caching.Hybrid) with a 2 s expiry and a tag per event; call RemoveByTagAsync after holds change.
  7. Live updates: ASP.NET Core SignalR (built-in), one group per event: push "held", "booked", "free again". Use Azure SignalR Service (Microsoft.Azure.SignalR) to scale out.
  8. Sweeper: BackgroundService with new PeriodicTimer(TimeSpan.FromSeconds(30), timeProvider) clears expired holds and pushes "free again" events.
  9. Big on-sale mode: Redis hold first with StringSetAsync(key, user, TimeSpan.FromMinutes(10), When.NotExists) (core trick 2), plus a queue-based waiting room; SQL confirm stays the final check.
  10. Deploy: App Service or Container Apps, Azure SQL (read replica for seat-map reads if needed), OpenTelemetry metrics for holds created, expired and confirmed.

Core trick 1: all-or-nothing hold with a conditional UPDATE

app.MapPost("/events/{eventId:int}/holds", async (int eventId, HoldRequest req, BookingDb db,
    TimeProvider clock, ClaimsPrincipal user, CancellationToken ct) =>
{
    var now = clock.GetUtcNow();
    var me = user.FindFirstValue(ClaimTypes.NameIdentifier)!;
    await using var tx = await db.Database.BeginTransactionAsync(ct);

    var held = await db.Seats
        .Where(s => s.EventId == eventId && req.SeatIds.Contains(s.Id) && s.BookedBy == null
                 && (s.HoldExpiresAt == null || s.HoldExpiresAt <= now))      // free or expired
        .ExecuteUpdateAsync(s => s
            .SetProperty(x => x.HeldBy, me)
            .SetProperty(x => x.HoldExpiresAt, now.AddMinutes(10)), ct);

    if (held != req.SeatIds.Length)                    // someone else got at least one seat
    {
        await tx.RollbackAsync(ct);                    // all or nothing
        return Results.Conflict("One or more seats are no longer free");
    }
    await tx.CommitAsync(ct);
    return Results.Ok(new { expiresAt = now.AddMinutes(10) });
});

Core trick 2: Redis hold with expiry, and a safe release

// SET hold:7:A3 alice NX PX 600000  -> only the first caller wins
bool got = await redis.StringSetAsync($"hold:{eventId}:{seatId}", userId,
                                      TimeSpan.FromMinutes(10), When.NotExists);
if (!got) return Results.Conflict("Seat is held by someone else");

// Release only if the hold is still yours (compare-and-delete in one step)
await redis.ScriptEvaluateAsync(
    "if redis.call('GET', KEYS[1]) == ARGV[1] then return redis.call('DEL', KEYS[1]) end return 0",
    [$"hold:{eventId}:{seatId}"], [userId]);

Testing scenarios

TypeScenarioExpected resultHow to test (.NET)
✅ PositiveHold 2 free seats200 + expiresAt = now + 10 minWebApplicationFactory + Testcontainers MsSql; FakeTimeProvider fixes "now"
✅ PositiveConfirm within the holdSeats "booked"; one Booking rowIntegration test, fake payment callback
✅ PositiveCancel a holdSeats free at once; SignalR "free again" sentAssert DB + a test SignalR client (Microsoft.AspNetCore.SignalR.Client)
✅ PositiveHold a seat after someone else's hold expired200FakeTimeProvider.Advance(TimeSpan.FromMinutes(10)) (sample below)
❌ NegativeHold a seat another user holds409 ConflictTwo test users via a test auth handler
❌ NegativeConfirm after the hold expired and the seat was taken410 Gone; refund startedAdvance fake time; second user holds; first confirms
❌ NegativeConfirm someone else's hold403; seats unchangedIntegration test
❌ Negative7 seats, or duplicate seat IDs400 validation errorUnit test the validator; Verify snapshot of ProblemDetails
⚠️ Edge50 users race for the same seatExactly 1 gets 200, 49 get 409Task.WhenAll against a real SQL container (not the in-memory provider)
⚠️ EdgeRequest at exactly HoldExpiresAtSeat counts as free (<=), matching the docsAdvance fake time by exactly 10:00
⚠️ EdgeOverlapping requests A1+A2 vs A2+A3One wins both seats; the other gets none (no half holds)Parallel requests; assert no user holds just one seat
⚠️ EdgePayment callback arrives twiceOne booking (unique PaymentId)Replay the callback
⚠️ EdgeDB times out during hold500/503 and no half-held seats (transaction rolled back)Polly chaos latency/fault strategy, or pause the container

Sample test: expiry without real waiting (Microsoft.Extensions.TimeProvider.Testing)

[Fact]
public async Task Hold_expires_after_ten_minutes()
{
    var ct = TestContext.Current.CancellationToken;
    var clock = new FakeTimeProvider(DateTimeOffset.Parse("2026-10-16T09:00:00Z"));
    await using var app = factory.WithWebHostBuilder(b =>
        b.ConfigureTestServices(s => s.AddSingleton<TimeProvider>(clock)));
    var alice = app.CreateClientFor("alice");          // helper: test auth handler sets the user
    var bob = app.CreateClientFor("bob");
    var seat12 = new { seatIds = new[] { 12 } };

    var first = await alice.PostAsJsonAsync("/events/7/holds", seat12, ct);
    Assert.Equal(HttpStatusCode.OK, first.StatusCode);

    var tooEarly = await bob.PostAsJsonAsync("/events/7/holds", seat12, ct);
    Assert.Equal(HttpStatusCode.Conflict, tooEarly.StatusCode);

    clock.Advance(TimeSpan.FromMinutes(10));           // ten minutes pass instantly
    var later = await bob.PostAsJsonAsync("/events/7/holds", seat12, ct);
    Assert.Equal(HttpStatusCode.OK, later.StatusCode);
}
The .NET stack for this use case
ComponentLibrary / APINotes
Hold / confirmEF Core 10 ExecuteUpdateAsync in a transactionConditional UPDATE is the lock
Timebuilt-in TimeProvider; tests: Microsoft.Extensions.TimeProvider.TestingFakeTimeProvider.Advance
Fast holdsStackExchange.Redis StringSetAsync(..., When.NotExists)Redis TTL uses Redis's clock, not TimeProvider
Seat map cacheMicrosoft.Extensions.Caching.HybridShort expiry + tag invalidation
Live updatesbuilt-in SignalR; Microsoft.Azure.SignalR to scaleGroup per event

Test concurrency against a real database (Testcontainers). The EF Core in-memory provider has no row locks, so race tests pass there and fail in production.

  • A hold is "mine until time X". A conditional UPDATE (or SET NX PX) grants it atomically.
  • Make expiry lazy: an expired hold is free in every query; the sweeper only tidies up.
  • Multi-seat holds are all or nothing, in one transaction.
  • Inject TimeProvider so expiry tests take milliseconds, not minutes.

Two people. One seat. Both pressed "Pay". Who gets it? 🎟️

🔒 Give a 10-minute hold with one conditional UPDATE (or Redis SET NX PX)

⏳ Let holds expire lazily: an expired hold is simply "free"

🧩 Multi-seat holds are all or nothing

✅ Confirm re-checks the hold; late payments get an automatic refund

🧪 .NET tip: inject TimeProvider, then FakeTimeProvider.Advance(10 min) in tests

How long should a hold last on your platform?

#SystemDesign #DotNet #EFCore #Testing #Architecture

22.3 Search autocomplete (typeahead) Added

Suggest the best 5–10 completions while the user types. Precompute the answers per prefix, cache hard, and send fewer requests from the client.

Problem statement

A large store or banking app has a search box: products, merchants, help articles. Every keystroke could be a request, so a naive "LIKE 'hom%'" query on the main database is slow and expensive. Users expect suggestions to appear as fast as they type.

Functional

  • Top 10 suggestions for a prefix, most popular first
  • Case- and accent-insensitive ("Café" = "cafe")
  • New trending terms show up within ~15 minutes
  • Never suggest blocked or offensive terms
  • Nice to have: typo tolerance, personal history

Non-functional

  • Traffic: 10M daily users × 5 searches × ~6 requests (after debounce) ≈ 300M/day ≈ 3,500/s, peak ~15,000/s
  • Latency: server p99 < 20 ms; end to end < 100 ms
  • Availability: 99.9% (if it fails, search still works)
  • Freshness: minutes are fine

Constraints & assumptions: ~5M distinct terms from search logs; English first; prefixes capped at 10 characters for the precomputed lists; Azure hosting with a CDN in front.

Solution

Key decisionTrade-off
Precompute top 10 per prefix (a trie flattened into one Redis sorted set per prefix).Reads are one call and already ranked; costs memory (a few GB) and a rebuild job.
Client debounce (200–300 ms) + cancel stale calls.Cuts requests several times over; adds a small delay before the first suggestion.
Cache short prefixes at the edge (60 s). One- and two-letter prefixes are the hottest.Suggestions can be up to a minute old.
Batch rebuild from logs, written to a new version and switched with one pointer key.Trending terms appear after the next batch, not instantly; readers never see half-built data.
Managed search (Azure AI Search suggester) when you need typo tolerance and filters.Less code, fuzzy matching built in; higher cost and latency than a Redis lookup.

Steps to implement in .NET 10

  1. Set up: minimal API; Redis via Aspire.StackExchange.Redis (client integration) or plain StackExchange.Redis.
  2. Log searches: send each final query as an event to Event Hubs (Azure.Messaging.EventHubs). Never log personal data in the term.
  3. Aggregate: a timer job (Azure Functions timer trigger, or a BackgroundService) counts term frequency with time decay, and drops blocked terms.
  4. Build the index: for each term and each prefix up to 10 chars, SortedSetAddAsync then trim with SortedSetRemoveRangeByRankAsync(key, 0, -11) to keep the top 10 (core trick 1). Use batching/pipelining; write to ac:v43:*, then set ac:current = v43.
  5. Normalise once, everywhere: Trim(), ToLowerInvariant(), remove accents (string.Normalize(NormalizationForm.FormD) and drop non-spacing marks). Same function in the indexer and the API.
  6. Suggest endpoint: validate 1–50 chars, read SortedSetRangeByRankAsync(key, 0, 9, Order.Descending). For prefixes longer than 10 chars, read the 10-char list and filter with StartsWith.
  7. Cache: built-in output caching (AddOutputCache, .CacheOutput(p => p.Expire(...).SetVaryByQuery("q"))) and a Cache-Control: public, max-age=60 header for the CDN (Azure Front Door).
  8. Protect: built-in rate limiter per IP; if Redis fails, return an empty list (200), not an error.
  9. Client: debounce 250 ms and cancel the previous request (AbortController in JS, a new CancellationTokenSource per keystroke in Blazor/MAUI).
  10. Optional managed option: Azure AI Search (Azure.Search.Documents) with a SearchSuggester on the index and SearchClient.AutocompleteAsync / SuggestAsync with UseFuzzyMatching = true (core trick 2).

Core trick 1: top 10 per prefix in Redis sorted sets

// Indexer: put a term into the top-10 list of each of its prefixes
async Task IndexTermAsync(IDatabase db, string version, string term, double score)
{
    var t = Normalise(term);
    for (int len = 1; len <= Math.Min(10, t.Length); len++)
    {
        var key = $"ac:{version}:{t[..len]}";
        await db.SortedSetAddAsync(key, t, score);
        await db.SortedSetRemoveRangeByRankAsync(key, 0, -11);      // keep only the top 10
    }
}

// API: one Redis call per uncached keystroke
app.MapGet("/suggest", async (string q, IConnectionMultiplexer mux) =>
{
    var norm = Normalise(q);
    if (norm.Length is 0 or > 50) return Results.BadRequest();
    var db = mux.GetDatabase();
    var version = await db.StringGetAsync("ac:current");             // switched after each rebuild
    var hits = await db.SortedSetRangeByRankAsync($"ac:{version}:{norm[..Math.Min(10, norm.Length)]}",
                                                  0, 9, Order.Descending);
    return Results.Ok(hits.Select(h => h.ToString()).Where(h => h.StartsWith(norm)));
}).CacheOutput(p => p.Expire(TimeSpan.FromSeconds(60)).SetVaryByQuery("q"));

Core trick 2: Azure AI Search with typo tolerance

// Index definition includes: Suggesters = { new SearchSuggester("sg", "name") }
var response = await searchClient.AutocompleteAsync("hom", "sg", new AutocompleteOptions
{
    Mode = AutocompleteMode.OneTermWithContext,
    UseFuzzyMatching = true,          // "hmoe" still finds "home"
    Size = 8
});
var suggestions = response.Value.Results.Select(r => r.Text);

Testing scenarios

TypeScenarioExpected resultHow to test (.NET)
✅ Positive"hom" with seeded popular termsHighest score first, max 10 itemsWebApplicationFactory + Testcontainers Redis; Verify snapshot of the JSON
✅ Positive"HOM" and " hom " (case, spaces)Same result as "hom"xUnit [Theory] with [InlineData]
✅ PositiveNew trending term after a rebuildAppears once the pointer switchesRun the indexer in the test, then query
✅ PositiveSecond identical requestServed from output cache (Redis not called)Count calls with a Redis wrapper or OpenTelemetry in-memory exporter
❌ NegativeEmpty q or 51+ chars400[Theory] integration test
❌ NegativeBlocked/offensive term in the logsNever suggestedUnit test the aggregator's filter
❌ NegativeRedis down200 with an empty list, fast (no 500, no hang)Stop the container, or Polly chaos fault/latency; assert response < 200 ms
❌ NegativeSpecial characters (*, %, \, emoji)Treated as plain text; no error[Theory] with odd inputs
⚠️ EdgeAccents: "café" vs "cafe"Both return the same listUnit test Normalise on both sides
⚠️ EdgePrefix longer than 10 charsUses the 10-char list, filtered by full prefixIntegration test with a 15-char query
⚠️ EdgeQuery during a rebuildOld or new version, never a mix or emptyRun indexer and queries in parallel; assert every response is non-empty
⚠️ EdgeUser types 6 letters in 300 ms1 request sent (debounce), older calls cancelledPlaywright for .NET (Microsoft.Playwright) counting network calls
⚠️ Edge15,000 req/s peakServer p99 < 20 ms; cache hit rate > 80%k6 or NBomber (commercial licence) against a staging slot

Sample test: ranking snapshot with Verify (Verify.XunitV3)

public sealed class SuggestTests(SuggestApiFactory factory) : IClassFixture<SuggestApiFactory>
{
    [Fact]
    public async Task Returns_top_terms_by_popularity()
    {
        var ct = TestContext.Current.CancellationToken;
        await factory.IndexAsync(("home loan", 900), ("home insurance", 700),
                                 ("homepage", 50), ("holiday", 400));     // runs the real indexer

        var client = factory.CreateClient();
        var result = await client.GetFromJsonAsync<string[]>("/suggest?q=HOM", ct);

        await Verify(result);   // snapshot: ["home loan", "home insurance", "homepage"]
    }
}
The .NET stack for this use case
ComponentLibrary / APINotes
Prefix indexStackExchange.Redis sorted setsSortedSetRangeByRankAsync; SortedSetRangeByValueAsync for plain A–Z prefix lists
Managed optionAzure.Search.DocumentsSuggesters, AutocompleteAsync, fuzzy matching
Cachingbuilt-in Output caching; Microsoft.AspNetCore.OutputCaching.StackExchangeRedis to share itVary by q; CDN with Cache-Control
Search logsAzure.Messaging.EventHubsFeed the batch rebuild
TestsVerify.XunitV3, Testcontainers.Redis, Microsoft.PlaywrightSnapshot ranking; browser test for debounce

A single sorted set with ZRANGEBYLEX (SortedSetRangeByValueAsync) finds all terms with a prefix, but in A–Z order. For "most popular first", precompute per prefix as above.

  • Do the work at write time: a ready-sorted top 10 per prefix makes reads one lookup.
  • The cheapest request is the one never sent: debounce and cancel on the client.
  • Cache short prefixes hard; rebuild in a new version and switch a pointer.
  • Autocomplete must fail soft: an empty list, never an error.

Your search box sends a request on every keystroke. At 10M users, that's a lot of "h", "ho", "hom". ⌨️

🌳 Precompute the top 10 for every prefix (a trie, flattened into Redis sorted sets)

⏱️ Debounce 250 ms and cancel the old request

🗄️ Cache short prefixes at the CDN for 60 seconds

🔄 Rebuild from search logs every 15 minutes, switch versions atomically

🧯 If Redis fails, return an empty list. Search still works.

Fast typeahead is mostly about not doing work at read time.

#SystemDesign #DotNet #Redis #Search #Performance

22.4 Proximity service: find nearby ATMs, branches or drivers Added

"What is near me?" Static places (ATMs, branches) live in a spatial database; fast-moving things (drivers, couriers) live in an in-memory geo index that is updated every few seconds.

Problem statement

A banking app needs "nearest ATM or branch" with filters. A delivery or ride app needs "available drivers within 3 km", where every driver moves and reports a new location every few seconds. Scanning every row and computing distances is too slow, and moving objects create a heavy write load.

Functional

  • Nearest N ATMs/branches within a radius, with filters (deposits, open now, accessible)
  • Drivers report location every ~4 s
  • Find available drivers within 3 km, nearest first
  • Hide drivers that stopped reporting

Non-functional

  • Places: ~20k locations, 2,000 queries/s, p95 < 100 ms
  • Drivers: 200k online × 1 update / 4 s = 50,000 writes/s; 5,000 searches/s, p99 < 50 ms
  • Freshness: driver location at most ~10 s old
  • Availability: 99.95%

Constraints & assumptions: WGS84 latitude/longitude (SRID 4326); distances in metres; the customer's location is never stored or logged in full; driver history is not kept in the hot store.

Solution

Key decisionTrade-off
Two stores for two problems. Places in SQL (geography + spatial index); drivers in Redis GEO (in memory).Each fits its load (rich filters vs 50k writes/s), but you run two technologies.
Shard drivers by city/region key (drivers:syd, drivers:mel).Small sets and spread load; a rider near a boundary must query two keys.
Heartbeat key with a TTL per driver. GEO members can't expire on their own.One extra key and an MGET per search; a cleanup job removes stale members.
Client-side throttling: send only if moved > 50 m or 4 s passed.Far fewer writes; position can be ~50 m off.
Cache place searches by rounded location (e.g. 3 decimals ≈ 110 m).High hit rate in busy areas; recompute exact distances for display.

Steps to implement in .NET 10

  1. Set up: minimal API; Microsoft.EntityFrameworkCore.SqlServer.NetTopologySuite (or Npgsql.EntityFrameworkCore.PostgreSQL.NetTopologySuite for PostGIS); StackExchange.Redis.
  2. Model places: Atm with NetTopologySuite.Geometries.Point Location mapped to SQL Server geography; register with UseSqlServer(cs, x => x.UseNetTopologySuite()).
  3. Spatial index: add CREATE SPATIAL INDEX with migrationBuilder.Sql(...) in a migration (create it in SQL; don't rely on HasIndex for spatial indexes).
  4. Nearest places query: IsWithinDistance + OrderBy(Distance) + Take(10) (core trick 1). Validate lat −90..90, lon −180..180, radius capped at 50 km.
  5. Driver updates: GeoAddAsync("drivers:{city}", lon, lat, driverId) plus StringSetAsync($"hb:{driverId}", 1, TimeSpan.FromSeconds(15)).
  6. Nearby drivers: GeoSearchAsync(key, lon, lat, new GeoSearchCircle(3, GeoUnit.Kilometers), count: 20, order: Order.Ascending, options: GeoRadiusOptions.WithDistance), then drop members with no heartbeat (core trick 2).
  7. Cleanup: BackgroundService removes stale members with GeoRemoveAsync every 30 s.
  8. Live tracking: after a match, stream the driver's position to the rider through a SignalR group (Azure SignalR Service to scale).
  9. Cache place results: HybridCache keyed by rounded lat/lon + filters, 5-minute expiry (places rarely change).
  10. Deploy & protect privacy: Container Apps autoscaling on HTTP load; round coordinates in logs and traces; OpenTelemetry for write rate and search latency.

Core trick 1: nearest ATMs with EF Core + NetTopologySuite

builder.Services.AddDbContext<PlacesDb>(o =>
    o.UseSqlServer(cs, sql => sql.UseNetTopologySuite()));

// Nearest 10 ATMs within 2 km that accept deposits
var here = new Point(lon, lat) { SRID = 4326 };            // x = longitude, y = latitude!
var atms = await db.Atms
    .Where(a => a.AcceptsDeposits && a.Location.IsWithinDistance(here, 2_000))  // metres on geography
    .OrderBy(a => a.Location.Distance(here))
    .Take(10)
    .Select(a => new { a.Id, a.Name, Metres = a.Location.Distance(here) })
    .ToListAsync(ct);

Core trick 2: moving drivers with Redis GEO + heartbeat

// Driver app, every ~4 s
await db.GeoAddAsync("drivers:syd", lon, lat, driverId);
await db.StringSetAsync($"hb:{driverId}", 1, TimeSpan.FromSeconds(15));      // "still online"

// Rider: 20 closest drivers within 3 km, nearest first
var near = await db.GeoSearchAsync("drivers:syd", lon, lat,
    new GeoSearchCircle(3, GeoUnit.Kilometers),
    count: 20, order: Order.Ascending, options: GeoRadiusOptions.WithDistance);

var alive = await db.StringGetAsync(near.Select(n => (RedisKey)$"hb:{n.Member}").ToArray());
var result = near.Zip(alive)
    .Where(x => x.Second.HasValue)                                           // skip stale drivers
    .Select(x => new { id = x.First.Member.ToString(), km = x.First.Distance })
    .Take(5);

Testing scenarios

TypeScenarioExpected resultHow to test (.NET)
✅ Positive3 seeded ATMs, 2 within 2 kmThe 2 near ones, nearest first, with distancesWebApplicationFactory + Testcontainers MsSql (sample below)
✅ PositiveFilter "accepts deposits"Only matching ATMsIntegration test
✅ PositiveDriver reports, rider searchesDriver found with correct distance (± 1%)Testcontainers Redis
✅ PositiveDriver moves closerNew position and order on the next searchTwo updates, then search
❌ Negativelat = 91 or lon = 181400 with ProblemDetails[Theory] with bad coordinates
❌ NegativeRadius 500 km400 (max 50 km)[Theory]
❌ NegativeNothing within the radius200 with an empty list (not 404)Search in the ocean
❌ NegativeDriver stops sending updatesHidden once the heartbeat expiresConfigure TTL = 1 s in the test, wait 1.5 s (Redis time can't be faked)
⚠️ EdgeLatitude/longitude swapped in codeTest fails loudlyUse asymmetric real points (Sydney: lat −33.87, lon 151.21)
⚠️ EdgePoint exactly on the radiusIncluded (documented rule)Seed a point at a computed 2,000 m
⚠️ EdgeNear the 180° meridian (e.g. Fiji) or a city shard borderNeighbours on the other side foundSeed both sides; query both shard keys
⚠️ Edge1,000 drivers at one airportMax 20 returned, fastBulk GeoAddAsync, assert count and latency
⚠️ Edge50,000 updates/sRedis CPU < 70%, search p99 < 50 msk6 or NBomber (commercial licence) driving the update endpoint

Sample test: real distances, real order (SQL Server container supports geography)

[Fact]
public async Task Nearest_atms_are_sorted_by_real_distance()
{
    var ct = TestContext.Current.CancellationToken;
    await factory.SeedAtmsAsync(
        ("Town Hall",     151.2070, -33.8731),       // ~0.3 km from the user
        ("Circular Quay", 151.2108, -33.8615),       // ~1.6 km
        ("Parramatta",    151.0036, -33.8150));      // ~20 km: outside the radius

    var client = factory.CreateClient();
    var atms = await client.GetFromJsonAsync<AtmDto[]>(
        "/atms/near?lat=-33.8757&lon=151.2069&radiusM=2000", ct);

    Assert.Equal(new[] { "Town Hall", "Circular Quay" }, atms!.Select(a => a.Name));
    Assert.True(atms[0].Metres < atms[1].Metres);
}
The .NET stack for this use case
ComponentLibrary / APINotes
PlacesMicrosoft.EntityFrameworkCore.SqlServer.NetTopologySuitePoint, IsWithinDistance, Distance (metres on geography)
Places (PostGIS)Npgsql.EntityFrameworkCore.PostgreSQL.NetTopologySuiteSame LINQ, PostGIS underneath
Moving objectsStackExchange.Redis GeoAddAsync, GeoSearchAsyncGEOSEARCH needs Redis 6.2+
Live trackingbuilt-in SignalR; Microsoft.Azure.SignalROne group per trip
Managed alternativeAzure Maps (REST / Azure.Maps.Search)Geocoding addresses to coordinates

NetTopologySuite uses (x, y) = (longitude, latitude). new Point(lat, lon) is the most common proximity bug, and it silently returns wrong results.

  • Static places: spatial index in SQL/PostGIS. Moving objects: Redis GEO in memory.
  • GEO members don't expire; pair them with a heartbeat key and a cleanup job.
  • Shard by region, and query neighbours near borders.
  • Test with real, asymmetric coordinates to catch lat/lon swaps.

"Find the nearest ATM" and "find the nearest driver" sound the same. They are not. 📍

🏦 ATMs barely move: SQL geography + a spatial index + EF Core NetTopologySuite

🚗 Drivers move every 4 s: Redis GEOADD / GEOSEARCH in memory

💓 Add a heartbeat key with a TTL, or ghosts stay on the map

🗺️ Shard by city; check the neighbour shard near borders

🐛 #1 bug: Point(lat, lon) instead of Point(lon, lat)

Same question, two very different write patterns, two stores.

#SystemDesign #DotNet #Geospatial #Redis #Architecture

22.5 Large file upload & processing (chunked, resumable, scanned) Added

Let clients upload big files straight to blob storage in chunks they can resume, keep the bytes away from your API servers, and process a file only after a malware scan says it is clean.

Problem statement

Customers upload loan documents, statements, photos and videos, some of them several GB, often from a phone on a weak network. A single HTTP POST through the API ties up servers, fails at 90% and starts again from zero, and lets untrusted files reach back-office systems.

Functional

  • Upload files up to 5 GB with a progress bar
  • Resume after a network drop without starting over
  • Check type and size; scan for malware
  • Process clean files (OCR, thumbnails), then notify the user
  • Delete abandoned uploads automatically

Non-functional

  • Volume: 200k uploads/day, average 10 MB ≈ 2 TB/day; ~20 uploads/s at peak
  • Resume: a drop loses at most one 8 MiB block
  • Speed: 95% of files scanned and processed within 5 minutes
  • Availability: 99.9%; zone-redundant storage
  • Security: nothing is opened before a clean scan

Constraints & assumptions: Azure Blob Storage; browser and mobile clients; files are untrusted until scanned; managed identities everywhere (no storage keys in code); retention rules apply to stored documents.

Solution

Key decisionTrade-off
Direct to blob with a short, write-only SAS. The API never touches the bytes.Scales without app servers; you must validate size and type after upload and keep the SAS tight (one blob, 1 h, no read).
Block blobs as resumable chunks: Put Block, then Put Block List.Free resume (uncommitted blocks are kept for up to a week); the client must track block IDs.
tus protocol (tusdotnet) instead when you need a standard resumable protocol, ready clients (tus-js-client, Uppy) or non-Azure storage.Mature and open, but the bytes flow through your servers.
Scan before use: Defender for Storage on-upload scanning + Event Grid; only "No threats found" files move to the clean container.Per-GB cost and a short delay; blob index tags alone aren't tamper-proof, so act on events.
Event-driven processing with retries and a dead-letter queue.The UX is async: show "scanning..." and notify when ready.

Steps to implement in .NET 10

  1. Storage: containers incoming (untrusted) and clean. Enable Defender for Storage on-upload malware scanning and send results to an Event Grid custom topic. Add a lifecycle rule that deletes old blobs in incoming.
  2. Packages: Azure.Storage.Blobs, Azure.Identity (DefaultAzureCredential / managed identity), Microsoft.Extensions.Azure (AddAzureClients).
  3. Start an upload: validate name, declared size (≤ 5 GB) and allowed types; save an Upload row as Pending; return a user delegation SAS for one blob with Create + Write only, valid 1 hour (core trick 1).
  4. Client upload: browsers use @azure/storage-blob; .NET clients (MAUI, desktop, workers) use BlockBlobClient.StageBlockAsync per 8 MiB block and CommitBlockListAsync at the end, saving progress after each block (core trick 2).
  5. Alternative: tusdotnet with app.MapTus("/files", ...) and the community Azure store Xtensible.TusDotNet.Azure; raise Kestrel's MaxRequestBodySize (default ~30 MB) for the tus route.
  6. Scan result handler: Azure Functions isolated worker with Microsoft.Azure.Functions.Worker.Extensions.EventGrid ([EventGridTrigger]). Read scanResultType: clean → server-side copy to clean (StartCopyFromUriAsync) and mark Ready; malicious → delete or quarantine and alert; not scanned → keep blocked for review.
  7. Validate content: check the real size and the file's magic bytes (a renamed .exe is not a PDF) before processing.
  8. Process: send a Service Bus message (Azure.Messaging.ServiceBus) per clean file; the worker does OCR (Azure AI Document Intelligence) or thumbnails, idempotent by upload id, with a dead-letter queue.
  9. Notify & status: GET /uploads/{id} plus a SignalR or push notification when Ready or Rejected.
  10. Clean up & observe: a nightly job expires Pending uploads older than 24 h; alerts on "Malicious" results and dead-letter count; OpenTelemetry across API, Function and worker.

Core trick 1: a write-only, one-blob, 1-hour user delegation SAS

app.MapPost("/uploads", async (NewUpload req, BlobServiceClient blobs, TimeProvider clock, UploadsDb db) =>
{
    if (req.Size is <= 0 or > 5L * 1024 * 1024 * 1024) return Results.BadRequest("Max 5 GB");
    if (!AllowedTypes.Contains(req.ContentType)) return Results.BadRequest("File type not allowed");

    var id = Guid.CreateVersion7();
    var blob = blobs.GetBlobContainerClient("incoming").GetBlobClient(id.ToString());
    var now = clock.GetUtcNow();
    UserDelegationKey key = await blobs.GetUserDelegationKeyAsync(now.AddMinutes(-5), now.AddHours(1));

    var sas = new BlobSasBuilder
    {
        BlobContainerName = "incoming", BlobName = blob.Name, Resource = "b",
        StartsOn = now.AddMinutes(-5), ExpiresOn = now.AddHours(1)
    };
    sas.SetPermissions(BlobSasPermissions.Create | BlobSasPermissions.Write);    // no read, no list

    var url = new BlobUriBuilder(blob.Uri) { Sas = sas.ToSasQueryParameters(key, blobs.AccountName) }.ToUri();
    db.Uploads.Add(new Upload(id, req.FileName, req.Size, UploadStatus.Pending));
    await db.SaveChangesAsync();
    return Results.Ok(new { id, uploadUrl = url });
});

Core trick 2: resumable upload in 8 MiB blocks (.NET client)

var blob = new BlockBlobClient(uploadUrl);                       // SAS URL from the API
const int BlockSize = 8 * 1024 * 1024;
var ids = Enumerable.Range(0, progress.BlocksSent).Select(BlockId).ToList();
file.Position = (long)progress.BlocksSent * BlockSize;           // resume: skip blocks already sent
var buffer = new byte[BlockSize];

int n;
while ((n = await file.ReadAtLeastAsync(buffer, BlockSize, throwOnEndOfStream: false)) > 0)
{
    var id = BlockId(ids.Count);
    await blob.StageBlockAsync(id, new MemoryStream(buffer, 0, n));   // SDK retries transient errors
    ids.Add(id);
    await progress.SaveAsync(ids.Count);                         // remember after every block
}
await blob.CommitBlockListAsync(ids);                            // the blob appears only now

static string BlockId(int i) => Convert.ToBase64String(BitConverter.GetBytes(i));  // same length

Testing scenarios

TypeScenarioExpected resultHow to test (.NET)
✅ PositiveUpload a small PDF, commit, clean scan eventStatus Ready; file in cleanTestcontainers Azurite; post a sample Event Grid event to the handler
✅ PositiveUpload drops at 45%, then resumesFinal file's SHA-256 equals the originalAzurite + a "flaky" stream that throws mid-way (sample below)
✅ PositiveSAS lets the client write its blob201 on Put Block / Put Block ListIntegration test with the returned URL
✅ PositiveProcessing finishedUser notified once; status endpoint shows ReadyTest SignalR client or fake notifier
❌ NegativeDeclared size 6 GB, or type application/x-msdownload400; no SAS issued[Theory] on POST /uploads
❌ NegativeEICAR test file uploadedMalicious → quarantined, never processed, alert raisedIn a real Azure test account with Defender on (Azurite doesn't scan); locally, a "Malicious" sample event
❌ NegativeUse the SAS to read or list, or after it expired403Call download/list with the SAS; build a SAS with ExpiresOn in the past
❌ Negative.exe renamed to .pdfRejected by the magic-byte checkUnit test the validator with real file headers
⚠️ Edge0-byte file; exactly 5 GB400; accepted (boundary is inclusive)[Theory] on the validator (no real 5 GB upload needed)
⚠️ EdgeCommit lists a block that was never uploadedCommit fails (InvalidBlockList); blob unchangedAzurite integration test
⚠️ EdgeEvent Grid delivers the scan result twiceCopied and processed onceInvoke the handler twice with the same event
⚠️ EdgeScan result "Not scanned" or timed outFile stays blocked; flagged for review/retrySample event with scanResultType = "Not Scanned"
⚠️ EdgeUpload abandoned for 24 hPending row expired; blob removed by lifecycle ruleFakeTimeProvider.Advance(TimeSpan.FromHours(24)) on the cleanup job

Sample test: a resumed upload gives the identical file (Testcontainers.Azurite)

public sealed class UploadTests : IAsyncLifetime
{
    private readonly AzuriteContainer _azurite =
        new AzuriteBuilder("mcr.microsoft.com/azure-storage/azurite:3.37.0").Build();
    public async ValueTask InitializeAsync() => await _azurite.StartAsync();
    public ValueTask DisposeAsync() => _azurite.DisposeAsync();

    [Fact]
    public async Task Resumed_upload_produces_the_identical_file()
    {
        var ct = TestContext.Current.CancellationToken;
        var container = new BlobContainerClient(_azurite.GetConnectionString(), "incoming");
        await container.CreateIfNotExistsAsync(cancellationToken: ct);
        var blob = container.GetBlockBlobClient("big.bin");
        var data = RandomNumberGenerator.GetBytes(20 * 1024 * 1024);          // 20 MiB = 3 blocks

        var uploader = new ResumableUploader(blob, blockSize: 8 * 1024 * 1024);  // code under test
        await Assert.ThrowsAsync<IOException>(() =>
            uploader.UploadAsync(new FlakyStream(data, failAfterBytes: 9_000_000), ct));
        await uploader.UploadAsync(new MemoryStream(data), ct);               // resume

        var stored = (await blob.DownloadContentAsync(ct)).Value.Content.ToArray();
        Assert.Equal(SHA256.HashData(data), SHA256.HashData(stored));
    }
}
The .NET stack for this use case
ComponentLibrary / APINotes
Storage + SASAzure.Storage.Blobs (BlockBlobClient, BlobSasBuilder)User delegation SAS via Azure.Identity
Resumable (tus)tusdotnet; Azure store Xtensible.TusDotNet.Azure (community)MIT; bytes pass through your app
Malware scanDefender for Storage on-upload scanning (Azure feature, no package)Results via Event Grid, index tags, Log Analytics
Scan handlerMicrosoft.Azure.Functions.Worker.Extensions.EventGridIsolated worker; copy clean files server-side
ProcessingAzure.Messaging.ServiceBusRetries + dead-letter queue
TestsTestcontainers.Azurite, FakeTimeProviderAzurite emulates storage but does not scan

Defender scanning has a maximum file size, a per-account throughput limit and a scan timeout. Check the current limits for your file sizes, and treat "Not scanned" as "not safe".

  • Keep bytes off your API: hand out a short, write-only, one-blob SAS.
  • Block blobs give you resumable uploads for free: stage blocks, commit the list.
  • Untrusted until scanned: act on the Event Grid scan result, then move to a clean container.
  • Make every step idempotent; events and messages can arrive twice.

A 3 GB upload fails at 90% on a train. Your user starts again from zero. Again. 📤

🔑 API hands out a 1-hour, write-only SAS for ONE blob; bytes never touch your servers

🧱 Upload 8 MiB blocks, commit the list at the end; a drop costs one block, not the file

🛡️ Defender for Storage scans on upload; Event Grid tells you the result

✅ Only "No threats found" files move to the clean container for processing

🔁 Every handler is idempotent; events can arrive twice

Resumable, scalable and safe: three properties, one design.

#Azure #DotNet #SystemDesign #CloudSecurity #Architecture

22.6 Distributed job scheduler Added

"Design a service that runs thousands of scheduled jobs (cron and one-off) across several nodes, on time, and never with a double effect." The hard parts: only one node fires each job, and nothing is lost when a node dies.

Problem statement

A bank runs thousands of scheduled tasks: monthly statements, direct-debit files, reminder emails, data exports. Today they're Task Scheduler entries on one VM. If that VM dies, jobs silently don't run, and nobody can see what ran when.

Functional

  • Register jobs by API: cron, one-off at a time, or a fixed interval
  • Run on any healthy node; retry failures with backoff
  • Pause, resume and "run now"; full run history and status
  • Schedules in a named time zone (Australia/Sydney, with daylight saving)

Non-functional

  • 50,000 job definitions; peak 2,000 firings a minute (on the hour)
  • Fire within 5 s of the due time (p99)
  • A dead node's jobs picked up by another node within 30 s
  • Each firing has its business effect once; 99.9% available; 90 days of history

Constraints & assumptions:

Solution

DecisionChoiceTrade-off
EngineQuartz.NET 4 with a clustered SQL job store. A DB row lock lets exactly one node take each firingNo extra infrastructure, but the DB is the limit at very high firing rates (split into several named schedulers)
DeliveryAt-least-once + an idempotency key (job name + scheduled fire time)Simple and safe, but every job must check the key
Where work runsThe job only enqueues a message; workers do the heavy workScheduler stays light and fast; one more hop to monitor
TimeStore UTC; cron in a named time zone; TimeProvider in codeTestable, but DST gaps and misfires need an explicit policy
AlternativesFunctions timer trigger for a few fixed jobs; Hangfire for background jobs with a dashboardLess to run, but less control at 50k jobs (Hangfire Pro is commercial)

Steps to implement in .NET 10

  1. Create a Worker Service (dotnet new worker) and add Quartz 4.x plus Microsoft.Data.SqlClient. From 4.0, DI and hosting live in the core package; Quartz.Extensions.Hosting is now an empty forwarding package.
  2. Create the Quartz tables in Azure SQL from the scripts in the Quartz repo, run as a migration step in the pipeline.
  3. Configure AddQuartz with UsePersistentStore(s => { s.UseSqlServer(...); s.UseClustering(); }). Every node uses the same InstanceName and a unique instance id (GenerateInstanceId = true).
  4. Write each job as an IJob with [DisallowConcurrentExecution]. Build a run key from ScheduledFireTimeUtc and insert it into a JobRuns table with a unique index.
  5. Hand long work to Azure.Messaging.ServiceBus with MessageId = run key, so duplicate detection drops repeats.
  6. Add an admin API (minimal APIs + Entra ID roles) that calls IScheduler: ScheduleJob, PauseJob, TriggerJob.
  7. Wire OpenTelemetry (Quartz 4 emits spans and metrics) and health checks; alert on misfires and failed runs.
  8. Deploy 3+ replicas. Set WaitForJobsToComplete and HostOptions.ShutdownTimeout so a rolling deploy drains running jobs.
builder.AddQuartz(q =>
{
    q.ConfigureScheduler(o => { o.InstanceName = "bank-jobs"; o.GenerateInstanceId = true; });
    q.UsePersistentStore(store =>
    {
        store.UseSqlServer(builder.Configuration.GetConnectionString("Quartz")!);
        store.UseClustering();                      // DB row locks: one node per firing
    });
    q.ScheduleJob<StatementJob>(t => t
        .WithIdentity("monthly-statements")
        .WithCronSchedule("0 0 2 1 * ?", c => c.InTimeZone(
            TimeZoneInfo.FindSystemTimeZoneById("Australia/Sydney"))));
});
builder.AddQuartzHostedService(o => o.WaitForJobsToComplete = true);
[DisallowConcurrentExecution]
public sealed class StatementJob(JobRunStore runs, ServiceBusSender sender, TimeProvider clock) : IJob
{
    public async ValueTask Execute(IJobExecutionContext context, CancellationToken ct = default)
    {
        // A retried or recovered firing has the same key, so the effect happens once
        string runKey = $"{context.JobDetail.Key}:{context.ScheduledFireTimeUtc:O}";
        if (!await runs.TryStartAsync(runKey, clock.GetUtcNow(), ct)) return;   // unique index on RunKey

        await sender.SendMessageAsync(new ServiceBusMessage(runKey) { MessageId = runKey }, ct);
        await runs.CompleteAsync(runKey, clock.GetUtcNow(), ct);
    }
}

Testing scenarios

TypeScenarioExpected resultHow to test (.NET)
✅ PositiveMonthly cron at 02:00 Sydney timeNext fire time is 02:00 local on the 1st, stored as UTCxUnit v3: CronExpression with the Sydney time zone, assert GetNextValidTimeAfter
✅ PositiveOne-off job scheduled by API for +2 minutesFires within 5 s of due; history row CompletedWebApplicationFactory + Testcontainers.MsSql
✅ Positive"Run now" from the admin APIRuns at once; audit row records who triggered itWebApplicationFactory with a test auth handler
✅ Positive3 nodes, 1,000 jobs due in the same minuteEach fires once; work spreads across nodesThree hosts in one test against Testcontainers.MsSql; count JobRuns
❌ NegativeJob's downstream call failsRetried with backoff, then Failed + alertPolly chaos AddChaosFault in the job's HttpClient pipeline
❌ NegativeDatabase unreachable for 30 sNodes retry; no duplicate firings; they catch up afterTestcontainers: PauseAsync / UnpauseAsync the SQL container
❌ NegativeInvalid cron expression sent to the API400 Problem Details; nothing storedWebApplicationFactory
❌ NegativeUser without the Scheduler role calls "run now"403; job not triggeredWebApplicationFactory with a test auth handler
⚠️ EdgeNode dies in the middle of a jobAnother node recovers it after the check-in timeout; run key stops a double effectStop one host without draining; assert one completed run
⚠️ EdgeDST start (first Sunday in October): 02:30 doesn't exist in SydneyJob fires once per the misfire policy, not skipped silentlyCronExpression across the DST date; FakeTimeProvider for the job's own logic
⚠️ EdgeThe same firing is executed twiceSecond run is a no-opUnit test: call Execute twice with the same fire time (sample below)
⚠️ EdgeWhole scheduler down for 2 hoursMissed firings run once (or are skipped) per policy, not 120 timesStop all hosts, move past several due times, restart; count runs
public class StatementJobTests
{
    [Fact]
    public async Task Same_firing_twice_sends_one_message()
    {
        var clock = new FakeTimeProvider(new DateTimeOffset(2026, 11, 1, 2, 0, 0, TimeSpan.FromHours(11)));
        var runs = new InMemoryJobRunStore();               // unique RunKey, like the SQL index
        var sender = new FakeServiceBusSender();             // subclass of ServiceBusSender that records messages
        var job = new StatementJob(runs, sender, clock);
        var context = TestContexts.ForFiring("monthly-statements", clock.GetUtcNow());   // your helper

        await job.Execute(context, TestContext.Current.CancellationToken);
        await job.Execute(context, TestContext.Current.CancellationToken);   // retry or recovery

        Assert.Single(sender.Sent);
    }
}
The .NET stack for this design
ComponentLibrary / APINotes
SchedulerQuartz 4.x (clustered ADO.NET job store)4.x targets net10.0; DI, hosting and OpenTelemetry built in
Simpler optionsFunctions timer trigger (Microsoft.Azure.Functions.Worker.Extensions.Timer); Hangfire.Core check licenceTimer triggers run as a singleton. Hangfire core is LGPL; Hangfire Pro is commercial
Leader electionDistributedLock.Azure (blob lease) or DistributedLock.SqlServerFor your own "tick" loop when you don't want Quartz
Hand-offAzure.Messaging.ServiceBusDuplicate detection on MessageId = run key
Timebuilt-in TimeProvider, PeriodicTimer; Microsoft.Extensions.TimeProvider.TestingFakeTimeProvider moves time in tests

Quartz 4.0 needs a database schema migration from 3.x, and daylight-saving firing behaviour changed. Read the migration guide before upgrading a live cluster.

  • Exactly-once firing is a myth. Use at-least-once firing + idempotent jobs.
  • Clustered Quartz uses DB row locks: same instance name on every node, a unique instance id per node.
  • The scheduler fires; a queue and workers do the heavy work.
  • Store UTC, schedule in a named time zone, and test DST and misfires.

"Run this job every night at 2 am" sounds easy. Until you have 3 servers. ⏰

🔒 Only one node may fire each job: clustered Quartz uses a DB row lock

💀 A node dies mid-job? Another one recovers it after a check-in timeout

🔁 That means a job can run twice, so make it idempotent (job + fire time = key)

📨 The scheduler only enqueues; workers do the heavy lifting

🌏 Daylight saving: 2:30 am doesn't exist one Sunday in October. Test it.

At-least-once + idempotency beats chasing exactly-once. How do you schedule jobs today?

#SystemDesign #DotNet #Quartz #Architecture #InterviewPrep

22.7 Web crawler Added

"Design a crawler that fetches a billion public pages a month, politely, without fetching the same page twice." A giant queue (the frontier) plus three rules: obey robots.txt, limit each host, remember what you've seen.

Problem statement

A search or knowledge product needs fresh copies of public web pages. The crawler starts from seed URLs, follows links, and keeps pages up to date, without overloading any website or getting stuck in endless link loops.

Functional

  • Start from seeds; fetch HTML; extract links and text
  • Follow links within allowed domains; obey robots.txt and Crawl-delay
  • Recrawl pages that change; skip unchanged ones
  • Store raw HTML and extracted text for indexing

Non-functional

  • 1B pages/month ≈ 400 pages/s average, 1,000/s peak
  • Default 1 request/s per host
  • 100 KB average page ≈ 100 TB/month raw (≈25 TB compressed)
  • Seen set of 10B URLs: a Bloom filter at 1% false positives ≈ 12 GB

Constraints & assumptions:

Solution

DecisionChoiceTrade-off
FrontierKafka topic keyed by host, so one worker owns each host; per-host Channel<T> queues inside the workerPoliteness stays local and simple; very big hosts make partitions uneven
PolitenessPartitionedRateLimiter keyed by host (token bucket 1/s, or the site's Crawl-delay)Safe for websites; big sites take longer to crawl
Seen URLsNormalise the URL, then a Redis Bloom filter12 GB instead of hundreds of GB; ~1% of new URLs are wrongly skipped
Duplicate contentSHA-256 of normalised text for exact copies; SimHash for near copiesSaves storage and index noise; extra CPU per page
FreshnessAdaptive recrawl interval per page + conditional GET (ETag / Last-Modified)Hot pages stay fresh; needs a per-URL metadata store

Steps to implement in .NET 10

  1. Create a Worker Service. Register a named HttpClient with IHttpClientFactory, AddStandardResilienceHandler() (Microsoft.Extensions.Http.Resilience), automatic decompression, a 10 s timeout and a User-Agent with a contact URL.
  2. Build the frontier: a Kafka topic urls keyed by host (Confluent.Kafka); inside each worker, bounded System.Threading.Channels per host.
  3. robots.txt: fetch once per host, cache with HybridCache, and parse with TurnerSoftware.RobotsExclusionTools (or a small RFC 9309 parser of your own).
  4. Politeness: a PartitionedRateLimiter<Uri> keyed by host, plus a cap of 1–2 concurrent requests per host.
  5. Parse with AngleSharp: a[href] links, canonical link, meta robots (noindex, nofollow) and body text. Resolve relative links with new Uri(baseUri, href) and normalise them.
  6. Seen set: StackExchange.Redis sending BF.ADD / BF.EXISTS (Redis 8 includes Bloom filters; on managed Redis, check your tier supports them).
  7. Store compressed HTML in Blob Storage (Azure.Storage.Blobs) keyed by URL hash, and per-URL metadata (ETag, last crawl, content hash, next due) in Cosmos DB or Table Storage.
  8. Emit OpenTelemetry metrics (pages/s, errors per host, queue depth). Run N workers on AKS with KEDA scaling on Kafka consumer lag.
// One token bucket per host: 1 request per second, the rest wait in a queue
PartitionedRateLimiter<Uri> perHost = PartitionedRateLimiter.Create<Uri, string>(uri =>
    RateLimitPartition.GetTokenBucketLimiter(uri.Host, _ => new TokenBucketRateLimiterOptions
    {
        TokenLimit = 1, TokensPerPeriod = 1, ReplenishmentPeriod = TimeSpan.FromSeconds(1),
        QueueLimit = 1_000, QueueProcessingOrder = QueueProcessingOrder.OldestFirst,
        AutoReplenishment = true
    }));

async Task CrawlAsync(Uri url, HttpClient http, CancellationToken ct)
{
    using RateLimitLease lease = await perHost.AcquireAsync(url, 1, ct);
    if (!lease.IsAcquired) { await frontier.RequeueLaterAsync(url, ct); return; }

    using var response = await http.GetAsync(url, HttpCompletionOption.ResponseHeadersRead, ct);
    if (!response.IsSuccessStatusCode || response.Content.Headers.ContentType?.MediaType != "text/html") return;

    var doc = await new HtmlParser().ParseDocumentAsync(await response.Content.ReadAsStreamAsync(ct), ct);
    foreach (var href in doc.QuerySelectorAll("a[href]").Select(a => a.GetAttribute("href")))
        if (Uri.TryCreate(url, href, out var link) && link.Scheme is "http" or "https")
            await frontier.OfferAsync(Normalise(link), ct);      // Bloom filter check inside
}

Testing scenarios

TypeScenarioExpected resultHow to test (.NET)
✅ PositivePage with 10 links on an allowed hostPage stored; 10 new URLs offered to the frontierxUnit v3 + WireMock.Net fake site; in-memory frontier
✅ Positiverobots.txt disallows /private/private/* is never requestedWireMock.Net request log (sample below)
✅ PositiveSite sets Crawl-delay: 2Requests to that host are at least 2 s apartWireMock.Net log timestamps over a short crawl
✅ PositiveRecrawl of an unchanged page304 Not Modified; no parse, no new blobWireMock.Net returns 304 when If-None-Match matches
❌ NegativeHost returns 429 / 503 with Retry-AfterThat host pauses; URL retried later; other hosts carry onWireMock.Net faults; assert other hosts' requests continue
❌ NegativeServer accepts the connection but never answersTimes out after 10 s; the host slot is freedWireMock.Net WithDelay; resilience timeout
❌ NegativeLink to a 500 MB PDFSkipped from the Content-Type / Content-Length headers, body not downloadedWireMock.Net + byte counter on the handler
❌ NegativeBroken HTML, javascript: and mailto: linksParser doesn't crash; invalid links droppedxUnit theory over HTML fixture files with AngleSharp
⚠️ EdgeSpider trap: endless calendar "next month" links, session ids in URLsMax depth and max URLs per host stop it; normaliser strips session paramsUnit tests on Normalise + a generated infinite site in WireMock.Net
⚠️ EdgeSame page via http/https, trailing slash, utm_ params, canonical tagStored onceUnit tests on URL normalisation + content hash
⚠️ EdgeRedirect loop A → B → AStops after 5 redirects; URL marked as errorWireMock.Net redirect responses
⚠️ EdgeBloom filter false positivesMeasured rate stays near 1%; sitemaps help recover missed URLsTestcontainers.Redis with a Redis 8 image; add 1M URLs, probe 1M new ones
[Fact]
public async Task Disallowed_paths_are_never_fetched()
{
    using var site = WireMockServer.Start();
    site.Given(Request.Create().WithPath("/robots.txt"))
        .RespondWith(Response.Create().WithBody("User-agent: *\nDisallow: /private"));
    site.Given(Request.Create().WithPath("/"))
        .RespondWith(Response.Create().WithHeader("Content-Type", "text/html")
            .WithBody("<a href='/private/a'>x</a> <a href='/public/b'>y</a>"));
    site.Given(Request.Create().WithPath("/public/b")).RespondWith(Response.Create().WithStatusCode(200));

    var crawler = TestCrawler.Create();                      // real pipeline, in-memory frontier
    await crawler.RunAsync(new Uri(site.Url! + "/"), maxPages: 10, TestContext.Current.CancellationToken);

    Assert.DoesNotContain(site.LogEntries, e => e.RequestMessage.Path.StartsWith("/private"));
    Assert.Contains(site.LogEntries, e => e.RequestMessage.Path == "/public/b");
}
The .NET stack for this design
ComponentLibrary / APINotes
HTTPbuilt-in IHttpClientFactory + Microsoft.Extensions.Http.ResilienceTimeouts, retries with jitter, circuit breaker per host
Frontierbuilt-in System.Threading.Channels + Confluent.KafkaKafka key = host, so one worker owns each host
Politenessbuilt-in System.Threading.RateLimiting; TurnerSoftware.RobotsExclusionToolsThe robots.txt library is small and last released in 2022; a tiny own parser is also fine
HTML parsingAngleSharpStandards-based HTML5 parser with CSS selectors
Seen setStackExchange.Redis (BF.ADD, BF.EXISTS)Bloom filters are built into Redis 8
StorageAzure.Storage.Blobs + Microsoft.Azure.CosmosRaw pages in blobs; per-URL metadata in Cosmos DB
  • Do the maths: 1B pages/month ≈ 400 pages/s and ≈ 100 TB of raw HTML.
  • Politeness is per host: shard the frontier by host and rate-limit each host.
  • Normalise URLs before the Bloom filter, and hash content to catch copies under different URLs.
  • Defend against spider traps with depth limits and per-host URL caps.

A web crawler is just a loop: fetch, parse, enqueue. At 400 pages a second, the loop is the easy part. 🕷️

🤝 Be polite: one request per second per host, and obey robots.txt

🗂️ Shard the queue by host, so one worker owns each website

🧠 Remember 10B URLs in about 12 GB with a Bloom filter

🪤 Spider traps (endless calendars, session ids) need depth and URL caps

♻️ Conditional GET (ETag) means unchanged pages cost almost nothing

Which part would you deep-dive in an interview?

#SystemDesign #DotNet #WebCrawler #Architecture #InterviewPrep

22.8 Real-time fraud detection Added

"Score every payment for fraud in under 100 ms, block the obvious ones, and send the doubtful ones to analysts." A fast inline decision, backed by a payment stream that keeps the features fresh.

Problem statement

A bank processes card and real-time account-to-account payments. Real-time payments can't be pulled back once they settle, so the fraud decision must happen before the money moves, without making checkout feel slow.

Functional

  • Score each payment: approve, step-up (extra check), decline, or approve + review
  • Rules (velocity, new payee, impossible travel, known mule accounts) + an ML model
  • Analysts work cases; their outcomes become training labels
  • Change rules and thresholds without a deploy

Non-functional

  • Decision p99 under 100 ms; 3,000 payments/s at peak (500 average)
  • 99.99% available; a safe fallback if scoring fails
  • Every decision audited with rule and model versions
  • Few false declines: a wrong decline costs a customer

Constraints & assumptions:

Solution

DecisionChoiceTrade-off
Inline vs streamSynchronous gRPC decision; a stream consumer computes long-window features and casesLatency stays bounded; stream features can lag a few seconds (hot counters are updated inline)
Rules + modelHard rules override (sanctions, known mule); model score bands map to actionsExplainable and safe; two things to tune and govern
FeaturesRedis: INCR + expiry for fixed windows, sorted sets for sliding windowsSub-millisecond reads; memory cost, so every key needs a TTL
Model servingIn-process: ML.NET PredictionEnginePool or ONNX Runtime, hot-reloadedAbout 1 ms, no network hop; the model must fit in memory
Failure mode80 ms budget, then a rules-only fallback decisionPayments keep flowing in an outage; a little more fraud risk while degraded

Steps to implement in .NET 10

  1. Create an ASP.NET Core gRPC service (Grpc.AspNetCore) with a Score(PaymentRequest) contract agreed with the payment switch.
  2. Features with StackExchange.Redis: velocity counters via StringIncrementAsync + KeyExpireAsync in one transaction; sliding windows via SortedSetAddAsync, SortedSetRemoveRangeByScoreAsync, SortedSetLengthAsync.
  3. Rules as plain C# classes, with thresholds in IOptionsMonitor<FraudRules> fed by Azure App Configuration, so changes apply without a deploy.
  4. Train offline (ML.NET with Microsoft.ML.LightGbm, or Python to ONNX). Serve with Microsoft.Extensions.ML: AddPredictionEnginePool<TxnFeatures, FraudScore>().FromFile("fraud", path, watchForChanges: true), or Microsoft.ML.OnnxRuntime.
  5. Publish every decision to Event Hubs (Azure.Messaging.EventHubs) or Kafka, partition key = card token. An EventProcessorClient consumer updates long-window features and opens cases.
  6. Case management: review-band decisions create a case; the analyst's outcome (fraud / genuine) is stored as a label for retraining.
  7. Wrap scoring in a Microsoft.Extensions.Resilience pipeline: 80 ms timeout + fallback to rules-only. Record latency histograms, decision mix and model version with OpenTelemetry.
  8. Deploy across zones (min 3 replicas, zone-redundant Redis). Run a new model in shadow mode (score, don't act) before switching it on.
public sealed class FraudScorer(
    IDatabase redis, PredictionEnginePool<TxnFeatures, FraudScore> model, RuleSet rules)
{
    public async Task<Decision> ScoreAsync(Payment p)
    {
        string key = $"vel:{p.CardToken}:10m";
        ITransaction tx = redis.CreateTransaction();
        Task<long> count10m = tx.StringIncrementAsync(key);
        _ = tx.KeyExpireAsync(key, TimeSpan.FromMinutes(10), ExpireWhen.HasNoExpiry);
        await tx.ExecuteAsync();

        var f = new TxnFeatures { Amount = (float)p.Amount, Count10m = await count10m,
                                  NewPayee = p.IsNewPayee, MerchantRisk = p.MerchantRisk };

        if (rules.HardBlock(p, f) is { } reason) return Decision.Decline(reason);   // e.g. known mule account

        float score = model.Predict("fraud", f).Probability;
        return score switch
        {
            >= 0.90f => Decision.Decline("model"),
            >= 0.60f => Decision.ApproveAndReview(score),     // async case for an analyst
            _        => Decision.Approve(score)
        };
    }
}

Testing scenarios

TypeScenarioExpected resultHow to test (.NET)
✅ PositiveNormal payment from a known deviceApprove in under 100 msWebApplicationFactory + gRPC client + Testcontainers.Redis
✅ Positive6th payment on one card within 10 minutesVelocity rule steps the payment upTestcontainers.Redis (sample below)
✅ PositiveScore in the review bandApproved; decision event published; case createdTestcontainers.Kafka: consume the topic and assert the event
✅ PositiveNew model file deployedPool reloads without a restart; decisions carry the new model versionIntegration test replaces the file watched by FromFile(..., watchForChanges: true)
❌ NegativeRedis unreachableRules-only fallback within the budget; alert raisedPause the Redis container, or Polly chaos AddChaosFault
❌ NegativeCorrupt model filePrevious model stays loaded; error loggedUnit/integration test with a broken zip
❌ NegativeMalformed request (negative amount, missing token)gRPC InvalidArgument; nothing storedWebApplicationFactory + gRPC client
❌ NegativeThe same event delivered twice from the streamLong-window features not double counted (dedupe by payment id)Testcontainers.Kafka: publish twice, assert one increment
⚠️ EdgeBurst across a fixed-window boundary (5 at 10:09, 5 at 10:11)Sliding-window rule still catches 10 in 10 minutesSorted-set timestamps from TimeProvider; FakeTimeProvider moves time
⚠️ EdgeCard used in Sydney, then London 30 minutes laterImpossible-travel rule flags itxUnit theory with location fixtures
⚠️ EdgeSale-day peak of 3,000 payments/sp99 under 100 ms, no timeoutsk6 or Azure Load Testing; or NBomber commercial licence
⚠️ EdgeScore exactly 0.60 or 0.90Falls in the documented band (bounds inclusive)xUnit [Theory] with boundary values
public sealed class VelocityTests : IAsyncLifetime
{
    private readonly RedisContainer _redis = new RedisBuilder().WithImage("redis:8").Build();
    public async ValueTask InitializeAsync() => await _redis.StartAsync();
    public async ValueTask DisposeAsync() => await _redis.DisposeAsync();

    [Fact]
    public async Task Sixth_payment_in_ten_minutes_is_stepped_up()
    {
        var mux = await ConnectionMultiplexer.ConnectAsync(_redis.GetConnectionString());
        var scorer = TestScorers.RulesOnly(mux.GetDatabase());     // real Redis, stub model
        var card = $"tok_{Guid.NewGuid():N}";

        Decision last = null!;
        for (int i = 0; i < 6; i++)
            last = await scorer.ScoreAsync(TestPayments.For(card, amount: 49.95m));

        Assert.Equal(Outcome.StepUp, last.Outcome);
    }
}
The .NET stack for this design
ComponentLibrary / APINotes
Decision APIGrpc.AspNetCoreBinary, HTTP/2, low overhead for the payment switch
FeaturesStackExchange.RedisCounters with TTL; sorted sets for sliding windows
ModelMicrosoft.Extensions.ML (PredictionEnginePool) + Microsoft.ML.LightGbm; or Microsoft.ML.OnnxRuntimeThread-safe pooled engines; hot reload from file or URI
StreamAzure.Messaging.EventHubs.Processor or Confluent.KafkaPartition by card token to keep one card's events in order
FallbackMicrosoft.Extensions.ResilienceTimeout + fallback strategy around scoring

Watch for training-serving skew: compute features with the same code (or the same definitions) offline and online, or the model sees different numbers in production.

  • Real-time payments can't be recalled, so decide before the money moves.
  • Keep the inline path tiny: Redis features + in-process model + rules, under 100 ms.
  • Hard rules override the model; score bands map to approve / review / step-up / decline.
  • Plan the fallback: a rules-only decision beats blocking every payment.

You have 100 milliseconds to decide if a payment is fraud. Go. ⏱️

⚡ Inline: Redis velocity counters + an in-process model, about 3 ms of work

🚫 Hard rules first: known mule accounts never reach the model

🎚️ Score bands: approve, step-up, review, decline

🌊 The payment stream updates long-window features and analyst cases in the background

🛟 Scoring down? Fall back to rules. Never block every payment.

Real-time payments can't be pulled back. That's why this has to be fast and right.

#FraudDetection #Payments #SystemDesign #DotNet #Banking

22.9 Customer onboarding (KYC) workflow Added

"Design digital onboarding: verify a new customer's identity, screen them, ask a human when needed, open the account, and prove every step later." A long-running workflow that may take three minutes or three weeks.

Problem statement

Customers open an account in the app. Under Australia's AML/CTF Act (reformed obligations from 31 March 2026), a bank must complete initial customer due diligence before it provides the service, and keep the records for 7 years after the relationship ends. Most applicants should be done in minutes; a few need an analyst.

Functional

  • Capture details, an ID document photo and a selfie
  • Extract document fields; verify the document with the issuer (e.g. the government Document Verification Service, via a gateway or identity provider)
  • Liveness and face match; sanctions and PEP screening; risk rating
  • Auto-approve low risk; analyst review for medium and high risk
  • Open the account and issue a card; the customer can resume later

Non-functional

  • 80% of applicants auto-decided in under 3 minutes
  • 10,000 applications a day; peak 50 starts a minute
  • Workflows survive restarts and can wait up to 30 days
  • Every step in an append-only audit trail, kept 7 years after the relationship ends
  • PII encrypted; documents stored in an Australian region

Constraints & assumptions:

Solution

DecisionChoiceTrade-off
Workflow engineDurable Functions / Durable Task SDK on the Durable Task Scheduler (Temporal is a good alternative)State, timers and retries survive restarts; orchestrator code must be deterministic (no DateTime.Now, no I/O)
Human stepWait for an external event, racing a durable timerWaits for days at no cost; you must design reminders and expiry
Side effectsSaga with compensations (close account if card issue fails)No distributed transaction; compensations must be idempotent
AuditSeparate append-only store (Azure SQL ledger table or an event store), not the workflow historyTamper-evident and queryable for 7 years; one more write per step
VendorsAdapters behind interfaces, idempotency keys, standard resilienceSafe retries and swappable vendors; more code up front

Steps to implement in .NET 10

  1. Create an isolated-worker Azure Functions app with Microsoft.Azure.Functions.Worker.Extensions.DurableTask, backed by the Durable Task Scheduler. (In containers, use Microsoft.DurableTask.Worker.AzureManaged instead.)
  2. API: POST /applications saves the application, uploads images to private Blob Storage (Australian region), and calls ScheduleNewOrchestrationInstanceAsync with instance id = application id.
  3. Activities: extract fields with Azure.AI.DocumentIntelligence (prebuilt-idDocument covers passports and US licences; use a custom model or your identity provider for Australian licences); verify, screen and rate risk through typed HttpClients with AddStandardResilienceHandler().
  4. Orchestrator: chain the activities with TaskOptions.FromRetryPolicy; for review, race WaitForExternalEvent<ReviewDecision> against CreateTimer.
  5. Analyst UI/API calls RaiseEventAsync(instanceId, "ReviewDecision", decision), protected by an Entra ID app role; require a second approver for high risk.
  6. Saga: OpenAccount then IssueCard; on failure, run CloseAccount in the catch. Each activity is idempotent (key = application id + step).
  7. Audit: every activity appends to an Azure SQL table created with LEDGER = ON (APPEND_ONLY = ON). Never update or delete rows.
  8. Deploy; show stuck instances on a dashboard (custom status); purge old workflow history once the audit trail holds what you need.
public sealed class KycOrchestrator : TaskOrchestrator<Application, KycOutcome>
{
    public override async Task<KycOutcome> RunAsync(TaskOrchestrationContext ctx, Application app)
    {
        var retry = TaskOptions.FromRetryPolicy(new RetryPolicy(5, TimeSpan.FromSeconds(10), backoffCoefficient: 2));
        IdResult id = await ctx.CallActivityAsync<IdResult>("VerifyIdentity", app, retry);
        Risk risk   = await ctx.CallActivityAsync<Risk>("ScreenAndRate", new ScreenInput(app, id), retry);
        if (risk.Level != RiskLevel.Low)
        {
            using var cts = new CancellationTokenSource();
            Task timeout = ctx.CreateTimer(ctx.CurrentUtcDateTime.AddDays(5), cts.Token);
            Task<ReviewDecision> review = ctx.WaitForExternalEvent<ReviewDecision>("ReviewDecision");
            if (await Task.WhenAny(review, timeout) == timeout) return KycOutcome.Expired();
            cts.Cancel();   if (!review.Result.Approved) return KycOutcome.Declined(review.Result.Reason);
        }
        string accountId = await ctx.CallActivityAsync<string>("OpenAccount", app, retry);
        try { await ctx.CallActivityAsync("IssueCard", accountId, retry); }
        catch (TaskFailedException)
        {
            await ctx.CallActivityAsync("CloseAccount", accountId, retry);      // compensate
            return KycOutcome.Failed("card");
        }
        return KycOutcome.Approved(accountId);
    }
}

Testing scenarios

TypeScenarioExpected resultHow to test (.NET)
✅ PositiveLow-risk applicant, every check passesApproved and account opened without a human; audit has an entry per stepDurableTaskTestHost with fake activities
✅ PositiveMedium risk; analyst approves on day 2Workflow waits, then completes ApprovedDurableTaskTestHost + RaiseEventAsync (sample below)
✅ PositiveCustomer closes the app and resumes laterSame instance continues; no second workflowWebApplicationFactory: second POST returns current status
✅ PositivePassport photo processedName, date of birth, number and expiry extracted with good confidenceContract test against Document Intelligence with synthetic sample passports
❌ NegativeDocument expired or details don't match the issuerDeclined with a reason the customer can fixFake verify activity returns a mismatch
❌ NegativeSanctions matchWorkflow stops and escalates to financial crime; no account; no wording that could tip off the customerDurableTaskTestHost + assert the customer message template
❌ NegativeIdentity vendor returns 503 for 10 minutesActivity retries with backoff, then the workflow continuesWireMock.Net vendor stub, or Polly chaos AddChaosFault
❌ NegativeUser without the analyst role tries to approve403; no event raisedWebApplicationFactory with a test auth handler
⚠️ EdgeAnalyst never respondsExpires after 5 days; customer told to restartMoq a TaskOrchestrationContext whose timer wins; or Temporal's WorkflowEnvironment.StartTimeSkippingAsync if you chose Temporal
⚠️ EdgeCard issue fails after the account was openedCloseAccount runs once; audit shows both stepsDurableTaskTestHost with a failing IssueCard fake
⚠️ EdgeAnalyst double-clicks Approve (event raised twice)Second event is ignored; one accountDurableTaskTestHost: raise the event twice
⚠️ EdgeNew code deployed while 2,000 workflows are mid-flightIn-flight instances keep running correctlyOrchestration versioning; run old and new side by side in a staging slot
[Fact]
public async Task Medium_risk_waits_for_analyst_then_opens_account()
{
    await using var host = await DurableTaskTestHost.StartAsync(tasks =>
    {
        tasks.AddOrchestrator<KycOrchestrator>();
        tasks.AddActivityFunc<Application, IdResult>("VerifyIdentity", (_, app) => IdResult.Verified(app.Name));
        tasks.AddActivityFunc<ScreenInput, Risk>("ScreenAndRate", (_, _) => new Risk(RiskLevel.Medium));
        tasks.AddActivityFunc<Application, string>("OpenAccount", (_, _) => "ACC-001");
        tasks.AddActivityFunc<string, bool>("IssueCard", (_, _) => true);
    });

    string id = await host.Client.ScheduleNewOrchestrationInstanceAsync(nameof(KycOrchestrator), TestApps.Medium());
    await host.Client.RaiseEventAsync(id, "ReviewDecision", new ReviewDecision(Approved: true, Reason: null));

    var result = await host.Client.WaitForInstanceCompletionAsync(id, getInputsAndOutputs: true);
    Assert.Equal(OrchestrationRuntimeStatus.Completed, result.RuntimeStatus);
    Assert.Equal("ACC-001", result.ReadOutputAs<KycOutcome>()!.AccountId);
}
The .NET stack for this design
ComponentLibrary / APINotes
WorkflowMicrosoft.Azure.Functions.Worker.Extensions.DurableTask or Microsoft.DurableTask.Worker.AzureManaged; alternative TemporalioTimers, external events, retries and sagas as plain C#
ID documentsAzure.AI.DocumentIntelligence (prebuilt-idDocument)Passports worldwide and US licences; other IDs need a custom model or provider
Vendor callsbuilt-in IHttpClientFactory + Microsoft.Extensions.Http.ResilienceTyped clients per vendor; idempotency key header
Audit trailEF Core 10 / Microsoft.Data.SqlClient on an Azure SQL append-only ledger tableDatabase-verified tamper evidence
TestingMicrosoft.DurableTask.InProcessTestHost preview; Temporalio.Testing (inside Temporalio)In-memory engine; Temporal's test server can skip time

Tipping off: if you report a suspicious matter, the customer must not be told. Keep decline messages neutral and route screening hits to the financial crime team.

  • Long-running + human steps = a durable workflow, not a status column and a cron job.
  • Human approval is an external event racing a timer; design the reminder and the expiry.
  • Use compensations for side effects you can't roll back, and make them idempotent.
  • Audit lives in its own append-only store, kept 7 years after the relationship ends.

Opening a bank account takes 3 minutes for most people and 3 weeks for a few. Your design has to handle both. 🏦

🔎 Verify the document with the issuer, then liveness and face match

🚩 Screen for sanctions and PEPs, then rate the risk

👩‍💼 Medium risk? The workflow sleeps until an analyst decides, or 5 days pass

↩️ Card failed after the account opened? Compensate: close the account

📜 Every step in an append-only audit trail, kept for 7 years

Durable workflows turn "weeks of waiting" into a few lines of C#.

#KYC #Banking #DurableFunctions #DotNet #SystemDesign

22.10 Enterprise RAG assistant Added

"Let staff ask questions over policies, procedures and wikis, get grounded answers with citations, and never see a document they aren't allowed to open." RAG at enterprise scale: the hard parts are permissions, freshness and proving quality.

Problem statement

A company has 2 million documents across SharePoint, Confluence and file shares, and 20,000 staff. People waste time hunting for the right policy. The assistant must answer from these documents only, show its sources, and respect each document's permissions.

Functional

  • Ingest new, changed and deleted documents, and permission changes
  • Chunk, enrich and embed; hybrid retrieval trimmed to the user's groups
  • Answers with clickable citations; "I don't know" when nothing fits
  • Thumbs up/down feedback; an evaluation dashboard for the owners

Non-functional

  • 2M docs ≈ 20M chunks; at 1,536 dimensions × 4 bytes ≈ 120 GB of vectors (before quantisation)
  • Peak 20 questions/s; first token under 2 s; p95 full answer under 8 s
  • Permission removals reflected within 15 minutes; content changes within 1 hour
  • Groundedness at least 4/5 on the golden set; data stays in region

Constraints & assumptions:

Solution

DecisionChoiceTrade-off
PermissionsStore allowed group ids on every chunk; filter with search.in on every query (native ACL support in AI Search is in preview)Unauthorised text never reaches the model; you must sync permission changes quickly
RetrievalHybrid (keywords + vectors) + semantic ranker (17.10)Finds codes and acronyms; adds a few hundred ms and cost
ChunkingStructure-aware, with metadata (17.5)Better citations and filters; more ingestion work
CitationsModel must cite source numbers; the API checks each one was retrievedTrustworthy answers; sometimes a refusal or a retry
QualityGolden-set evals in CI + online feedback (17.14); content safety on input and outputCatches regressions from model, prompt or chunking changes; LLM-judge cost

Steps to implement in .NET 10

  1. Create an ASP.NET Core API with Entra ID auth (Microsoft.Identity.Web). Register IChatClient and IEmbeddingGenerator from Microsoft.Extensions.AI, using Azure.AI.OpenAI + Microsoft.Extensions.AI.OpenAI (AsIChatClient()).
  2. Design the Azure AI Search index (Azure.Search.Documents): content, contentVector (HNSW), title, url, headingPath, lastModified, and group_ids as a filterable, non-retrievable string collection; add a semantic configuration and a vectorizer.
  3. Build the ingestion worker: source change feeds (e.g. Microsoft Graph delta for SharePoint); Azure.AI.DocumentIntelligence layout to Markdown; chunk with Microsoft.Extensions.DataIngestion preview or your own splitter; embed; MergeOrUpload and delete.
  4. Query path: read the user's group ids from the token (use Microsoft Graph when the token has too many groups), build the search.in filter, run hybrid + semantic search.
  5. Prompt: system rules ("answer only from the sources, cite [n], say you don't know"), numbered sources; stream with GetStreamingResponseAsync; validate citations before showing them.
  6. Guardrails: Azure.AI.ContentSafety on output, Prompt Shields (REST) on the question and documents, per-user rate limits (Microsoft.AspNetCore.RateLimiting), no prompts or PII in logs.
  7. Evaluation: a golden Q&A set run as xUnit tests with Microsoft.Extensions.AI.Evaluation.Quality, cached and reported with Microsoft.Extensions.AI.Evaluation.Reporting; fail the PR if scores drop.
  8. Deploy with private endpoints for Search and the model; trace tokens, latency and "I don't know" rate with OpenTelemetry; review low-rated answers weekly.
app.MapPost("/ask", async (AskRequest req, ClaimsPrincipal user, SearchClient search,
                           IChatClient chat, CancellationToken ct) =>
{
    string groups = string.Join(",", user.FindAll("groups").Select(c => c.Value));   // Graph on overage
    var options = new SearchOptions
    {
        Size = 8,
        Filter = $"group_ids/any(g: search.in(g, '{groups}', ','))",            // security trimming
        QueryType = SearchQueryType.Semantic,
        SemanticSearch = new() { SemanticConfigurationName = "default" },
        VectorSearch = new() { Queries = { new VectorizableTextQuery(req.Question)
            { KNearestNeighborsCount = 50, Fields = { "contentVector" } } } }
    };
    var hits = new List<Chunk>();
    await foreach (var r in (await search.SearchAsync<Chunk>(req.Question, options, ct)).Value.GetResultsAsync())
        hits.Add(r.Document);
    if (hits.Count == 0) return Results.Ok(Answer.DontKnow);
    string sources = string.Join("\n", hits.Select((h, i) => $"[{i + 1}] {h.Title}: {h.Content}"));
    ChatResponse reply = await chat.GetResponseAsync(
        [new(ChatRole.System, Prompts.GroundedAnswer),
         new(ChatRole.User, $"Sources:\n{sources}\n\nQuestion: {req.Question}")],
        cancellationToken: ct);
    return Results.Ok(Answer.WithValidatedCitations(reply.Text, hits));   // drop or retry unknown [n]
});

Testing scenarios

TypeScenarioExpected resultHow to test (.NET)
✅ PositiveQuestion answered by a policy the user can readGrounded answer with the right citationMicrosoft.Extensions.AI.Evaluation.Quality GroundednessEvaluator on the golden set (sample below)
✅ PositivePolicy updated in SharePointNew answer within 1 hour; old chunks goneIntegration test: ingestion worker against a dedicated test search index
✅ PositiveQuery with a code ("form KYC-07")Exact document in the top 3RetrievalEvaluator or a hit-rate@k check over labelled queries
✅ PositiveStreaming answerFirst token under 2 sWebApplicationFactory with a fake streaming IChatClient; k6 in staging for real latency
❌ NegativeUser asks about a document they can't access (HR salary bands)No text from it is retrieved; answer is "I don't know"Two test users in different groups against the test index; assert retrieved chunk ids
❌ NegativeA document says "ignore previous instructions and..."Flagged by Prompt Shields; answer unaffectedRed-team set with Microsoft.Extensions.AI.Evaluation.Safety preview (indirect attack)
❌ NegativeModel endpoint returns 429 or times outRetry with backoff, then a friendly error; no half answerFake IChatClient that throws; Polly chaos strategies
❌ NegativeAbusive or harmful questionBlocked with a polite messageAzure.AI.ContentSafety test set; safety evaluators
⚠️ EdgeNothing relevant in the index"I don't know" + who to contact; no invented answerUnanswerable questions in the golden set; assert refusal
⚠️ EdgeUser in hundreds of groups (token group overage)Groups fetched from Graph; filter still worksUnit test with an overage claim and a fake Graph client
⚠️ EdgeModel cites [9] when only 8 sources were sentCitation removed or answer regeneratedUnit test on Answer.WithValidatedCitations
⚠️ EdgeA user loses access to a documentNot retrievable after the next permission sync (within 15 minutes)Integration: change the ACL, run sync, query as that user
[Theory]
[MemberData(nameof(GoldenSet.Questions), MemberType = typeof(GoldenSet))]
public async Task Answers_are_grounded_in_retrieved_sources(string question)
{
    var (answer, sources, judge) = await RagTestHarness.AskAsync(question);   // real pipeline, test index

    IEvaluator evaluator = new GroundednessEvaluator();
    EvaluationResult result = await evaluator.EvaluateAsync(
        [new ChatMessage(ChatRole.User, question)], answer,
        new ChatConfiguration(judge),
        additionalContext: [new GroundednessEvaluatorContext(sources)]);

    NumericMetric groundedness = result.Get<NumericMetric>(GroundednessEvaluator.GroundednessMetricName);
    Assert.True(groundedness.Value >= 4, $"Groundedness {groundedness.Value} for: {question}");
}
The .NET stack for this design
ComponentLibrary / APINotes
ModelsMicrosoft.Extensions.AI + Azure.AI.OpenAI / Microsoft.Extensions.AI.OpenAIIChatClient and IEmbeddingGenerator; swap providers without code changes
RetrievalAzure.Search.Documents; or Microsoft.Extensions.VectorData.Abstractions + CommunityToolkit.VectorData.AzureAISearchHybrid, semantic ranker and OData security filters. The old Microsoft.SemanticKernel.Connectors.* vector packages are deprecated
IngestionMicrosoft.Extensions.DataIngestion preview + Azure.AI.DocumentIntelligenceReaders, chunkers, enrichers and a vector store writer
GuardrailsAzure.AI.ContentSafetyHarm categories and blocklists; Prompt Shields over REST
EvaluationMicrosoft.Extensions.AI.Evaluation.Quality, .Reporting; .Safety previewGroundedness, relevance, retrieval; cached runs and HTML reports in CI
IdentityMicrosoft.Identity.WebValidated token; group claims or a Graph lookup

Never let the model decide who may see what. Trim results in the search query, from the validated token, before any text reaches the prompt.

  • Security trimming happens at retrieval, from the user's token, never in the prompt.
  • Ingestion must handle deletes and permission changes, not just new documents.
  • Hybrid + semantic ranking + structure-aware chunks give the best retrieval baseline.
  • Prove quality with a golden set in CI, and validate every citation.

The hardest part of an enterprise RAG assistant isn't the LLM. It's permissions. 🔐

👥 Store allowed group ids on every chunk and filter every query by the user's groups

🔄 Sync deletes and permission changes, not just new documents

🔎 Hybrid search + semantic ranker: codes and meaning both matter

📎 Make the model cite sources, then check every citation was really retrieved

📊 Golden-set evals in CI catch quality drops before users do

If the assistant can see it, assume the user can too. Trim before you prompt.

#RAG #GenAI #DotNet #EnterpriseAI #SystemDesign

⚡ Quick-fire Q&A

Flash sale: how do you guarantee you never oversell?

Make "check stock and take stock" one atomic operation: a Redis Lua script, or SQL UPDATE ... SET Stock = Stock - @qty WHERE Stock >= @qty and check the rows affected. Add idempotency keys so retries don't buy twice, and prove it with a parallel test (Task.WhenAll) against a real Redis or SQL container.

Booking: how do temporary holds expire without a timer per seat?

Store an expiry time on the hold and treat any hold with HoldExpiresAt <= now as free in every query (lazy expiry). A periodic sweeper only tidies up and pushes "free again" events. In Redis, SET key value NX PX 600000 gives the same thing in one command. Test it with FakeTimeProvider.Advance.

Autocomplete: why not run LIKE 'abc%' on the database?

At thousands of requests per second it is too slow and it can't rank by popularity cheaply. Precompute the top 10 per prefix (Redis sorted sets or a trie) from search logs, cache short prefixes at the edge, and debounce on the client. Use Azure AI Search suggesters if you need typo tolerance.

Proximity: why use two different stores for ATMs and drivers?

ATMs rarely change and need rich filters, so a SQL/PostGIS spatial index fits. Drivers move every few seconds (tens of thousands of writes per second), so an in-memory Redis GEO index fits, with a heartbeat TTL to hide drivers who went offline.

File upload: how do you make a 5 GB upload resumable and safe?

Give the client a short, write-only SAS for one blob, upload 8 MiB blocks (Put Block) and commit the list at the end (Put Block List), so a drop only re-sends one block. Scan with Defender for Storage, react to the Event Grid result, and process only files marked "No threats found".

Job scheduler: how do you stop two nodes running the same job?

Let one shared store decide. Clustered Quartz.NET takes a database row lock when it acquires a trigger, so only one node fires each firing. Because a crashed node's job is recovered and may run again, also make every job idempotent with a key such as job name + scheduled fire time.

Web crawler: how do you stay polite and avoid re-fetching pages?

Shard the frontier by host so one worker owns each site, and rate-limit per host (about 1 request/s, or the site's Crawl-delay) after checking robots.txt. Normalise URLs and check a Bloom filter before enqueueing, and use conditional GET (ETag / Last-Modified) on recrawls.

Fraud detection: how do you decide in under 100 ms?

Keep the inline path tiny: read and update velocity counters in Redis, run hard rules, then score with an in-process model (ML.NET or ONNX Runtime). Publish the decision to a stream and do everything else (long-window features, cases, retraining labels) asynchronously. If scoring times out, fall back to rules only.

KYC onboarding: how do you model a human approval that can take days?

Use a durable workflow (Durable Functions / Durable Task, or Temporal). The orchestrator waits for an external "ReviewDecision" event racing a durable timer, so it costs nothing while it waits and expires cleanly. Side effects like opening an account use saga compensations, and every step writes to an append-only audit store.

Enterprise RAG: how do you stop the assistant leaking documents?

Trim at retrieval. Store the allowed group ids on every chunk, read the user's groups from their validated Entra token, and filter every search with them (e.g. search.in in Azure AI Search). Unauthorised text never reaches the prompt, and permission changes are synced into the index quickly.

← 21 · Design Case Studies