Ten real problems, solved the .NET 10 way. Each one follows the same path: the problem statement (with numbers), the solution as an animated flow plus the key trade-offs, the steps to build it with the exact APIs and NuGet packages, and the test scenarios: positive, negative and edge cases, with the .NET tool for each.
These are implementation-focused. The classic architecture case studies (URL shortener, rate limiter, chat, notifications, feed, ledger) live in 21 Design Case Studies.
22.1 Flash-sale order & inventory without overselling Added
Thousands of people press "Buy" in the same second for a few thousand items. Take the stock in one atomic step, answer fast, and do the slow work (payment, order rows) in the background.
Problem statement
A retailer (or a bank's rewards store) sells 5,000 limited items at 12:00 sharp. About 200,000 shoppers arrive in the first minute. A normal "read stock, then write stock" flow sells the same last item to many people, and the database melts under the spike.
Functional
Show remaining stock (can be a few seconds old)
Reserve and place an order; pay within 10 minutes
Limit: 2 items per customer
Release stock if payment fails or times out
A double-click or retry must not buy twice
Non-functional
Peak: 50,000 requests/s on "Buy" for about 2 minutes
Latency: reserve answer p99 < 200 ms
Correctness: 0 oversold items, ever
Availability: 99.95% during the sale window
Durability: no accepted order is lost
Constraints & assumptions: one Azure region; SQL Server holds the real stock and orders; Redis is available; the payment provider is external and slow (1–3 s); the sale time is known in advance, so you can pre-scale.
Solution
Key decision
Trade-off
Gate in Redis, truth in SQL. A Lua script checks and decrements stock in one atomic step in memory.
Very fast, but Redis and SQL can drift, so SQL keeps a final WHERE Stock >= @qty guard and a job reconciles.
Reserve now, write later. The API answers 202 and a queue absorbs the spike.
The shopper sees "pending" for a moment, and you need a timeout to release unpaid stock.
Idempotency key per checkout. Retries return the first result.
One extra Redis key per order (kept 24 h).
Admission control. Rate limit or a waiting room in front of "Buy".
Some shoppers wait or get 429, but the system stays up.
Pre-scale before 12:00. Set minimum replicas ahead of time.
You pay for idle capacity for a few minutes; autoscale reacts too slowly for a 2-minute spike.
Steps to implement in .NET 10
Set up:dotnet new webapi (minimal APIs) plus an Aspire 13 AppHost with Aspire.Hosting.Redis, Aspire.Hosting.SqlServer and Aspire.Hosting.Azure.ServiceBus, so Redis, SQL and the Service Bus emulator start locally with one command.
Model: EF Core 10 (Microsoft.EntityFrameworkCore.SqlServer) with Product (Id, Stock, [Timestamp] byte[] RowVersion), Order (PK = orderId) and an outbox table. RowVersion makes admin edits (price, stock top-ups) fail with DbUpdateConcurrencyException instead of silently overwriting.
Load stock into Redis before the sale: StringSetAsync("stock:{sku}", 5000) with StackExchange.Redis.
Reserve endpoint: run the Lua script with IDatabase.ScriptEvaluateAsync (core trick 1). It also adds the order to a sorted set scored by its expiry time, so the sweeper can find unpaid reservations.
Idempotency: require an Idempotency-Key header (an IETF draft name, widely used). Check it inside the same Lua script, so "seen before?" and "take stock" can't race.
PublishOrderReserved with Azure.Messaging.ServiceBus. Set MessageId = orderId and turn on duplicate detection on the queue.
Order worker: a BackgroundService with ServiceBusProcessor writes the order using the SQL guard (core trick 2), in one transaction with the outbox row.
Admission control: built-in Microsoft.AspNetCore.RateLimiting (AddRateLimiter, token bucket per client, 429 + Retry-After), plus Azure Front Door WAF rate rules at the edge.
Release expired reservations: a BackgroundService using new PeriodicTimer(TimeSpan.FromSeconds(15), timeProvider) reads expired entries (SortedSetRangeByScoreAsync up to "now"), returns their stock and removes them. Inject TimeProvider so tests can fake time.
Deploy & observe: Azure Container Apps with min replicas raised before the sale, Azure Managed Redis, Azure.Monitor.OpenTelemetry.AspNetCore for traces and metrics (reserved, sold out, released). Run the load test a day before.
Core trick 1: atomic reserve + idempotency in one Lua script
const string ReserveLua = """
-- KEYS: 1 stock, 2 customer count, 3 idempotency key, 4 reservations (sorted set)
if redis.call('EXISTS', KEYS[3]) == 1 then return -2 end -- seen before
local qty, limit = tonumber(ARGV[1]), tonumber(ARGV[2])
local stock = tonumber(redis.call('GET', KEYS[1]) or '0')
if stock < qty then return -1 end -- sold out
if tonumber(redis.call('GET', KEYS[2]) or '0') + qty > limit then return -3 end
redis.call('DECRBY', KEYS[1], qty)
redis.call('INCRBY', KEYS[2], qty)
redis.call('SET', KEYS[3], ARGV[3], 'EX', 86400) -- remember orderId 24 h
redis.call('ZADD', KEYS[4], ARGV[4], ARGV[3]) -- score = expiry time
return stock - qty
""";
var tag = "{sku-123}"; // hash tag: all keys land on the same cluster slot
var left = (long)await db.ScriptEvaluateAsync(ReserveLua,
[$"stock:{tag}", $"cust:{tag}:{customerId}", $"idem:{tag}:{idemKey}", $"resv:{tag}"],
[qty, 2, orderId.ToString(), clock.GetUtcNow().AddMinutes(10).ToUnixTimeSeconds()]);
return left switch
{
-1 => Results.Conflict("Sold out"),
-2 => Results.Accepted($"/orders/{await db.StringGetAsync($"idem:{tag}:{idemKey}")}"),
-3 => Results.UnprocessableEntity("Limit is 2 per customer"),
_ => Results.Accepted($"/orders/{orderId}")
};
Core trick 2: the SQL guard in the worker (EF Core ExecuteUpdateAsync)
await using var tx = await db.Database.BeginTransactionAsync(ct);
if (await db.Orders.AnyAsync(o => o.Id == msg.OrderId, ct)) return; // redelivered message
var rows = await db.Products
.Where(p => p.Id == msg.ProductId && p.Stock >= msg.Qty) // final guard
.ExecuteUpdateAsync(s => s.SetProperty(p => p.Stock, p => p.Stock - msg.Qty), ct);
if (rows == 0) // Redis and SQL disagreed
{
await tx.RollbackAsync(ct);
await ReleaseInRedisAsync(msg, ct);
return;
}
db.Orders.Add(Order.AwaitingPayment(msg.OrderId, msg.CustomerId, msg.ProductId, msg.Qty));
await db.SaveChangesAsync(ct); // order + outbox row together
await tx.CommitAsync(ct);
Testing scenarios
Type
Scenario
Expected result
How to test (.NET)
✅ Positive
Buy 1 when stock is 10
202 + orderId; Redis stock 9; order row "AwaitingPayment"
xunit.v3, Microsoft.AspNetCore.Mvc.Testing, Testcontainers.Redis; NBombercommercial licence or k6
NBomber is free only for personal use
Never "read stock, then write stock" in two calls. Whatever store you use, the check and the decrement must be one atomic operation.
Check-and-decrement in one atomic step: a Redis Lua script or UPDATE ... WHERE Stock >= @qty.
Answer 202 fast; a queue and a worker do the slow part.
Idempotency key + reservation expiry turn retries and abandoned carts into non-events.
Prove it with a Task.WhenAll race test, not with hope.
5,000 items. 200,000 shoppers. 12:00 sharp. How do you sell exactly 5,000? 🛒
⚛️ Check and take stock in ONE atomic step (Redis Lua or a conditional SQL UPDATE)
📨 Answer 202 in milliseconds, let a queue write the orders
🔑 Idempotency keys: a double-click never buys twice
⏳ Unpaid after 10 minutes? The stock goes back automatically
🧪 Test it: 500 parallel requests for 100 items must give exactly 100 orders
Overselling is a design bug, not bad luck.
#SystemDesign #DotNet #Redis #Ecommerce #Testing
22.2 Seat / appointment booking with temporary holds Added
Let a user hold a seat or time slot for a few minutes while they pay, block everyone else, and release it automatically if they walk away.
Problem statement
Concert tickets, flight seats and branch appointments all have the same shape: a limited set of named slots. Users need a short, exclusive hold while they fill in details or pay. Without holds, two people pay for the same seat; with holds that never expire, seats look "sold out" when they are not.
Functional
See free, held and booked seats/slots
Hold 1–6 seats for 10 minutes, all or nothing
Confirm (after payment) or cancel a hold
Holds release by themselves when time runs out
Show a countdown; other screens update live
Non-functional
Peak: 2,000 hold requests/s at a big on-sale (appointments: ~50/s)
Latency: hold p99 < 300 ms; seat map p95 < 100 ms
Correctness: zero double bookings
Availability: 99.9%
Freshness: seat map may be 2–3 s stale; hold/confirm never
Constraints & assumptions: SQL Server is the source of truth; up to 60,000 seats per event; app servers are NTP-synced (seconds of skew don't matter for a 10-minute hold); all times are stored in UTC (DateTimeOffset).
Solution
Key decision
Trade-off
Hold columns on the seat row + a conditional UPDATE.WHERE BookedBy IS NULL AND (HoldExpiresAt IS NULL OR HoldExpiresAt <= @now).
One source of truth and transactional with booking; SQL row locks become the bottleneck at very big on-sales.
Lazy expiry. An expired hold counts as free in every query; no exact-time job is needed.
Expired values stay in the row until a sweeper or the next hold overwrites them.
All-or-nothing multi-seat holds in one transaction.
One taken seat fails the whole request; the UI must suggest alternatives.
Redis SET NX PX in front for huge on-sales (plus a waiting room).
Much faster, but now two stores must agree; SQL still does the final confirm.
10-minute hold, one extension.
Longer holds mean fewer failed payments but more "fake sold out" seats.
Steps to implement in .NET 10
Set up: minimal API + Aspire AppHost (Aspire.Hosting.SqlServer); EF Core 10 with Microsoft.EntityFrameworkCore.SqlServer.
Model:Seat (Id, EventId, HeldBy, HoldExpiresAt as DateTimeOffset?, BookedBy, [Timestamp] RowVersion), index on (EventId, Id). A Booking row per confirmed payment with a unique PaymentId.
Time: register builder.Services.AddSingleton(TimeProvider.System) and never call DateTime.UtcNow directly. Tests swap in FakeTimeProvider.
Hold endpoint: one ExecuteUpdateAsync inside a transaction; if fewer rows changed than seats asked for, roll back (core trick 1). Reject duplicate seat IDs and more than 6 seats up front (400).
Confirm endpoint:UPDATE ... SET BookedBy = @me WHERE HeldBy = @me AND HoldExpiresAt > @now, idempotent on PaymentId. If the hold expired but the seat is still free, book it; if someone else has it, trigger an automatic refund.
Seat map reads:HybridCache (Microsoft.Extensions.Caching.Hybrid) with a 2 s expiry and a tag per event; call RemoveByTagAsync after holds change.
Live updates: ASP.NET Core SignalR (built-in), one group per event: push "held", "booked", "free again". Use Azure SignalR Service (Microsoft.Azure.SignalR) to scale out.
Sweeper:BackgroundService with new PeriodicTimer(TimeSpan.FromSeconds(30), timeProvider) clears expired holds and pushes "free again" events.
Big on-sale mode: Redis hold first with StringSetAsync(key, user, TimeSpan.FromMinutes(10), When.NotExists) (core trick 2), plus a queue-based waiting room; SQL confirm stays the final check.
Deploy: App Service or Container Apps, Azure SQL (read replica for seat-map reads if needed), OpenTelemetry metrics for holds created, expired and confirmed.
Core trick 1: all-or-nothing hold with a conditional UPDATE
app.MapPost("/events/{eventId:int}/holds", async (int eventId, HoldRequest req, BookingDb db,
TimeProvider clock, ClaimsPrincipal user, CancellationToken ct) =>
{
var now = clock.GetUtcNow();
var me = user.FindFirstValue(ClaimTypes.NameIdentifier)!;
await using var tx = await db.Database.BeginTransactionAsync(ct);
var held = await db.Seats
.Where(s => s.EventId == eventId && req.SeatIds.Contains(s.Id) && s.BookedBy == null
&& (s.HoldExpiresAt == null || s.HoldExpiresAt <= now)) // free or expired
.ExecuteUpdateAsync(s => s
.SetProperty(x => x.HeldBy, me)
.SetProperty(x => x.HoldExpiresAt, now.AddMinutes(10)), ct);
if (held != req.SeatIds.Length) // someone else got at least one seat
{
await tx.RollbackAsync(ct); // all or nothing
return Results.Conflict("One or more seats are no longer free");
}
await tx.CommitAsync(ct);
return Results.Ok(new { expiresAt = now.AddMinutes(10) });
});
Core trick 2: Redis hold with expiry, and a safe release
// SET hold:7:A3 alice NX PX 600000 -> only the first caller wins
bool got = await redis.StringSetAsync($"hold:{eventId}:{seatId}", userId,
TimeSpan.FromMinutes(10), When.NotExists);
if (!got) return Results.Conflict("Seat is held by someone else");
// Release only if the hold is still yours (compare-and-delete in one step)
await redis.ScriptEvaluateAsync(
"if redis.call('GET', KEYS[1]) == ARGV[1] then return redis.call('DEL', KEYS[1]) end return 0",
[$"hold:{eventId}:{seatId}"], [userId]);
Confirm after the hold expired and the seat was taken
410 Gone; refund started
Advance fake time; second user holds; first confirms
❌ Negative
Confirm someone else's hold
403; seats unchanged
Integration test
❌ Negative
7 seats, or duplicate seat IDs
400 validation error
Unit test the validator; Verify snapshot of ProblemDetails
⚠️ Edge
50 users race for the same seat
Exactly 1 gets 200, 49 get 409
Task.WhenAll against a real SQL container (not the in-memory provider)
⚠️ Edge
Request at exactly HoldExpiresAt
Seat counts as free (<=), matching the docs
Advance fake time by exactly 10:00
⚠️ Edge
Overlapping requests A1+A2 vs A2+A3
One wins both seats; the other gets none (no half holds)
Parallel requests; assert no user holds just one seat
⚠️ Edge
Payment callback arrives twice
One booking (unique PaymentId)
Replay the callback
⚠️ Edge
DB times out during hold
500/503 and no half-held seats (transaction rolled back)
Polly chaos latency/fault strategy, or pause the container
Sample test: expiry without real waiting (Microsoft.Extensions.TimeProvider.Testing)
[Fact]
public async Task Hold_expires_after_ten_minutes()
{
var ct = TestContext.Current.CancellationToken;
var clock = new FakeTimeProvider(DateTimeOffset.Parse("2026-10-16T09:00:00Z"));
await using var app = factory.WithWebHostBuilder(b =>
b.ConfigureTestServices(s => s.AddSingleton<TimeProvider>(clock)));
var alice = app.CreateClientFor("alice"); // helper: test auth handler sets the user
var bob = app.CreateClientFor("bob");
var seat12 = new { seatIds = new[] { 12 } };
var first = await alice.PostAsJsonAsync("/events/7/holds", seat12, ct);
Assert.Equal(HttpStatusCode.OK, first.StatusCode);
var tooEarly = await bob.PostAsJsonAsync("/events/7/holds", seat12, ct);
Assert.Equal(HttpStatusCode.Conflict, tooEarly.StatusCode);
clock.Advance(TimeSpan.FromMinutes(10)); // ten minutes pass instantly
var later = await bob.PostAsJsonAsync("/events/7/holds", seat12, ct);
Assert.Equal(HttpStatusCode.OK, later.StatusCode);
}
built-in SignalR; Microsoft.Azure.SignalR to scale
Group per event
Test concurrency against a real database (Testcontainers). The EF Core in-memory provider has no row locks, so race tests pass there and fail in production.
A hold is "mine until time X". A conditional UPDATE (or SET NX PX) grants it atomically.
Make expiry lazy: an expired hold is free in every query; the sweeper only tidies up.
Multi-seat holds are all or nothing, in one transaction.
Inject TimeProvider so expiry tests take milliseconds, not minutes.
Two people. One seat. Both pressed "Pay". Who gets it? 🎟️
🔒 Give a 10-minute hold with one conditional UPDATE (or Redis SET NX PX)
⏳ Let holds expire lazily: an expired hold is simply "free"
🧩 Multi-seat holds are all or nothing
✅ Confirm re-checks the hold; late payments get an automatic refund
🧪 .NET tip: inject TimeProvider, then FakeTimeProvider.Advance(10 min) in tests
Suggest the best 5–10 completions while the user types. Precompute the answers per prefix, cache hard, and send fewer requests from the client.
Problem statement
A large store or banking app has a search box: products, merchants, help articles. Every keystroke could be a request, so a naive "LIKE 'hom%'" query on the main database is slow and expensive. Users expect suggestions to appear as fast as they type.
Functional
Top 10 suggestions for a prefix, most popular first
Availability: 99.9% (if it fails, search still works)
Freshness: minutes are fine
Constraints & assumptions: ~5M distinct terms from search logs; English first; prefixes capped at 10 characters for the precomputed lists; Azure hosting with a CDN in front.
Solution
Key decision
Trade-off
Precompute top 10 per prefix (a trie flattened into one Redis sorted set per prefix).
Reads are one call and already ranked; costs memory (a few GB) and a rebuild job.
Client debounce (200–300 ms) + cancel stale calls.
Cuts requests several times over; adds a small delay before the first suggestion.
Cache short prefixes at the edge (60 s). One- and two-letter prefixes are the hottest.
Suggestions can be up to a minute old.
Batch rebuild from logs, written to a new version and switched with one pointer key.
Trending terms appear after the next batch, not instantly; readers never see half-built data.
Managed search (Azure AI Search suggester) when you need typo tolerance and filters.
Less code, fuzzy matching built in; higher cost and latency than a Redis lookup.
Steps to implement in .NET 10
Set up: minimal API; Redis via Aspire.StackExchange.Redis (client integration) or plain StackExchange.Redis.
Log searches: send each final query as an event to Event Hubs (Azure.Messaging.EventHubs). Never log personal data in the term.
Aggregate: a timer job (Azure Functions timer trigger, or a BackgroundService) counts term frequency with time decay, and drops blocked terms.
Build the index: for each term and each prefix up to 10 chars, SortedSetAddAsync then trim with SortedSetRemoveRangeByRankAsync(key, 0, -11) to keep the top 10 (core trick 1). Use batching/pipelining; write to ac:v43:*, then set ac:current = v43.
Normalise once, everywhere:Trim(), ToLowerInvariant(), remove accents (string.Normalize(NormalizationForm.FormD) and drop non-spacing marks). Same function in the indexer and the API.
Suggest endpoint: validate 1–50 chars, read SortedSetRangeByRankAsync(key, 0, 9, Order.Descending). For prefixes longer than 10 chars, read the 10-char list and filter with StartsWith.
Cache: built-in output caching (AddOutputCache, .CacheOutput(p => p.Expire(...).SetVaryByQuery("q"))) and a Cache-Control: public, max-age=60 header for the CDN (Azure Front Door).
Protect: built-in rate limiter per IP; if Redis fails, return an empty list (200), not an error.
Client: debounce 250 ms and cancel the previous request (AbortController in JS, a new CancellationTokenSource per keystroke in Blazor/MAUI).
Optional managed option: Azure AI Search (Azure.Search.Documents) with a SearchSuggester on the index and SearchClient.AutocompleteAsync / SuggestAsync with UseFuzzyMatching = true (core trick 2).
Core trick 1: top 10 per prefix in Redis sorted sets
// Indexer: put a term into the top-10 list of each of its prefixes
async Task IndexTermAsync(IDatabase db, string version, string term, double score)
{
var t = Normalise(term);
for (int len = 1; len <= Math.Min(10, t.Length); len++)
{
var key = $"ac:{version}:{t[..len]}";
await db.SortedSetAddAsync(key, t, score);
await db.SortedSetRemoveRangeByRankAsync(key, 0, -11); // keep only the top 10
}
}
// API: one Redis call per uncached keystroke
app.MapGet("/suggest", async (string q, IConnectionMultiplexer mux) =>
{
var norm = Normalise(q);
if (norm.Length is 0 or > 50) return Results.BadRequest();
var db = mux.GetDatabase();
var version = await db.StringGetAsync("ac:current"); // switched after each rebuild
var hits = await db.SortedSetRangeByRankAsync($"ac:{version}:{norm[..Math.Min(10, norm.Length)]}",
0, 9, Order.Descending);
return Results.Ok(hits.Select(h => h.ToString()).Where(h => h.StartsWith(norm)));
}).CacheOutput(p => p.Expire(TimeSpan.FromSeconds(60)).SetVaryByQuery("q"));
Core trick 2: Azure AI Search with typo tolerance
// Index definition includes: Suggesters = { new SearchSuggester("sg", "name") }
var response = await searchClient.AutocompleteAsync("hom", "sg", new AutocompleteOptions
{
Mode = AutocompleteMode.OneTermWithContext,
UseFuzzyMatching = true, // "hmoe" still finds "home"
Size = 8
});
var suggestions = response.Value.Results.Select(r => r.Text);
Testing scenarios
Type
Scenario
Expected result
How to test (.NET)
✅ Positive
"hom" with seeded popular terms
Highest score first, max 10 items
WebApplicationFactory + Testcontainers Redis; Verify snapshot of the JSON
✅ Positive
"HOM" and " hom " (case, spaces)
Same result as "hom"
xUnit [Theory] with [InlineData]
✅ Positive
New trending term after a rebuild
Appears once the pointer switches
Run the indexer in the test, then query
✅ Positive
Second identical request
Served from output cache (Redis not called)
Count calls with a Redis wrapper or OpenTelemetry in-memory exporter
❌ Negative
Empty q or 51+ chars
400
[Theory] integration test
❌ Negative
Blocked/offensive term in the logs
Never suggested
Unit test the aggregator's filter
❌ Negative
Redis down
200 with an empty list, fast (no 500, no hang)
Stop the container, or Polly chaos fault/latency; assert response < 200 ms
❌ Negative
Special characters (*, %, \, emoji)
Treated as plain text; no error
[Theory] with odd inputs
⚠️ Edge
Accents: "café" vs "cafe"
Both return the same list
Unit test Normalise on both sides
⚠️ Edge
Prefix longer than 10 chars
Uses the 10-char list, filtered by full prefix
Integration test with a 15-char query
⚠️ Edge
Query during a rebuild
Old or new version, never a mix or empty
Run indexer and queries in parallel; assert every response is non-empty
⚠️ Edge
User types 6 letters in 300 ms
1 request sent (debounce), older calls cancelled
Playwright for .NET (Microsoft.Playwright) counting network calls
⚠️ Edge
15,000 req/s peak
Server p99 < 20 ms; cache hit rate > 80%
k6 or NBomber (commercial licence) against a staging slot
Sample test: ranking snapshot with Verify (Verify.XunitV3)
public sealed class SuggestTests(SuggestApiFactory factory) : IClassFixture<SuggestApiFactory>
{
[Fact]
public async Task Returns_top_terms_by_popularity()
{
var ct = TestContext.Current.CancellationToken;
await factory.IndexAsync(("home loan", 900), ("home insurance", 700),
("homepage", 50), ("holiday", 400)); // runs the real indexer
var client = factory.CreateClient();
var result = await client.GetFromJsonAsync<string[]>("/suggest?q=HOM", ct);
await Verify(result); // snapshot: ["home loan", "home insurance", "homepage"]
}
}
.NET 10 The .NET stack for this use case
Component
Library / API
Notes
Prefix index
StackExchange.Redis sorted sets
SortedSetRangeByRankAsync; SortedSetRangeByValueAsync for plain A–Z prefix lists
Managed option
Azure.Search.Documents
Suggesters, AutocompleteAsync, fuzzy matching
Caching
built-in Output caching; Microsoft.AspNetCore.OutputCaching.StackExchangeRedis to share it
A single sorted set with ZRANGEBYLEX (SortedSetRangeByValueAsync) finds all terms with a prefix, but in A–Z order. For "most popular first", precompute per prefix as above.
Do the work at write time: a ready-sorted top 10 per prefix makes reads one lookup.
The cheapest request is the one never sent: debounce and cancel on the client.
Cache short prefixes hard; rebuild in a new version and switch a pointer.
Autocomplete must fail soft: an empty list, never an error.
Your search box sends a request on every keystroke. At 10M users, that's a lot of "h", "ho", "hom". ⌨️
🌳 Precompute the top 10 for every prefix (a trie, flattened into Redis sorted sets)
⏱️ Debounce 250 ms and cancel the old request
🗄️ Cache short prefixes at the CDN for 60 seconds
🔄 Rebuild from search logs every 15 minutes, switch versions atomically
🧯 If Redis fails, return an empty list. Search still works.
Fast typeahead is mostly about not doing work at read time.
#SystemDesign #DotNet #Redis #Search #Performance
22.4 Proximity service: find nearby ATMs, branches or drivers Added
"What is near me?" Static places (ATMs, branches) live in a spatial database; fast-moving things (drivers, couriers) live in an in-memory geo index that is updated every few seconds.
Problem statement
A banking app needs "nearest ATM or branch" with filters. A delivery or ride app needs "available drivers within 3 km", where every driver moves and reports a new location every few seconds. Scanning every row and computing distances is too slow, and moving objects create a heavy write load.
Functional
Nearest N ATMs/branches within a radius, with filters (deposits, open now, accessible)
Drivers report location every ~4 s
Find available drivers within 3 km, nearest first
Hide drivers that stopped reporting
Non-functional
Places: ~20k locations, 2,000 queries/s, p95 < 100 ms
Drivers: 200k online × 1 update / 4 s = 50,000 writes/s; 5,000 searches/s, p99 < 50 ms
Freshness: driver location at most ~10 s old
Availability: 99.95%
Constraints & assumptions: WGS84 latitude/longitude (SRID 4326); distances in metres; the customer's location is never stored or logged in full; driver history is not kept in the hot store.
Solution
Key decision
Trade-off
Two stores for two problems. Places in SQL (geography + spatial index); drivers in Redis GEO (in memory).
Each fits its load (rich filters vs 50k writes/s), but you run two technologies.
Shard drivers by city/region key (drivers:syd, drivers:mel).
Small sets and spread load; a rider near a boundary must query two keys.
Heartbeat key with a TTL per driver. GEO members can't expire on their own.
One extra key and an MGET per search; a cleanup job removes stale members.
Client-side throttling: send only if moved > 50 m or 4 s passed.
Far fewer writes; position can be ~50 m off.
Cache place searches by rounded location (e.g. 3 decimals ≈ 110 m).
High hit rate in busy areas; recompute exact distances for display.
Steps to implement in .NET 10
Set up: minimal API; Microsoft.EntityFrameworkCore.SqlServer.NetTopologySuite (or Npgsql.EntityFrameworkCore.PostgreSQL.NetTopologySuite for PostGIS); StackExchange.Redis.
Model places:Atm with NetTopologySuite.Geometries.Point Location mapped to SQL Server geography; register with UseSqlServer(cs, x => x.UseNetTopologySuite()).
Spatial index: add CREATE SPATIAL INDEX with migrationBuilder.Sql(...) in a migration (create it in SQL; don't rely on HasIndex for spatial indexes).
Nearest places query:IsWithinDistance + OrderBy(Distance) + Take(10) (core trick 1). Validate lat −90..90, lon −180..180, radius capped at 50 km.
Driver updates:GeoAddAsync("drivers:{city}", lon, lat, driverId) plus StringSetAsync($"hb:{driverId}", 1, TimeSpan.FromSeconds(15)).
Nearby drivers:GeoSearchAsync(key, lon, lat, new GeoSearchCircle(3, GeoUnit.Kilometers), count: 20, order: Order.Ascending, options: GeoRadiusOptions.WithDistance), then drop members with no heartbeat (core trick 2).
Cleanup:BackgroundService removes stale members with GeoRemoveAsync every 30 s.
Live tracking: after a match, stream the driver's position to the rider through a SignalR group (Azure SignalR Service to scale).
Cache place results:HybridCache keyed by rounded lat/lon + filters, 5-minute expiry (places rarely change).
Deploy & protect privacy: Container Apps autoscaling on HTTP load; round coordinates in logs and traces; OpenTelemetry for write rate and search latency.
Core trick 1: nearest ATMs with EF Core + NetTopologySuite
builder.Services.AddDbContext<PlacesDb>(o =>
o.UseSqlServer(cs, sql => sql.UseNetTopologySuite()));
// Nearest 10 ATMs within 2 km that accept deposits
var here = new Point(lon, lat) { SRID = 4326 }; // x = longitude, y = latitude!
var atms = await db.Atms
.Where(a => a.AcceptsDeposits && a.Location.IsWithinDistance(here, 2_000)) // metres on geography
.OrderBy(a => a.Location.Distance(here))
.Take(10)
.Select(a => new { a.Id, a.Name, Metres = a.Location.Distance(here) })
.ToListAsync(ct);
Core trick 2: moving drivers with Redis GEO + heartbeat
// Driver app, every ~4 s
await db.GeoAddAsync("drivers:syd", lon, lat, driverId);
await db.StringSetAsync($"hb:{driverId}", 1, TimeSpan.FromSeconds(15)); // "still online"
// Rider: 20 closest drivers within 3 km, nearest first
var near = await db.GeoSearchAsync("drivers:syd", lon, lat,
new GeoSearchCircle(3, GeoUnit.Kilometers),
count: 20, order: Order.Ascending, options: GeoRadiusOptions.WithDistance);
var alive = await db.StringGetAsync(near.Select(n => (RedisKey)$"hb:{n.Member}").ToArray());
var result = near.Zip(alive)
.Where(x => x.Second.HasValue) // skip stale drivers
.Select(x => new { id = x.First.Member.ToString(), km = x.First.Distance })
.Take(5);
22.5 Large file upload & processing (chunked, resumable, scanned) Added
Let clients upload big files straight to blob storage in chunks they can resume, keep the bytes away from your API servers, and process a file only after a malware scan says it is clean.
Problem statement
Customers upload loan documents, statements, photos and videos, some of them several GB, often from a phone on a weak network. A single HTTP POST through the API ties up servers, fails at 90% and starts again from zero, and lets untrusted files reach back-office systems.
Functional
Upload files up to 5 GB with a progress bar
Resume after a network drop without starting over
Check type and size; scan for malware
Process clean files (OCR, thumbnails), then notify the user
Delete abandoned uploads automatically
Non-functional
Volume: 200k uploads/day, average 10 MB ≈ 2 TB/day; ~20 uploads/s at peak
Resume: a drop loses at most one 8 MiB block
Speed: 95% of files scanned and processed within 5 minutes
Availability: 99.9%; zone-redundant storage
Security: nothing is opened before a clean scan
Constraints & assumptions: Azure Blob Storage; browser and mobile clients; files are untrusted until scanned; managed identities everywhere (no storage keys in code); retention rules apply to stored documents.
Solution
Key decision
Trade-off
Direct to blob with a short, write-only SAS. The API never touches the bytes.
Scales without app servers; you must validate size and type after upload and keep the SAS tight (one blob, 1 h, no read).
Block blobs as resumable chunks: Put Block, then Put Block List.
Free resume (uncommitted blocks are kept for up to a week); the client must track block IDs.
tus protocol (tusdotnet) instead when you need a standard resumable protocol, ready clients (tus-js-client, Uppy) or non-Azure storage.
Mature and open, but the bytes flow through your servers.
Scan before use: Defender for Storage on-upload scanning + Event Grid; only "No threats found" files move to the clean container.
Per-GB cost and a short delay; blob index tags alone aren't tamper-proof, so act on events.
Event-driven processing with retries and a dead-letter queue.
The UX is async: show "scanning..." and notify when ready.
Steps to implement in .NET 10
Storage: containers incoming (untrusted) and clean. Enable Defender for Storage on-upload malware scanning and send results to an Event Grid custom topic. Add a lifecycle rule that deletes old blobs in incoming.
Start an upload: validate name, declared size (≤ 5 GB) and allowed types; save an Upload row as Pending; return a user delegation SAS for one blob with Create + Write only, valid 1 hour (core trick 1).
Client upload: browsers use @azure/storage-blob; .NET clients (MAUI, desktop, workers) use BlockBlobClient.StageBlockAsync per 8 MiB block and CommitBlockListAsync at the end, saving progress after each block (core trick 2).
Alternative:tusdotnet with app.MapTus("/files", ...) and the community Azure store Xtensible.TusDotNet.Azure; raise Kestrel's MaxRequestBodySize (default ~30 MB) for the tus route.
Scan result handler: Azure Functions isolated worker with Microsoft.Azure.Functions.Worker.Extensions.EventGrid ([EventGridTrigger]). Read scanResultType: clean → server-side copy to clean (StartCopyFromUriAsync) and mark Ready; malicious → delete or quarantine and alert; not scanned → keep blocked for review.
Validate content: check the real size and the file's magic bytes (a renamed .exe is not a PDF) before processing.
Process: send a Service Bus message (Azure.Messaging.ServiceBus) per clean file; the worker does OCR (Azure AI Document Intelligence) or thumbnails, idempotent by upload id, with a dead-letter queue.
Notify & status:GET /uploads/{id} plus a SignalR or push notification when Ready or Rejected.
Clean up & observe: a nightly job expires Pending uploads older than 24 h; alerts on "Malicious" results and dead-letter count; OpenTelemetry across API, Function and worker.
Core trick 1: a write-only, one-blob, 1-hour user delegation SAS
app.MapPost("/uploads", async (NewUpload req, BlobServiceClient blobs, TimeProvider clock, UploadsDb db) =>
{
if (req.Size is <= 0 or > 5L * 1024 * 1024 * 1024) return Results.BadRequest("Max 5 GB");
if (!AllowedTypes.Contains(req.ContentType)) return Results.BadRequest("File type not allowed");
var id = Guid.CreateVersion7();
var blob = blobs.GetBlobContainerClient("incoming").GetBlobClient(id.ToString());
var now = clock.GetUtcNow();
UserDelegationKey key = await blobs.GetUserDelegationKeyAsync(now.AddMinutes(-5), now.AddHours(1));
var sas = new BlobSasBuilder
{
BlobContainerName = "incoming", BlobName = blob.Name, Resource = "b",
StartsOn = now.AddMinutes(-5), ExpiresOn = now.AddHours(1)
};
sas.SetPermissions(BlobSasPermissions.Create | BlobSasPermissions.Write); // no read, no list
var url = new BlobUriBuilder(blob.Uri) { Sas = sas.ToSasQueryParameters(key, blobs.AccountName) }.ToUri();
db.Uploads.Add(new Upload(id, req.FileName, req.Size, UploadStatus.Pending));
await db.SaveChangesAsync();
return Results.Ok(new { id, uploadUrl = url });
});
var blob = new BlockBlobClient(uploadUrl); // SAS URL from the API
const int BlockSize = 8 * 1024 * 1024;
var ids = Enumerable.Range(0, progress.BlocksSent).Select(BlockId).ToList();
file.Position = (long)progress.BlocksSent * BlockSize; // resume: skip blocks already sent
var buffer = new byte[BlockSize];
int n;
while ((n = await file.ReadAtLeastAsync(buffer, BlockSize, throwOnEndOfStream: false)) > 0)
{
var id = BlockId(ids.Count);
await blob.StageBlockAsync(id, new MemoryStream(buffer, 0, n)); // SDK retries transient errors
ids.Add(id);
await progress.SaveAsync(ids.Count); // remember after every block
}
await blob.CommitBlockListAsync(ids); // the blob appears only now
static string BlockId(int i) => Convert.ToBase64String(BitConverter.GetBytes(i)); // same length
Testing scenarios
Type
Scenario
Expected result
How to test (.NET)
✅ Positive
Upload a small PDF, commit, clean scan event
Status Ready; file in clean
Testcontainers Azurite; post a sample Event Grid event to the handler
✅ Positive
Upload drops at 45%, then resumes
Final file's SHA-256 equals the original
Azurite + a "flaky" stream that throws mid-way (sample below)
✅ Positive
SAS lets the client write its blob
201 on Put Block / Put Block List
Integration test with the returned URL
✅ Positive
Processing finished
User notified once; status endpoint shows Ready
Test SignalR client or fake notifier
❌ Negative
Declared size 6 GB, or type application/x-msdownload
400; no SAS issued
[Theory] on POST /uploads
❌ Negative
EICAR test file uploaded
Malicious → quarantined, never processed, alert raised
In a real Azure test account with Defender on (Azurite doesn't scan); locally, a "Malicious" sample event
❌ Negative
Use the SAS to read or list, or after it expired
403
Call download/list with the SAS; build a SAS with ExpiresOn in the past
❌ Negative
.exe renamed to .pdf
Rejected by the magic-byte check
Unit test the validator with real file headers
⚠️ Edge
0-byte file; exactly 5 GB
400; accepted (boundary is inclusive)
[Theory] on the validator (no real 5 GB upload needed)
⚠️ Edge
Commit lists a block that was never uploaded
Commit fails (InvalidBlockList); blob unchanged
Azurite integration test
⚠️ Edge
Event Grid delivers the scan result twice
Copied and processed once
Invoke the handler twice with the same event
⚠️ Edge
Scan result "Not scanned" or timed out
File stays blocked; flagged for review/retry
Sample event with scanResultType = "Not Scanned"
⚠️ Edge
Upload abandoned for 24 h
Pending row expired; blob removed by lifecycle rule
FakeTimeProvider.Advance(TimeSpan.FromHours(24)) on the cleanup job
Sample test: a resumed upload gives the identical file (Testcontainers.Azurite)
public sealed class UploadTests : IAsyncLifetime
{
private readonly AzuriteContainer _azurite =
new AzuriteBuilder("mcr.microsoft.com/azure-storage/azurite:3.37.0").Build();
public async ValueTask InitializeAsync() => await _azurite.StartAsync();
public ValueTask DisposeAsync() => _azurite.DisposeAsync();
[Fact]
public async Task Resumed_upload_produces_the_identical_file()
{
var ct = TestContext.Current.CancellationToken;
var container = new BlobContainerClient(_azurite.GetConnectionString(), "incoming");
await container.CreateIfNotExistsAsync(cancellationToken: ct);
var blob = container.GetBlockBlobClient("big.bin");
var data = RandomNumberGenerator.GetBytes(20 * 1024 * 1024); // 20 MiB = 3 blocks
var uploader = new ResumableUploader(blob, blockSize: 8 * 1024 * 1024); // code under test
await Assert.ThrowsAsync<IOException>(() =>
uploader.UploadAsync(new FlakyStream(data, failAfterBytes: 9_000_000), ct));
await uploader.UploadAsync(new MemoryStream(data), ct); // resume
var stored = (await blob.DownloadContentAsync(ct)).Value.Content.ToArray();
Assert.Equal(SHA256.HashData(data), SHA256.HashData(stored));
}
}
Defender scanning has a maximum file size, a per-account throughput limit and a scan timeout. Check the current limits for your file sizes, and treat "Not scanned" as "not safe".
Keep bytes off your API: hand out a short, write-only, one-blob SAS.
Block blobs give you resumable uploads for free: stage blocks, commit the list.
Untrusted until scanned: act on the Event Grid scan result, then move to a clean container.
Make every step idempotent; events and messages can arrive twice.
A 3 GB upload fails at 90% on a train. Your user starts again from zero. Again. 📤
🔑 API hands out a 1-hour, write-only SAS for ONE blob; bytes never touch your servers
🧱 Upload 8 MiB blocks, commit the list at the end; a drop costs one block, not the file
🛡️ Defender for Storage scans on upload; Event Grid tells you the result
✅ Only "No threats found" files move to the clean container for processing
🔁 Every handler is idempotent; events can arrive twice
Resumable, scalable and safe: three properties, one design.
"Design a service that runs thousands of scheduled jobs (cron and one-off) across several nodes, on time, and never with a double effect." The hard parts: only one node fires each job, and nothing is lost when a node dies.
Problem statement
A bank runs thousands of scheduled tasks: monthly statements, direct-debit files, reminder emails, data exports. Today they're Task Scheduler entries on one VM. If that VM dies, jobs silently don't run, and nobody can see what ran when.
Functional
Register jobs by API: cron, one-off at a time, or a fixed interval
Run on any healthy node; retry failures with backoff
Pause, resume and "run now"; full run history and status
Schedules in a named time zone (Australia/Sydney, with daylight saving)
Non-functional
50,000 job definitions; peak 2,000 firings a minute (on the hour)
Fire within 5 s of the due time (p99)
A dead node's jobs picked up by another node within 30 s
Each firing has its business effect once; 99.9% available; 90 days of history
Constraints & assumptions:
Jobs are short (under 15 minutes). Long work is handed to a queue.
Azure SQL is available; the app runs as containers (AKS or Container Apps).
Delivery is at-least-once. Exactly-once firing isn't realistic, so jobs must be idempotent.
Solution
Decision
Choice
Trade-off
Engine
Quartz.NET 4 with a clustered SQL job store. A DB row lock lets exactly one node take each firing
No extra infrastructure, but the DB is the limit at very high firing rates (split into several named schedulers)
Delivery
At-least-once + an idempotency key (job name + scheduled fire time)
Simple and safe, but every job must check the key
Where work runs
The job only enqueues a message; workers do the heavy work
Scheduler stays light and fast; one more hop to monitor
Time
Store UTC; cron in a named time zone; TimeProvider in code
Testable, but DST gaps and misfires need an explicit policy
Alternatives
Functions timer trigger for a few fixed jobs; Hangfire for background jobs with a dashboard
Less to run, but less control at 50k jobs (Hangfire Pro is commercial)
Steps to implement in .NET 10
Create a Worker Service (dotnet new worker) and add Quartz 4.x plus Microsoft.Data.SqlClient. From 4.0, DI and hosting live in the core package; Quartz.Extensions.Hosting is now an empty forwarding package.
Create the Quartz tables in Azure SQL from the scripts in the Quartz repo, run as a migration step in the pipeline.
Configure AddQuartz with UsePersistentStore(s => { s.UseSqlServer(...); s.UseClustering(); }). Every node uses the same InstanceName and a unique instance id (GenerateInstanceId = true).
Write each job as an IJob with [DisallowConcurrentExecution]. Build a run key from ScheduledFireTimeUtc and insert it into a JobRuns table with a unique index.
Hand long work to Azure.Messaging.ServiceBus with MessageId = run key, so duplicate detection drops repeats.
Add an admin API (minimal APIs + Entra ID roles) that calls IScheduler: ScheduleJob, PauseJob, TriggerJob.
Wire OpenTelemetry (Quartz 4 emits spans and metrics) and health checks; alert on misfires and failed runs.
Deploy 3+ replicas. Set WaitForJobsToComplete and HostOptions.ShutdownTimeout so a rolling deploy drains running jobs.
builder.AddQuartz(q =>
{
q.ConfigureScheduler(o => { o.InstanceName = "bank-jobs"; o.GenerateInstanceId = true; });
q.UsePersistentStore(store =>
{
store.UseSqlServer(builder.Configuration.GetConnectionString("Quartz")!);
store.UseClustering(); // DB row locks: one node per firing
});
q.ScheduleJob<StatementJob>(t => t
.WithIdentity("monthly-statements")
.WithCronSchedule("0 0 2 1 * ?", c => c.InTimeZone(
TimeZoneInfo.FindSystemTimeZoneById("Australia/Sydney"))));
});
builder.AddQuartzHostedService(o => o.WaitForJobsToComplete = true);
[DisallowConcurrentExecution]
public sealed class StatementJob(JobRunStore runs, ServiceBusSender sender, TimeProvider clock) : IJob
{
public async ValueTask Execute(IJobExecutionContext context, CancellationToken ct = default)
{
// A retried or recovered firing has the same key, so the effect happens once
string runKey = $"{context.JobDetail.Key}:{context.ScheduledFireTimeUtc:O}";
if (!await runs.TryStartAsync(runKey, clock.GetUtcNow(), ct)) return; // unique index on RunKey
await sender.SendMessageAsync(new ServiceBusMessage(runKey) { MessageId = runKey }, ct);
await runs.CompleteAsync(runKey, clock.GetUtcNow(), ct);
}
}
Testing scenarios
Type
Scenario
Expected result
How to test (.NET)
✅ Positive
Monthly cron at 02:00 Sydney time
Next fire time is 02:00 local on the 1st, stored as UTC
xUnit v3: CronExpression with the Sydney time zone, assert GetNextValidTimeAfter
✅ Positive
One-off job scheduled by API for +2 minutes
Fires within 5 s of due; history row Completed
WebApplicationFactory + Testcontainers.MsSql
✅ Positive
"Run now" from the admin API
Runs at once; audit row records who triggered it
WebApplicationFactory with a test auth handler
✅ Positive
3 nodes, 1,000 jobs due in the same minute
Each fires once; work spreads across nodes
Three hosts in one test against Testcontainers.MsSql; count JobRuns
❌ Negative
Job's downstream call fails
Retried with backoff, then Failed + alert
Polly chaos AddChaosFault in the job's HttpClient pipeline
❌ Negative
Database unreachable for 30 s
Nodes retry; no duplicate firings; they catch up after
Testcontainers: PauseAsync / UnpauseAsync the SQL container
❌ Negative
Invalid cron expression sent to the API
400 Problem Details; nothing stored
WebApplicationFactory
❌ Negative
User without the Scheduler role calls "run now"
403; job not triggered
WebApplicationFactory with a test auth handler
⚠️ Edge
Node dies in the middle of a job
Another node recovers it after the check-in timeout; run key stops a double effect
Stop one host without draining; assert one completed run
⚠️ Edge
DST start (first Sunday in October): 02:30 doesn't exist in Sydney
Job fires once per the misfire policy, not skipped silently
CronExpression across the DST date; FakeTimeProvider for the job's own logic
⚠️ Edge
The same firing is executed twice
Second run is a no-op
Unit test: call Execute twice with the same fire time (sample below)
⚠️ Edge
Whole scheduler down for 2 hours
Missed firings run once (or are skipped) per policy, not 120 times
Stop all hosts, move past several due times, restart; count runs
public class StatementJobTests
{
[Fact]
public async Task Same_firing_twice_sends_one_message()
{
var clock = new FakeTimeProvider(new DateTimeOffset(2026, 11, 1, 2, 0, 0, TimeSpan.FromHours(11)));
var runs = new InMemoryJobRunStore(); // unique RunKey, like the SQL index
var sender = new FakeServiceBusSender(); // subclass of ServiceBusSender that records messages
var job = new StatementJob(runs, sender, clock);
var context = TestContexts.ForFiring("monthly-statements", clock.GetUtcNow()); // your helper
await job.Execute(context, TestContext.Current.CancellationToken);
await job.Execute(context, TestContext.Current.CancellationToken); // retry or recovery
Assert.Single(sender.Sent);
}
}
.NET 10 The .NET stack for this design
Component
Library / API
Notes
Scheduler
Quartz 4.x (clustered ADO.NET job store)
4.x targets net10.0; DI, hosting and OpenTelemetry built in
Quartz 4.0 needs a database schema migration from 3.x, and daylight-saving firing behaviour changed. Read the migration guide before upgrading a live cluster.
Exactly-once firing is a myth. Use at-least-once firing + idempotent jobs.
Clustered Quartz uses DB row locks: same instance name on every node, a unique instance id per node.
The scheduler fires; a queue and workers do the heavy work.
Store UTC, schedule in a named time zone, and test DST and misfires.
"Run this job every night at 2 am" sounds easy. Until you have 3 servers. ⏰
🔒 Only one node may fire each job: clustered Quartz uses a DB row lock
💀 A node dies mid-job? Another one recovers it after a check-in timeout
🔁 That means a job can run twice, so make it idempotent (job + fire time = key)
📨 The scheduler only enqueues; workers do the heavy lifting
🌏 Daylight saving: 2:30 am doesn't exist one Sunday in October. Test it.
At-least-once + idempotency beats chasing exactly-once. How do you schedule jobs today?
"Design a crawler that fetches a billion public pages a month, politely, without fetching the same page twice." A giant queue (the frontier) plus three rules: obey robots.txt, limit each host, remember what you've seen.
Problem statement
A search or knowledge product needs fresh copies of public web pages. The crawler starts from seed URLs, follows links, and keeps pages up to date, without overloading any website or getting stuck in endless link loops.
Functional
Start from seeds; fetch HTML; extract links and text
Follow links within allowed domains; obey robots.txt and Crawl-delay
100 KB average page ≈ 100 TB/month raw (≈25 TB compressed)
Seen set of 10B URLs: a Bloom filter at 1% false positives ≈ 12 GB
Constraints & assumptions:
Public pages only, no logins. Identify the bot with a User-Agent and a contact URL.
robots.txt follows RFC 9309. Cache it per host for up to 24 hours.
No JavaScript rendering in v1 (a headless browser only for an allow-list of sites).
Solution
Decision
Choice
Trade-off
Frontier
Kafka topic keyed by host, so one worker owns each host; per-host Channel<T> queues inside the worker
Politeness stays local and simple; very big hosts make partitions uneven
Politeness
PartitionedRateLimiter keyed by host (token bucket 1/s, or the site's Crawl-delay)
Safe for websites; big sites take longer to crawl
Seen URLs
Normalise the URL, then a Redis Bloom filter
12 GB instead of hundreds of GB; ~1% of new URLs are wrongly skipped
Duplicate content
SHA-256 of normalised text for exact copies; SimHash for near copies
Saves storage and index noise; extra CPU per page
Freshness
Adaptive recrawl interval per page + conditional GET (ETag / Last-Modified)
Hot pages stay fresh; needs a per-URL metadata store
Steps to implement in .NET 10
Create a Worker Service. Register a named HttpClient with IHttpClientFactory, AddStandardResilienceHandler() (Microsoft.Extensions.Http.Resilience), automatic decompression, a 10 s timeout and a User-Agent with a contact URL.
Build the frontier: a Kafka topic urls keyed by host (Confluent.Kafka); inside each worker, bounded System.Threading.Channels per host.
robots.txt: fetch once per host, cache with HybridCache, and parse with TurnerSoftware.RobotsExclusionTools (or a small RFC 9309 parser of your own).
Politeness: a PartitionedRateLimiter<Uri> keyed by host, plus a cap of 1–2 concurrent requests per host.
Parse with AngleSharp: a[href] links, canonical link, meta robots (noindex, nofollow) and body text. Resolve relative links with new Uri(baseUri, href) and normalise them.
Seen set: StackExchange.Redis sending BF.ADD / BF.EXISTS (Redis 8 includes Bloom filters; on managed Redis, check your tier supports them).
Store compressed HTML in Blob Storage (Azure.Storage.Blobs) keyed by URL hash, and per-URL metadata (ETag, last crawl, content hash, next due) in Cosmos DB or Table Storage.
Emit OpenTelemetry metrics (pages/s, errors per host, queue depth). Run N workers on AKS with KEDA scaling on Kafka consumer lag.
// One token bucket per host: 1 request per second, the rest wait in a queue
PartitionedRateLimiter<Uri> perHost = PartitionedRateLimiter.Create<Uri, string>(uri =>
RateLimitPartition.GetTokenBucketLimiter(uri.Host, _ => new TokenBucketRateLimiterOptions
{
TokenLimit = 1, TokensPerPeriod = 1, ReplenishmentPeriod = TimeSpan.FromSeconds(1),
QueueLimit = 1_000, QueueProcessingOrder = QueueProcessingOrder.OldestFirst,
AutoReplenishment = true
}));
async Task CrawlAsync(Uri url, HttpClient http, CancellationToken ct)
{
using RateLimitLease lease = await perHost.AcquireAsync(url, 1, ct);
if (!lease.IsAcquired) { await frontier.RequeueLaterAsync(url, ct); return; }
using var response = await http.GetAsync(url, HttpCompletionOption.ResponseHeadersRead, ct);
if (!response.IsSuccessStatusCode || response.Content.Headers.ContentType?.MediaType != "text/html") return;
var doc = await new HtmlParser().ParseDocumentAsync(await response.Content.ReadAsStreamAsync(ct), ct);
foreach (var href in doc.QuerySelectorAll("a[href]").Select(a => a.GetAttribute("href")))
if (Uri.TryCreate(url, href, out var link) && link.Scheme is "http" or "https")
await frontier.OfferAsync(Normalise(link), ct); // Bloom filter check inside
}
"Score every payment for fraud in under 100 ms, block the obvious ones, and send the doubtful ones to analysts." A fast inline decision, backed by a payment stream that keeps the features fresh.
Problem statement
A bank processes card and real-time account-to-account payments. Real-time payments can't be pulled back once they settle, so the fraud decision must happen before the money moves, without making checkout feel slow.
Functional
Score each payment: approve, step-up (extra check), decline, or approve + review
Rules (velocity, new payee, impossible travel, known mule accounts) + an ML model
Analysts work cases; their outcomes become training labels
Change rules and thresholds without a deploy
Non-functional
Decision p99 under 100 ms; 3,000 payments/s at peak (500 average)
99.99% available; a safe fallback if scoring fails
Every decision audited with rule and model versions
Few false declines: a wrong decline costs a customer
Constraints & assumptions:
PCI DSS: the fraud store holds card tokens, never full card numbers.
The model is trained offline (ML.NET, or Python exported to ONNX) and served in .NET.
The payment switch calls the fraud service synchronously; everything else can be async.
Solution
Decision
Choice
Trade-off
Inline vs stream
Synchronous gRPC decision; a stream consumer computes long-window features and cases
Latency stays bounded; stream features can lag a few seconds (hot counters are updated inline)
Rules + model
Hard rules override (sanctions, known mule); model score bands map to actions
Explainable and safe; two things to tune and govern
Features
Redis: INCR + expiry for fixed windows, sorted sets for sliding windows
Sub-millisecond reads; memory cost, so every key needs a TTL
Model serving
In-process: ML.NET PredictionEnginePool or ONNX Runtime, hot-reloaded
About 1 ms, no network hop; the model must fit in memory
Failure mode
80 ms budget, then a rules-only fallback decision
Payments keep flowing in an outage; a little more fraud risk while degraded
Steps to implement in .NET 10
Create an ASP.NET Core gRPC service (Grpc.AspNetCore) with a Score(PaymentRequest) contract agreed with the payment switch.
Features with StackExchange.Redis: velocity counters via StringIncrementAsync + KeyExpireAsync in one transaction; sliding windows via SortedSetAddAsync, SortedSetRemoveRangeByScoreAsync, SortedSetLengthAsync.
Rules as plain C# classes, with thresholds in IOptionsMonitor<FraudRules> fed by Azure App Configuration, so changes apply without a deploy.
Train offline (ML.NET with Microsoft.ML.LightGbm, or Python to ONNX). Serve with Microsoft.Extensions.ML: AddPredictionEnginePool<TxnFeatures, FraudScore>().FromFile("fraud", path, watchForChanges: true), or Microsoft.ML.OnnxRuntime.
Publish every decision to Event Hubs (Azure.Messaging.EventHubs) or Kafka, partition key = card token. An EventProcessorClient consumer updates long-window features and opens cases.
Case management: review-band decisions create a case; the analyst's outcome (fraud / genuine) is stored as a label for retraining.
Wrap scoring in a Microsoft.Extensions.Resilience pipeline: 80 ms timeout + fallback to rules-only. Record latency histograms, decision mix and model version with OpenTelemetry.
Deploy across zones (min 3 replicas, zone-redundant Redis). Run a new model in shadow mode (score, don't act) before switching it on.
public sealed class FraudScorer(
IDatabase redis, PredictionEnginePool<TxnFeatures, FraudScore> model, RuleSet rules)
{
public async Task<Decision> ScoreAsync(Payment p)
{
string key = $"vel:{p.CardToken}:10m";
ITransaction tx = redis.CreateTransaction();
Task<long> count10m = tx.StringIncrementAsync(key);
_ = tx.KeyExpireAsync(key, TimeSpan.FromMinutes(10), ExpireWhen.HasNoExpiry);
await tx.ExecuteAsync();
var f = new TxnFeatures { Amount = (float)p.Amount, Count10m = await count10m,
NewPayee = p.IsNewPayee, MerchantRisk = p.MerchantRisk };
if (rules.HardBlock(p, f) is { } reason) return Decision.Decline(reason); // e.g. known mule account
float score = model.Predict("fraud", f).Probability;
return score switch
{
>= 0.90f => Decision.Decline("model"),
>= 0.60f => Decision.ApproveAndReview(score), // async case for an analyst
_ => Decision.Approve(score)
};
}
}
Long-window features not double counted (dedupe by payment id)
Testcontainers.Kafka: publish twice, assert one increment
⚠️ Edge
Burst across a fixed-window boundary (5 at 10:09, 5 at 10:11)
Sliding-window rule still catches 10 in 10 minutes
Sorted-set timestamps from TimeProvider; FakeTimeProvider moves time
⚠️ Edge
Card used in Sydney, then London 30 minutes later
Impossible-travel rule flags it
xUnit theory with location fixtures
⚠️ Edge
Sale-day peak of 3,000 payments/s
p99 under 100 ms, no timeouts
k6 or Azure Load Testing; or NBombercommercial licence
⚠️ Edge
Score exactly 0.60 or 0.90
Falls in the documented band (bounds inclusive)
xUnit [Theory] with boundary values
public sealed class VelocityTests : IAsyncLifetime
{
private readonly RedisContainer _redis = new RedisBuilder().WithImage("redis:8").Build();
public async ValueTask InitializeAsync() => await _redis.StartAsync();
public async ValueTask DisposeAsync() => await _redis.DisposeAsync();
[Fact]
public async Task Sixth_payment_in_ten_minutes_is_stepped_up()
{
var mux = await ConnectionMultiplexer.ConnectAsync(_redis.GetConnectionString());
var scorer = TestScorers.RulesOnly(mux.GetDatabase()); // real Redis, stub model
var card = $"tok_{Guid.NewGuid():N}";
Decision last = null!;
for (int i = 0; i < 6; i++)
last = await scorer.ScoreAsync(TestPayments.For(card, amount: 49.95m));
Assert.Equal(Outcome.StepUp, last.Outcome);
}
}
.NET 10 The .NET stack for this design
Component
Library / API
Notes
Decision API
Grpc.AspNetCore
Binary, HTTP/2, low overhead for the payment switch
Features
StackExchange.Redis
Counters with TTL; sorted sets for sliding windows
Model
Microsoft.Extensions.ML (PredictionEnginePool) + Microsoft.ML.LightGbm; or Microsoft.ML.OnnxRuntime
Thread-safe pooled engines; hot reload from file or URI
Stream
Azure.Messaging.EventHubs.Processor or Confluent.Kafka
Partition by card token to keep one card's events in order
Fallback
Microsoft.Extensions.Resilience
Timeout + fallback strategy around scoring
Watch for training-serving skew: compute features with the same code (or the same definitions) offline and online, or the model sees different numbers in production.
Real-time payments can't be recalled, so decide before the money moves.
Keep the inline path tiny: Redis features + in-process model + rules, under 100 ms.
Hard rules override the model; score bands map to approve / review / step-up / decline.
Plan the fallback: a rules-only decision beats blocking every payment.
You have 100 milliseconds to decide if a payment is fraud. Go. ⏱️
⚡ Inline: Redis velocity counters + an in-process model, about 3 ms of work
🚫 Hard rules first: known mule accounts never reach the model
🎚️ Score bands: approve, step-up, review, decline
🌊 The payment stream updates long-window features and analyst cases in the background
🛟 Scoring down? Fall back to rules. Never block every payment.
Real-time payments can't be pulled back. That's why this has to be fast and right.
"Design digital onboarding: verify a new customer's identity, screen them, ask a human when needed, open the account, and prove every step later." A long-running workflow that may take three minutes or three weeks.
Problem statement
Customers open an account in the app. Under Australia's AML/CTF Act (reformed obligations from 31 March 2026), a bank must complete initial customer due diligence before it provides the service, and keep the records for 7 years after the relationship ends. Most applicants should be done in minutes; a few need an analyst.
Functional
Capture details, an ID document photo and a selfie
Extract document fields; verify the document with the issuer (e.g. the government Document Verification Service, via a gateway or identity provider)
Liveness and face match; sanctions and PEP screening; risk rating
Auto-approve low risk; analyst review for medium and high risk
Open the account and issue a card; the customer can resume later
Non-functional
80% of applicants auto-decided in under 3 minutes
10,000 applications a day; peak 50 starts a minute
Workflows survive restarts and can wait up to 30 days
Every step in an append-only audit trail, kept 7 years after the relationship ends
PII encrypted; documents stored in an Australian region
Constraints & assumptions:
Identity, liveness and screening are external vendor APIs with rate limits and outages.
Analysts work business hours; every decision must be explainable.
You record what you verified (document type, number, result), not necessarily a copy of the document.
Solution
Decision
Choice
Trade-off
Workflow engine
Durable Functions / Durable Task SDK on the Durable Task Scheduler (Temporal is a good alternative)
State, timers and retries survive restarts; orchestrator code must be deterministic (no DateTime.Now, no I/O)
Human step
Wait for an external event, racing a durable timer
Waits for days at no cost; you must design reminders and expiry
Side effects
Saga with compensations (close account if card issue fails)
No distributed transaction; compensations must be idempotent
Audit
Separate append-only store (Azure SQL ledger table or an event store), not the workflow history
Tamper-evident and queryable for 7 years; one more write per step
Vendors
Adapters behind interfaces, idempotency keys, standard resilience
Safe retries and swappable vendors; more code up front
Steps to implement in .NET 10
Create an isolated-worker Azure Functions app with Microsoft.Azure.Functions.Worker.Extensions.DurableTask, backed by the Durable Task Scheduler. (In containers, use Microsoft.DurableTask.Worker.AzureManaged instead.)
API: POST /applications saves the application, uploads images to private Blob Storage (Australian region), and calls ScheduleNewOrchestrationInstanceAsync with instance id = application id.
Activities: extract fields with Azure.AI.DocumentIntelligence (prebuilt-idDocument covers passports and US licences; use a custom model or your identity provider for Australian licences); verify, screen and rate risk through typed HttpClients with AddStandardResilienceHandler().
Orchestrator: chain the activities with TaskOptions.FromRetryPolicy; for review, race WaitForExternalEvent<ReviewDecision> against CreateTimer.
Analyst UI/API calls RaiseEventAsync(instanceId, "ReviewDecision", decision), protected by an Entra ID app role; require a second approver for high risk.
Saga: OpenAccount then IssueCard; on failure, run CloseAccount in the catch. Each activity is idempotent (key = application id + step).
Audit: every activity appends to an Azure SQL table created with LEDGER = ON (APPEND_ONLY = ON). Never update or delete rows.
Deploy; show stuck instances on a dashboard (custom status); purge old workflow history once the audit trail holds what you need.
public sealed class KycOrchestrator : TaskOrchestrator<Application, KycOutcome>
{
public override async Task<KycOutcome> RunAsync(TaskOrchestrationContext ctx, Application app)
{
var retry = TaskOptions.FromRetryPolicy(new RetryPolicy(5, TimeSpan.FromSeconds(10), backoffCoefficient: 2));
IdResult id = await ctx.CallActivityAsync<IdResult>("VerifyIdentity", app, retry);
Risk risk = await ctx.CallActivityAsync<Risk>("ScreenAndRate", new ScreenInput(app, id), retry);
if (risk.Level != RiskLevel.Low)
{
using var cts = new CancellationTokenSource();
Task timeout = ctx.CreateTimer(ctx.CurrentUtcDateTime.AddDays(5), cts.Token);
Task<ReviewDecision> review = ctx.WaitForExternalEvent<ReviewDecision>("ReviewDecision");
if (await Task.WhenAny(review, timeout) == timeout) return KycOutcome.Expired();
cts.Cancel(); if (!review.Result.Approved) return KycOutcome.Declined(review.Result.Reason);
}
string accountId = await ctx.CallActivityAsync<string>("OpenAccount", app, retry);
try { await ctx.CallActivityAsync("IssueCard", accountId, retry); }
catch (TaskFailedException)
{
await ctx.CallActivityAsync("CloseAccount", accountId, retry); // compensate
return KycOutcome.Failed("card");
}
return KycOutcome.Approved(accountId);
}
}
Testing scenarios
Type
Scenario
Expected result
How to test (.NET)
✅ Positive
Low-risk applicant, every check passes
Approved and account opened without a human; audit has an entry per step
In-memory engine; Temporal's test server can skip time
Tipping off: if you report a suspicious matter, the customer must not be told. Keep decline messages neutral and route screening hits to the financial crime team.
Long-running + human steps = a durable workflow, not a status column and a cron job.
Human approval is an external event racing a timer; design the reminder and the expiry.
Use compensations for side effects you can't roll back, and make them idempotent.
Audit lives in its own append-only store, kept 7 years after the relationship ends.
Opening a bank account takes 3 minutes for most people and 3 weeks for a few. Your design has to handle both. 🏦
🔎 Verify the document with the issuer, then liveness and face match
🚩 Screen for sanctions and PEPs, then rate the risk
👩💼 Medium risk? The workflow sleeps until an analyst decides, or 5 days pass
↩️ Card failed after the account opened? Compensate: close the account
📜 Every step in an append-only audit trail, kept for 7 years
Durable workflows turn "weeks of waiting" into a few lines of C#.
"Let staff ask questions over policies, procedures and wikis, get grounded answers with citations, and never see a document they aren't allowed to open." RAG at enterprise scale: the hard parts are permissions, freshness and proving quality.
Problem statement
A company has 2 million documents across SharePoint, Confluence and file shares, and 20,000 staff. People waste time hunting for the right policy. The assistant must answer from these documents only, show its sources, and respect each document's permissions.
Functional
Ingest new, changed and deleted documents, and permission changes
Chunk, enrich and embed; hybrid retrieval trimmed to the user's groups
Answers with clickable citations; "I don't know" when nothing fits
Thumbs up/down feedback; an evaluation dashboard for the owners
Non-functional
2M docs ≈ 20M chunks; at 1,536 dimensions × 4 bytes ≈ 120 GB of vectors (before quantisation)
Peak 20 questions/s; first token under 2 s; p95 full answer under 8 s
Permission removals reflected within 15 minutes; content changes within 1 hour
Groundedness at least 4/5 on the golden set; data stays in region
Constraints & assumptions:
Models run in an Azure region you approve; staff sign in with Microsoft Entra ID.
Documents can contain prompt-injection text; treat retrieved text as data, never as instructions.
Restricted-label content is excluded at ingestion (see 7.2).
Solution
Decision
Choice
Trade-off
Permissions
Store allowed group ids on every chunk; filter with search.in on every query (native ACL support in AI Search is in preview)
Unauthorised text never reaches the model; you must sync permission changes quickly
Model must cite source numbers; the API checks each one was retrieved
Trustworthy answers; sometimes a refusal or a retry
Quality
Golden-set evals in CI + online feedback (17.14); content safety on input and output
Catches regressions from model, prompt or chunking changes; LLM-judge cost
Steps to implement in .NET 10
Create an ASP.NET Core API with Entra ID auth (Microsoft.Identity.Web). Register IChatClient and IEmbeddingGenerator from Microsoft.Extensions.AI, using Azure.AI.OpenAI + Microsoft.Extensions.AI.OpenAI (AsIChatClient()).
Design the Azure AI Search index (Azure.Search.Documents): content, contentVector (HNSW), title, url, headingPath, lastModified, and group_ids as a filterable, non-retrievable string collection; add a semantic configuration and a vectorizer.
Build the ingestion worker: source change feeds (e.g. Microsoft Graph delta for SharePoint); Azure.AI.DocumentIntelligence layout to Markdown; chunk with Microsoft.Extensions.DataIngestionpreview or your own splitter; embed; MergeOrUpload and delete.
Query path: read the user's group ids from the token (use Microsoft Graph when the token has too many groups), build the search.in filter, run hybrid + semantic search.
Prompt: system rules ("answer only from the sources, cite [n], say you don't know"), numbered sources; stream with GetStreamingResponseAsync; validate citations before showing them.
Guardrails: Azure.AI.ContentSafety on output, Prompt Shields (REST) on the question and documents, per-user rate limits (Microsoft.AspNetCore.RateLimiting), no prompts or PII in logs.
Evaluation: a golden Q&A set run as xUnit tests with Microsoft.Extensions.AI.Evaluation.Quality, cached and reported with Microsoft.Extensions.AI.Evaluation.Reporting; fail the PR if scores drop.
Deploy with private endpoints for Search and the model; trace tokens, latency and "I don't know" rate with OpenTelemetry; review low-rated answers weekly.
Groundedness, relevance, retrieval; cached runs and HTML reports in CI
Identity
Microsoft.Identity.Web
Validated token; group claims or a Graph lookup
Never let the model decide who may see what. Trim results in the search query, from the validated token, before any text reaches the prompt.
Security trimming happens at retrieval, from the user's token, never in the prompt.
Ingestion must handle deletes and permission changes, not just new documents.
Hybrid + semantic ranking + structure-aware chunks give the best retrieval baseline.
Prove quality with a golden set in CI, and validate every citation.
The hardest part of an enterprise RAG assistant isn't the LLM. It's permissions. 🔐
👥 Store allowed group ids on every chunk and filter every query by the user's groups
🔄 Sync deletes and permission changes, not just new documents
🔎 Hybrid search + semantic ranker: codes and meaning both matter
📎 Make the model cite sources, then check every citation was really retrieved
📊 Golden-set evals in CI catch quality drops before users do
If the assistant can see it, assume the user can too. Trim before you prompt.
#RAG #GenAI #DotNet #EnterpriseAI #SystemDesign
⚡ Quick-fire Q&A
Flash sale: how do you guarantee you never oversell?
Make "check stock and take stock" one atomic operation: a Redis Lua script, or SQL UPDATE ... SET Stock = Stock - @qty WHERE Stock >= @qty and check the rows affected. Add idempotency keys so retries don't buy twice, and prove it with a parallel test (Task.WhenAll) against a real Redis or SQL container.
Booking: how do temporary holds expire without a timer per seat?
Store an expiry time on the hold and treat any hold with HoldExpiresAt <= now as free in every query (lazy expiry). A periodic sweeper only tidies up and pushes "free again" events. In Redis, SET key value NX PX 600000 gives the same thing in one command. Test it with FakeTimeProvider.Advance.
Autocomplete: why not run LIKE 'abc%' on the database?
At thousands of requests per second it is too slow and it can't rank by popularity cheaply. Precompute the top 10 per prefix (Redis sorted sets or a trie) from search logs, cache short prefixes at the edge, and debounce on the client. Use Azure AI Search suggesters if you need typo tolerance.
Proximity: why use two different stores for ATMs and drivers?
ATMs rarely change and need rich filters, so a SQL/PostGIS spatial index fits. Drivers move every few seconds (tens of thousands of writes per second), so an in-memory Redis GEO index fits, with a heartbeat TTL to hide drivers who went offline.
File upload: how do you make a 5 GB upload resumable and safe?
Give the client a short, write-only SAS for one blob, upload 8 MiB blocks (Put Block) and commit the list at the end (Put Block List), so a drop only re-sends one block. Scan with Defender for Storage, react to the Event Grid result, and process only files marked "No threats found".
Job scheduler: how do you stop two nodes running the same job?
Let one shared store decide. Clustered Quartz.NET takes a database row lock when it acquires a trigger, so only one node fires each firing. Because a crashed node's job is recovered and may run again, also make every job idempotent with a key such as job name + scheduled fire time.
Web crawler: how do you stay polite and avoid re-fetching pages?
Shard the frontier by host so one worker owns each site, and rate-limit per host (about 1 request/s, or the site's Crawl-delay) after checking robots.txt. Normalise URLs and check a Bloom filter before enqueueing, and use conditional GET (ETag / Last-Modified) on recrawls.
Fraud detection: how do you decide in under 100 ms?
Keep the inline path tiny: read and update velocity counters in Redis, run hard rules, then score with an in-process model (ML.NET or ONNX Runtime). Publish the decision to a stream and do everything else (long-window features, cases, retraining labels) asynchronously. If scoring times out, fall back to rules only.
KYC onboarding: how do you model a human approval that can take days?
Use a durable workflow (Durable Functions / Durable Task, or Temporal). The orchestrator waits for an external "ReviewDecision" event racing a durable timer, so it costs nothing while it waits and expires cleanly. Side effects like opening an account use saga compensations, and every step writes to an append-only audit store.
Enterprise RAG: how do you stop the assistant leaking documents?
Trim at retrieval. Store the allowed group ids on every chunk, read the user's groups from their validated Entra token, and filter every search with them (e.g. search.in in Azure AI Search). Unauthorised text never reaches the prompt, and permission changes are synced into the index quickly.