DevOps, deployment strategies & observability

15.1 CI/CD pipelines Added

CI: merge often, and build and test automatically. CD: every change that passes is releasable (delivery) or released automatically (deployment).

# the core of any .NET pipeline
dotnet restore                                   # NuGet audit warns about vulnerable packages
dotnet build -c Release --no-restore -warnaserror
dotnet test  -c Release --no-build
dotnet publish src/Orders.Api -c Release /t:PublishContainer \
    -p:ContainerRegistry=myacr.azurecr.io -p:ContainerRepository=orders-api

15.2 Source control, branching & code review Added

How a team shares one codebase: where changes live, how they get reviewed, and how they get back into main quickly and safely.

Three branching models

Why trunk-based + feature flags suits CI/CD: "continuous integration" means merging with everyone else's code at least daily. A branch that lives two weeks isn't integrated, so conflicts and surprises pile up at the end. Flags split deploy from release, so unfinished code can merge safely. DORA lists trunk-based development as a capability linked to better delivery performance.

Pull requests & code review

Commits, protected branches & ownership

Conventional Commits (spec 1.0.0): a typed, machine-readable message. Tools use it to pick the next version and write the changelog (15.4).

fix(transfers): reject negative amounts           → PATCH
feat(payments): add PayTo mandate status endpoint → MINOR
feat(api)!: remove v1 /accounts route             → MAJOR

BREAKING CHANGE: clients must call /v2/accounts
# other types: docs, test, refactor, perf, build, ci, chore

<!-- Directory.Build.props at the repo root: applies to every project -->
<Project>
  <PropertyGroup>
    <Nullable>enable</Nullable>
    <TreatWarningsAsErrors>true</TreatWarningsAsErrors>
    <EnforceCodeStyleInBuild>true</EnforceCodeStyleInBuild>
    <AnalysisLevel>latest-recommended</AnalysisLevel>
  </PropertyGroup>
</Project>

15.3 Testing strategy Added

Many fast, cheap tests at the bottom, a few slow, realistic ones at the top, and some checks that only make sense in production.

Both shapes say the same thing: buy the most confidence for the least test time. The anti-pattern is the "ice-cream cone": mostly manual and E2E tests.

Contract testing (consumer-driven)

• Each consumer states what it actually uses. The provider proves it still meets every contract before it deploys.
• It replaces most slow, flaky cross-team E2E tests between microservices.

Performance testing: four different questions

Chaos engineering

// xUnit v3: the real API in memory + a real Postgres in Docker
public sealed class PaymentsApiTests : IAsyncLifetime
{
    private readonly PostgreSqlContainer _db = new PostgreSqlBuilder("postgres:17").Build();
    private WebApplicationFactory<Program> _factory = null!;

    public async ValueTask InitializeAsync()
    {
        await _db.StartAsync();
        _factory = new WebApplicationFactory<Program>().WithWebHostBuilder(b =>
            b.UseSetting("ConnectionStrings:Payments", _db.GetConnectionString()));
    }

    [Fact]
    public async Task Create_payment_returns_201()
    {
        var client = _factory.CreateClient();
        var response = await client.PostAsJsonAsync("/payments",
            new { FromAccount = "123", ToAccount = "456", Amount = 50m },
            TestContext.Current.CancellationToken);

        Assert.Equal(HttpStatusCode.Created, response.StatusCode);
    }

    public async ValueTask DisposeAsync()
    {
        await _factory.DisposeAsync();
        await _db.DisposeAsync();
    }
}

15.4 Versioning & release management Added

A version number is a promise about compatibility. Release management decides how a tested artifact gets that number, its release notes, and its path to production.

Semantic Versioning (SemVer 2.0.0)

Release trains vs continuous delivery

Many banks run both: continuous delivery for internal services and APIs, and a train for mobile app releases and core banking changes.

Build once, promote the same artifact

• Artifact immutability: a published version never changes. Never rebuild or overwrite 1.8.0; ship 1.8.1. nuget.org won't accept the same version twice (you can only unlist).
• Deploy containers by digest (@sha256:…), not a moving tag like latest.
• Environment promotion moves the artifact you tested, never a fresh build. Config and secrets come from each environment (15.1).
• Keep provenance: who built it, from which commit, signed and with an SBOM (14.10).

Changelogs & release notes

Automate it: Conventional Commits (15.2) let tools such as release-please or semantic-release work out the next version, write the changelog and create the release. In .NET, MinVer, Nerdbank.GitVersioning or GitVersion stamp the version into the build.

<!-- library .csproj (or Directory.Build.props) -->
<PropertyGroup>
  <MinVerTagPrefix>v</MinVerTagPrefix>             <!-- tag v1.5.0 => 1.5.0 -->
  <EnablePackageValidation>true</EnablePackageValidation>
  <PackageValidationBaselineVersion>1.4.0</PackageValidationBaselineVersion>
  <IncludeSymbols>true</IncludeSymbols>
  <SymbolPackageFormat>snupkg</SymbolPackageFormat>
</PropertyGroup>
<PropertyGroup Condition="'$(GITHUB_ACTIONS)' == 'true'">
  <ContinuousIntegrationBuild>true</ContinuousIntegrationBuild>
</PropertyGroup>
<ItemGroup>
  <PackageReference Include="MinVer" Version="8.0.0" PrivateAssets="all" />
</ItemGroup>

# pipeline, after tagging the release commit with v1.5.0
dotnet pack -c Release -o out
dotnet nuget push "out/*.nupkg" -s https://api.nuget.org/v3/index.json -k $NUGET_KEY

15.5 Deployment strategies

Ways to release a new version that trade off downtime, risk, cost and rollback speed.

15.6 Feature flags (toggles) Added

Ship code turned off, then turn it on for chosen users without redeploying. Deploy ≠ release.

builder.Configuration.AddAzureAppConfiguration(o =>
    o.Connect(new Uri("https://my-config.azconfig.io"), new DefaultAzureCredential())
     .UseFeatureFlags());                         // flags refresh without a restart
builder.Services.AddAzureAppConfiguration();
builder.Services.AddFeatureManagement().WithTargeting();   // users, groups, % rollout

var app = builder.Build();
app.UseAzureAppConfiguration();

app.MapGet("/checkout", async (IFeatureManager features) =>
    await features.IsEnabledAsync("NewCheckout") ? NewCheckout() : OldCheckout());

15.7 Infrastructure as Code & GitOps Added

Define infrastructure in versioned code files. The tool makes reality match the code.

• Declarative ("what I want": Terraform, Bicep, ARM, CloudFormation) vs imperative ("steps to run": scripts, Pulumi in code).
• Benefits: repeatable environments, review and audit trail, disaster recovery by redeploying, no snowflake servers.
• Drift = manual changes outside code. Detect it and stop it.
• Immutable infrastructure: replace, don't patch in place.
• GitOps: Git is the source of truth, and an agent in the cluster (Argo CD, Flux) pulls and reconciles continuously.

// Aspire AppHost: the app and its infrastructure in one C# file
var builder = DistributedApplication.CreateBuilder(args);

var db = builder.AddAzurePostgresFlexibleServer("pg").AddDatabase("orders");
builder.AddProject<Projects.Orders_Api>("api").WithReference(db);

builder.Build().Run();
// locally: runs everything with the dashboard;  "azd up": provisions Azure and deploys

15.8 Containers & Kubernetes Added

A container packages an app with its dependencies. Kubernetes runs many containers across many machines: it schedules, heals, scales and connects them.

VM vs container

VM = [App][Libs][Guest OS] on a hypervisor. Container = [App][Libs] on the shared host OS kernel → lighter, starts in seconds.

Kubernetes at a glance

Deployment → ReplicaSet → Pods (1+ containers). Service = stable IP/DNS for pods · Ingress = L7 routing in · ConfigMap / Secret = config · HPA = auto-scale pods. Control plane: API server · scheduler · controller manager · etcd (state). Worker nodes: kubelet + container runtime.

builder.Services.AddHealthChecks()
    .AddCheck("self", () => HealthCheckResult.Healthy(), tags: ["live"])
    .AddDbContextCheck<AppDbContext>(tags: ["ready"]);  // Microsoft.Extensions.Diagnostics.HealthChecks.EntityFrameworkCore
builder.Services.Configure<HostOptions>(o => o.ShutdownTimeout = TimeSpan.FromSeconds(25));

var app = builder.Build();
app.MapHealthChecks("/health/live",  new() { Predicate = r => r.Tags.Contains("live") });   // restart if failing
app.MapHealthChecks("/health/ready", new() { Predicate = r => r.Tags.Contains("ready") });  // no traffic if failing

15.9 Observability Added

Monitoring tells you that something is wrong. Observability lets you ask why, from the telemetry the system emits.

The W3C traceparent header is propagated on every call, so all these spans share one trace ID (4bf9…). Child spans (└) sit inside their parent's time. Here the SQL INSERT is the bottleneck.

builder.Services.AddOpenTelemetry()
    .ConfigureResource(r => r.AddService("orders-api"))
    .WithTracing(t => t.AddAspNetCoreInstrumentation()
                       .AddHttpClientInstrumentation()
                       .AddSource("Orders"))                   // our own ActivitySource
    .WithMetrics(m => m.AddAspNetCoreInstrumentation()
                       .AddHttpClientInstrumentation()
                       .AddMeter("Orders", "System.Runtime"))  // our Meter + runtime metrics
    .WithLogging()
    .UseOtlpExporter();   // endpoint from OTEL_EXPORTER_OTLP_ENDPOINT

// a custom span (Source is a static field: new ActivitySource("Orders"))
using var activity = Source.StartActivity("PlaceOrder");
activity?.SetTag("order.channel", "web");

15.10 Logging Added

A log is a timestamped record of something that happened. Logs explain why something went wrong, and they're only useful if you can search and correlate them.

Log levels: use them consistently

Structured vs unstructured

✗ Unstructured  "Payment failed for cust 42 amount 800 because timeout"
✓ Structured    {"ts":"2026-10-16T01:02:03Z","level":"ERROR","service":"payments","env":"prod",
                 "traceId":"4bf92f…","correlationId":"c-91a…","event":"PaymentFailed",
                 "paymentId":"7f3a","reason":"FraudServiceTimeout","durationMs":2004}
→ query it: where event=="PaymentFailed" | summarize count() by reason, bin(ts, 5m)

public static partial class Log
{
    [LoggerMessage(LogLevel.Error, "Payment {PaymentId} failed: {Reason}")]
    public static partial void PaymentFailed(this ILogger logger, string paymentId, string reason);
}

// every log line inside the scope carries CorrelationId
using (logger.BeginScope(new Dictionary<string, object> { ["CorrelationId"] = correlationId }))
{
    logger.PaymentFailed("7f3a", "FraudServiceTimeout");   // fields, not one long string
}

15.11 Metrics Added

Metrics are numbers measured over time. They're cheap to store, fast to query, and ideal for dashboards, trends and alerts. They tell you that something is wrong.

Metric types

What to measure

100 requests: 95 take 100 ms and 5 take 3,000 ms. The average (245 ms) looks fine and hides the slow tail. p95 = 100 ms, but p99 = 3,000 ms: 1 in 20 users waits 3 s.

public sealed class PaymentMetrics
{
    private readonly Counter<long> _processed;
    private readonly Histogram<double> _duration;

    public PaymentMetrics(IMeterFactory meterFactory)
    {
        var meter = meterFactory.Create("Payments");
        _processed = meter.CreateCounter<long>("payments.processed", unit: "{payment}");
        _duration  = meter.CreateHistogram<double>("payments.duration", unit: "s");
    }

    // keep tag values bounded: status and channel, never paymentId
    public void Record(string status, string channel, double seconds)
    {
        var tags = new TagList { { "status", status }, { "channel", channel } };
        _processed.Add(1, tags);
        _duration.Record(seconds, tags);
    }
}
// builder.Services.AddSingleton<PaymentMetrics>();  and  .AddMeter("Payments") in OpenTelemetry

15.12 Alerting Added

An alert should page a human only when something needs action now. Every page should be urgent, actionable and tied to user impact.

Types of alert rules

SLO burn-rate alerting (multi-window)

SLO 99.9% → error budget 0.1% per 30 days (≈ 43 min). Burn rate = actual error rate ÷ allowed error rate. 1× uses the budget up in exactly 30 days.

Each PAGE rule needs a long AND a short window to both be over the threshold. The short window confirms it's still happening → fewer false alarms.

Severity & routing

15.13 Incident management Added

How a team detects, responds to and learns from production incidents.

• MTTD (time to detect), MTTR (time to restore), MTBF (time between failures).
• Mitigate first (restore service), find the root cause second.
• Postmortem template and on-call health: 15.14. Change-caused incidents feed DORA's change fail rate and recovery time: 15.15.

15.14 SRE practices Added

Site Reliability Engineering runs operations as a software problem: agree how reliable a service must be, spend the leftover risk on purpose, and automate the repetitive work away.

SLI vs SLO vs SLA and the error-budget maths are in 3.10. Burn-rate alerts are in 15.12. This section covers the team practices around them.

Error budget policy

A written agreement, signed off by product and engineering before anything goes wrong, saying what happens as the budget runs down.

• Define thresholds and actions, who decides, and the exceptions (security patches, legal deadlines).
• Name an escalation path for disagreements (e.g. head of engineering).
• If the budget is always untouched, the SLO may be too loose, or you could take more risk. If it's always gone, renegotiate the SLO or invest in reliability.

Toil and the 50% rule

On-call health

Blameless postmortems

• Blameless: assume people acted sensibly with what they knew. Fix the system, not the person. Blame makes people hide information.
• Triggers: any Sev 1/2, data loss, a big error-budget burn, or an incident that monitoring missed.
• Share postmortems widely and review open action items regularly. An unfinished action item is a repeat incident waiting to happen.

Production readiness review (PRR)

A checklist a service passes before it goes live or before SRE takes on its pager.

Game days

Planned exercises that inject a real failure (zone outage, slow dependency, expired certificate) while the on-call team responds as if it were live. They test runbooks, alerts and people, not just code. How to run them, plus chaos principles and tools, is in 15.16.

// SLO: 99% of requests faster than 300 ms, so 0.3 s must be a bucket edge
builder.Services.AddOpenTelemetry().WithMetrics(m => m
    .AddAspNetCoreInstrumentation()
    .AddView("http.server.request.duration",
        new ExplicitBucketHistogramConfiguration
        {
            Boundaries = [0.05, 0.1, 0.3, 0.5, 1, 2.5, 5]
        }));
// SLI = requests with duration <= 0.3 s / all requests, over 30 days (PromQL / KQL)

15.15 DORA metrics & engineering effectiveness Added

DORA's research-backed metrics measure software delivery outcomes, how fast and how safely changes reach production, so teams can see whether their improvements work.

The five metrics (DORA, 2024 onwards)

These are the original "four keys" plus rework rate. DORA keeps finding that speed and stability aren't a trade-off: top performers do well on all of them. The lever behind all five is small batches.

Performance clusters

Latest (2025 DORA report): DORA dropped the single elite-to-low ladder. It now gives benchmarks per metric and describes seven team archetypes, from "Foundational challenges" to "Harmonious high-achievers", that also factor in well-being, friction and product performance. DORA's advice: aim for elite improvement, not an elite label.

Measuring them from pipelines and incidents

• Tag every deploy with service, version, commit SHA and environment. Link each incident to the change that caused it.
• Measure one service at a time, use medians, and watch trends over months.
• Start with a conversation or the free DORA Quick Check (dora.dev) before building a big data pipeline.

SPACE framework in brief

Forsgren, Storey and others (2021): developer productivity can't be one number. Pick metrics from at least three of these dimensions, mixing surveys and system data.

Anti-patterns

var meter = new Meter("Delivery");   // or IMeterFactory in a DI-based app
var deploys = meter.CreateCounter<long>("delivery.deployments");
var leadTime = meter.CreateHistogram<double>("delivery.change_lead_time", unit: "h");

// the pipeline POSTs here after every production deployment
app.MapPost("/deployments", (DeploymentEvent e) =>
{
    var tags = new TagList { { "service", e.Service }, { "result", e.Result } };
    deploys.Add(1, tags);
    foreach (var committedAt in e.CommitTimes)
        leadTime.Record((e.DeployedAt - committedAt).TotalHours, tags);
    return Results.Accepted();
});

// add .AddMeter("Delivery") to OpenTelemetry. Change fail rate =
// deployments{result="failed"} / deployments, calculated in the dashboard
record DeploymentEvent(string Service, string Result, DateTimeOffset DeployedAt,
    DateTimeOffset[] CommitTimes);

15.16 Chaos engineering and game days Added

Break things on purpose, in a controlled experiment, to prove that the system and the team cope the way you think they do.

The short version is in 15.3. SLOs, error budgets and the error budget policy are in 15.14. This section is the how-to.

The experiment loop

How it works

Common experiments

Non-production first, then production

• Code and CI: in-process faults (Polly chaos) in integration tests.
• Staging: platform faults with production-like load.
• Production: off-peak, tiny blast radius, people watching, abort automated. Only production shows the real config, traffic and dependencies.
• Continuous: proven experiments run on a schedule or in the pipeline.

Game days and the error budget

• Game day: a planned exercise. A facilitator injects the fault; on-call responds with the real alerts and runbooks; a scribe times detection and mitigation.
• Afterwards: blameless review, fix the runbooks and alerts, add action items.
• Error budget: experiments spend it on purpose. No budget left? No chaos. Use the burn rate as the abort condition.

Tools

When to use / when not

// Chaos on one dependency, off unless config turns it on (staging or a game day)
bool chaosOn = builder.Configuration.GetValue<bool>("Chaos:Enabled");

builder.Services.AddHttpClient<FxRatesClient>(c => c.BaseAddress = new Uri("https://fx.example.com"))
    .AddResilienceHandler("fx", pipeline =>
    {
        // 1. Normal resilience: this is what the experiment tests
        pipeline.AddRetry(new HttpRetryStrategyOptions { MaxRetryAttempts = 3 });
        pipeline.AddTimeout(TimeSpan.FromSeconds(2));          // per try

        // 2. Chaos goes last, closest to the real call
        if (chaosOn)
        {
            pipeline.AddChaosLatency(0.10, TimeSpan.FromSeconds(3));   // 10% of calls +3 s
            pipeline.AddChaosOutcome(0.05, () =>
                new HttpResponseMessage(HttpStatusCode.ServiceUnavailable)); // 5% get a 503
        }
    });
// Hypothesis: retries + a 2 s per-try timeout keep FX success above 99.5%