Section 02

Networking & communication protocols

How bytes move between systems: the layers, the transport (TCP/UDP), the web protocols (HTTP/1.1, 2, 3), and how to pick the right protocol for real-time, request/response, messaging or file exchange.

2.1 OSI & TCP/IP layers Added

A network is a stack of layers. Each layer does one job and hands off to the next.

#OSI layerJobExamplesTCP/IP model
7ApplicationWhat the app speaksHTTP, gRPC, SMTP, DNS, MQTTApplication
6PresentationEncoding, encryptionTLS, JSON, Protobuf
5SessionKeep a conversation goingSessions, RPC setup
4TransportDelivery between portsTCP, UDP (QUIC runs on UDP)Transport
3NetworkRouting between networksIP, ICMPInternet
2Data linkDelivery on the local linkEthernet, MAC, Wi-FiLink
1PhysicalBits on wire or airFibre, copper, radio
  • Mnemonic (7→1): "All People Seem To Need Data Processing".
  • In system design, "L4" and "L7" come up all the time. An L4 load balancer sees IP and port. An L7 load balancer sees URLs, headers and cookies.
How to implement it
LayerLibrary / API
7 ApplicationHttpClient / Kestrel (HTTP), Grpc.Net.Client, System.Net.Dns, MailKit (SMTP), MQTTnet
6 Presentationbuilt-in SslStream (System.Net.Security) for TLS; System.Text.Json / Google.Protobuf for encoding
4 Transportbuilt-in System.Net.Sockets (Socket, TcpClient, UdpClient); System.Net.Quic for QUIC
3 Networkbuilt-in IPAddress, System.Net.NetworkInformation (Ping, NetworkInterface)
Fast byte handlingbuilt-in System.IO.Pipelines: what Kestrel uses to parse protocols without extra copies

2.2 TCP vs UDP Added

TCP is a reliable, ordered phone call. UDP is fire-and-forget postcards: fast, but some may get lost.

TCPUDP
ConnectionHandshake firstNone
ReliabilityRetransmits lost packets, keeps orderNo guarantee
Speed / overheadHigherLower
Used byHTTP/1.1, HTTP/2, DB connections, SFTP, AMQPDNS, video/voice, gaming, QUIC (HTTP/3)
  • TCP head-of-line blocking: one lost packet holds up everything behind it.
  • QUIC builds reliability on top of UDP, per stream, so one lost packet doesn't block the other streams.
How to implement it
Library / APIPackageUse it for
TcpListener / TcpClientbuilt-in System.Net.SocketsReliable, ordered byte streams (custom protocols, legacy host links)
UdpClientbuilt-in System.Net.SocketsFire-and-forget datagrams (metrics, discovery, streaming)
Socketbuilt-in System.Net.SocketsLow-level control when the helpers above are not enough
QuicListener / QuicConnectionbuilt-in System.Net.QuicQUIC: reliable streams over UDP, the base of HTTP/3. Needs msquic (built into Windows 11 / Server 2022; a package on Linux)
Pipelinesbuilt-in System.IO.PipelinesParse a TCP stream into messages without buffer juggling
// TCP: connect first, then a reliable ordered stream
using var tcp = new TcpClient();
await tcp.ConnectAsync("core-banking.internal", 9000, ct);
await using var stream = tcp.GetStream();
await stream.WriteAsync("PING\n"u8.ToArray(), ct);

// UDP: no connection, no delivery guarantee
using var udp = new UdpClient();
await udp.SendAsync("payments.count:1|c"u8.ToArray(), "statsd.internal", 8125, ct);

TCP is a stream, not messages: one ReadAsync can return half a message or two. Add framing (length prefix or delimiter), or let System.IO.Pipelines help.

2.3 HTTP/1.1 vs HTTP/2 vs HTTP/3 Added

Same HTTP meaning (methods, status codes, headers), faster ways of sending it.

HTTP/1.1 (1997)

  • Text-based
  • Keep-alive connections
  • One request at a time per connection, so browsers open about 6 connections
  • Head-of-line blocking

HTTP/2 (2015)

  • Binary framing
  • Multiplexing: many streams on one TCP connection
  • Header compression (HPACK)
  • gRPC runs on it
  • Still blocks at the TCP level

HTTP/3 (2022)

  • Runs on QUIC over UDP
  • TLS 1.3 built in, faster setup (1-RTT, 0-RTT on resume)
  • No TCP head-of-line blocking
  • Connection survives a network change (Wi-Fi → 4G)
  • HTTPS = HTTP + TLS. In production, always use TLS 1.2 or later (prefer 1.3).
  • CDNs and gateways often speak HTTP/2 or 3 to clients and HTTP/1.1 to backends.
How to implement it
Library / APIPackageUse it for
Kestrelbuilt-in HttpProtocols.Http1AndHttp2AndHttp3Serve HTTP/1.1, 2 and 3 on one endpoint (HTTP/3 needs TLS)
HttpClientbuilt-in DefaultRequestVersion / DefaultVersionPolicyAsk for HTTP/2 or 3 and choose whether to fall back
IHttpClientFactoryMicrosoft.Extensions.Http (included in ASP.NET Core)Named/typed clients with pooled, recycled handlers. The standard way to call HTTP APIs
SocketsHttpHandlerbuilt-in System.Net.HttpConnection pooling settings: PooledConnectionLifetime, MaxConnectionsPerServer, EnableMultipleHttp2Connections
builder.WebHost.ConfigureKestrel(k =>
    k.ListenAnyIP(5001, listen =>
    {
        listen.Protocols = HttpProtocols.Http1AndHttp2AndHttp3;
        listen.UseHttps();                      // HTTP/2 and HTTP/3 in browsers need TLS
    }));

builder.Services.AddHttpClient("payments", c =>
{
    c.BaseAddress = new Uri("https://payments.internal/");
    c.DefaultRequestVersion = HttpVersion.Version30;
    c.DefaultVersionPolicy = HttpVersionPolicy.RequestVersionOrLower; // fall back to 2 or 1.1
});

2.4 DNS Added

The internet's phone book: it turns api.bank.com into an IP address.

Browser / OS cache
→
Recursive resolverISP / 8.8.8.8
→
Root server"ask .com"
→
TLD server"ask bank.com NS"
→
AuthoritativeA → 20.1.2.3

Record types

  • A / AAAA: name → IPv4 / IPv6
  • CNAME: alias to another name
  • MX: mail server
  • TXT: verification, SPF/DKIM
  • NS: which servers are authoritative

Points to remember

  • TTL controls caching. Lower it before a migration or cutover.
  • DNS can load balance and route by geography (Azure Traffic Manager, Route 53).
  • DNS failover is slow because clients cache old answers.
  • Private DNS zones resolve private endpoints inside a VNet.
How to implement it
Library / APIPackageUse it for
Dnsbuilt-in System.Net.DnsResolve a name to IPs with the OS resolver (GetHostAddressesAsync)
DnsClientDnsClientQuery any record type (MX, TXT, SRV, NS) and pick the DNS server
Connection lifetimebuilt-in SocketsHttpHandler.PooledConnectionLifetimeMake long-lived HttpClient instances reconnect, so they pick up DNS changes after a failover
Manage zonesAzure.ResourceManager.Dns / Azure.ResourceManager.PrivateDnsCreate and update public and private DNS records from code or a pipeline
var lookup = new LookupClient();                      // DnsClient
var result = await lookup.QueryAsync("bank.com", QueryType.MX);
foreach (var mx in result.Answers.MxRecords())
    Console.WriteLine($"{mx.Preference} {mx.Exchange}");

// A singleton HttpClient that still follows DNS changes
var http = new HttpClient(new SocketsHttpHandler
{
    PooledConnectionLifetime = TimeSpan.FromMinutes(2)
});

A static HttpClient without PooledConnectionLifetime keeps its old connections and never sees a DNS change. IHttpClientFactory avoids this by recycling handlers.

2.5 Communication protocols: the catalogue

Pick a protocol by the interaction you need: request/response, streaming, real-time push, messaging or file transfer.

ProtocolPatternTransport / formatBest forWatch out
REST over HTTPRequest/responseHTTP + JSONPublic and partner APIs, CRUDOver/under-fetching, chatty calls
SOAPRequest/responseHTTP/JMS + XML (WSDL)Legacy enterprise, strict contracts, WS-SecurityVerbose, heavy tooling
GraphQLQuery languageHTTP + JSONFront-ends needing flexible data shapesCaching, query cost, N+1 queries
gRPCUnary + streaming RPCHTTP/2 + ProtobufInternal service-to-service, low latencyBrowser support needs gRPC-Web
WebSocketFull-duplex, persistentUpgrade from HTTP over TCPChat, live trading, collaborationStateful connections are harder to scale
SSEServer → client streamHTTP, text/event-streamLive feeds, notifications, LLM token streamingOne direction only
WebhooksEvent callback (HTTP POST)HTTP + JSONNotifying partners when something happensRetries, signature verification, replay
AMQPMessaging (queues/topics)TCP, binaryRabbitMQ, Azure Service BusBroker operations
MQTTLightweight pub/subTCP, tiny headersIoT, mobile, unreliable networksLimited enterprise features
Kafka protocolEvent streaming logTCP, binaryHigh-volume streams, replayPartitions, ordering, operations
SFTP / FTPS / MFTFile transferSSH / TLSBatch files, bank and partner exchangePolling, partial files, latency
AS2 / AS4Signed B2B messagesHTTP + S/MIMEEDI, B2B with receipts (MDN)Certificate management
SMTPStore and forwardTCPEmail notificationsNot for system integration

Pick by need

Need an answer now?REST / gRPC
Push to a browser live?WebSocket / SSE
Tell a partner something happened?Webhook / event
Decouple internal systems?AMQP / Kafka
Big daily volume?SFTP batch file
  • JMS is a Java API, not a wire protocol. AMQP is a wire protocol.
  • gRPC is great between services. REST is better for public, discoverable APIs.
  • For any protocol, ask: how does it authenticate, retry, order messages and get monitored?
How to implement it
ProtocolPackageNotes
RESTbuilt-in minimal APIs / controllers, HttpClient; Microsoft.AspNetCore.OpenApiDefault choice for public and partner APIs
gRPCGrpc.AspNetCore, Grpc.Net.Client, Grpc.Net.ClientFactoryBrowser access: Grpc.AspNetCore.Web; expose as REST too: Microsoft.AspNetCore.Grpc.JsonTranscoding
GraphQLHotChocolate.AspNetCoreServer, with DataLoader to avoid N+1 queries
SOAPServer: CoreWCF.Http. Client: System.ServiceModel.Http + dotnet-svcutilsvcutil generates the client from the WSDL
WebSocket / SSEbuilt-in UseWebSockets, TypedResults.ServerSentEvents; SignalR (built-in)See 2.6 for details
AMQP / Kafka / MQTTAzure.Messaging.ServiceBus, RabbitMQ.Client, Confluent.Kafka, MQTTnetMessaging and event streaming clients
SFTP / FTPS / SMTPSSH.NET, FluentFTP, MailKitFile transfer and email. Don't use the old SmtpClient; Microsoft recommends MailKit

AS2/AS4 has no well-known free .NET library. It is usually handled by an MFT or B2B gateway product, not app code.

REST, gRPC, GraphQL, WebSocket, SSE, Webhooks, AMQP, Kafka, SFTP… which one? 🤔

Ask one question: what interaction do I need?

✅ Answer now → REST / gRPC ✅ Live push → WebSocket / SSE ✅ "Something happened" → Webhook / Event ✅ Decouple → Queue ✅ Bulk → File

The protocol follows the interaction, not the hype.

#Integration #APIs #Architecture

2.6 Short polling vs Long polling vs SSE vs WebSocket vs Webhook Added

Five ways to get updates from a server, from "keep asking" to "the server calls you".

  • Webhooks need HMAC signature checks, retries with backoff, idempotent handlers and fast 2xx replies (do the work async).
  • WebSockets and SSE need sticky connections or a pub/sub backplane (for example Redis or Azure SignalR) to scale out.
  • SSE reconnects automatically, which makes it great for streaming LLM tokens.
How to implement it
TechniqueLibrary / APINotes
Pollingbuilt-in minimal API endpoint + ETag / 304Plain GET. Return 304 when nothing changed
SSE (server)built-in TypedResults.ServerSentEvents (new in ASP.NET Core 10)Stream any IAsyncEnumerable<T> as text/event-stream
SSE (client)System.Net.ServerSentEvents (SseParser)Read an SSE stream from .NET (e.g. LLM token streams)
WebSocketbuilt-in app.UseWebSockets(), ClientWebSocketRaw two-way messages; you handle reconnects and scale-out
SignalRbuilt-in ASP.NET Core SignalR; scale-out: Microsoft.Azure.SignalR or Microsoft.AspNetCore.SignalR.StackExchangeRedisPicks WebSocket / SSE / long polling for you, adds groups and reconnects
Webhookbuilt-in HMACSHA256 + CryptographicOperations.FixedTimeEqualsVerify the signature before trusting the payload
// SSE: push price updates to the browser
app.MapGet("/prices", (PriceFeed feed, CancellationToken ct) =>
    TypedResults.ServerSentEvents(feed.StreamAsync(ct), eventType: "price"));

// Webhook: verify the HMAC signature, reply 2xx fast, process later
app.MapPost("/webhooks/payments", async (HttpRequest req, IConfiguration cfg, Channel<byte[]> inbox) =>
{
    using var ms = new MemoryStream();
    await req.Body.CopyToAsync(ms);
    var body = ms.ToArray();
    var expected = HMACSHA256.HashData(Encoding.UTF8.GetBytes(cfg["Webhook:Secret"]!), body);
    var given = Convert.FromHexString(req.Headers["X-Signature"].ToString());
    if (!CryptographicOperations.FixedTimeEquals(expected, given))
        return Results.Unauthorized();
    await inbox.Writer.WriteAsync(body);         // a worker does the real work
    return Results.Ok();
});

Compare signatures with FixedTimeEquals, never == or SequenceEqual, which leak timing. Also check a timestamp header to block replays.

2.7 Data formats & financial message standards Added

The protocol carries the message. The format defines what the message looks like.

FormatTypeNotes
JSONTextDefault for web APIs. Validate with JSON Schema
XMLTextSOAP, legacy systems, XSD validation
ProtobufBinarygRPC. Small and fast, schema in .proto
AvroBinaryKafka + Schema Registry, handles schema changes well
ParquetColumnarAnalytics, data lakes
CSV / fixed-widthTextBatch files. Agree on headers, encoding, control totals
ISO 20022XML financial messagingPayments (pacs, pain, camt). Replacing SWIFT MT; used in Australia's NPP
ISO 8583Binary card messagingCard authorisations, ATM/POS
How to implement it
FormatPackageNotes
JSONbuilt-in System.Text.JsonUse source generation for speed and Native AOT. JsonSchemaExporter creates a JSON Schema from a type
XML / ISO 20022built-in System.Xml (XmlSerializer, XmlReader + XmlSchemaSet)Generate classes from the ISO 20022 XSDs (pacs, pain, camt) with xsd.exe or dotnet-xscgen, then validate against the XSD
ProtobufGoogle.Protobuf + Grpc.ToolsGrpc.Tools generates C# from .proto files at build time
AvroApache.Avro + Confluent.SchemaRegistry.Serdes.AvroKafka messages with schemas checked by Schema Registry
Parquet / CSVParquet.Net, CsvHelperAnalytics files and batch files
// System.Text.Json source generation: no reflection, AOT-safe, faster
[JsonSerializable(typeof(Payment))]
[JsonSourceGenerationOptions(PropertyNamingPolicy = JsonKnownNamingPolicy.CamelCase)]
internal partial class AppJsonContext : JsonSerializerContext { }

var json = JsonSerializer.Serialize(payment, AppJsonContext.Default.Payment);

// ISO 20022: validate the XML against the official XSD before using it
var settings = new XmlReaderSettings { ValidationType = ValidationType.Schema };
settings.Schemas.Add(null, "pacs.008.001.08.xsd");
using var reader = XmlReader.Create(stream, settings);    // throws on an invalid message
var doc = (Document)new XmlSerializer(typeof(Document)).Deserialize(reader)!;

ISO 8583 (cards) has no standard Microsoft or widely used open-source .NET library. It is normally handled by the card switch or a vendor SDK.

⚡ Quick-fire Q&A

Why is gRPC faster than REST/JSON?

Binary Protobuf payloads, HTTP/2 multiplexing, persistent connections and generated client stubs.

WebSocket or SSE for a live dashboard?

SSE if data only flows server → client (simpler, plain HTTP, auto-reconnect). WebSocket if the client also sends frequent messages.

How do you secure webhooks?

HTTPS only, an HMAC signature with a shared secret (or mTLS), timestamps to block replays, an allowlist, idempotent processing and a quick 2xx acknowledgement.

← 01 · Foundations