Chapter 1: Networking Fundamentals

TCP connections begin with a three-step handshake: the client sends a SYN (synchronize) packet, the server responds with SYN-ACK (synchronize-acknowledge), and the client confirms with ACK. This takes one full round-trip before any application data can flow.

For API calls to distant servers, this initial handshake adds measurable latency. With a 100 ms round-trip time (RTT) to the server, that's 100 ms of pure handshake before the client can send its first byte of inference data, and the response arrives about one RTT after that. HTTP/2 and HTTP/3 mitigate this by multiplexing requests over a single connection.

TCP uses sequence numbers and acknowledgments to guarantee that every byte arrives in the correct order. Each byte of data is assigned a sequence number, and the receiver sends ACKs confirming which bytes it has received. If the sender doesn't receive an ACK within a timeout period, it retransmits the lost data.

This is why downloading a 5GB model file always completes correctly — even if individual packets are lost due to network congestion, TCP automatically retransmits them. The tradeoff is that a single lost packet can stall the entire stream (head-of-line blocking) until the retransmission arrives.

TCP adjusts its sending rate based on both the receiver's capacity (flow control) and network conditions (congestion control). The receiver advertises a window size indicating how much data it can buffer. The sender also maintains a congestion window that starts small (slow start) and grows exponentially until packet loss is detected, then backs off.

This slow-start behavior means that initial throughput on a new connection is low and ramps up over time. For transferring large datasets, this is a brief warm-up period. But for short-lived connections (individual API calls), you may never reach full bandwidth — another reason to use persistent connections.

UDP sends independent packets (datagrams) with no connection state, no ordering guarantees, and no retransmission. Each datagram is self-contained with source/destination ports and a simple checksum. The lack of overhead makes UDP faster and lighter than TCP.

UDP is ideal for real-time applications where receiving stale data is worse than losing data entirely. In a real-time video inference pipeline, a dropped frame from 100ms ago is irrelevant — you need the current frame. Retransmitting old frames would only add latency and waste bandwidth.

Choosing between TCP and UDP depends on your ML workload's requirements. Use TCP for model downloads (every byte must arrive), API calls (request-response requires reliability), database queries (data integrity is critical), and SSH (interactive sessions need ordering). Use UDP for real-time video inference (latest frame matters most), sensor data streaming (occasional loss is acceptable), and some custom distributed training communication layers where the application handles reliability.

In practice, most ML infrastructure uses TCP because reliability is almost always required. UDP appears mainly in specialized real-time inference pipelines and some high-performance computing frameworks that implement their own reliability layer on top of UDP for finer control.

A TCP connection moves through states: LISTEN on the server, SYN_SENT and SYN_RECEIVED during the handshake, ESTABLISHED while data flows, then FIN_WAIT, CLOSE_WAIT and LAST_ACK as each side closes. The side that closes first enters TIME_WAIT, which Linux holds for a fixed 60 seconds so delayed packets from the old connection cannot corrupt a new one with the same addresses and ports. A pile of CLOSE_WAIT means your application forgot to close sockets. A pile of TIME_WAIT on a client that opens a fresh connection per request can exhaust its ephemeral ports. Inspect states with ss -tan.

HTTP defines a set of request methods that indicate the desired action. GET retrieves data without side effects. POST sends data to create a resource or trigger processing (inference requests). PUT replaces a resource entirely. PATCH applies partial updates. DELETE removes a resource. HEAD retrieves headers only — useful for checking if a model file exists without downloading it.

For ML APIs, the most common pattern is POST for inference (sending input data and receiving predictions) and GET for metadata (listing available models, checking health status). Well-designed APIs use methods consistently: POST /predict for inference, GET /models for listing, DELETE /models/{id} for cleanup.

HTTP status codes are three-digit numbers that communicate the result of a request. 2xx codes indicate success (200 OK, 201 Created). 4xx codes indicate client errors (400 Bad Request for malformed input, 401 Unauthorized for missing API keys, 403 Forbidden, 404 Not Found, 429 Too Many Requests for rate limiting). 5xx codes indicate server errors (500 Internal Server Error, 503 Service Unavailable when the model is still loading).

For ML APIs, 429 (rate limited) and 503 (model loading) are especially important to handle. Implement exponential backoff for 429 responses and readiness probes for 503 during model warm-up.

Headers are name-value pairs that carry metadata about a request or response. Authorization carries credentials such as a bearer token. Content-Type declares the body format, for example application/json or multipart/form-data, and Accept states what format the client wants back. Content-Length, Cache-Control and Retry-After shape transfer, caching and retries, and a request-ID header lets you follow one call through several services. RFC 6648 deprecated the old convention of prefixing custom headers with X-, so newer standards drop it, as in RateLimit-Limit, though many APIs still send X-RateLimit-Remaining. Header names are case-insensitive, and HTTP/2 sends them lowercase.

HTTPS runs TLS between TCP and HTTP. In the handshake the server presents a certificate chain that the client checks against trusted authorities and the requested hostname, and both sides run an ephemeral Diffie-Hellman exchange, usually ECDHE, to derive session keys. Bulk data is then encrypted with a fast authenticated cipher such as AES-GCM or ChaCha20-Poly1305. TLS 1.3 removed static RSA key exchange and needs one round trip for the handshake, versus two for TLS 1.2. SSL is the obsolete predecessor, kept only in names. Let's Encrypt issues free certificates with automated renewal, so production model APIs have no excuse to skip HTTPS.

POST, PUT and PATCH requests carry a body, and the Content-Type header tells the server how to parse it. ML APIs use JSON for structured inputs such as text and parameters, multipart/form-data for file uploads, and sometimes raw binary. JSON is costly for numeric arrays: on the test machine, a 224 by 224 by 3 float32 tensor is 602,112 bytes raw but 3.15 MB as a JSON list, 5.2 times larger, and parsing it took about 34 ms. The same image as uint8 pixels is 150,528 bytes raw, and as a JPEG far less. Send compressed images or binary tensors, not floats as text.

HTTP/1.1 reuses TCP connections by default (Keep-Alive), avoiding the overhead of repeated handshakes. However, HTTP/1.1 processes requests sequentially on each connection — a slow response blocks subsequent requests on that connection.

HTTP/2 solves this with multiplexing: multiple requests and responses flow over a single TCP connection simultaneously. This dramatically reduces latency for batch inference calls where you send many predictions at once. HTTP/3 goes further by using QUIC (UDP-based) to eliminate head-of-line blocking at the transport layer.

An A record maps a domain name directly to an IPv4 address. For example, api.mymodel.com → 203.0.113.50. This is the most common DNS record type you'll configure when deploying ML services. A single domain can have multiple A records pointing to different IP addresses, providing basic round-robin load balancing.

When you deploy a model API on a cloud server, you create an A record pointing your custom domain to the server's public IP. If the server's IP changes (e.g., after restarting a cloud instance without an Elastic IP), you need to update the A record.

A CNAME (Canonical Name) record creates an alias that points to another domain name rather than directly to an IP address. For example, inference.myapp.com → my-gpu-server.us-east-1.compute.amazonaws.com. This is useful for pointing your custom domain to a cloud load balancer or CDN endpoint.

CNAME records add an extra DNS lookup (resolving the target domain), but they decouple your domain from specific IP addresses. If the cloud provider changes the load balancer's IP, your CNAME still works because it points to a name, not an address.

TTL specifies how long (in seconds) a DNS record should be cached by resolvers and clients before re-querying. A low TTL (60 seconds) allows fast failover — if you change a server IP, traffic redirects within a minute. A high TTL (86400 seconds / 24 hours) reduces DNS query load but makes changes slow to propagate.

For ML services: use a low TTL (60-300s) during deployments or migrations when you might need to quickly switch servers. Use a higher TTL (3600-86400s) for stable production services to reduce DNS lookup latency.

DNS results are cached at multiple levels: the browser cache, the operating system's resolver cache, your local DNS resolver, your ISP's recursive resolver, and potentially other intermediate resolvers. Each level respects the TTL value from the authoritative response.

This multi-level caching means DNS changes are never instant — old records persist at each cache layer until their TTL expires. When migrating an ML API to a new server, reduce the TTL to 60 seconds several hours before the migration, perform the switch, and then raise the TTL back up once you've confirmed stability.

When multiple A records exist for the same domain, DNS resolvers typically rotate through them, distributing requests across servers. This provides a basic form of load balancing for model inference endpoints without requiring a dedicated load balancer.

However, round-robin DNS has limitations: it doesn't check server health (traffic goes to crashed servers), doesn't account for server load (busy and idle servers get equal traffic), and client-side caching can cause uneven distribution. For production ML services, use a proper load balancer (Nginx, HAProxy, cloud LB) instead.

Within cloud VPCs, internal DNS resolves private hostnames (gpu-worker-1.internal) to private IP addresses without exposing them to the internet. AWS Route53 Private Hosted Zones, GCP Cloud DNS private zones, and similar services let you create internal DNS records accessible only within your network.

This is essential for ML infrastructure where multiple services communicate internally: the training coordinator, parameter server, data loading service, and monitoring stack can all have human-readable internal DNS names instead of hard-coded IP addresses.

IANA divides the 65,536 port numbers into three ranges. Well-known ports, 0 to 1023, belong to standard protocols: 22 for SSH, 53 for DNS, 80 for HTTP and 443 for HTTPS. Registered ports, 1024 to 49151, are assigned to applications, such as 3306 for MySQL, 5432 for PostgreSQL and 6379 for Redis. Dynamic ports above that are for temporary client use, though Linux actually picks client ports from 32768 to 60999. On Linux, binding below 1024 traditionally requires root or the CAP_NET_BIND_SERVICE capability. Standard numbers let firewalls and engineers recognize traffic at a glance, but anything can listen on any port.

The ML ecosystem uses several well-known registered ports: 8888 (Jupyter Notebook/Lab), 6006 (TensorBoard), 8000/8080 (FastAPI, Flask, and generic web servers), 8501 (TensorFlow Serving REST API), 8500 (TensorFlow Serving gRPC), 7860 (Gradio), 5000 (MLflow tracking server), 9090 (Prometheus metrics), and 3000 (Grafana dashboards).

When setting up a GPU server, you'll need to open these ports in the firewall for the specific tools you're using. A typical research setup might need SSH (22), Jupyter (8888), TensorBoard (6006), and a model server (8000).

iptables is the standard Linux firewall that filters network packets based on configurable rules. It organizes rules into chains: INPUT (incoming traffic), OUTPUT (outgoing traffic), and FORWARD (traffic being routed through). Each rule specifies conditions (source IP, destination port, protocol) and an action (ACCEPT, DROP, REJECT).

Rules are evaluated in order — the first matching rule wins. A common pattern is to ACCEPT specific ports (SSH, HTTPS, model server), then DROP everything else as a catch-all. Always place ACCEPT rules before DROP rules, or your allowed traffic will be blocked.

Cloud-level firewalls (AWS Security Groups, GCP Firewall Rules, Azure NSGs) control traffic to cloud instances. They operate at the cloud infrastructure level, separate from the instance's own iptables. Security groups are stateful — if you allow inbound traffic on a port, the corresponding outbound response is automatically allowed.

Always prefer security groups over iptables in cloud environments. They're easier to manage, can be shared across instances, and provide a clear audit trail. Reserve iptables for edge cases where you need finer-grained control.

Port forwarding redirects traffic from one port to another. SSH tunneling (the -L flag) is the most common form for ML engineers: ssh -L 8888:localhost:8888 gpu-server forwards your local port 8888 to the remote server's port 8888. You can then access Jupyter on the GPU server by opening localhost:8888 in your local browser.

This is more secure than exposing ports through firewalls because the traffic is encrypted through the SSH tunnel. You can forward multiple ports simultaneously: -L 8888:localhost:8888 -L 6006:localhost:6006 for Jupyter and TensorBoard.

Services bound to 127.0.0.1 (localhost) only accept connections from the same machine. Services bound to 0.0.0.0 accept connections from any network interface — meaning they're accessible from outside if the firewall allows it.

For development, bind to 127.0.0.1 for safety. For production ML servers that need to accept external requests, bind to 0.0.0.0 but protect with a firewall. A common mistake is running a model server bound to 127.0.0.1 and wondering why external clients can't connect, or binding to 0.0.0.0 without a firewall and exposing the service to the entire internet.

SSH authentication uses asymmetric cryptography. The private key stays on your machine (~/.ssh/id_ed25519) and must never be shared. The public key goes on remote servers (~/.ssh/authorized_keys). ed25519 keys are recommended over RSA — they provide equivalent security with shorter key lengths and faster operations.

The private key can be protected with a passphrase that must be entered each time the key is used. ssh-agent caches the decrypted key in memory so you don't re-enter the passphrase for every connection within a session.

The ~/.ssh/config file defines host aliases with all connection parameters, eliminating repetitive command-line flags. You can specify the hostname, username, identity file, port number, port forwarding rules, and connection keep-alive settings for each server.

This file makes SSH scripting much cleaner and reduces errors. Instead of remembering complex connection strings, you define them once and use a short alias. The config also supports wildcard patterns (Host *.internal) for applying settings to groups of hosts.

SSH local port forwarding tunnels a remote port to your local machine, creating an encrypted channel for accessing remote web services. The command ssh -L 8888:localhost:8888 gpu-server forwards your local port 8888 to port 8888 on the GPU server.

This is the standard way to access Jupyter notebooks, TensorBoard, MLflow, and Gradio interfaces running on remote servers. The traffic is encrypted through the SSH tunnel, so you don't need to expose these ports through firewalls.

scp copies files over SSH; since OpenSSH 9.0 it uses the SFTP protocol underneath, but it still copies whole files and cannot resume. rsync compares source and destination first: by default it skips files whose size and modification time match, and for changed files it uses a rolling checksum to send only the differing blocks. With --partial, or -P, which also shows progress, an interrupted transfer keeps the partial file and the next run reuses it. The usual command is rsync -avhP gpu:~/checkpoints/ ./local/, where -a preserves permissions and times. The trailing slash matters: it copies the directory's contents, not the directory.

sshfs mounts a remote filesystem locally via SSH using the FUSE (Filesystem in Userspace) framework. After mounting, you can browse remote files with your local file manager, open them in your local IDE, and run local tools on remote data as if it were on your machine.

This is particularly useful for editing training scripts and configuration files on a remote server using VS Code or another local editor. The command sshfs gpu:~/project /mnt/remote-project makes the remote project directory appear as a local directory.

ssh-agent is a daemon that caches decrypted private keys in memory, eliminating the need to re-enter your passphrase for every SSH connection. ssh-add loads keys into the agent. Agent forwarding (-A) passes your local keys to remote servers, letting you chain SSH connections without copying private keys to intermediate machines.

Use agent forwarding carefully — any user with root access on the intermediate server can use your forwarded keys while you're connected. For high-security environments, use ProxyJump instead of agent forwarding.

REST APIs organize endpoints around resources (nouns), not actions (verbs). A well-designed ML API uses paths like /v1/models/resnet50/predict rather than /v1/runPrediction. Version your API with a path prefix (/v1/, /v2/) to allow breaking changes without disrupting existing clients.

Return consistent JSON structures with clear status and data fields. Error responses should include a machine-readable error code, a human-readable message, and optionally a details field for debugging. This consistency makes your API predictable and easy to integrate with.

JSON is the default format for REST APIs because it is human-readable, self-describing and supported by every language, which makes debugging with curl easy. Its weakness is numeric data. On the test machine, a 224 by 224 by 3 float32 tensor took 602,112 bytes in memory but 3.15 MB as a JSON array, about 5.2 times larger, because each float printed as about 20 characters. Parsing it took about 34 ms, against under a microsecond to reinterpret the raw bytes. For images, send the compressed file, base64-encoded or as multipart; for tensors, use a binary format such as protobuf or raw bytes.

Protocol Buffers is Google's schema-based binary serialization format. You declare messages in a .proto file with typed, numbered fields, and the compiler generates code for Python, Go, C++ and other languages. On the wire, each field is a field number and type tag followed by the value: integers use variable-length encoding, so small numbers take one byte, and repeated floats are packed at 4 bytes each. Messages are typically several times smaller than JSON for numeric data and much faster to parse. Field numbers are the contract: you may add fields, but never reuse or renumber old ones, or old clients misread new data.

gRPC supports four communication patterns: unary (single request, single response — like REST), server streaming (one request, many responses — ideal for progressive inference results), client streaming (many requests, one response — for sending data in chunks), and bidirectional streaming (both sides send messages concurrently).

Server streaming is particularly useful for ML: the client sends an image, and the server streams back detection results as they're found, or streams intermediate attention maps during generation. Bidirectional streaming enables real-time video inference where frames flow in and detections flow out simultaneously.

API keys are the simplest authentication method — a secret string passed in the Authorization header. They're easy to implement but hard to scope (an API key typically grants full access) and revoke (changing the key affects all users).

JWT (JSON Web Tokens) are stateless tokens that contain encoded user information and permissions. OAuth2 provides scoped permissions with token expiration and refresh. For service-to-service communication within a cluster, mutual TLS (mTLS) ensures both sides verify each other's identity using certificates.

Rate limiting restricts the number of API requests a client can make within a time window to prevent abuse and ensure fair resource allocation. Standard headers communicate limits: X-RateLimit-Limit (max requests per window), X-RateLimit-Remaining (requests left), and X-RateLimit-Reset (when the window resets).

When a client exceeds the limit, the server returns 429 Too Many Requests. The client should implement exponential backoff: wait 1s, then 2s, then 4s, up to a maximum delay. For ML APIs, rate limiting is essential because each inference request consumes expensive GPU compute.

All-reduce is the fundamental communication pattern in data-parallel training. Every GPU sends its gradient tensor to every other GPU and receives the aggregated (summed) result. The naive approach of sending all data to one node and broadcasting back is bandwidth-wasteful.

Ring all-reduce is bandwidth-optimal: each of N GPUs sends and receives only 2*(N-1)/N of the total data. The GPUs form a logical ring, and data is passed in two phases — scatter-reduce and all-gather — each taking N-1 steps. This makes the communication cost independent of the number of GPUs (in terms of bandwidth), scaling linearly with the gradient size alone.

NCCL (NVIDIA Collective Communication Library, pronounced 'nickel') is NVIDIA's optimized library for GPU-to-GPU communication. It automatically detects the network topology (NVLink connections, PCIe layout, InfiniBand adapters) and selects the best algorithm for each collective operation (all-reduce, broadcast, all-gather, reduce-scatter).

NCCL supports multiple backends: NVLink for intra-machine GPU communication, InfiniBand with GPUDirect RDMA for inter-machine transfers (bypassing the CPU entirely), and TCP/IP sockets as a fallback. Setting NCCL_DEBUG=INFO reveals which topology and algorithms NCCL detected, which is essential for diagnosing performance problems.

Bandwidth drops sharply at each level of a GPU cluster. Within a GPU, HBM on an A100 delivers about 2 TB/s. Between GPUs in one server, NVLink gives 600 GB/s per A100, and PCIe Gen4 about 64 GB/s. Between servers, each GPU typically gets one 100 to 400 Gbps InfiniBand or EFA link, roughly 12 to 50 GB/s, and storage over the network is slower still. The hierarchy dictates placement: tensor parallelism, which exchanges activations inside every layer, stays within the NVLink domain, while pipeline and data parallelism, which communicate less often or can overlap, span servers.

Network can bottleneck data loading, not just gradient synchronization. If your DataLoader fetches training data from network-attached storage (NFS, S3, GCS) slower than the GPU processes batches, the GPU sits idle waiting for data. This is especially common with large image datasets or video training data.

Solutions include: pre-downloading data to local NVMe SSDs, using data format optimizations (WebDataset, TFRecord) that enable sequential reads, increasing DataLoader workers and prefetch buffers, and using cloud instances with high network bandwidth (AWS p4d instances have 400 Gbps networking).

Object stores such as S3 scale across many connections, not within one, so large transfers are split into parts that move in parallel. boto3's TransferConfig switches to multipart above multipart_threshold, 8 MB by default, uses multipart_chunksize parts, also 8 MB, and runs max_concurrency threads, 10 by default. Downloads use byte-range requests the same way. A single TCP flow is also capped: AWS limits single-flow traffic to 5 Gbps between instances outside a cluster placement group, so one stream cannot fill a 100 Gbps instance. For large checkpoints, raise concurrency and chunk size, and spread hot objects across prefixes.

In cloud GPU clusters, physical proximity matters. Placement groups (AWS) or compact placement policies (GCP) ensure that GPU instances are physically close — ideally on the same rack or connected to the same network switch. This minimizes inter-node latency and maximizes bandwidth.

Cross-region training (GPUs in different datacenters) is rarely practical due to WAN latency of 10-100ms versus LAN latency of 0.01-0.1ms. Even cross-availability-zone communication within a region adds measurable latency that impacts gradient synchronization.