Chapter 6: Orchestration & Scaling
Deploy, scale, and manage ML workloads with Kubernetes, Helm, and distributed training infrastructure
Chapter Overview
Container orchestration is the discipline of managing containerized workloads across clusters of machines. For ML systems, orchestration solves the critical challenge of running inference APIs with high availability, scaling training jobs across GPU clusters, and managing the infrastructure complexity that comes with production ML.
Kubernetes has become the industry standard for orchestration. It provides a declarative model where you specify what you want (3 replicas of my model server, each with 1 GPU and 4 GB RAM) and the system continuously works to make reality match your specification. When a GPU crashes, Kubernetes restarts the Pod on a healthy node. When traffic spikes, the autoscaler adds replicas. When you deploy a new model, rolling updates ensure zero downtime.
This chapter builds from foundations to production-grade infrastructure. We start with why orchestration matters (single-server limits), then cover Kubernetes core concepts (Pods, Deployments, Services), configuration management (ConfigMaps, Secrets), autoscaling (HPA, VPA, cluster autoscaler), load balancing (Ingress, TLS), Helm packaging, and finally distributed training orchestration. By the end, you will understand how to operate ML workloads at scale with reliability, efficiency, and automation.
Chapter Roadmap
Click any topic to jump in
Why Orchestration?
Vertical limits, single points of failure, and the pets-vs-cattle mindset that motivates declarative orchestration.
Kubernetes Core
Pods, nodes, namespaces and the declarative API surface that defines every Kubernetes workload.
Exposing pods and externalising state
Deployments & Services
Rolling updates, label selectors and the four service types that expose pods to traffic.
Config & Secrets
Externalising configuration safely with ConfigMaps, Secrets, sealed secrets and external operators.
Scaling behaviour meets request routing
Scaling & Autoscaling
HPA, VPA, cluster autoscaler and the GPU-specific challenges of scale-to-zero.
Load Balancing
L4 vs L7, ingress controllers, TLS termination and circuit breaking patterns at the edge.
Helm & EKS
Templating clusters with Helm and provisioning managed Kubernetes on AWS with eksctl.
Distributed Training
Data and model parallelism, NCCL all-reduce, the PyTorch operator and Ray on Kubernetes.
Sign up to unlock this chapter
This chapter is part of PixelBank Premium. Create a free account, then upgrade to read the full lesson — concepts, walkthroughs, and exercises.