Chapter 3: Docker & Containerization

Read-only templates with a layered filesystem — each layer is a cached diff (base OS, Python, pip packages, your code). Images are immutable once built. When you docker pull python:3.11-slim, you download a series of layers that stack to form a complete filesystem.

Layers are shared between images. If two images both start FROM python:3.11-slim, they share the base layers and only differ in the layers added on top. This saves disk space and speeds up pulls.

Running instances of images with a writable layer on top — isolated processes with their own PID namespace, network, and filesystem. When you docker run an image, Docker creates a thin writable layer over the read-only image layers.

Any files the container creates or modifies live in this writable layer. When the container is removed, this layer is deleted. This is why containers are ephemeral by design — they can be created and destroyed quickly without affecting the underlying image.

docker run creates a container from an image and starts it, pulling the image first if it is not local. The flags you will use most: -d runs it in the background, -it attaches an interactive terminal, --rm deletes the container when it exits, --name gives it a stable name, -p host:container publishes a port, -v or --mount attaches storage, -e sets environment variables, and --gpus exposes GPUs. Anything after the image name replaces the image's default command. Most problems show up here first, so when a container exits immediately, run it again without -d, or read docker logs, to see why.

docker ps lists containers. By default it shows only running ones with their ID, image, command, age, status, published ports and name. Add -a to include stopped containers, which is where a container that crashed at start-up will be hiding with an Exited status and its exit code. -q prints only IDs for scripting, such as docker rm on everything returned by docker ps -aq. --filter narrows by status, label, name or ancestor image, and --format prints chosen fields. -s adds each container's writable-layer size, which is how you spot a container quietly filling its layer with logs or checkpoints.

docker exec runs an extra command inside a container that is already running, joining its namespaces and cgroups. docker exec -it trainer bash opens a shell, and docker exec trainer nvidia-smi checks the GPU without stopping training. It only works on running containers, and it starts a new process rather than attaching to the main one (docker attach does that). Anything you change this way is lost when the container is replaced, so exec is for diagnosis, not configuration. Minimal images may have no bash, or no shell at all, so use sh or a debug sidecar instead.

A container moves through created, running, paused, exited and removed. docker create makes one, start runs it, pause freezes its processes with the cgroup freezer, and stop sends SIGTERM to process 1, then SIGKILL after a grace period, 10 seconds by default on Linux. An exited container keeps its writable layer, logs and exit code until docker rm deletes it. Exit code 137 means it was killed by SIGKILL, often from docker stop or the out-of-memory killer, and 143 means it exited on SIGTERM. A restart policy such as unless-stopped restarts containers that exit.

Lazydocker is an open-source terminal UI for Docker and Docker Compose, written in Go by Jesse Duffield. It shows containers, Compose services, images, volumes and networks in panels, with each container's logs, CPU and memory graphs and configuration beside them. Single keys restart, stop, remove, attach or open a shell, and it can prune unused objects. It talks to the same Docker API as the CLI, so it works over SSH against a remote GPU box. It is a viewer, not a replacement for scripts: everything it does maps to a docker command you can still automate in CI.

FROM chooses the base image every later instruction builds on, and it must come first (only ARG may precede it). The choice sets the image's floor size and much of its attack surface. python:3.11 is about 1.1 GB because it includes compilers and development headers, while python:3.11-slim is about 125 MB. Alpine is smaller still but uses musl instead of glibc, so many prebuilt Python wheels do not apply and packages compile from source. Pin a specific tag, never latest, and for full reproducibility pin the digest as well, so an upstream rebuild of the same tag cannot change your image.

Execute commands during build — each RUN creates a layer, so chain commands with && to reduce layers. Always clean up package manager caches in the same RUN command (e.g., rm -rf /var/lib/apt/lists/*) to avoid storing temporary files in the layer.

Bad: three separate RUN commands for apt update, apt install, and rm cache (3 layers, cache saved in first layer). Good: one chained RUN command that does all three (1 layer, cache never persisted).

COPY adds files from the build context, or with --from from another stage or image, into the image. Its cache key is a checksum of the copied files' contents and metadata, so the step and everything after it rebuild only when those files change. The standard ML pattern is to copy the dependency manifest first (requirements.txt, then pip install) and the source code later, so a code edit leaves the slow install cached. Use --chown to set ownership in the same step instead of a later chown RUN, which would duplicate the files in a new layer. Prefer COPY to ADD unless you need ADD's archive extraction.

WORKDIR sets the working directory for every later RUN, COPY, CMD and ENTRYPOINT in the stage, creating the directory if it does not exist. It is the right replacement for RUN cd /app, because each RUN starts a fresh shell: a cd inside one RUN does not carry into the next instruction. Relative WORKDIR paths are resolved against the previous WORKDIR, so use absolute paths to keep the Dockerfile readable. It also becomes the container's default directory at run time, so docker exec and the default command start there unless overridden with -w.

Both define what runs when the container starts. ENTRYPOINT is the fixed executable, and CMD supplies default arguments that anything after the image name in docker run replaces. ENTRYPOINT python3 with CMD train.py runs python3 train.py, while docker run img infer.py runs python3 infer.py. Use the exec form, a JSON array, for both. The shell form wraps the command in /bin/sh -c, which may leave the shell as process 1 so your program never receives SIGTERM, and a shell-form ENTRYPOINT ignores CMD and docker run arguments entirely. Override an ENTRYPOINT at run time with --entrypoint.

docker build checks each instruction against the cache. A RUN is reused if its command text and parent layer match a previous build, and a COPY or ADD is reused only if the copied files' checksums also match. The first miss invalidates that step and every step after it, even ones that would produce the same result. So order instructions from least to most frequently changed: base image, system packages, dependency manifests and install, then source code. A cached RUN apt-get update can also be stale, because the builder does not know the package index changed, which is why it is chained with the install.

The build context is every file in the directory you pass to docker build, and it is sent to the builder before any instruction runs. .dockerignore lists patterns, in the same style as .gitignore, to leave out of it. In ML repositories, ignore .git, data directories, checkpoints such as *.pt or *.safetensors, virtual environments, pycache, notebooks' outputs and local .env files. Excluded files cannot be copied, so COPY . . cannot accidentally ship a dataset or a secret. It also shrinks the image: in a test, a 200 MB data folder made COPY . . produce a 217 MB image instead of 7.8 MB.

Hadolint is a Dockerfile linter. It parses the Dockerfile into a syntax tree and checks it against about 70 rules numbered DL3000 and up, such as pinning versions in apt-get and pip installs, using --no-install-recommends, avoiding latest tags, preferring COPY to ADD, and setting a non-root USER. It also runs ShellCheck on the shell code inside RUN instructions, catching quoting bugs before they reach a build. Run it locally with docker run --rm -i hadolint/hadolint and redirect the Dockerfile to its standard input, and add it to CI so violations fail a pull request. Rules can be ignored per line or in a config file.

Dive is an open-source tool for exploring an image layer by layer. For each layer it shows the instruction that created it, its size, and a file tree of what it added, changed or removed. It also computes an efficiency score and lists wasted space: files that were written in one layer and overwritten or deleted in a later one but still ship in the image. Typical finds in ML images are pip and apt caches, test data, model checkpoints copied by COPY . ., and build tools. Run it in CI with the CI=true environment variable and a threshold to fail builds that waste too much space.

A single-stage Dockerfile keeps everything it ever installed, so an image that compiles code also ships the compiler. Multi-stage builds separate building from running. Each FROM starts a new stage, which you name with AS builder. A later stage starts from a clean base and copies in only the outputs it needs with COPY --from=builder. Only the last stage is tagged as the image unless you pick another with --target, which is also handy for a dev or test stage. BuildKit builds only the stages the target depends on, and runs independent stages in parallel.

The builder stage holds everything needed to produce artifacts and nothing needs to be small. It typically starts from a full image such as python:3.11 or a CUDA devel image, installs compilers, headers and build tools, and then builds: compiling wheels for packages with C or CUDA extensions, installing dependencies into a self-contained directory, or exporting a model. Two common patterns are pip install --prefix=/install and creating a virtual environment at /opt/venv, both of which leave a single directory to copy out. Order it for caching like any Dockerfile, with dependency manifests copied before source code.

The runtime stage is what you ship, so it should contain only what the program needs at run time. Start from a slim base, such as python:3.11-slim, a CUDA runtime image or a distroless image, install only runtime system libraries, and copy in the built environment, the application code and any model files. It has no compiler, no package caches and no build secrets. Set a non-root USER, a HEALTHCHECK if your platform uses it, and an exec-form ENTRYPOINT here. The base must be compatible with the builder: wheels built against glibc on Debian will not load on Alpine's musl.

COPY --from copies files from another stage, named or numbered, or from any image, into the current stage. COPY --from=builder /opt/venv /opt/venv moves a whole built virtual environment, and COPY --from=builder /install /usr/local moves packages installed with --prefix. Copying from an image works too, for example taking a static binary such as uv or tini from its official image without inheriting that image. Only the files you name cross over, so be explicit. Copying a whole build directory can pull in caches and object files and undo much of the saving.

Multi-stage builds remove what was needed only to build: compilers such as gcc and gfortran, development headers, package-manager caches and intermediate files. How much that saves depends on what dominates the image. For a compiled Go or Rust service it can be over 90 percent. For an ML image, the remaining frameworks and CUDA libraries usually dominate, so the saving is often a quarter to a half. Smaller images pull and start faster, which matters most for autoscaling. They also cost less to store and contain fewer packages that vulnerability scanners will flag.

BuildKit is Docker's build engine and the default since Docker Engine 23. It turns the Dockerfile into a dependency graph, runs independent stages in parallel and skips stages the target does not need. It adds RUN --mount: type=cache keeps pip, apt or model download caches between builds without baking them into a layer, and type=secret exposes a token, for a private package index, only during that RUN so it never lands in the image. Through docker buildx it can export cache to a registry or to GitHub Actions, so CI runners reuse layers, and build images for several CPU architectures.

The build stage is a good place for one-time model work: exporting a PyTorch model to ONNX, compiling it with TensorRT, or quantizing weights to INT8. The builder carries the full training framework, which can be several gigabytes. The runtime stage then needs only a lighter inference runtime such as ONNX Runtime plus the converted model. Two cautions. TensorRT engines are tied to the GPU architecture and TensorRT version they were built with, so build them on, or for, the deployment GPU. And always check the converted model's accuracy against the original before shipping it.

A named volume is a directory that Docker creates and manages, usually under /var/lib/docker/volumes, and mounts into containers with -v name:/path or --mount. It outlives every container that uses it, so a training job can write checkpoints to /ckpt, crash or be replaced, and the next container resumes from the same files. Writes bypass the overlay filesystem, so large files avoid copy-up costs. A new, empty volume mounted over an image directory is first filled with that directory's contents, which is how database images seed themselves. Volumes are removed only explicitly, with docker volume rm, docker compose down -v, or prune --volumes.

A bind mount maps an existing host file or directory into the container, for example -v ./src:/app/src, and changes on either side appear on the other at once. That makes it the standard development setup: edit code on the host and a reload server inside the container picks it up. Bind mounts also give containers access to host datasets without copying them. They tie the container to the host's directory layout, and the container sees host files with their real owners and permissions. Add :ro when the container should only read, such as for datasets, so a bug cannot overwrite them.

Both put data outside the container's writable layer, but they suit different jobs. A named volume is managed by Docker, independent of the host's directory layout, works the same on Linux and Docker Desktop, and is easy to back up or share between containers by name. It suits databases, caches and checkpoints. A bind mount points at a path you choose, so you can see and edit the files directly, which suits source code and existing datasets. Bind mounts expose host paths and permissions, so avoid them in production images, and never bind-mount sensitive host directories such as /etc into a container.

-p 8080:8000 publishes container port 8000 on host port 8080, so localhost:8080 reaches the service. Docker implements it with NAT rules in iptables or nftables and, by default, a small docker-proxy process. Without -p, the service is reachable only from other containers on its network. By default the port binds on all host interfaces, which can expose a notebook or database to the whole network and bypass host firewall rules. Use -p 127.0.0.1:8888:8888 for local-only tools. EXPOSE in a Dockerfile only documents a port; it publishes nothing by itself.

docker network create ml-net creates a user-defined bridge network. Containers attached to it get private addresses and can reach each other by container name or alias through Docker's embedded DNS server, so the API connects to http://model-server:5000 with no hard-coded IP. Each network is isolated from the others, so a database on a backend network cannot be reached from a container on a frontend network unless that container joins both. Other drivers exist: host shares the host's network stack, none gives no network, and overlay spans several hosts in Swarm.

Docker's built-in network named bridge, on the docker0 interface and the 172.17.0.0/16 subnet by default, is where containers go when you do not choose one. It has a key limitation: it has no name-based discovery. In a test, a container on the default bridge failed to resolve another by name ('bad address'), while on a user-defined bridge the same lookup worked at once. Containers there can still talk by IP address, or through the legacy --link flag. Docker's documentation calls user-defined bridges superior, so create one for any multi-container setup. Compose does this automatically for each project.

Volume drivers let a Docker volume live somewhere other than the local disk. The built-in local driver can already mount NFS or CIFS shares with docker volume create and driver options, and plugins add other backends, such as cloud file systems or distributed storage. For ML, the use case is shared data: several GPU hosts mount one dataset or checkpoint store read-only instead of copying it to each machine. Network storage is slower than local NVMe and shares one server's bandwidth among every client, so for heavy training I/O many teams copy or cache the hot data locally first.

The Compose file, now preferably named compose.yaml, declares an application's services, networks, volumes, configs and secrets. Each service lists its image or build context, command, ports, volumes, environment, health check, dependencies and resource limits. It is declarative: docker compose up compares the file with what is running and creates, recreates or leaves services as needed, so rerunning it after a small edit restarts only the changed service. The top-level version key is obsolete and ignored. Several files can be layered with -f, for example a base file plus a dev override that adds bind mounts.

A service is a container definition that Compose can run one or more copies of. It either names an image, such as redis:7-alpine, or a build section pointing at a Dockerfile, and adds ports, volumes, environment, command, health check, restart policy and deploy settings such as memory limits or GPU reservations. The service name doubles as its DNS name on the project network. docker compose up --scale worker=3 runs three replicas behind that name. Profiles let a service, such as a GPU trainer or a debugging tool, start only when its profile is requested.

depends_on controls the order in which Compose creates and starts services, and the reverse order in which it stops them. In its short form it only waits for the dependency's container to start, not for the program inside to be ready, so an API can boot before its database accepts connections. The long form adds a condition: service_started, service_healthy, which waits for the dependency's healthcheck to pass, or service_completed_successfully, which waits for a one-off job such as a migration or model download to exit with code 0. restart: true restarts the dependent service when the dependency is restarted.

Compose creates a network for each project, named after the project with a _default suffix, and attaches every service to it. Each service is reachable by its service name, so if the model server service is called model, other services connect to http://model:5000. Inside the network, services talk on container ports, so only ports for the outside world need publishing. You can declare extra networks to separate tiers: put the database on a backend network shared only with the API, and the API also on a frontend network with the proxy, so the proxy cannot reach the database directly.

Compose uses environment variables in two separate ways that are easy to confuse. Interpolation happens when the file is parsed: a reference to a variable in the YAML is filled in from your shell or from a .env file next to the Compose file. Container environment is what the program sees at run time, set with the environment key or loaded from files with env_file. A .env file therefore does not reach containers unless you also use env_file or pass its variables through. Keep real secrets out of the Compose file and out of Git, and prefer Compose secrets, mounted as files under /run/secrets, for passwords and keys.

docker compose up creates networks and volumes, builds or pulls images, and starts every service. Add -d to run in the background, --build to rebuild images first, and a service name to start just that service and its dependencies. docker compose down stops and removes the project's containers and networks, but keeps named volumes unless you add -v, which deletes them. Use that only when you mean to wipe databases and checkpoints. Day to day, docker compose ps shows status, logs -f follows every service's output, exec runs commands in a service, and watch syncs source edits into running containers.

Portainer is a web interface for managing Docker, Swarm and Kubernetes environments. The Community Edition runs as a container from portainer/portainer-ce, serves its UI over HTTPS on port 9443 with a self-signed certificate, and stores its settings in a volume. It controls Docker by mounting /var/run/docker.sock, which is equivalent to root access on the host, so protect its login, keep it off the public internet and update it. It shows containers, logs, stats and volumes and can deploy Compose stacks from a Git repository, which helps teammates who manage a shared GPU server but do not use the CLI.

The NVIDIA Container Toolkit lets containers use NVIDIA GPUs without installing the driver inside the image. When a container asks for GPUs, its runtime hook, or the newer CDI device specification, mounts the device nodes such as /dev/nvidia0 and /dev/nvidiactl and injects the host driver's user-space libraries, such as libcuda.so and libnvidia-ml.so, plus tools like nvidia-smi. The image provides everything above the driver: the CUDA runtime, cuDNN, NCCL and your framework. Install it on the host after the driver, then run nvidia-ctk runtime configure for Docker and restart the daemon so the runtime is registered.

--gpus tells Docker which host GPUs a container may use. --gpus all exposes every GPU, --gpus 2 exposes any two, and --gpus followed by device=0,1 in double quotes inside single quotes exposes those specific GPUs, by index or UUID. The quoting is needed because the value contains a comma. Inside the container, visible GPUs are renumbered from 0, so code never needs to know host indices. A capabilities option controls which driver libraries are injected, and Compose requests GPUs under deploy.resources.reservations.devices. Pinning workloads to separate GPUs is how several experiments share one 8-GPU machine without contention.

NVIDIA's nvidia/cuda images come in three main flavors. base has the minimal CUDA runtime library, runtime adds the CUDA math libraries such as cuBLAS and cuFFT, and devel adds nvcc, headers and static libraries for compiling. None contains the driver, which comes from the host. cudnn variants add cuDNN. For the 12.4.1 Ubuntu 22.04 tags, Docker Hub reports compressed sizes of about 32 MB for base, 1.4 GB for runtime, 2.1 GB for cudnn-runtime and 3.6 GB for devel. Note that PyTorch's pip wheels bundle their own CUDA libraries, so they need only a slim base, not a runtime image.

Running nvidia-smi in a container is the standard first test of GPU plumbing: docker run --rm --gpus all with an nvidia/cuda base image and the command nvidia-smi. It should list the GPUs with the host's driver version and the highest CUDA version that driver supports. The binary itself is injected from the host, so a failure points at the host setup. 'Could not select device driver' means the toolkit is not installed or Docker was not restarted, and 'executable file not found' usually means --gpus was missing. Remember that its CUDA Version line describes the driver, not the CUDA toolkit inside the image.

Two versions must fit together: the host driver and the CUDA runtime in the image. A driver supports every CUDA version up to the one nvidia-smi reports, so newer drivers run older images. Running a newer CUDA minor version on an older driver is possible within a major release through minor version compatibility, CUDA 12.x on drivers 525 and up, with some features unavailable, such as PTX JIT. But nvidia/cuda images set NVIDIA_REQUIRE_CUDA, and the toolkit refuses to start an image whose requirement the driver does not meet. Framework builds add a third constraint: the GPU architectures compiled into the wheel.

Multi-GPU training in a container needs more than --gpus all. Docker gives each container a 64 MB /dev/shm by default (a test container showed exactly 65,536 KB), but PyTorch DataLoader workers pass batches through shared memory and NCCL uses it between processes on the same node, so jobs crash with bus errors or shared-memory errors. Raise it with --shm-size=16g, or share the host's IPC namespace with --ipc=host. For training across nodes, use --network host or a fast container network, set NCCL_SOCKET_IFNAME to the right interface, and expose InfiniBand devices when the cluster has them.

nvidia-docker was NVIDIA's original approach: a wrapper command, and later a nvidia-docker2 package that registered an nvidia runtime with the Docker daemon. Docker 19.03 added the --gpus flag and the NVIDIA Container Toolkit replaced both, so the old packages are deprecated and no longer maintained. When old scripts use nvidia-docker run or --runtime=nvidia, replace them with docker run --gpus all after installing the toolkit. Some environments, such as older Kubernetes nodes, still configure the nvidia runtime as a default, which is why you may still see the name in configuration files.

A registry stores images as content-addressed blobs (compressed layers and configs) plus manifests that list them. Clients talk to it over the OCI distribution HTTP API: push uploads only blobs the registry does not already have, and pull downloads only blobs missing locally. Images are grouped into repositories and named by tags or by digest. Private registries add authentication, access policies, vulnerability scanning, retention rules and replication. For ML teams the main design questions are where the registry sits relative to the GPUs, since pulling multi-gigabyte images across regions is slow and billed, and who is allowed to push production tags.

Docker Hub is Docker's public registry and the default when an image name has no registry host: python:3.11-slim means docker.io/library/python:3.11-slim. It hosts Docker Official Images, verified publisher images such as nvidia/cuda, and community repositories. Pulls are rate-limited: unauthenticated clients get 100 pulls per 6 hours per IPv4 address or IPv6 /64, and free Personal accounts get 200, while paid plans are unlimited under fair use. CI fleets behind one NAT address hit the anonymous limit quickly, so log in with an access token, or mirror the bases you use into a private registry or pull-through cache.

Amazon Elastic Container Registry is AWS's managed private registry. Access is controlled with IAM, so ECS tasks, EKS nodes, SageMaker jobs and CI roles pull with their own role permissions instead of stored passwords. The docker CLI needs a token from aws ecr get-login-password, valid for 12 hours, or the ECR credential helper. ECR can scan images for vulnerabilities, expire old images with lifecycle policies, make tags immutable, replicate across regions and accounts, and act as a pull-through cache for Docker Hub and other registries. Pulls within the same region are free of data-transfer charges, and storage is billed per GB-month.

A tag is a human-readable, mutable pointer to an image manifest. my-model:v1.2.3 can be moved to a different image by pushing again, which makes tags convenient for people but unreliable as a deployment record. The common practice is to push several tags per build, such as the semantic version, the git commit SHA and an environment tag like staging, and to deploy by digest, my-model followed by @sha256 and the hash, which can never change. Avoid latest in production: it is simply the default tag, not the newest image, and it moves whenever anyone pushes without a tag.

Semantic Versioning names releases MAJOR.MINOR.PATCH. A PATCH release fixes bugs without changing behavior callers depend on, MINOR adds backward-compatible features, and MAJOR may break compatibility. For a model-serving image, the contract is the API: input and output schema, preprocessing and meaning of scores. A retrained model with the same schema might be a MINOR or PATCH bump, while a new output format or label set is MAJOR. Publishing the same digest as 1.4.2, 1.4 and 1 lets consumers choose how much change to accept automatically, with only the full version guaranteed never to move.

docker push uploads an image to the registry named in its tag, and docker pull downloads one. Both work per layer. A push first asks which blobs the registry already has and uploads only the missing ones, then the manifest. A pull fetches the manifest, then any layers not present locally, then unpacks them. So after the first push of a 5 GB ML image, a code-only change usually pushes one small layer in seconds. Authenticate with docker login or a credential helper first. The daemon uploads up to 5 layers and downloads up to 3 layers in parallel by default.

Skopeo is a command-line tool from the containers project for working with images in registries without a Docker daemon or local image storage. skopeo inspect reads an image's manifest and config, including digest, layers, labels and architecture, without pulling it. skopeo copy moves images between registries, or between a registry and a directory or archive, transferring layers directly, which is how teams mirror Docker Hub bases into a private registry. skopeo sync copies whole repositories, and skopeo list-tags lists available tags. Because it needs no daemon or root, it fits well in locked-down CI runners.

crane is a command-line tool from Google's go-containerregistry library for talking to registries directly. crane digest prints an image's digest, crane ls lists tags, crane manifest and crane config show the raw JSON, and crane copy moves images between registries. It can also change images without a daemon or rebuild: crane append adds a layer, crane mutate edits labels, entrypoint or environment, and crane tag adds a tag remotely without moving any layers. It is a single static binary, so CI jobs use it to resolve tags to digests before deploying and to promote a tested image by retagging it.