Loading...
Master the Linux command line for managing GPU servers, automating ML workflows, and maintaining production infrastructure.
Linux is the operating system of AI. Nearly every GPU server, cloud instance, HPC cluster, and container orchestration platform runs Linux. Your ability to navigate the filesystem, manage processes, write automation scripts, and configure services directly determines how productive you are as an ML engineer. Engineers who are comfortable on the command line ship experiments faster, debug problems more quickly, and build more reliable infrastructure.
This chapter covers the essential Linux skills for AI engineering, organized in order of daily usage. You will start with the filesystem — understanding where things live and how permissions work. Then move to the commands you will use hundreds of times a day for file manipulation, searching, and piping. Process management teaches you how to keep training jobs alive and monitor GPU resources. Users and permissions set up shared infrastructure for team collaboration. Shell scripting automates everything you do manually. Package management prevents dependency hell. Cron and systemd handle scheduling and service management. And environment variables keep your configuration secure and portable.
Each topic is taught through the lens of real ML workflows — not abstract system administration. You will learn to organize training projects, analyze logs with command-line tools, manage multi-user GPU servers, write experiment launcher scripts, and deploy model servers as auto-restarting services. These are the skills that distinguish an ML engineer who gets things done from one who spends hours fighting with infrastructure.
Click any topic to jump in
FHS hierarchy, inodes, permission triplets, symlinks — the on-disk model every other Linux concept builds on.
ls, find, grep, pipes, redirection — the core verbs of getting work done at the shell.
PIDs, signals, nohup, tmux, nvidia-smi — how to launch, inspect, and recover long-running training jobs.
From files to running programs
Users, groups, sudo, umask, ownership — the security model that keeps multi-tenant GPU boxes sane.
Shebangs, variables, conditionals, loops, error handling — gluing tools into reproducible pipelines.
apt vs pip vs conda, virtual envs, requirements files — controlling exactly which versions actually run.
Scheduled jobs and managed services — how to make scripts run on a schedule and survive reboots.
export, .env, CUDA_VISIBLE_DEVICES, secrets management — the configuration channel between OS, shell, and apps.
The Linux file system is a hierarchical tree rooted at /. Every file, directory, device, and process has a place in this tree. As an AI engineer working on remote GPU servers, you'll spend significant time navigating this hierarchy — finding where CUDA libraries are installed, managing disk space for large datasets, and setting correct permissions on model files.
The key directories serve specific purposes. /home contains user directories where you'll keep your projects and virtual environments. /tmp provides temporary storage that's cleared on reboot — useful for intermediate training artifacts. /var holds variable data like logs (/var/log). /etc contains system configuration files. /usr houses user-installed programs and libraries. /opt is where many GPU drivers and CUDA toolkits are installed (/opt/cuda, /opt/nvidia). Understanding this layout helps you find things quickly on any Linux machine.
File permissions in Linux follow the owner-group-others model. Each file has read (r=4), write (w=2), and execute (x=1) permissions for three categories. A training script needs execute permission (chmod +x train.sh). Model files shared across a team need group read permissions. SSH keys must be restricted to owner-only (chmod 600 ~/.ssh/id_ed25519) or SSH will refuse to use them.
Tools like tree visualize directory structures, ncdu (NCurses Disk Usage) provides an interactive way to find what's consuming disk space — critical when your GPU server's disk fills up with model checkpoints and you need to quickly identify the largest files. The df and du commands show disk usage at filesystem and directory levels respectively.
The entire Linux filesystem is a single tree rooted at /. There are no drive letters like Windows. Every directory serves a specific purpose: /home stores user data and personal projects, /tmp provides temporary storage cleared on reboot, /var holds variable data like logs and caches, /etc contains system-wide configuration files, /usr houses installed programs and libraries, /opt stores optional software like CUDA, /proc exposes process information as virtual files, and /dev represents hardware devices including GPUs.
Knowing this layout means you can find things on any Linux machine. CUDA toolkit? Check /usr/local/cuda or /opt/cuda. System logs? /var/log. GPU device files? /dev/nvidia*. This consistency across all Linux distributions is one of the platform's strengths.
The Filesystem Hierarchy Standard (FHS) is a tree where every file lives at exactly one canonical path. Lookup is where is the depth — the kernel walks / → etc → nginx → nginx.conf doing one inode lookup per component. The dentry cache (dcache) hashes recently-resolved paths, dropping repeat lookups to .
You SSH into a new GPU server for the first time. You need to verify CUDA is installed, check available GPU devices, find how much disk space is free, and locate your home directory. What commands do you run?
Every file in Linux has three sets of permissions — for the owner, the group, and others. Each set can include read (r=4), write (w=2), and execute (x=1). The numeric representation sums these values: 755 means owner has rwx (7), group has r-x (5), others have r-x (5).
Critical permission values for ML work: 600 for SSH keys (owner read-write only — SSH refuses keys with looser permissions), 755 for scripts you want to execute, 775 for shared project directories (group-writable), and 644 for regular files (owner read-write, everyone else read-only).
The rwx mode is encoded in 9 bits (3 sets of 3), giving possible permission combinations. With the setuid/setgid/sticky bits added, that's 12 bits, hence the 4-digit octal mode (e.g., 0755). Each octal digit packs r=4, w=2, x=1 — so chmod 755 mentally decomposes to rwxr-xr-x in .
You get 'Permission denied' when running ssh -i ~/.ssh/gpu_key ubuntu@server even though the key file exists. ls -la ~/.ssh/gpu_key shows -rw-rw-r-- 1 alice alice. What's wrong?
The chmod command changes file permissions. It accepts either symbolic notation (chmod +x script.sh adds execute for all, chmod g+w file adds group write) or numeric notation (chmod 755 script.sh sets owner=rwx, group=rx, others=rx).
The -R flag applies permissions recursively to directories: chmod -R 775 /data/datasets/ sets all files and subdirectories to be group-writable. Be careful with recursive chmod on directories containing a mix of files and executables — you might accidentally make data files executable or remove execute permission from scripts.
chmod is a single inode update — regardless of file size. Recursive chmod -R walks the directory tree in syscalls for files, each one a separate lchmod() syscall costing – µs of kernel time. On a million-file dataset, that's seconds-to-minutes — long enough to matter on shared filesystems where contention amplifies it .
The chown command changes file ownership, setting both the user and group owner. The syntax is chown user:group file. This is critical when files created by root need to be accessed by service accounts, or when a new team member needs to take ownership of a project directory.
Common scenario: you install a Python package with sudo, and the resulting files are owned by root. Your training script can't write to those directories. Fix: sudo chown -R $USER:$USER ~/miniconda3/ transfers ownership back to you.
chown writes 8 bytes (uid + gid) to the inode. The interesting cost is on networked filesystems like NFS or Lustre, where it triggers a metadata RPC to the server — ms per file. Recursive chown on files over NFS is 1000 seconds of pure RPC stalls. Always prefer find -exec chown + to batch syscalls.
Symbolic links (symlinks) are shortcuts that point to another file or directory. Created with ln -s target link_name, they appear as aliases in the filesystem. A common ML use case: keep large datasets on a separate high-capacity volume while maintaining a clean project structure with ln -s /mnt/data-volume/imagenet ~/project/data/imagenet.
Symlinks are also used extensively for managing CUDA versions (/usr/local/cuda is often a symlink to /usr/local/cuda-12.2) and for Python virtual environments. The -s flag creates a symbolic (soft) link; without it, ln creates a hard link, which has different semantics.
A symlink is a tiny inode (typically <60 bytes) holding a path string. Resolution adds 1 syscall per symlink in the path. The kernel caps symlink resolution at 40 hops (MAXSYMLINKS) to prevent infinite loops — exceed it and you get ELOOP. Hard links bypass this entirely: they're just additional dentries pointing to the same inode, at all access times.
GPU servers frequently run out of disk space from accumulated model checkpoints, training logs, and dataset caches. df -h shows filesystem-level usage in human-readable format. du -sh * shows the size of each item in the current directory. ncdu provides an interactive browser where you can navigate directories and see sizes in real-time — press 'd' to delete files directly.
A common cleanup workflow: find checkpoints older than a week (find ./checkpoints -name '*.pt' -mtime +7), identify the largest files (du -sh * | sort -rh | head), and remove all but the best N checkpoints. Automate this with a cron job to prevent disk-full emergencies during overnight training runs.
du walks every inode in time and returns block-level usage; df reads filesystem superblock metadata in . They disagree when files are deleted but still held open by a process: the kernel keeps the inode, df reports the bytes as used, du can't see them. lsof | grep deleted finds the culprit. ML training logs are the canonical example.
You join a team where three ML engineers share a GPU server. Each engineer has their own home directory, but they need shared access to datasets in /data/ and model checkpoints in /models/. Currently, Alice's training scripts can't read Bob's checkpoints, and Carol's datasets are only accessible with sudo. Design a permission scheme that allows all three engineers to read and write shared resources while keeping home directories private.