Chapter 4: Cloud & AWS Essentials

An instance type name reads family, generation, attributes and size. In p4d.24xlarge, p is the GPU-training family, 4 the generation, d means local NVMe disks, and 24xlarge the size. Common families are t and m for general purpose, c for compute-optimized, r for memory-optimized, and p, g and inf or trn for accelerators. Within a family, each size step roughly doubles vCPUs, memory and price. GPU counts do not follow the size names, though: g5.xlarge through g5.16xlarge all have one A10G, g5.12xlarge has four, and g5.48xlarge has eight. Read the spec table rather than inferring from the name, and pick the family whose strongest resource is your bottleneck.

The GPU families split into training machines and inference machines. Training: p3 (V100 16 or 32 GB), p4d (8x A100 40 GB), p4de (8x A100 80 GB), and p5 (8x H100 80 GB). Their GPUs are linked by NVLink and NVSwitch, and the instances by EFA networking, both of which multi-GPU training needs. Inference and light training: g5 (A10G 24 GB, g5.xlarge about 1 dollar an hour) and g6 (L4 24 GB, about 0.80). Choose by memory first, since the model, optimizer state and activations must fit. Then choose by interconnect if you train across GPUs, and finally by price per useful hour.

60-90% cheaper than on-demand — AWS can reclaim with 2-minute warning. Perfect for checkpoint-based ML training. When AWS needs the capacity back, your instance gets a 2-minute warning via the instance metadata endpoint.

The strategy: save checkpoints to S3 every N epochs. If the spot instance is interrupted, launch a new one and resume from the latest checkpoint. For a 10-hour training job, you might be interrupted once or twice, adding 15-30 minutes but saving 70-90% on compute cost.

The three EC2 pricing models trade commitment for discount. On-demand is the list price, billed per second, with no commitment. Reserved Instances and Savings Plans exchange a 1- or 3-year commitment for up to about 72 percent off, and suit the baseline you will certainly run, such as an always-on inference fleet. Spot uses spare capacity at a discount that varies by instance type and zone (often 50 to 90 percent), but AWS can reclaim it with two minutes' notice. Use it for checkpointed training and batch jobs. A typical mix commits the steady baseline, runs bursts on demand and puts fault-tolerant work on spot.

An AMI is the boot image an instance launches from: an OS root volume snapshot plus launch settings. AWS's Deep Learning AMIs come with NVIDIA drivers, CUDA, cuDNN, NCCL, EFA drivers and PyTorch already installed and tested together. That saves hours of driver debugging and gives every team member the same stack. You can also bake your own AMI from a configured instance, adding your dependencies, so autoscaled instances start ready instead of running a long setup script. The tradeoffs are size and staleness. A large AMI boots more slowly because blocks load lazily from S3, and a baked image misses security patches unless you rebuild it on a schedule.

A key pair is how SSH proves your identity to a Linux instance. AWS stores the public key and injects it into the instance's authorized_keys at launch. You keep the private key, which AWS does not keep a copy of. Lose it and you lose SSH access to instances that only trust that key. Protect it: chmod 400 the file, since SSH refuses keys that others can read, and never commit it or share it between people. Increasingly you can skip long-lived keys entirely. EC2 Instance Connect pushes a short-lived key per session, and SSM Session Manager gives a shell with no open port 22.

A security group is a stateful virtual firewall attached to an instance's network interface. It contains only allow rules, each naming a protocol, port range and source (a CIDR block or another security group). Anything not allowed is dropped. Stateful means replies to allowed traffic flow back automatically, so you do not write return rules. A new group allows all outbound traffic and no inbound traffic. Open only what you need: SSH from your IP or VPN range, never 22 to 0.0.0.0/0, and your API port from the load balancer's group. Referencing groups instead of IP ranges keeps rules correct as instances come and go.

boto3 is the AWS SDK for Python. It exposes every EC2 API, so a script can launch an instance with run_instances, wait for it with a waiter, tag it, and terminate it when training ends. That makes it the glue for automated ML workflows, such as a nightly job that launches a GPU spot instance, runs training from a container, uploads the model to S3 and terminates. Two pitfalls: list calls return pages, so use paginators, and always terminate in a finally block, or a crash leaves a GPU instance running.

A bucket is the top-level container for objects, and the unit where you set region, access policy, versioning, encryption and lifecycle rules. Bucket names are global across all AWS accounts, because they appear in DNS names like name.s3.amazonaws.com. A name like 'datasets' is long gone, so use names such as company-ml-datasets. A bucket lives in one region, which matters for latency, transfer cost and data-residency law. Accounts get up to 10,000 general-purpose buckets by default. Teams usually split buckets by access boundary, such as raw data, curated datasets and production models, and use prefixes inside each bucket for projects. Block Public Access should stay on.

An object is a blob of bytes plus metadata, stored under a key such as models/v2/resnet50.onnx. Objects can be up to 50 TB. The key space is flat: models/ is not a directory, just a shared prefix, which is why 'renaming a folder' means copying and deleting every object under it. Objects are immutable. To change one you write a new version of the whole object, and there are no appends or in-place edits. S3 has given strong read-after-write consistency since December 2020, so a read after a successful PUT always returns the new data.

Storage classes trade storage price against access cost. In us-east-1, Standard costs about 0.023 dollars per GB-month with no retrieval fee, for data you read often. Standard-IA costs about 0.0125, but adds a per-GB retrieval fee and a 30-day minimum, for data read about monthly. Glacier Instant Retrieval costs 0.004 and Glacier Flexible Retrieval 0.0036, the latter with restores taking minutes to hours. Deep Archive costs 0.00099, with restores taking up to 12 hours and a 180-day minimum. Intelligent-Tiering moves objects automatically for a small monitoring fee. Choose by how often you read the data, not just how cold it feels. Repeatedly reading archived data quickly erases the savings.

A lifecycle configuration is a set of bucket rules that act on objects by age, filtered by prefix or tag. Transition actions move objects to cheaper classes, and expiration actions delete them. For example, logs/ moves to Standard-IA after 30 days, to Glacier Flexible Retrieval after 90, and expires after 365. Constraints matter: objects must spend 30 days in Standard before moving to an IA class. Each transition request has a cost, so moving millions of tiny objects can cost more than it saves. Archive classes bill minimum durations if you delete early. With versioning on, add rules for noncurrent versions too.

With versioning enabled, every overwrite of a key creates a new version instead of replacing the bytes. A delete adds a delete marker that hides the object without removing earlier versions. You can list versions, read any one by its version ID, and restore an old one by copying it back on top. For ML artifacts this guards against the common accident of a training job writing model.pt over yesterday's good model. Every version is billed, though, so a checkpoint overwritten hourly accumulates hundreds of full copies. Pair versioning with a lifecycle rule that expires noncurrent versions after N days. For deliberate model versions, put the version in the key.

aws s3 sync copies a directory tree between local disk and S3, or between buckets, transferring only files that are new or differ in size or modification time. Transfers run in parallel and large files use multipart upload, so syncing a dataset directory is far faster than a loop of single copies. Flags shape the result: --exclude and --include filter by glob (for example, skipping pycache and .pyc files), --delete removes destination files missing from the source, and --dryrun shows what would happen. Run --dryrun before any --delete, because a sync from the wrong directory with --delete will wipe the destination's extra files.

s3fs-fuse mounts a bucket as a local directory, so unmodified code can call open() on S3 objects. It is convenient for browsing, small scripts and legacy tools, but the mapping from files to objects leaks. Each small read becomes an HTTP request with tens of milliseconds of latency. A random write or append rewrites the whole object. That makes it a poor training data loader: thousands of small images are read at network latency instead of SSD speed. For training, stage data to local NVMe, read large shards sequentially, or use a purpose-built client such as Mountpoint for Amazon S3.

The boto3 S3 client gives scripts and pipelines direct access: upload_file and download_file for files, get_object and put_object for bytes, and list_objects_v2 (through a paginator, since each call returns at most 1,000 keys) for listings. upload_file and download_file use the transfer manager, which splits large files into parts and moves them concurrently. Tune it with TransferConfig through multipart_threshold, multipart_chunksize and max_concurrency. Generate presigned URLs to let a client upload or download one object without AWS credentials. On EC2, SageMaker or Lambda, let boto3 pick up the role's temporary credentials instead of passing keys.

An IAM user is a long-lived identity inside one account, with a console password, up to two access keys for the CLI and SDKs, or both. Each person gets their own user, never a shared login, so CloudTrail can attribute every action and offboarding means disabling one identity. Access keys never expire on their own, which makes them the most dangerous credential AWS issues. AWS's current guidance is to avoid IAM users for people where possible. Use federation or IAM Identity Center, which hand out temporary credentials, and keep users for the rare tool that can only take static keys.

A role is an identity with permissions but no long-term credentials. A trusted principal assumes it through STS and receives temporary credentials (an access key, secret and session token) that expire, typically after 1 hour and at most 12. Its trust policy says who may assume it, such as the EC2 service, a Lambda function, a GitHub OIDC identity or another account, and its permission policies say what it can do. Attach a role to a training instance through an instance profile, and boto3 fetches and refreshes credentials from the metadata service automatically. Nothing is written to disk, so there is nothing to commit, copy or forget to rotate.

A policy is a JSON document of statements. Each statement has an Effect (Allow or Deny), Actions such as s3:GetObject, Resources given as ARNs like arn:aws:s3:::ml-datasets/, and optional Conditions, such as requiring MFA or a source VPC endpoint. Identity policies attach to users, groups and roles. Resource policies, such as bucket policies, attach to the resource and name principals. AWS evaluates every applicable policy together. An explicit deny anywhere wins, otherwise any allow grants access, and anything not allowed is implicitly denied. A frequent bug comes from resource scope: s3:ListBucket applies to the bucket ARN, while object actions apply to bucket/.

Grant minimum permissions needed — start with zero access, add permissions as needed. This is the most important security concept in cloud computing. Instead of starting with admin access and trying to restrict it, start with no access and add only what each user or service needs.

For ML teams, this means: training instances can read data and write models, inference endpoints can read models and serve predictions, and only administrators can modify infrastructure.

AWS managed policies, such as AmazonS3ReadOnlyAccess or AmazonSageMakerFullAccess, are written and updated by AWS for common job functions. They are quick to attach and stay current as services add actions, but they are broad by design. AmazonS3FullAccess grants every S3 action on every bucket in the account. Customer managed policies are ones you write: reusable, versioned and scoped to your buckets, regions and tags. Inline policies are embedded in a single identity and deleted with it. A sensible path is to prototype with an AWS managed policy, then replace it with a scoped customer policy, using IAM Access Analyzer to generate one from the actions CloudTrail shows were actually used.

A group is a collection of IAM users that share attached policies, for example ml-engineers, data-scientists or admins. You manage permissions on the group, and membership decides who gets them. Adding a new hire to data-scientists grants the full set at once, and removing someone revokes it. Without groups, policies end up attached to individuals, and over time each person's permissions drift into a unique, unreviewable mix. Groups cannot be nested, cannot be principals in a resource policy, and cannot be assumed like roles. In organizations that use IAM Identity Center, permission sets assigned to directory groups play the same part for federated users.

aws-vault is an open-source CLI that stores your long-term AWS access keys in the operating system's keychain instead of plaintext in ~/.aws/credentials. For each command it requests short-lived STS credentials, optionally after an MFA prompt and a role assumption, and passes them to the process as environment variables, as in aws-vault exec ml-dev -- python train.py. Malware or a careless backup that reads your home directory finds no permanent key, and anything it captures from a running process expires within the hour. The tool protects keys on developer laptops. For servers and CI, use roles and OIDC instead, so there are no long-term keys to store at all.

Multi-factor authentication requires a second proof, such as a code from an authenticator app or a hardware security key, in addition to a password or access key. Enable it on the root user first, then on every human user. In IAM policies, the aws:MultiFactorAuthPresent condition key lets you require MFA for dangerous actions, for example denying ec2:TerminateInstances or s3:DeleteBucket without it. Phishing-resistant factors (FIDO2 security keys and passkeys) are stronger than one-time codes, because they are bound to the real sign-in domain and cannot be relayed through a fake login page. MFA does not protect access keys, which is another reason to prefer temporary credentials.

A VPC is your private network inside one AWS region. When you create it you choose an IPv4 CIDR block, such as 10.0.0.0/16, which holds 65,536 addresses. Every subnet, instance and interface in the VPC takes its private address from that range. Nothing is reachable from the internet by default. Public access exists only where you add an internet gateway, a route to it and a public IP. Plan the CIDR before you build. It is hard to change later, and if two VPCs or an office network overlap, they cannot be peered or connected with a VPN.

A subnet is a slice of the VPC's CIDR block that lives in exactly one availability zone. 'Public' and 'private' describe routing, not a setting. A public subnet's route table sends 0.0.0.0/0 to an internet gateway, and a private subnet's either has no default route or sends it to a NAT gateway. The standard pattern is one public and one private subnet per zone, across at least two zones. Load balancers, bastions and NAT gateways go in the public subnets. Training instances, model servers and databases go in the private ones.

In a VPC, security groups are the main per-instance firewall, and their key feature is that rules can reference other security groups instead of IP ranges. A rule such as 'allow TCP 5000 from api-sg' admits any interface that carries api-sg, wherever it is and however many there are, so autoscaled instances need no rule changes. Groups are stateful, so replies to allowed traffic return automatically, and they hold only allow rules, which are combined: traffic passes if any rule matches. Give each tier its own group (load balancer, API, model server, database) and chain them, so each tier accepts traffic only from the one in front of it.

A network ACL is a stateless firewall attached to a subnet, evaluated before traffic reaches any security group. Rules are numbered and checked in order, the first match wins, and unlike security groups they can deny as well as allow. Stateless means return traffic is not remembered. If you allow inbound HTTPS, you must also allow outbound replies on the ephemeral ports (1024 to 65535). Forgetting this is the classic NACL outage. The default NACL allows everything, while a newly created custom NACL denies everything until you add rules. Most teams leave NACLs permissive and use them only for coarse subnet-wide blocks, such as denying a known-bad CIDR.

An internet gateway is the VPC component that connects public subnets to the internet. It is horizontally scaled and highly available, and AWS imposes no bandwidth limit or hourly charge for it. It does two jobs. It is the target of a subnet's 0.0.0.0/0 route, and it performs one-to-one translation between an instance's private IP and its public or Elastic IP. Both conditions are needed for internet access: a subnet route to the gateway and a public IP on the instance. Either alone does nothing. One gateway attaches per VPC. Private instances that only need outbound access should use a NAT gateway instead.

A NAT gateway lets instances in private subnets start outbound connections, for pip install, apt or calls to external APIs, while blocking any connection started from the internet. It sits in a public subnet with an Elastic IP, and private route tables send 0.0.0.0/0 to it. Each one lives in a single zone, so put one per zone to survive a zone failure and to avoid cross-zone transfer. It starts at 5 Gbps and scales automatically to 100 Gbps. It is also a classic cost trap: about 0.045 dollars an hour plus 0.045 per GB processed, including S3 traffic, unless a gateway endpoint carries that traffic instead.

A VPC endpoint gives private subnets access to an AWS service without the internet, an internet gateway or a NAT gateway. Gateway endpoints exist for S3 and DynamoDB. They are entries in route tables and cost nothing. Interface endpoints, built on PrivateLink, place elastic network interfaces in your subnets for services such as ECR, CloudWatch, STS and SageMaker, and are billed per hour and per GB. Endpoint policies and bucket conditions like aws:sourceVpce let you require that a bucket only be reached through your endpoint. For ML, the S3 gateway endpoint is nearly mandatory: it removes NAT charges on dataset reads and keeps the traffic on AWS's network.

Writing the network in Terraform turns dozens of console clicks into reviewed code. A typical VPC needs the VPC itself, a public and a private subnet per zone, route tables and their associations, an internet gateway, NAT gateways, an S3 endpoint and security groups. That is easily 20 to 30 resources, wired together by references. The community terraform-aws-modules/vpc module packages that pattern, so dev, staging and production can share one definition and differ only in inputs such as the CIDR and the number of NAT gateways. terraform plan shows a reviewer exactly which rule or route changes, which is how you catch an accidental 0.0.0.0/0 before it ships.

A Lambda function is a handler, such as def handler(event, context) in Python, plus configuration: runtime, memory, timeout, environment variables and an execution role. Each invocation gets an event (an HTTP request, an S3 notification, a queue batch) and returns a result. Lambda runs the handler in an execution environment, a Firecracker microVM, that it creates on demand, reuses for later calls and discards when idle. Concurrency scales by adding environments, up to a default limit of 1,000 per region. Pricing is 0.20 dollars per million requests plus about 0.0000167 dollars per GB-second. Functions must be stateless between calls, so keep state in S3, DynamoDB or a queue.

A trigger connects an event source to a function, and the source sets the invocation mode. Synchronous sources, such as API Gateway, function URLs and the SDK's RequestResponse calls, wait for the result and return errors to the caller. Asynchronous sources, such as S3 notifications, EventBridge schedules and SNS, put the event on an internal queue, and Lambda retries failures twice before sending them to a dead-letter destination. Poll-based sources, such as SQS, Kinesis and DynamoDB streams, have Lambda read batches, and a failing batch is retried. Design handlers to be idempotent, because retries can deliver an event twice.

A cold start happens when no idle execution environment is available, so Lambda must create one. That means starting a microVM, downloading the code or image, starting the runtime and running your initialization code before the handler. Afterwards the environment stays warm for an unpublished, variable period and serves later calls without that cost. Cold starts are around 100 to 300 ms for a small Python function, but importing PyTorch or loading a model can add seconds. Load the model at module level, outside the handler, so the cost is paid once per environment rather than per request. Use provisioned concurrency to keep environments initialized when latency matters.

A layer is a zip archive of libraries, a custom runtime or data that Lambda extracts into /opt for any function that references it. Layers let several functions share one copy of numpy, scikit-learn or a common utilities package, so deployment packages stay small and dependency upgrades happen in one place. They do not raise the size limit: function code plus all layers must fit in 250 MB unzipped. That is too small for PyTorch, and model weights do not belong in a layer anyway. Load weights from S3, or switch to a container image, which allows up to 10 GB. Layers are versioned, so pin a version number.

Memory is the one sizing knob. It runs from 128 MB to 10,240 MB, and CPU is allocated in proportion: at 1,769 MB a function gets the equivalent of one full vCPU, and at the top setting about six. Timeout runs from 1 second to 15 minutes. For inference, more memory often lowers cost, because the extra CPU shortens each call by more than the price per millisecond rises. AWS Lambda Power Tuning measures this across settings. Anything that needs a GPU, more than 10 GB of RAM or runs longer than 15 minutes belongs on ECS, Batch or SageMaker.

Lambda can run a container image of up to 10 GB from ECR instead of a zip package. That lifts the 250 MB limit, so PyTorch, ONNX Runtime and their native libraries fit, and you can build with the same Dockerfile workflow you use elsewhere. The image must implement the Lambda Runtime API. The simplest route is an AWS base image such as public.ecr.aws/lambda/python, or adding the runtime interface client to your own. Lambda caches and lazily loads image chunks, so a large image does not necessarily start slower than a zip. Initialization code that imports heavy libraries or downloads weights still dominates cold starts.

The AWS Serverless Application Model is an open-source framework for defining serverless applications in a template.yaml. It is an extension of CloudFormation: resource types such as AWS::Serverless::Function and AWS::Serverless::Api expand at deploy time into the full set of functions, roles, permissions and event mappings. sam build packages code and dependencies, sam local invoke and sam local start-api run functions in a local container for testing, and sam deploy creates or updates the CloudFormation stack. Because the template lives in Git and deploys through CloudFormation, changes are reviewable and roll back as a unit. It is AWS-only, which is the main tradeoff.

The Serverless Framework is a third-party CLI that deploys functions from a serverless.yml file describing functions, events and resources. Running serverless deploy packages the code and, on AWS, generates and applies a CloudFormation stack, much as SAM does. Its strengths are a large plugin ecosystem, for example serverless-python-requirements for building native dependencies in Docker, and support for several cloud providers from one configuration format. Recent major versions need a Serverless account and a paid license above a free tier for larger organizations, so check the terms before standardizing on it. For AWS-only teams, SAM or Terraform avoids that dependency.

ECR hosts private repositories of OCI and Docker images. Each image is addressed by a repository URI, account.dkr.ecr.region.amazonaws.com/name, plus a tag or digest. Authentication uses IAM: aws ecr get-login-password gives docker a 12-hour token, and AWS services pull with their own roles, so no registry password is stored anywhere. Images are encrypted at rest, can be scanned on push for known CVEs, and can be replicated to other regions or accounts. Storage costs about 0.10 dollars per GB-month. Pulls are free within the same region, but pulls from private subnets still go through NAT unless you add the ECR and S3 endpoints.

A lifecycle policy is a set of prioritized JSON rules that ECR applies to a repository to expire images automatically. Rules select images by tag status (tagged, untagged, or tags matching a prefix or pattern) and by either count (imageCountMoreThan) or age (sinceImagePushed). For example, keep the 20 most recent images tagged with a release prefix, and expire untagged images after 1 day. A CI pipeline that pushes an image per commit needs this, or storage grows without limit. Use the console's preview to test a policy first, and never select by age alone on tags production depends on.

A training job is one managed, ephemeral run. You specify the image, instance type and count, input channels in S3, hyperparameters, an output path and an IAM role. SageMaker provisions the instances, pulls the image, copies each channel to /opt/ml/input/data/<channel> (or streams it in fast-file or pipe mode), runs your entry point, then tars /opt/ml/model and uploads it to S3 before releasing the instances. Billing covers only the instances' running time, which removes the forgotten-instance failure. Managed spot training uses spare capacity at up to 90 percent off. Set a checkpoint S3 URI and resume from /opt/ml/checkpoints, because an interrupted job restarts from scratch otherwise.

A real-time endpoint hosts one or more models on instances you choose, behind an HTTPS API that you call with InvokeEndpoint. You pay per instance-hour whether or not traffic arrives. An endpoint configuration defines production variants, each a model on an instance type with an instance count and traffic weight, which enables A/B tests and canary shifts between model versions. Application Auto Scaling adjusts instance counts on metrics such as invocations per instance. Updating an endpoint uses blue-green deployment, with optional automatic rollback on CloudWatch alarms. For rare or bursty traffic, consider serverless inference or asynchronous inference instead, which can scale down to zero instances.

The Estimator is the SageMaker Python SDK class that defines a training job. You construct it with an image URI, or a framework class such as PyTorch with a framework version and entry_point script, plus a role, instance type and count, hyperparameters, and options such as use_spot_instances, max_wait and checkpoint_s3_uri. Calling estimator.fit({'train': 's3://bucket/train'}) submits the job, streams its logs and returns when it finishes. estimator.model_data then holds the S3 URI of model.tar.gz, and estimator.deploy creates an endpoint from it. The abstraction is convenient, but it hides real API calls. Read the resulting job in the console to understand what it actually ran.

Bring your own container means SageMaker runs your Docker image instead of one of its framework images. The contract is a set of paths and commands. For training, SageMaker runs the image with the argument train. Your code reads hyperparameters from /opt/ml/input/config/hyperparameters.json and data from /opt/ml/input/data/<channel>, writes the model to /opt/ml/model, and writes a failure message to /opt/ml/output/failure. For hosting, it runs serve, and your server must answer GET /ping and POST /invocations on port 8080. The sagemaker-training-toolkit implements the training side for you. BYOC is how you get unusual dependencies or custom CUDA builds while keeping managed spot, logging and lifecycle.

SageMaker Studio is a web IDE for ML, with JupyterLab and code-editor spaces running on instances you pick, CPU or GPU, plus panels for experiments, training jobs, pipelines, the model registry and endpoints. Spaces store work on a persistent volume, while the compute behind them can be stopped and restarted on a different instance type. That makes it easy to prototype on a small CPU instance and switch to a GPU only for the step that needs it. The common failure is cost. A GPU space left running overnight bills like any instance, so configure idle shutdown.

The SageMaker Python SDK is a high-level library over the SageMaker APIs. Estimators and framework classes launch training jobs, Model and Predictor classes deploy and call endpoints, Processing and Transformer classes run preprocessing and batch inference, and the workflow module defines SageMaker Pipelines. It handles plumbing such as uploading a local source directory to S3, picking framework image URIs for a region, and polling jobs, all of which you would otherwise script with boto3. The SDK's major versions have changed APIs, so pin the version in requirements.txt. For infrastructure that should be reproducible and reviewed, such as endpoints and roles, prefer Terraform or CloudFormation.

On-demand is EC2's list price: no commitment, no upfront payment, billed per second for Linux after a one-minute minimum. Examples in us-east-1 are about 21.96 dollars an hour for a p4d.24xlarge, 1.01 for a g5.xlarge and 0.04 for a t3.medium. You pay for every second an instance is running, whether or not it does useful work. On-demand fits development, experiments of unknown length and short spikes. It also serves as the reference point for every other model's discount. Prices change, and AWS cut P4 and P5 prices by up to 33 and 45 percent in 2025, so read current figures rather than old notes.

Spot instances use EC2's spare capacity at a discount that varies by instance type, zone and time. The Spot Instance Advisor recently showed about 56 percent off for p4d.24xlarge and 86 percent for p3.2xlarge. In return, AWS can reclaim an instance with a two-minute warning, delivered through instance metadata and EventBridge. Spot pays only if interruptions are cheap. Checkpoint to S3 every 15 to 30 minutes, watch for the warning and save immediately, and resume automatically. Spread requests across several instance types and zones, because the scarcest GPU types are also the most frequently reclaimed.

A Reserved Instance is a billing discount, not a machine. You commit to an instance type in a region (Standard RI) or an exchangeable family (Convertible RI) for 1 or 3 years, paying all, part or nothing upfront, and matching usage is billed at the reduced rate. Savings can reach about 72 percent for a 3-year all-upfront Standard RI. The catch is that you pay for every committed hour whether or not it is used. That makes RIs right for a steady base load, such as an inference fleet that never scales below four instances, and wrong for experiments.

A Savings Plan commits you to a fixed amount of compute spend per hour for 1 or 3 years. Matching usage up to that amount is billed at discounted rates, and anything above it at on-demand. Compute Savings Plans apply to EC2 in any family or region, and to Fargate and Lambda, with savings of up to 66 percent. EC2 Instance Savings Plans lock to one family in one region in exchange for up to 72 percent. The flexibility is the point: when you migrate from p4d to p5, a Compute Savings Plan keeps applying, while a Standard Reserved Instance for p4d would sit unused.

Billing alerts tell you about spend before the invoice does. AWS Budgets lets you set cost or usage budgets, monthly, quarterly, daily or for specific tags, with notifications by email or SNS at actual or forecasted thresholds. Budget actions can go further and apply an IAM deny policy or stop specific instances when a threshold is crossed. The limit is latency. Budgets data refreshes up to three times a day, so an alert can arrive 8 to 12 hours after the spend. Pair budgets with faster controls, such as idle-instance shutdown and CloudWatch alarms on usage.

Cost Explorer is the billing console's analysis tool. It charts costs and usage over time, filters and groups them by service, linked account, region, instance type, usage type, purchase option or cost-allocation tag, and forecasts the coming months from past spend. Grouping by usage type is how you find surprises such as NAT gateway bytes, cross-region transfer or EBS snapshots. Grouping by a project or owner tag answers who spent what, provided tags are activated as cost-allocation tags and applied consistently. The same console offers rightsizing and Savings Plans recommendations. Review it weekly, and treat untagged spend as a bug to fix.

Right-sizing means matching the instance to measured usage rather than guessing large. Collect CPU, memory, GPU utilization and GPU memory over representative periods. CloudWatch covers CPU, but memory needs the CloudWatch agent and GPU metrics need DCGM or nvidia-smi exporters. Then pick the smallest type that meets the latency or throughput target. AWS Compute Optimizer suggests changes for EC2, EBS, Lambda and ECS. For ML, the common wins are serving a model that fits in 24 GB on one g5 or g6 instead of a p-family node, halving an instance whose CPU sits below 40 percent, and moving to a newer generation that delivers more work per dollar.

Infracost is an open-source tool that reads Terraform code or a plan, looks up cloud list prices, and reports the monthly cost of the resources and the difference a change makes. In CI it posts that diff as a pull-request comment, for example '+16,000 dollars a month: aws_instance.trainer changed from g5.xlarge to p4d.24xlarge', so reviewers see the cost of a change next to the code. It prices from list rates and resource attributes, so usage-based charges such as data transfer, requests and NAT processing need a usage file of estimates. It also ignores your discounts, so treat its numbers as a list-price guide.