All cheat sheets
Serverless & AIcheat sheet

SageMaker Managed Spot Training

One boolean puts SageMaker training jobs on Spot capacity at 50–90% off, with interruption handling and checkpoint-resume automated. The only real requirement: your script must actually write checkpoints.

Last reviewed: July 11, 2026

Prefer step by step? Take the guided SageMaker Managed Spot Training walkthrough — free, anonymous, interactive.Have an AI assistant? Jump to the copy-paste audit prompt at the bottom of this sheet.

TL;DR: Spot is the same hardware at 50–90% off because AWS can reclaim it on 2 minutes' notice — fatal for web servers, nearly free money for training jobs that checkpoint. Managed Spot Training automates the whole dance: flip use_spot_instances=True, write checkpoints to /opt/ml/checkpoints/, and SageMaker resumes interrupted jobs from S3 on fresh capacity. The trade is wall-clock predictability, which is why SLA-bound production retrains stay On-Demand.

The numbers

Instance On-Demand Typical Spot Discount
ml.p3.2xlarge ~$3.80/hr ~$1.14–1.90/hr 50–70%
ml.p3.16xlarge ~$28/hr ~$8.40–14/hr 50–70%
ml.c5.4xlarge ~$0.85/hr ~$0.26–0.43/hr 50–70%

Field examples: a vision researcher's 24–36 h jobs dropped from $300–400 to $90–120 per run; a 200-configuration hyperparameter sweep went from $1,200 to $240 — stragglers don't matter when jobs run in parallel.

Do this

  1. Wire checkpoints first — model weights, optimizer state, epoch/step counters — into /opt/ml/checkpoints/; SageMaker syncs that path to S3 continuously. PyTorch/TensorFlow/MXNet all support this natively.

  2. Flip the estimator flags:

    estimator = Estimator(
    ...,
    use_spot_instances=True,
    max_run=7200,        # expected training time
    max_wait=14400,      # ~2x max_run: interruption + re-queue headroom
    checkpoint_s3_uri="s3://my-bucket/checkpoints/",
)

  3. Test the resume path before trusting it: run a short job, kill it manually, restart, confirm it picks up mid-training. Five minutes now prevents the "resumed from epoch 0 after six hours" disaster.

  4. Prioritize by dollars: long GPU jobs and hyperparameter sweeps first; they're where percentage discounts become real money.

  5. Verify realized savings in the console — BillableTimeInSeconds vs training time shows exactly what Spot saved per job.

Gotchas

  • No checkpoints = no savings, just lost progress on every interruption. This is the entire failure mode.
  • max_wait must exceed max_run generously — a tight window means interrupted jobs fail instead of resuming.
  • Spot scarcity is real at peak times (re:Invent week, quarter-end) — GPU capacity especially; expect longer waits.
  • Sub-10-minute jobs lose the savings to provisioning overhead.
  • SLA-bound retrains don't belong here: a fraud-detection team's 1-hour jobs stretched to 2–3 hours under interruption and blew a 30-minute deploy SLA — they moved that one workload back to On-Demand and kept Spot for everything else. Hybrid is the right answer.

Skip this if

  • Training must complete by a hard deadline every time — pay for On-Demand predictability on that job.
  • Your framework genuinely can't checkpoint (rare today).
  • Training is a rounding error next to inference — endpoints can't use Spot; that half of the bill is SageMaker Savings Plans territory.

Run this audit with your AI assistant

Paste this into Claude, ChatGPT, or any agent that can run the AWS CLI with read-only credentials. It audits your account for exactly the waste this sheet describes — and changes nothing.

You are auditing an AWS account's SageMaker training spend for Managed
Spot opportunities. Use the AWS CLI with READ-ONLY credentials. Do not
create, modify, or delete anything.

1. Pull recent training jobs: aws sagemaker list-training-jobs
   (last 60–90 days), then describe-training-job on a sample — capture
   InstanceType, TrainingTimeInSeconds, BillableTimeInSeconds,
   EnableManagedSpotTraining, CheckpointConfig.
2. Findings:
   a. On-Demand jobs ≥ 1 hour → prime Spot candidates. Estimate
      savings at 50–70% of their instance-hours (GPU types like
      ml.p3/ml.g5 carry the biggest absolute dollars).
   b. Jobs already on Spot: compare BillableTime vs TrainingTime for
      realized savings; missing CheckpointConfig on Spot jobs is a
      risk finding (interruption = start over).
   c. Hyperparameter tuning jobs (list-hyper-parameter-tuning-jobs)
      on On-Demand — the textbook Spot case (parallel, no single
      job on a critical path).
   d. Short jobs (< ~10 min) on Spot — overhead may exceed savings;
      note but deprioritize.
3. SLA check to include in report: any training on a schedule feeding
   production deploys (names/tags suggesting prod retrain) should
   stay On-Demand or have generous max_wait margins.

Report: job-pattern table (pattern | instance | hrs/mo | On-Demand
$/mo | est. Spot $/mo), checkpoint-readiness gaps, and the exact
estimator changes (use_spot_instances, max_run, max_wait,
checkpoint_s3_uri). Change nothing.
Works with any assistant that can run shell commands.

Want the guided version?

The SageMaker Managed Spot Training walkthrough covers this topic interactively — it asks about your setup, branches to what’s relevant, and quizzes you on the tricky parts. Free and anonymous.

Start the walkthrough

More Serverless & AI cheat sheets

API Gateway HTTP APIs

HTTP APIs cost ~$1.00 per million requests versus REST's ~$3.50 — a 71% cut with lower latency and native JWT auth. Most REST APIs use none of the features that premium pays for; audit and migrate the eligible 80%.

App Runner Automatic Pausing

App Runner Automatic Pausing scales your service to zero when idle — you stop paying for provisioned compute and trade it for a 2–5s cold start on the first request. Business-hours apps save 60–70%, sporadic ones 80–95%.

AWS Lambda Power Tuning

Lambda bills GB-seconds, so memory and speed trade off — and for CPU-bound code, more memory is often cheaper. Power Tuning tests every memory setting against your real function in ~5 minutes and typically finds 20–50% savings.

GameLift Spot Fleets

Run multiplayer game servers on Spot capacity with an On-Demand buffer and let GameLift FleetIQ handle the 2-minute interruption choreography — typically 70–85% cheaper than pure On-Demand, invisible to players.