SageMaker Inference Recommender

TL;DR: SageMaker offers 50+ instance types to deploy a model on, and most teams pick one from a blog post, then never revisit it — the endpoint works, it's just twice the size it needs. Inference Recommender automates the answer: hand it your model artifact, container, and a realistic sample payload; it spins up candidate instances, pushes traffic, and ranks them by cost per inference. The benchmark costs ~$10–20 in temporary instance time and the savings compound for as long as the endpoint runs.

The numbers

• Cost-per-hour is the input cost; cost-per-inference is the output cost. A $0.10/hr instance doing 20 inf/s is more expensive per inference than a $0.40/hr instance doing 100 inf/s (throughput swings usually beat price swings).
• Inferentia is the recurring headline: a typical BERT deployment ran ml.g4dn.xlarge (T4 GPU) at $0.736/hr vs ml.inf1.xlarge at $0.362/hr — same throughput/latency, half the price. One team saved $3,200/mo on a single endpoint.
• Two modes: Instance Recommendation (auto-picks candidates, ~45 min) for first-time picks; Load Test (you specify instance types, traffic pattern, latency SLA, a few hours) before locking a high-volume endpoint.
• Field examples: a ResNet-50 image tagger swapped g4dn→inf1 for $3,200/yr off one endpoint; a BERT QA model on m5.2xlarge found inf1 41% cheaper per inference, saving ~$140/mo.

Do this

1. Register the model — S3 artifact + inference container image.
2. Build a realistic sample payload — production-shaped data, not a 1KB toy JSON (garbage in, irrelevant recommendation out).
3. Launch a job: Instance Recommendation for a fast first pick; Load Test with your real CloudWatch traffic pattern + latency SLA before high-volume commit. Include your current instance as the baseline so the report shows the dollar delta directly.
4. Always include Inferentia (inf1/inf2) in the candidate list — even when the model "sounds GPU-shaped." The tool tells you fast if it doesn't compile.
5. Benchmark the autoscale minimum — the base instance runs 24/7, so optimize it first. Re-run when traffic 2×'s, the SLA tightens, or the model architecture changes.

Gotchas

• The cheapest instance is usually the expensive deployment — chasing the lowest $/hr ignores throughput and overpays per inference.
• Report dollars are a snapshot — AWS prices drift and the math uses on-demand (not Spot) rates, so numbers are conservative; the ranking between instances is far more durable than the absolute prices.
• Inferentia isn't universal — very large LLMs, custom CUDA ops, and some vision models don't compile cleanly; benchmark rather than assume.
• Realistic payloads matter — a toy input tells you nothing about how the model handles real 50KB requests.

Skip this if

• You serve models off SageMaker (EKS, Lambda, raw EC2) — the cost-per-inference logic still holds, but you'll roll your own benchmarks.
• Traffic is truly sporadic (a few hundred req/hr, then silence) — scale-to-zero usually wins; see SageMaker Serverless Inference.
• You're hosting many small models on shared infra — see SageMaker Multi-Model Endpoints.