Rightsizing gets treated as a spring-cleaning project: someone opens a tool, accepts the recommendations, closes a few tickets, and declares the fleet "right-sized." Three months later the same instances are wrong again, or worse, one of the downsizes quietly started dropping requests and nobody connected it to the change. Rightsizing done well is not a project and it is not a tool's opinion accepted on faith. It is a repeatable decision with a specific shape, and most of the skill is in knowing where the easy answers lie to you.
What Compute Optimizer sees, and what it doesn't
AWS Compute Optimizer is the reasonable starting point. It ingests CloudWatch metrics and instance specifications over a lookback window (14 days by default), and returns rightsizing recommendations with risk ratings and projected savings. For CPU-bound and network-bound workloads it is genuinely useful and will catch the obviously over-provisioned instances you forgot about.
It has a blind spot large enough to cause an outage: memory. EC2 does not expose guest memory utilization to CloudWatch by default — the hypervisor can see your CPU and network, but it cannot see how full your heap is. Compute Optimizer therefore has no memory signal unless you install the CloudWatch agent to push mem_used_percent from inside the instance. Without that agent, the tool is reasoning about memory it cannot measure, and it will happily recommend downsizing an instance that looks CPU-idle but is running at 85% memory. Accept that recommendation and you have traded a few dollars a month for an OOM kill under load. The same blindness applies to disk utilization.
So the first rule is procedural: if a rightsizing recommendation touches memory-sensitive workloads, confirm the CloudWatch agent is installed and reporting before you trust it. A recommendation built on missing data is a guess wearing a tool's authority.
The p99-versus-average trap
Here is the mistake that sends people in both wrong directions at once. You pull up CPU utilization, you see it averages 18%, and you conclude the instance is four times too big. Sometimes true. Often a trap, because an average flattens exactly the peaks that sizing exists to survive.
Two failure modes bracket the correct answer:
- Size to the peak (the max) and you provision for the worst five minutes of the year all year long. The instance spends 99% of its life bored, and you pay for the Black Friday spike every ordinary Tuesday. This is the safe-feeling, expensive mistake.
- Size to the average and you are underwater during every routine daily peak. The mean looked comfortable; the workload is not the mean. This is the cheap-feeling mistake that shows up as latency and dropped requests during your busiest hour.
The right target is neither. You size to a considered percentile with headroom — commonly something like p95 or p99 of the relevant metric over a window that actually contains your peaks, plus a margin for burst and near-term growth. The percentile you choose is an SLO decision, not a default: a payments path might size to p99.9 with generous headroom because an incident is catastrophic, while a batch job that can slow down without anyone caring can size closer to p90. The point is that you look at the distribution, choose a point on it deliberately, and can say why.
Burst credits, and how a "cheap" instance throttles
The T-family instances — t3, t4g, and friends — are where a naive rightsizing decision detonates most spectacularly, because their price hides a mechanism. They are burstable: each instance has a baseline CPU level (a t3.medium's baseline is 20% of a vCPU, for example) and earns CPU credits while running below baseline. When load rises above baseline it spends credits to burst up to full speed. This is wonderful for spiky, mostly-idle workloads and a disaster for sustained ones.
Run a T instance above its baseline for long enough and the credit balance hits zero. At that point one of two things happens. In standard mode, the instance is throttled down to baseline — your "2 vCPU" instance is now delivering 20% of one, and your latency graph falls off a cliff for reasons the CPU utilization metric alone won't explain. In unlimited mode (the default for many T3 configurations), it keeps bursting but bills you surcharge credits at roughly $0.05 per vCPU-hour, so the "cheap" instance quietly runs up a variable bill that can exceed the fixed-price instance you were avoiding.
This is why a T instance that looked perfect in a demo throttles in production: the demo never sustained load long enough to drain the credits. Rightsizing onto a burstable instance is only correct when you have confirmed the workload's sustained utilization sits below baseline, with bursts short enough for the credit balance to recover between them. Otherwise the "cheaper" instance is more expensive in latency, in surprise charges, or both.
Observe before you cut
The through-line of every trap above is acting on too little observation. The methodology that avoids all of them:
- Gather representative data. A 14-day window that misses month-end batch, a marketing event, or a weekly cron peak is not representative. Cover the cycles your workload actually has.
- Look at the distribution, not a number. Averages hide peaks; peaks hide idleness. You want the shape.
- Cover every dimension, not just CPU. Memory (with the agent installed), disk throughput and IOPS, network, and any EBS volume limits. The binding constraint is often not the one you were watching.
- Change one variable at a time. Resize or change family or switch to burstable — not all three at once, or you will not know which change caused the regression you are now debugging.
- Canary and watch saturation signals. Roll the change to a subset, then watch p99 latency, error rate, and saturation (queue depth, credit balance, memory pressure) — not just cost. Keep a rollback one command away.
Cost is the reason for the change, but it is never the signal that tells you the change was safe. The signal is what happened to latency and errors at the peak after you cut.
Rightsizing is continuous
The reason the spring-cleaning framing fails is that workloads drift. Traffic grows. A dependency upgrade changes the memory profile. A new feature moves the CPU-to-memory ratio. A caching layer you added last quarter made half the fleet over-provisioned without anyone noticing. The "right" size is a moving target, so rightsizing is a loop tied into the broader cost practice, not a one-time sweep. Let the tools — Compute Optimizer, Cost Explorer's rightsizing recommendations — surface candidates continuously, and keep the actual decision human wherever the risk of getting it wrong is an incident rather than a few dollars.
The mental model, compressed
Strip it to five steps you can run every time:
- Measure the real distribution — the right metrics, over a window that contains your peaks, including memory and disk, not just CPU.
- Choose a percentile and headroom on purpose — an SLO decision, higher for workloads where failure hurts.
- Understand the instance's failure mode — burst-credit exhaustion, network caps, EBS throughput ceilings — before you commit to a family.
- Change one variable, canary, and watch saturation — with rollback ready and latency, not cost, as the safety signal.
- Re-evaluate on a cadence — because the right answer expires.
Run that loop and rightsizing stops being a guess you occasionally regret. It becomes what it should be: a small, boring, repeatable decision you can defend with the distribution you looked at and the percentile you chose.