TL;DR: Batch Operations is AWS's forklift for S3: give it a manifest (usually from S3 Inventory) and an operation — change storage class, copy cross-account, restore from Glacier, retag, delete — and it executes across millions of objects in parallel, retries transient failures, and hands you a completion report that doubles as an audit trail. At $0.25 per million objects, it replaces the hand-rolled script that crashes at hour eight.
The numbers
- Job cost: $0.25 per million objects, plus normal S3 request fees for the operation itself (a 10M-object class migration ≈ $2.50 + ~$50 in requests)
- Worked example from the source workflow: a 50M-object log bucket at $8K/month → Inventory manifest, Athena-filtered to 45M objects older than 90 days, one Deep Archive transition job → under $500/month
- Cross-account: 100 TB / ~30M objects moved in a two-day unattended job for a few dollars of Batch fees, versus a week of engineering and 3 AM pages for the DIY script
Do this
-
Generate the manifest: enable S3 Inventory on the source bucket (first report within ~48 h), optionally filter it with Athena ("older than 90 days", "prefix = logs/", "untagged only").
-
Set up the IAM role — the part everyone fumbles once: it needs read on the manifest, the operation permission on target objects, and write on the report destination.
-
Dry-run with a 100–1,000 object manifest end-to-end. Most teams hit one IAM error on the test job and zero on the real one — that's the right ratio.
-
Run the real job (console or
aws s3control create-job) and let the completion report tell you exactly what succeeded, with timestamps and error codes per object. -
Add the lifecycle rule afterward so the backlog can't re-accumulate — Batch Operations clears the past; lifecycle policies govern the future.
Gotchas
- No undo. Ten million deletes have no Ctrl+Z; the completion report is your only record. This is why the small-manifest dry run isn't optional.
- Glacier minimum-duration charges apply to migrations. Move data to Glacier Flexible (90-day min) or Deep Archive (180-day min) and change your mind next month — you pay the minimum anyway.
- Restores still take Glacier time. Batch orchestrates mass restores; it doesn't accelerate them.
- Lambda-per-object jobs are a Lambda bill. A million objects = a million invocations; right-size memory before you fan out.
- The completion report lands in S3, not the console — you download and parse it.
Skip this if
- The object count is small — dozens or hundreds of objects are a CLI loop, not a job.
- The work is incremental ("process each new upload") — that's S3 Event Notifications + Lambda.
- The logic is deeply conditional per object — Batch is for uniform actions across a known set; branching logic wants a custom script or the Lambda-invoke operation with care.