TL;DR: S3 Select runs a SQL SELECT ... WHERE ... inside S3 and streams back only the matching rows. You pay ~$0.002/GB to scan and $0.0007/GB for what's returned — instead of $0.09/GB egress on the whole file plus the compute to filter it. Need 200 MB out of a 10 GB file? ~$0.02 instead of ~$0.90. It also shrinks the Lambda that used to load the whole file into memory.
The numbers
| Approach (10 GB file, 200 MB needed) | Cost |
|---|---|
| Download all 10 GB, filter locally | ~$0.90 |
| S3 Select: scan 10 GB + return 200 MB | ~$0.02 |
Formats: CSV, JSON (use JSON Lines), and Parquet — Parquet is the double win, since columnar layout means unselected columns aren't even scanned. Field example: an IoT pipeline filtering 2 GB hourly CSVs down to ~20 MB of anomalies cut ~$1,200/month and let the Lambda shrink dramatically.
Do this
-
Find the "download-then-filter" pattern in your code: Lambdas with big memory settings reading large S3 files, jobs that
GetObjectgigabytes to extract rows. -
Replace the read with
SelectObjectContent:aws s3api select-object-content --bucket YOUR-BUCKET --key logs/big.csv \ --expression "SELECT * FROM s3object s WHERE s.status = 'ERROR'" \ --expression-type SQL \ --input-serialization '{"CSV":{"FileHeaderInfo":"USE"},"CompressionType":"GZIP"}' \ --output-serialization '{"JSON":{}}' /dev/stdout -
Make the
WHEREclause selective — the savings are proportional to what you don't return.SELECT *with no filter is paying scan fees for nothing. -
Prefer Parquet for new data you'll query this way; JSON Lines over multi-line JSON; standard, boring CSV over clever delimiters.
-
Cache repeated queries (ElastiCache/DynamoDB) — Select bills per call.
Gotchas
- The response is an event stream, not a JSON blob — SDKs handle it, but budget 30 minutes of docs-reading the first time.
- Compression: whole-object GZIP/BZIP2 is fine; anything compressed per-record/chunk inside the file is opaque to Select. Client-side encrypted objects are unqueryable.
- SQL is a subset: column projection, WHERE, LIKE, CAST, simple aggregates, LIMIT. No JOINs, subqueries, or window functions — one object at a time.
- Weird CSV breaks it: multi-line fields and exotic delimiters are the classic failure.
- Scan isn't free: unselective queries on huge files still bill $0.002/GB scanned.
Skip this if
- Files are tiny — just download them.
- You need the whole file anyway — no filter, no savings.
- The question spans many files/partitions with real SQL — that's Athena (the rule of thumb: Select inside your application, Athena at the query console).
- The data is unstructured (images, video, binaries) or streaming (Kinesis territory). For the same trick on archived data, see Glacier Select.