All cheat sheets
Databasescheat sheet

AWS Glue Job Bookmarks

Without bookmarks, a nightly Glue job rescans the entire S3 bucket to find a sliver of new data. One checkbox makes it track what it's already processed and touch only the delta — routinely cutting incremental-job runtime and cost 90%+.

Last reviewed: July 14, 2026

Prefer step by step? Take the guided AWS Glue Job Bookmarks walkthrough — free, anonymous, interactive.Have an AI assistant? Jump to the copy-paste audit prompt at the bottom of this sheet.

TL;DR: Without bookmarks, a nightly ETL reads every file in the bucket — yesterday's, last week's, three years ago's — to find the sliver of new data. Job Bookmarks are Glue's built-in state tracker: they remember which files (by path + last-modified), S3 partitions, and JDBC high-water marks have been processed, so the next run picks up where the last stopped and only new or changed data flows through. It's a one-checkbox, no-code-change fix that routinely turns full rescans into incremental processing — 90%+ off for append-only sources.

The numbers

  • Worked example — 5 TB of historical logs, 50 GB new/day: without bookmarks scans all 5 TB (3 hrs × 10 DPUs = 30 DPU-hrs ≈ $13.20/day); with bookmarks scans only 50 GB (10 min ≈ 1.67 DPU-hrs ≈ $0.73/day). ~$374/mo, ~$4,488/yr on one job.
  • First run = baseline: processes everything, records what it saw; from run two only the delta is touched (no "empty bookmark = empty job" failure).
  • Field examples: an 80 TB clickstream (200 GB new/day) went 6 hrs/$85/day → 12 min/$1.20/day ($2,514/mo saved); a 50M-row payment-table pull dropped to ~100K rows/run (~$14,000/yr); manufacturing telemetry saved >$60,000/yr on one hourly pipeline.

Do this

  1. Enable it — one setting: Job details → Advanced properties → Job bookmark → Enable, or CLI --job-bookmark-option job-bookmark-enable. State is stored internally, scoped per job.
  2. Point JDBC jobs at a monotonic column — a timestamp or auto-increment ID as the high-water mark; each run pulls only rows greater than the last bookmark.
  3. Partition S3 sources by date (year=/month=/day=) — bookmarks then skip whole partitions before listing files: faster startup, cheaper evaluation, easier debugging.
  4. Verify Glue-context reads — bookmarks apply to glueContext.create_dynamic_frame.from_catalog/from_options, not raw spark.read.parquet(...); convert raw-Spark reads on bookmarked sources.
  5. Reset deliberately when you need a full reprocess — Job details → Reset job bookmark, or aws glue reset-job-bookmark --job-name <name> (backfills, logic fixes, schema changes).

Gotchas

  • In-place file rewrites break the model — bookmarks track path + last-modified, so if upstream edits old files (rather than appending new ones), they get reprocessed. Works cleanly only on write-once/append-only data.
  • Raw Spark reads silently ignore bookmarks — the #1 reason "I enabled bookmarks but it still scans everything"; look here first.
  • Retroactively edited DB rows with no change signal won't be re-picked-up — bookmarks need a reliable "newer than" marker.

Skip this if

  • The job is a full-refresh dimension rebuild, an all-time running-total aggregation that needs the complete history each run, or a one-off migration/backfill that won't repeat.
  • The source mutates old records in place with no detectable change signal. Otherwise — for the ~90% of ETL that's "new data arrives, old data doesn't change" — it's a no-brainer, and pairs naturally with AWS Glue Flex Execution (34% off DPU-hours) and Glue Auto Scaling on the same workloads.

Run this audit with your AI assistant

Paste this into Claude, ChatGPT, or any agent that can run the AWS CLI with read-only credentials. It audits your account for exactly the waste this sheet describes — and changes nothing.

You are auditing an AWS account's Glue jobs for incremental-processing
(Job Bookmarks) savings. Use the AWS CLI with READ-ONLY credentials. Do
not create, modify, or delete anything — report findings and recommended
(unapplied) fixes only.

1. Inventory: aws glue get-jobs — capture DefaultArguments
   ["--job-bookmark-option"] (enable vs disable/absent) and worker/DPU
   footprint per job.
2. Reprocessing waste: aws glue get-job-runs per recurring job — if
   ExecutionTime is roughly CONSTANT run-over-run while source data
   grows, the job is rescanning history. Estimate wasted DPU-hours =
   (full-scan time - expected-delta time) × DPUs × runs/mo.
3. Source-shape fit: flag append-only / write-once S3 sources and JDBC
   sources with a monotonic timestamp/auto-increment column (bookmark
   candidates). Flag in-place-mutated files or retro-edited rows (NOT
   candidates).
4. Code trap: note that bookmarks only apply to glueContext
   create_dynamic_frame.from_catalog/from_options reads — raw
   spark.read.* ignores bookmarks (flag jobs to inspect).

Report a table: job | bookmarks on? | recurring? | runtime trend | source
shape | est. wasted $/mo | recommendation. Change nothing.
Works with any assistant that can run shell commands.

Want the guided version?

The AWS Glue Job Bookmarks walkthrough covers this topic interactively — it asks about your setup, branches to what’s relevant, and quizzes you on the tricky parts. Free and anonymous.

Start the walkthrough

More Databases cheat sheets

Amazon RDS Proxy

Connection count, not CPU, is often what forces an RDS upsize. RDS Proxy pools thousands of client connections (especially from Lambda) onto ~10–30 real ones, so a small instance stays small — the proxy fee typically covers 10–25% of the avoided upsize.

Athena Partition Projection

Instead of asking S3 which partitions exist before every query, Partition Projection lets Athena calculate the paths from a pattern you declare — near-instant query startup on tables with thousands of partitions, zero LIST metadata calls.

Athena Query Result Reuse

Athena caches identical query results in S3, so a repeated dashboard refresh or scheduled report returns instantly with zero bytes scanned. A workgroup checkbox that routinely cuts thousands off a scan-heavy bill.

Aurora I/O-Optimized

Aurora bills a sneaky third bucket — I/O operations at ~$0.20/million — that can dwarf compute and storage on I/O-heavy workloads. I/O-Optimized zeroes it out for a ~30–40% higher instance rate. A break-even calc, not a default.