Transformation at Scale: AWS Glue, EMR Serverless, and Lambda

Lessons 01 through 07 gave you a lake: S3 layout, file formats, open table formats, batch and streaming ingestion, and a catalog. None of that is yet curated. This lesson is about the layer that does the curating, the engines that read raw and bronze, clean it, deduplicate it, and write silver and gold, and about the three things that decide whether that layer is cheap, correct, or both: which engine you run, how much capacity you hold while it runs, and whether the second run produces duplicates.

In this lesson you will:

• model the Glue ETL job: anatomy, argument precedence, the 260 KB ceiling, triggers, concurrency quotas and the 2024 job-queuing behaviour;
• separate DynamicFrame from DataFrame and drive resolveChoice with the right action before any write;
• operate job bookmarks: Enable / Disable / Pause, what S3 and JDBC sources actually key on, why rewind never deletes your outputs, and how that differs from Iceberg MERGE;
• size workers: G.1X through G.16X, memory-optimized R workers, Ray's Z.2X, streaming G.025X, --executor-cores ceilings and auto scaling;
• author with Glue Studio visual ETL, notebooks and the script editor, and read the Glue version matrix as of October 2026: what each row costs and what it breaks;
• run EMR Serverless, applications, Workspaces, release pinning, metering, and know exactly when it beats Glue, plus Lambda for the control plane and never for the data plane;
• apply Spark lakehouse patterns: dynamic partition overwrite, AQE, skew, shuffle partitions and small-file compaction;
• write a worked PySpark Glue job from catalog read to Iceberg silver, and price every drill on the October 2026 rate card;
• study two AWS customer cases (Transport for NSW, Vanguard) plus a sourced 2026 update box;
• practise with 11 exam-style questions and five interactive checks.

---

1. The engine ladder: five paths from raw to silver

AWS positions a small number of transformation paths for lakehouse batch work, and they are not interchangeable. The ladder runs from most integrated, least flexible to most flexible, most operational:

Two properties of this ladder matter more than any feature list. First, only Glue ETL has job bookmarks wired into the job model: incremental processing is a first-class argument, not code you write. Second, Glue and EMR Serverless meter differently: DPU-hours versus vCPU-hours plus GB-hours, so a headline "per DPU" and "per vCPU" number can never be compared without normalizing memory (Section 13 does the arithmetic).

---

2. The Glue ETL job model: anatomy, arguments, triggers, queuing

2.1 What a job actually is

A Glue job is a definition plus runs. The definition is: script (or visual DAG), worker type and worker count, IAM role, optional connections and security settings, default arguments, timeout, retries, MaxConcurrentRuns, and the triggers that start it. A run is a bounded execution with a state and a retained history.

MaxConcurrentRuns defaults to 1, and job run history is retained for 90 days: two numbers worth memorizing, because a run that sits in WAITING is telling you about quota or trigger position, not about your code.

2.2 Arguments: three layers, one hard ceiling

Arguments are how a generic script becomes a specific pipeline. Three layers apply, in this precedence:

1. NonOverridableArguments (highest; you cannot beat these at run time)
2. run Arguments supplied at StartJobRun
3. DefaultArguments on the job

and the whole argument payload has a maximum size of 260 KB. That ceiling is the reason you never inline a megabyte of SQL or a base64 blob into arguments: put the SQL in an S3 object and reference it with --extra-py-files, --extra-jars, or --additional-python-modules.

The arguments that show up in almost every lakehouse job:

• 📚 Did you know? --executor-cores has a documented ceiling: it "should not exceed 2x the number of vCPUs on the worker type." On a G.1X (4 vCPUs) that means at most 8; on a G.2X, 16. Oversubscribing past that does not add parallelism; it thrashes memory and disk, which then looks like a skew problem but is not (AWS Glue job arguments reference, accessed Oct 2026).

2.3 Triggers: scheduled, conditional, on-demand

Glue triggers come in three flavours, Scheduled (a schedule), Conditional (on the status of a previous job or crawler) and On-demand, with lifecycle states CREATED, ACTIVATED, DEACTIVATED. Two hard rules bite in practice:

• you can attach a maximum of 2 crawlers to a single trigger;
• a conditional chain only works if the chain itself is trigger-started from a scheduled or on-demand root: a manually launched job will not fire downstream conditional triggers.

2.4 Queuing instead of failure at the concurrency ceiling

Before September 2024, hitting Max concurrent job runs per account, per-job concurrency or Max task DPUs per account failed the StartJobRun call outright, and teams built Step Functions retry loops to absorb it. AWS Glue job queuing (general availability, AWS Big Data Blog, 3 Sep 2024) changed the shape: "when the job runs are in Waiting state, the AWS Glue job queuing feature staggers jobs automatically whenever possible." The failure mode becomes a queue position, and capacity frees as other runs finish.

---

3. DynamicFrame versus DataFrame, and resolveChoice

3.1 The two representations

A Spark DataFrame needs a schema before data can be loaded; that is its contract. A Glue DynamicFrame deliberately breaks that contract, because real ingestion lands inconsistent data:

"each record is self-describing, so no schema is required initially. Instead, AWS Glue computes a schema on-the-fly when required, and explicitly encodes schema inconsistencies using a choice (or union) type."

That choice type is powerful and dangerous. It is powerful because a crawler-sampled JSON where price is a number in some rows and a string in others loads without failing. It is dangerous because the choice survives into your write path and silently degrades columns to a general type.

DynamicFrame gives you a toolbox of transforms, apply_mapping, drop_fields, filter, join, map, mergeDynamicFrame, relationalize, rename_field, resolveChoice, select_fields, spigot, split_fields, split_rows, unbox, union, unnest, unnest_ddb_json, write, of which relationalize is the one people forget: it "Converts a DynamicFrame into a form that fits within a relational database" by unnesting nested columns and pivoting array columns into a list of frames.

3.2 resolveChoice: settle the schema before you map

resolveChoice takes (field_path, action) specifications. The actions are:

You can also convert with toDF(options), where options is a list of ResolveOption objects: same idea, DataFrame-bound.

3.3 Worked example E1: a mixed-type price column

3.4 The Glue 5.0+ wrinkle you must plan for

The release notes state it plainly: "Glue Dynamic Frame / GlueContext-based table-level access control with AWS Lake Formation permissions supported in Glue 4.0 or before is not supported in Glue 5.0." A script that leans on DynamicFrame and GlueContext table-level Lake Formation enforcement is therefore a Glue ≤4.0 artifact. On 5.1 and 6.0 the direction of travel is Spark-native fine-grained access control (FGAC): plain DataFrames plus Lake Formation row filters and column masks. Migrating a 3.0-era resolveChoice + apply_mapping script without adopting FGAC means you keep the code and lose the enforcement.

---

4. Job bookmarks: incremental runs without re-reading everything

4.1 What a bookmark is

"AWS Glue tracks data that has already been processed during a previous run of an ETL job by persisting state information from the job run."

State is persisted per job and keyed by transformation_ctx: the label you attach to each source and sink in the script. Keep the source and its transformation_ctx stable; change a path without changing the key and the state no longer matches what you think it matches.

The three modes you set with --job-bookmark-option:

4.2 What each source type keys on

• Amazon S3: "AWS Glue job bookmarks check the last modified time of the objects to verify which objects need to be reprocessed." Supported formats are version-dependent: v0.9 supports JSON, CSV, Avro and XML; v1.0 and later add Parquet and ORC.
• JDBC: "AWS Glue by default uses the primary key as the bookmark key, provided that it is sequentially increasing or decreasing (with no gaps)." Surrogate, non-sequential keys need an explicit bookmark key.

4.3 The failure list: read this before you ship

• Disable is the default. Forgetting --job-bookmark-option=job-bookmark-enable silently full-reprocesses every run, and your bill says nothing about it.
• Do not change the data source property with the bookmark enabled. AWS lists this as a best practice for exactly the reason above: stale state plus a new path skips or double-reads files.
• Rewind and reset do not clean targets. "When you rewind or reset a bookmark, AWS Glue does not clean the target files… Only source files are tracked with job bookmarks." Idempotency on the write side is your job (Section 11).
• Pause is for backfill, not for steady state. It replays a run-ID window and does not advance: perfect for re-running 102 and 103, wrong as a daily mode.
• Large listings can exhaust the driver. Use the AWS Glue Amazon S3 file lister for large datasets rather than letting the driver enumerate millions of keys.

4.4 Worked example E2: the backfill drill

• 📚 Did you know? Bookmark state is not limited to JSON and CSV: from Glue 1.0 onward, S3 bookmarks also persist state for Parquet and ORC inputs, so the parquetized bronze you built in Lesson 05 is bookmark-friendly: the 0.9-era "JSON/CSV only" folklore is out of date (AWS Glue developer guide, monitor continuations, accessed Oct 2026).

---

5. Workers, DPUs and M-DPUs: where the bill and the OOM come from

5.1 The unit

"A DPU is a relative measure of processing power that consists of 4 vCPUs of compute capacity and 16 GB of memory."

And the memory-optimized variant: "Memory-Optimized DPUs (M-DPUs): R type workers use M-DPUs, which provide double the memory allocation for a given size compared to standard DPUs": 32 GB per 4 vCPU. Everything downstream (price, OOM triage, the Glue-versus-EMR comparison) flows from those two sentences.

5.2 Worker catalogue (as of Oct 2026)

5.3 Capacity: MaxCapacity is legacy, workers are current

• AllocatedCapacity is deprecated: "Use MaxCapacity instead."
• For Glue 2.0+ jobs: "you cannot specify a Maximum capacity. Instead, you should specify a Worker type and the Number of workers. Do not set MaxCapacity if using WorkerType and NumberOfWorkers."
• Where MaxCapacity still applies: pythonshell 0.0625 or 1 DPU; legacy glueetl/gluestreaming range 2 to 100 DPUs, default 10, with no fractional DPU allocation.
• Auto scaling: --enable-auto-scaling=true turns on auto scaling and per-worker billing, with NumberOfWorkers acting as the maximum.

5.4 Worked example E3: sizing and pricing a 400 GB batch

5.5 Worked example E4: the same job when memory, not speed, is the constraint

• 📚 Did you know? Disk is the quiet constraint: G.1X exposes 94 GB of disk but only about 44 GB of it is free: shuffle spills and large local intermediate data land there. If the Spark UI shows heavy spill on a G.1X, you are disk-bound rather than CPU-bound, and a G.4X (256 GB, about 230 GB free) or cloud shuffle storage is the fix (AWS Glue worker types, accessed Oct 2026).

---

6. Glue Studio: visual ETL, notebooks and the script editor over the same engine

Glue Studio is not a different service, it is the authoring and operations plane. AWS offers three ways in: Visual ETL (a visual interface focused on data flow), an interactive code notebook (Jupyter-based), and a script editor where you pick the engine, Python shell, Ray, Spark (Python) or Spark (Scala). The visual pitch is exact: "you can visually compose data transformation workflows and seamlessly run them on AWS Glue's Apache Spark-based serverless ETL engine… using a drag-and-drop interface without having to learn Spark or write code."

What makes it more than a diagram tool:

• The visual graph compiles through CodeGenConfigurationNodes: "The representation of a directed acyclic graph on which both the AWS Glue Studio visual component and AWS Glue Studio code generation is based." The canvas and the generated script are two views of one DAG.
• Two code flavours exist: "the original, or Classic version, and a newer, streamlined version." Read both before assuming the generated script is unreadable.
• The editor ships Script, Job details, Runs, Data quality, Schedules and Version Control tabs, plus node-level schema and sample data on the diagram.
• Data preview runs on interactive sessions: real DPU consumption, not a free hover preview.
• Job Notebooks give serverless notebook surfaces backed by AWS Glue Interactive Sessions, savable and schedulable as jobs; sample jobs include "Visual ETL job to join multiple sources", "Spark notebook using Pandas" and "Spark notebook using SQL".

---

7. The Glue version matrix (Oct 2026) and Glue for Ray

7.1 The matrix

Two tools sit under that table: an upgrade analysis tool that inspects Glue ≥2.0 jobs and recommends a migration target, and a Spark Upgrade Agent for moving existing jobs to 6.0.

7.2 Glue for Ray: brief, and what changed

Ray jobs use "the same AWS Glue jobs and interactive sessions that you use with AWS Glue for Spark", run on Graviton-based workers measured in M-DPUs (4 vCPU and 32 GB each), and today have exactly one worker type: Z.2X (2 M-DPU, 8 vCPU, 64 GB memory, 128 GB disk, up to 8 Ray workers from the autoscaler). The pitch is pandas without a rewrite: "run your pandas workflows against big data at enterprise scale: with only a few lines of code": plus auto scaling built into the Glue job model and Graviton price performance.

The October 2026 reality check: AWS Glue for Ray is no longer open to new customers (closed 30 April 2026); existing customers keep running jobs, no new features are planned beyond security and availability, and AWS documents a migration path to Ray on Amazon Elastic Kubernetes Service with an argument-mapping table (--min-workers → workerGroupSpecs.minReplicas, --pip-install → Ray runtime-env pip) and a ray job submit example. Two consequences for design: do not propose Ray to a team that is not already a customer, and do not build a new lakehouse critical path on a lane in maintenance mode.

---

8. EMR Serverless: the EMR release train without the cluster

8.1 Shape and setup

EMR Serverless is an application + runtime role + job submission model. You create an application, grant a job runtime role trusted by emr-serverless.amazonaws.com, point at S3, and optionally create an EMR Studio Workspace for interactive notebooks: "you need to specify an S3 bucket and the minimum service role for EMR Serverless to create a Workspace." Workspaces ship with a useful pre-installed set for Python and PySpark: ggplot, matplotlib, numpy, pandas, plotly, bokeh, scikit-learn, scipy.

Release selection is flexible and per-application. "With Amazon EMR 6.6.0 and higher, deploy EMR Serverless"; the current list runs from EMR Serverless 7.13.0 down to 6.6.0, plus the AWS runtime for Apache Spark (emr-spark-8.0.0) and its preview variant. That pinning freedom is the single biggest architectural difference from Glue: you choose the EMR runtime rather than the Spark version Glue happens to expose.

Capacity is elastic per run, not per cluster. You do not size a cluster; the application allocates workers for the run and you are billed by the second for what it uses, with pre-initialized capacity available when spin-up latency matters. Two 2025 additions shrink both cost and IAM sprawl: serverless storage (2 Dec 2025, release 7.12 and higher), "No need to configure local disk or manage instances", with cost reduced "by up to 20%", and inline runtime permissions (22 Jul 2025), which "grant temporary permissions to jobs at runtime without creating separate IAM roles per job."

8.2 EMR on EKS and provisioned EMR: one paragraph each

EMR on EKS runs EMR containers on Kubernetes clusters you already operate. Pricing is the EMR uplift on top of your Kubernetes and EC2 spend: $0.01012 per vCPU-hour plus $0.00111125 per GB-hour (on-demand, us-east-1 context), and "EMR on EKS cluster costs $0.10 per hour." Choose it when you have an EKS estate, need workload isolation per team namespace, or must mix EMR Spark with other containerized services.

Provisioned EMR is whole clusters: you pick instance types and counts, you own the scaling behaviour, and you pay for capacity whether it is busy or idle. It remains the defensible answer for GPUs, custom AMIs, sticky clusters and very large steady-state Spark. Treat any per-hour cluster markup arithmetic as something you re-verify against the live EC2 price list for your Region before printing it: the rates verified for this lesson are Glue DPU and M-DPU, EMR Serverless vCPU and GB, and the EMR on EKS uplift above.

8.3 When EMR Serverless beats Glue (and when it does not)

8.4 Worked example E5: the same 8 vCPU / 32 GB job on two meters

• 📚 Did you know? EMR Serverless advertises both a storage free tier, "The free tier is 20 GB of standard storage per month for 12 months from your initial use", and "Per-second billing (rounded up, with a minimum of 1 minute)." That 1-minute floor is why a six-second Spark invocation still costs a minute of vCPU and memory (AWS EMR pricing, accessed Oct 2026).

---

9. Lambda for light transforms: the control-plane scalpel

Lambda belongs in this lesson because it keeps showing up in lakehouse architectures: validating files on arrival, registering partitions, fanning out per-tenant work, promoting a staging prefix. It is not here as an ETL engine, because the quotas say it cannot be one:

Use Lambda for: event fan-out (S3 → EventBridge → Step Functions → Glue), file validation on landing, partition registration into the catalog, tiny S3-to-S3 fixes on a narrow key range, staged-promotion swaps, and the orchestration glue between jobs.

Do NOT use Lambda for: full table rewrites, joins that need a shuffle beyond /tmp, anything that approaches 15 minutes of steady compute, packages that will not fit 250 MB unzipped, or a fan-out that will exceed account concurrency and throttle. Each of those is a Glue or EMR Serverless job with a partition plan, not a function.

9.1 Worked example E6: the concurrency ceiling arrives before the timeout

---

10. Spark lakehouse patterns: partitions, AQE, skew, shuffle, small files

10.1 Dynamic partition overwrite

When a job rewrites a set of partitions rather than appending, you want the old objects in those partitions gone and everything else untouched. Spark expresses that as spark.sql.sources.partitionOverwriteMode=dynamic, set in Glue through --conf. Used with care it is a clean idempotency tool: run the job twice and the output is the same, because the partitions you wrote were replaced rather than duplicated.

Two caveats worth stating: partition overwrite is partition-scoped, so it does nothing for row-level updates (that is Iceberg MERGE territory), and the rename-algorithm option AWS documents for duplicate nested partition paths (--enable-rename-algorithm-v2) is documented as available only on AWS Glue version 1.0: do not assume it applies to a modern Glue 4.x or 5.x job.

10.2 AQE, skew and shuffle partitions: the three --conf dials

All three live in --conf, and all three change the shape of the plan at runtime rather than your code:

• AQE (adaptive query execution): lets Spark re-plan once shuffle boundaries are known: coalescing over-parallel shuffle partitions, converting sort-merge joins to broadcast joins when statistics allow, and repairing skewed joins.
• Shuffle partitions: the degree of post-shuffle parallelism. Too few on a big shuffle means stragglers and spill; too many on a small job means thousands of tiny tasks whose scheduling overhead dominates the work.
• Skew handling, one hot key makes a single task do all the work while 200 idle. The fix ladder: change or salt the key, enable adaptive skew handling, or repartition away from the offending column. Adding workers does not help single-key skew, the same single task still owns the key.

10.3 Small files: the compaction lifecycle

"Having thousands of small Amazon S3 objects in a data lake table increases metadata overhead and affects read performance."

The Glue Data Catalog ships managed compaction for Iceberg tables, which "performs compaction without interfering with concurrent queries and supports compaction only for Parquet format tables." It fires on a rule, not on a schedule:

• more than 100 files in a table or partition, and
• files smaller than 75% of the target file size, where the target comes from write.target-file-size-bytes (default 512 MB, so the trigger threshold is 384 MB).

Strategies: binpack (the Iceberg default: "Combines small files into larger files, typically targeting sizes over 100MB"), sort, and z-order for multi-dimensional pruning. Two sibling optimizers handle housekeeping: snapshot retention and orphan file deletion, with deleteFileThreshold defaulting to 100 deletes in a data file before compaction considers it.

Two facts surprise people: compaction is disabled by default for new tables, and it bills at the Glue ETL DPU rate ($0.44 per DPU-hour: compaction is its own line on the price list, not a free catalog feature).

10.4 Worked example E7: does the optimizer fire, and what does it cost?

• 📚 Did you know? Compaction is a product, not a script you maintain: the Data Catalog offers three table optimizers, Compaction, Snapshot retention and Orphan file deletion, configurable at catalog level through the Lake Formation console or the UpdateCatalog API, with CLI setup via aws glue create-table-optimizer --type compaction. Athena adds manual OPTIMIZE and VACUUM on the same tables (AWS Glue table optimizers, accessed Oct 2026).

---

11. Idempotency: bookmarks versus MERGE versus watermarks

Correctness at scale is the same question every time: if this run happens twice, is the table still right? The answer is never "bookmarks will handle it."

The checklist that falls out of the facts in this lesson:

1. Give every source and sink a stable transformation_ctx; it is the state key.
2. Enable bookmarks explicitly; disable is the default.
3. Never change a source property while a bookmark is live.
4. Pair bookmarks with a write-side idempotent strategy, dynamic partition overwrite, Iceberg MERGE, or a staging prefix, because only sources are tracked.
5. For Iceberg, prefer MERGE over blind appends; snapshot isolation gives readers that never see half-written state.
6. For streaming sinks, remember the maintenance window: "AWS Glue will restart the job within 3 hours of the specified maintenance window": design sinks for at-least-once plus an idempotent commit.

---

12. Worked PySpark Glue job outline: catalog read to Iceberg silver

Everything so far, in one job. This is the shape of a Glue 5.1 or 6.0 silver build: catalog source with bookmarks on, an explicit schema-settle step where the source carries Glue choice types, dedup on a natural key, an error quarantine, and an Iceberg write that is safe to repeat.

12.1 What to set alongside the code

12.2 Worked example E8: pricing this job

---

13. Cost math: DPU-hours, vCPU-seconds and the Flex discount

13.1 The October 2026 rate card

All Glue rates below come from the AWS Price List offer CSV for AWS Glue (publication 2026-10-06, effective 2026-10-01, us-east-1); EMR rates come from the EMR pricing page and the EMR Serverless pricing map (accessed 2026-10).

13.2 Worked example E9: three meters, one job

13.3 Catalog storage is a bill too

Data Catalog storage is $1.00 per 100,000 objects per month, first 1,000,000 objects free ($0.00001 each), plus $1.00 per million requests (first million free): a lake with **5,000,000** table and partition objects costs `(5,000,000 − 1,000,000) ×$0.00001 = $40.00 per month`. A high-cardinality partition column that explodes object counts is therefore a storage and catalog decision as much as a query-performance one (Lesson 07).

---

14. The decision matrix

2026 Updates (as of October 2026)

---

Real-World Case Studies

Case A, Transport for NSW: ~80% off data integration, and a lead time measured in weeks

Transport for NSW (NSW government transport agency, 2M+ passengers per day, 25,000+ employees). AWS's case study describes a legacy estate in which "CMAI used a third-party Extract, Transform, and Load (ETL) platform… to manage more than 400 ingestion workflows", a licensed platform that "Over time, this approach introduced two critical risks, cost in ongoing licensing and operating constraints." The team migrated those ETL workloads to AWS Glue, with AWS Step Functions for orchestration and monitoring, Amazon S3 as the data-lake foundation, and Amazon RDS retained for backward compatibility during convergence.

The reported outcomes, from the benefits strip of the case study: "reduction in data integration costs 80%", and "to build and deploy new ingestion jobs, down from several months Less than 3 weeks", with "More team members able to manage data workflows on cloud native tools, removing dependency on legacy platform expertise." The mechanism is named directly: "Because AWS Glue supports Python… the team was also able to distribute expertise across several existing team members", and AWS specialists trained "over 30 team members."

Lesson angle: serverless ETL is an organizational fix as much as a technical one. The ~80% saving is the headline, but the two structural wins are the ones that outlive the invoice: a lead time of less than 3 weeks replacing multi-month queue time, and pipeline ownership spreading beyond a single platform-skilled group. In this lesson's terms, TfNSW traded a license meter for DPU-hours (Section 13), a bespoke scheduler for triggers and Step Functions (Section 2), and a key-person dependency for Python skills.

Customer-reported figures from an AWS case-study page (undated, accessed 2026-10-11); not an AWS commitment. Keep the reported rounding: ~80 percent, "Less than 3 weeks".

Case B, Vanguard: 600+ Glue ETL jobs as the operating scale

Vanguard (asset management). In the AWS Big Data Blog of 18 March 2026, Vanguard's two-year modernization from a "fragmented 'data swamp' of Parquet files on Amazon Simple Storage Service" to a structured system published its operating numbers: "20 TB in data volume in Amazon Redshift and another 150 TB in S3 data lake", "600+ AWS Glue ETL jobs (a 30x increase)", "300+ tables (3x growth)", "250+ Amazon Redshift materialized views" and "500,000+ user queries/months." The lakehouse layer is explicit: "Vanguard is curating data products on the data lake using Apache Iceberg format and leveraging AWS Glue for metrics computation and data lake integration."

Lesson angle: 600+ Glue jobs is what "transformation at scale" actually looks like in production: at that count the concerns of Sections 2, 4 and 11 stop being theory: job queuing replaces fail-fast retries at the concurrency ceiling, stable transformation_ctx values matter because nobody can hand-audit 600 state keys, MERGE and managed compaction are load-bearing, and a move from $0.44 to$0.308 per DPU-hour is a line item you feel. Customer-reported figures from an AWS Big Data Blog (published 2026-03-18; accessed 2026-10-11); not an AWS commitment.

Practice Questions