Serverless Transformation with AWS Glue and DataBrew

Transformation is where a data engineer's judgment becomes visible. Ingestion can be bought as a managed pipe; storage can be rented by the gigabyte. But deciding which engine runs the job, how much capacity it holds while it runs, which records are processed on the second run, and where rejected rows go; that is design work, and it is exactly what DEA-C01 Domain 1 (Data Ingestion and Transformation, 34% of the exam; weights per the DEA-C01 exam guide, accessed Oct 2026) examines through the task statements "preparing data for transformation (AWS Glue DataBrew)", "using AWS Glue features to process data" and "using Lambda to automate data processing".

In this lesson you will:

• model the serverless Glue job: job types, anatomy, Data Catalog integration, run states and timeouts;
• separate DynamicFrame from DataFrame and drive resolveChoice → apply_mapping → toDF() in the right order;
• operate job bookmarks: Enable / Disable / Pause, job.init and job.commit, transformation_ctx as the state key, and the five ways they break;
• wire triggers into serial, conditional and scheduled topologies, and know the two hard trigger limits;
• size workers and DPU-hours on the October 2026 price list, including the G-versus-R memory trap and the crawler's 10-minute minimum;
• author with Glue Studio visual ETL: nodes, notebooks, the data-preview session and the Recipe node's import limits;
• pick DataBrew correctly: its six core objects, the 100-step recipe ceiling, recipe versus profile jobs, and its non-DPU metering;
• design error handling: ErrorsAsDynamicFrame, Spigot, EvaluateDataQuality, retries, timeout defaults and where error tables live;
• tune performance: partitioned writes, job run insights, input-file grouping, small-file arithmetic and Glue auto scaling;
• study four AWS customer case studies (BMW Group, PayU, Integral Ad Science and Nasdaq) and a sourced October 2026 update box;
• practise with 14 exam-style questions plus four interactive checks.

---

1. Glue ETL fundamentals: a serverless Spark engine with a catalog attached

1.1 What "serverless" means here

AWS Glue is a serverless ETL service: you define a job, AWS provisions and tears down the Spark (or shell) execution environment, and you pay only while the job runs, billed per second (jobs and interactive sessions bill in 1-second increments, rounded up, with a 1-minute minimum for Spark jobs: AWS Glue pricing, accessed Oct 2026). There is no cluster to keep warm, no AMI to patch, and no node to deprecate. The Data Catalog that crawlers populate is the glue between storage and compute: schemas live in the catalog, data lives in S3 (or JDBC sources), and jobs read both.

1.2 Job types and languages

Authoring routes: PySpark/Scala scripts, Python (shell or Ray), Glue Studio visual authoring, and interactive notebooks (Pandas on Spark, Spark SQL). A Glue job is assembled from: the command, worker type and count, an IAM role, optional connections, an optional security configuration, triggers, a timeout, retries, default arguments, and MaxConcurrentRuns.

1.3 Run states, timeouts and history

1.4 Job anatomy and Data Catalog integration

Two properties of that diagram are examinable:

1. The catalog is metadata, not storage. Crawlers bill separately from jobs (Section 5), and a job can run with or without a crawler in the same trigger chain.
2. Bookmark state lives outside the script. job.init reads it, job.commit writes it, and neither call is optional if you enabled bookmarks (Section 3).

• 📚 Did you know? Glue records every run with its state and keeps the 90 days of history, and WAITING simply means queued. Combined with MaxConcurrentRuns defaulting to 1, a run that appears "stuck" in WAITING is telling you about capacity or trigger position: not about your code (AWS Glue job-runs API, accessed Oct 2026).

---

2. DynamicFrame versus DataFrame: the single most-tested Glue concept

2.1 The two representations

2.2 Why choice types appear: the crawler sampling trap

AWS's own sample (Medicaid provider data) is the canonical illustration: a crawler that inspected only a 2 MB prefix inferred the provider-id column as long. A DataFrame built over the whole dataset met values the sample never saw and forced everything to string. A DynamicFrame correctly reported choice<long,string>: the rows genuinely disagreed. The bug is in the schema inference sample, not in the data, and the fix belongs in the script.

2.3 resolveChoice: settle the ambiguity

resolveChoice(specs=…, choice=…, database=…, table_name=…) offers four actions for a choice column (AWS API reference, accessed Oct 2026):

2.4 apply_mapping: rename, cast, nest, drop

apply_mapping(mappings, case_sensitive=False, …) takes a list of four-tuples:

Rules that cost marks on the exam: backtick a source name that contains a literal dot; a dotted target creates a nested field (provider.id); and case_sensitive defaults to False.

2.5 The order is not negotiable

2.6 Worked example E1: ambiguous provider IDs

2.7 Built-in DynamicFrame transforms worth naming

• Relationalize
• UnnestFrame
• Map
• FlatMap
• Join
• Spigot
• ErrorsAsDynamicFrame
• EvaluateDataQuality
• FlagDuplicatesInColumn
• SelectFields
• DropFields
• FillMissingValues
• mergeDynamicFrame
• FindMatches (AWS PySpark transforms reference, accessed Oct 2026)

The exam tests recognition: which transform routes rejects, which samples output, which tests rules.

---

3. Job bookmarks: incremental batch processing without duplicates

3.1 The three modes

3.2 How state is kept

The bookmark state is stored by the Glue service as roughly {job_name, run_id, run_number, attempt_number, states: {transformation_ctx: state}}, keyed by job name plus version. Your script participates through exactly two calls: job.init reads the state at start, job.commit writes it at the end. AWS's developer guide is blunt: "Bookmarks won't work without calling them."

And the key rule, verbatim from the AWS bookmarks page:

"The transformation_ctx serves as the key to search the bookmark state for a specific source in your script. For the bookmark to work properly, you should always keep the source and the associated transformation_ctx consistent."

3.3 Worked example E2: the delta arithmetic

3.4 When bookmarks break (the exam's favourite failure list)

• 📚 Did you know? AWS's own wording for the guarantee is that bookmarks "prevent duplicate processing and duplicate data": the phrase "exactly-once" is shorthand the exam world uses, but the honest model is at-least-once processing with de-duplicated outputs, because only sources are tracked and a failed run between job.init and job.commit leaves state untouched. Never claim a target-side guarantee you cannot point at in the state structure (AWS Glue developer guide, accessed Oct 2026).

---

4. Triggers: serial, conditional and scheduled topologies

4.1 The four trigger types

4.2 The rules that make conditional triggers work

1. Watched jobs must themselves have been started by a trigger. A job you started by hand and that then succeeded will not fire a conditional trigger.
2. The chain must descend from a single scheduled or on-demand root trigger.
3. Limits: maximum 2 crawlers per trigger; maximum 50 jobs per trigger (AWS Glue quotas, accessed Oct 2026).

Serial vs parallel in one line each: serial = J3 fires when J1 AND J2 both SUCCEEDED; fan-out = J4 fires when J1 OR J2 FAILED. That is the whole conditional-trigger grammar: plus the root-trigger rule, which is where most candidate mistakes live.

---

5. Worker types and DPU sizing: where the bill comes from

5.1 The unit

1 DPU = 4 vCPU and 16 GB of memory, and AWS states that "a single DPU is also referred to as a worker." Jobs on Glue 2.0 and later are sized with WorkerType + NumberOfWorkers: never with MaxCapacity (that legacy field belongs to Glue 1.0 and earlier; setting both is an error). Defaults: Spark jobs 10 DPU, streaming 2 DPU, both with a 2-DPU minimum.

5.2 Worker catalogue (as of Oct 2026)

M-DPU means "double the memory allocation for a given size compared to standard DPUs."

5.3 The October 2026 rate card

Baseline us-east-1 (N. Virginia); every figure as of Oct 2026 (Glue 6.0 rates announced 2026-08-21):

The four DPU-based meters of that table as an interactive chart (same units, so they are legitimately comparable: DataBrew's node-hour meter is deliberately not plotted, because it is a different unit):

Reading the chart: total cost = DPU × hours × bar height. The 6.0 bar is ~30% shorter than the 5.1 bar (the 2026-08-21 price cut), Flex buys ~34% off in exchange for a possible delayed start or interruption, and the R bar is the premium for double memory per DPU.

5.4 Worked example E3: AWS's own DPU-hour arithmetic

5.5 Worked example E4: Flex versus standard

5.6 Worked example E5: worker choice changes nothing about the price

5.7 Worked example E6: the crawler's 10-minute minimum

5.8 Auto scaling and the legacy MaxCapacity

For Glue 3.0 and later, auto scaling is enabled with --enable-auto-scaling=true on worker families G.1X…G.16X and R.1X…R.8X (G.025X for streaming); the Standard worker does not support it. When auto scaling is on, NumberOfWorkers is treated as the maximum, and batch jobs on Glue 4.0+ can be reviewed with the DPUSeconds CloudWatch metric. Data Catalog pricing adds its own meter: the first 1,000,000 objects and 1,000,000 requests are free each month, then $1.00 per 100,000 objects over the free tier and $1.00 per million requests over it (as of Oct 2026).

• 📚 Did you know? A DataBrew job is not priced in DPU-hours at all: it is $0.48 per node-hour with a 5-node default, and an interactive DataBrew session is $1.00 per 30-minute block: three separate interactions inside 75 minutes bill as three blocks = $3.00 (AWS Glue/DataBrew pricing, as of Oct 2026). Mixing the two meters is a classic cost-estimate trap.

---

6. Glue Studio: visual ETL over the same engine

6.1 What the visual layer actually is

Glue Studio lets you create, run and monitor Glue jobs on the serverless Spark engine "without having to learn Spark or write code." Each node in the canvas is one action, a source, a transform or a target, and Studio generates a PySpark script from the source schema plus the target location and schema. The script is real and editable in developer mode; the visual graph and the code are two views of the same job.

6.2 Sessions, previews and monitoring

• Data preview session: 2 DPUs for up to 30 minutes, auto-stops, billed at the standard DPU rate (as of Oct 2026).
• Monitoring: the Jobs page and run details, plus the job run monitoring dashboard with tiles for success rate and estimated DPU usage, links into CloudWatch, the Spark History Server and job-run insights.
• Logs: /aws-glue/jobs/logs-v2 (continuous logging), /aws-glue/jobs/output, /aws-glue/jobs/error; CloudWatch metrics arrive every 30 seconds as deltas.

6.3 The Data Preparation Recipe node: DataBrew inside Studio

You can author a recipe step inline in Studio or import a published DataBrew recipe into a Data Preparation Recipe node, subject to hard limits (AWS Glue Studio docs, accessed Oct 2026):

That bridge is why DataBrew and Glue Studio are on the exam together: a recipe authored by an analyst becomes a governed step inside an engineer's scheduled job, which then gains bookmarks, retries and auto scaling it never had as a standalone DataBrew job.

---

7. AWS Glue DataBrew: preparation and profiling without code

7.1 Six core objects, and the name that does not exist

AWS Prescriptive Guidance lists exactly six core concepts (accessed Oct 2026):

7.2 Recipes: bounded, versioned, point-and-click

• A recipe is an ordered list of RecipeAction objects (Action.Operation + Parameters + optional ConditionExpressions).
• Maximum 100 transformations per recipe; maximum 100 published versions per recipe (AWS quotas, accessed Oct 2026).
• Scale of the library: more than 250 point-and-click transformations (product page) / more than 200 recipe actions (developer guide).
• Exam-relevant operations: SPLIT, GROUP_BY, REPLACE_TEXT, REMOVE_VALUES, CHANGE_DATA_TYPE, PIVOT, UNPIVOT, TRANSPOSE, NEST, UNNEST, and Remove duplicates.

7.3 Jobs, schedules and data-quality rulesets

7.4 Worked example E7: pricing a DataBrew pipeline

7.5 When to pick DataBrew, when to pick Glue

• 📚 Did you know? AWS claims DataBrew can cut data-preparation time by up to 80% versus custom preparation work (AWS Glue DataBrew product page, accessed Oct 2026): a customer-facing claim, not a guarantee, and the examinable fact is the mechanism (project → dataset → recipe → recipe/profile job), not the percentage.

---

8. Error handling: reject rows, retries, and where failures go

8.1 The three in-script patterns

The rejected frame is then written to its own prefix; that is the error table pattern the exam means: bad rows land in s3://…/errors/, good rows land in the partitioned sink, and a downstream conditional trigger or analyst query can read each independently.

8.2 Retries, timeouts and run states

8.3 Worked example E8: the timeout default trap

8.4 Security configuration: four planes, one override rule

A security configuration is a named bundle attachable to a job, crawler or development endpoint, covering four planes: S3 data (SSE-S3 or SSE-KMS), CloudWatch Logs (SSE-KMS), job bookmarks (CSE-KMS: client-side) and data-quality assets. Three rules worth memorising:

1. "Currently, a security configuration overrides any server-side encryption (SSE-S3) setting that is passed as an ETL job parameter."
2. Only symmetric KMS keys are supported, and security configurations are not supported for Ray jobs.
3. The job role needs KMS permissions plus logs:AssociateKmsKey. Catalog encryption is a separate, account-level setting (CatalogEncryptionMode: DISABLED, SSE-KMS, SSE-KMS-WITH-SERVICE-ROLE).

---

9. Performance: insights, partitions and the small-file tax

9.1 Job run insights: the fastest path to a root cause

Available on Glue 2.0 and later, opt-in via the Studio checkbox "Generate job insights" or --enable-job-insights=true. When enabled, a failed run reports (AWS Glue docs, accessed Oct 2026):

• the failing script line,
• the last Spark action,
• the time-ordered exception stream, and
• a root cause with a recommended action.

Compare that with the bare failure An error occurred while calling o111.collectToPython: one is a diagnosis, the other is a symptom. Insights land under the same /aws-glue/jobs/logs-v2 log group you already ship to CloudWatch.

9.2 Partitioned writes and pushdown

Write partition keys with the sink (partitionKeys = ["year","month","day"]) so downstream readers only touch the partitions a query names. Partitioning is a write-time decision you cannot cheaply retrofit on a flat prefix, which is why the sink belongs in the design conversation, not in the cleanup sprint. Partition-aware outputs are also what makes crawlers cheap to run: the crawler registers partitions, and a partitioned table is what lets an engine skip files it will never read.

9.3 Worked example E9: small-file arithmetic

9.4 What to check before you buy more DPU

---

10. Connections, encryption and the version ladder

10.1 Connections are catalog objects, not job settings

"An AWS Glue connection is a Data Catalog object that stores login credentials, URI strings, virtual private cloud (VPC) information…"

Connections are therefore reusable across crawlers, jobs and development endpoints. Types include JDBC, NETWORK, MARKETPLACE, CUSTOM, KAFKA, MONGODB, SFTP (AWS notes SFTP is currently not supported) and the SaaS/database families (MYSQL, ORACLE, POSTGRESQL, SQLSERVER, SYNAPSE, TERADATA, OPENSEARCH, SALESFORCE, …). For JDBC you supply either HOST/PORT/JDBC_ENGINE or a full JDBC_CONNECTION_URL, plus USERNAME/PASSWORD or SECRET_ID, with SSL toggles. Inside a VPC, Glue creates an ENI; security groups need a self-referencing inbound rule for all TCP ports, and egress requires a NAT gateway or VPC endpoints. Limits: 1,000 connections per account, and ConnectionsList per job also caps at 1,000.

10.2 The version ladder (as of Oct 2026)

Glue 6.0 migration deltas that break old scripts: spark.sql.legacy.timeParserPolicy default changed; ANSI mode can fail implicit string→numeric casts; --additional-python-modules is deprecated in favour of --python-virtual-env; EMRFS is removed (S3A is the only S3 filesystem); and the AWS SDK for Java v1 is removed (SDK v2 only: boto3 is unaffected).

10.3 Choosing the engine in one table

10.4 Worked example E10: "the cheapest correct option"

2026 Updates (as of October 2026)

• 📚 Did you know? The exam guide's own removals are as examinable as its additions: AWS SCT, Cloud9 and CodeCommit left the in-scope list at v1.1, which is why "use Schema Conversion Tool" is now a distractor where "use Glue or native tooling" is the answer, and S3 Tables arriving in-scope is why Glue writing Iceberg (including Iceberg v3 features on 6.0) is fair game (DEA-C01 in-scope services page, accessed Oct 2026).
• 📚 Did you know? The rename confusion runs both ways in this topic: the DEA in-scope list still prints "Amazon Kinesis Data Firehose" while the product has been Amazon Data Firehose since 2024-02-09, "no other changes, including service endpoints, APIs, the AWS Command Line Interface (AWS CLI), the AWS Identity and Access Management (IAM) access policies, and Amazon CloudWatch metrics" (AWS What's New, 2024-02-09). IAM actions are still firehose:*, so either name is valid on the exam, never "correct" an option that uses the older one (AWS Firehose history page, accessed Oct 2026).

---

Real-World Case Studies

AWS publishes what these abstractions look like in production. Every figure below is customer- or AWS-claimed and unaudited, with the source named so you can check it: the examinable point is the pattern, not the marketing.

Case A, BMW Group: an on-premises lake becomes a governed cloud lake (Glue as the schema layer)

The examinable pattern is the separation AWS teaches: data lands in S3, schemas live in the Glue Data Catalog, transformation runs in Glue/EMR, and consumption (Athena, SageMaker) reads the same catalog. Producers and consumers are different accounts with different privileges; which is why the catalog, not the bucket, is the contract.

Case B, PayU: Glue ETL into a shared warehouse (transformation feeding consolidation)

"In 1 month, we cut down queries by 77 percent, which would have been a 6-month exercise in the previous environment.": PayU, AWS case study (accessed Oct 2026)

Read PayU as two separate levers: the platform move (Glue → S3 → Redshift with data sharing) is an architecture change worth $20,000/month (customer-claimed), while the demand rationalisation is a 77% query reduction. Answers that collapse both into "AWS saved 77% on cost" are reading the number wrong.

Case C, Integral Ad Science: the catalog as the authorization plane (Glue Data Catalog + Lake Formation)

"With Lake Formation tag-based access controls, IAS reduced hundreds of permission rules down to precisely two rules.": AWS Big Data Blog (accessed Oct 2026)

Read IAS as the same catalog this lesson describes (§1.4) wearing a second hat: the Glue Data Catalog holds the schema, and Lake Formation decides who may read which table, column or row of that schema. That is why "just add an S3 bucket policy" is the wrong instinct in an exam answer about a governed, multi-account lake, and why tags, not principals, are the scaling mechanism.

Case D, Nasdaq: an exchange batch load decoupled from query (S3 as the landing zone)

"We were able to easily support the jump from 30 billion records to 70 billion records a day because of the flexibility and scalability of Amazon S3 and Amazon Redshift.": Robert Hunt, VP Software Engineering, Nasdaq (AWS case study, accessed Oct 2026)

The examinable pattern is contention removed: loading writes to S3 while querying reads Redshift/Spectrum over the same storage, so the nightly deadline stops being a shared-capacity argument. It is the same architectural instinct as PayU's Glue → S3 → Redshift path, land once in the lake, transform with Glue, let each consumer read cheaply, and the reason partitioned, catalogued output (§9.2) matters more than raw throughput.

• 📚 Did you know? Neither case study mentions a code-free tool as the load path: both put Glue in the pipeline and kept humans out of the loop for the scheduled work. That is the shape of the exam's answer too: DataBrew for preparation and profiling, Glue for the scheduled, governed, incremental run (AWS case studies, accessed Oct 2026).
• 📚 Did you know? Nasdaq's own words describe a load deadline, not a query one: 90% of the daily load finished 5 hours sooner after moving to S3 + Redshift, and the 15 TB lake is queried where it sits instead of being copied: with S3 Object Lock holding the retention side of the exchange's compliance story (AWS Nasdaq case study, accessed Oct 2026). Numbers are AWS/customer-published and specific to Nasdaq's workload; do not generalise them into a universal "32% faster" claim.

---

Practice Questions

Comparative Verdict: how this topic compares on exam day

• Versus another cloud: this exam tests AWS services only: nothing in DEA-C01 asks you to compare Glue with a competitor's hosted Spark. Answer with an AWS service or an AWS-documented behaviour (DPU, quotas, trigger rules); an option that pivots to an unverified third-party benchmark is out of scope by construction.
• Versus self-managed / on-premises (your own EMR or Hadoop): Glue removes cluster provisioning, patching and idle capacity, you pay per second while the job runs ($0.44/DPU-h ≤5.1, $0.308/DPU-h on 6.0, as of Oct 2026) with a 1-minute minimum. The counter-case the exam respects: self-managed EMR remains defensible when you need fine-grained cluster control or a runtime Glue does not offer, the answer must fit the stated requirement, not a slogan.
• Versus another AWS service: Lambda wins for sub-15-minute event micro-transforms (GB-second metering, no DPU floor); Glue pythonshell for >15-minute single-process Python at 0.0625 DPU; Glue Spark for distributed GB: TB joins and partitioned lake writes; DataBrew for code-free preparation, cleansing and profiling; Amazon EMR when you need full Spark/Hive/Presto cluster control. Wrong answers almost always size the engine to habit instead of to volume, latency and who writes the logic.
• Versus a manual, human process: DataBrew exists because preparation is often analyst work, but scheduled, incremental, governed production runs belong to Glue with bookmarks, triggers, error tables and job insights. Choose the option with the least unmanaged manual effort: hand-edited CSVs and ad-hoc scripts are distractors, not bonus points.