Automating Data Quality in Pipelines

Data quality is the one Domain 3 topic that other topics silently depend on: a perfectly orchestrated pipeline that moves wrong rows on schedule is worse than no pipeline at all. The DEA-C01 exam guide states the requirement as a single skill, "Run data quality checks while processing the data (for example, checking for empty fields)" (Task 3.4.1, DEA-C01 exam guide, accessed Oct 2026), and everything examinable hangs off that word while. A weekly profile of the landing zone is a report; a check that runs inside the job, splits the rows and gates the next step is engineering.

In this lesson you will:

• read Task 3.4's five sub-statements and defend the "while processing" rule against end-of-pipeline distractors;
• map the six quality dimensions to AWS-native expressions and to the scenarios the exam writes;
• build the seven-check in-flight checklist, null/empty, schema, referential, duplicates, range/enum, freshness/watermark, volume, in DQDL, DataBrew and PySpark;
• operate AWS Glue Data Quality: rules, analyzers, rule sets, the 26 Studio rule types, row-level outcomes, the three failure actions and the publishing defaults;
• operate AWS Glue DataBrew quality: rule anatomy, rule sets that run in profile jobs, findings, the validation report, events, schedules and sampling;
• wire Glue job quality patterns: in-script assertions, error tables, the quarantine prefix pattern and streaming dead-letter destinations;
• detect schema drift: job bookmarks versus DynamicFrame resolveChoice, crawler SchemaChangePolicy, Data Catalog table versions and drift alerts;
• gate orchestration with EventBridge payloads and a Step Functions Choice on state and score, and publish a quality dashboard;
• decide quarantine versus fail-the-job, and why the exam guide wants in-flight over quality-at-the-end;
• study four AWS customer case studies, a sourced October 2026 update box and 14 exam-style questions plus four interactive checks.

---

1. Task 3.4: what "while processing" actually means

1.1 The five sub-statements

The trap is written into the first line: "profile the landing zone weekly" is a report on yesterday's data, not a check while the data is processed. Options built on "load everything, then profile on Fridays" are designed to fail 3.4.1.

1.2 Quality-at-the-end versus in-flight

Both have a place: an at-rest evaluation against the Data Catalog is legitimate for monitoring a table that is already landing. But when the question's skill line is 3.4.1, the answer is a check that runs during processing or that gates the next step.

1.3 Which tool: the three-way matrix

AWS's own Prescriptive Guidance table is the fastest "which tool" aid on this exam:

• 📚 Did you know? AWS Glue Data Quality is built on open-source Deequ, "a library that is used internally by Amazon to manage the quality of data lakes over 60 PB" (AWS Big Data Blog, 2023-06-06). That is why amazon-deequ-glue exists as an AWS sample, but Deequ-as-a-product, Great Expectations, dbt and Soda are not on the DEA-C01 in-scope list, so an option built around them is never the best answer.

---

2. The six dimensions, mapped to exam scenarios

2.1 Dimension → AWS expression → scenario

The data quality score is simply "the percentage of data quality rules that pass (result in true) when you evaluate a ruleset" (AWS Glue Data Quality docs, accessed Oct 2026): a number between 0 and 1.0, not a pass mark on its own.

2.2 Worked example E1: completeness that hides behind a string

A CSV column email arrives with 1,000,000 rows. In the raw file, 3,000 rows carry an empty field. Two rules are written over the same data:

Reading: the same 3,000 bad rows can pass or fail depending on the type the reader infers. That is why completeness rules are written after type coercion, at the post-cast stage boundary, and why "checking for empty fields" in 3.4.1 is a placement skill as much as a rule-writing skill.

2.3 Match the dimension to its exam signature

• 📚 Did you know? Of the six dimensions, uniqueness is the one most often "enforced" by a constraint that does not enforce: Amazon Redshift states that "Uniqueness, primary key, and foreign key constraints are informational only; they are not enforced by Amazon Redshift when you populate a table", while "Amazon Redshift does enforce NOT NULL column constraints" (Redshift docs, accessed Oct 2026). So INSERT of a duplicate primary key succeeds, INSERT of NULL into a NOT NULL column fails, and both behaviours are examinable as "which DDL prevents duplicate loads? None."

---

3. The in-flight checklist: seven checks, placed at stage boundaries

Every check below can be written in DQDL (Glue Data Quality), as a DataBrew rule, or in plain PySpark: placement is what earns the mark for 3.4.1.

3.1 Null / empty

3.2 Schema validation

ColumnExists, ColumnDataType, ColumnLength, ColumnNamesMatchPattern, SchemaMatch. An S3-target job default-enables an "at least one column" rule plus anomaly detection on column count: a failure lowers the score but does not fail the job (Glue tutorial, accessed Oct 2026). Schema rules answer validity; drift (Section 7) answers what changed since last time.

3.3 Referential integrity

3.4 Duplicates

3.5 Range / enum

3.6 Freshness / watermark

Bookmarks give incremental scope, not validation; they track processed files and rows and perform no schema check (Section 7). Streaming uses checkpoints, and Glue streaming's default window is 100 seconds.

3.7 Volume

RowCount between X and Y, RowCountMatch, AggregateMatch, CustomSql: "almost any type of data quality checks in SQL" (DQDL rule types, accessed Oct 2026). Volume rules are the cheapest way to catch "the source sent 4% of yesterday's rows" before downstream aggregates silently publish a small number.

3.8 Worked example E2: freshness against an SLA

An SLA says orders must be no more than 24 hours old when the curated table is published at 02:00 UTC.

Two signals, one decision: the DQDL rule bounds the rows, the watermark comparison bounds the load. Exam options that offer only one of them are usually testing whether you know which failure each one catches.

---

4. AWS Glue Data Quality: DQDL, rule sets, outcomes

4.1 Three objects, two entry points

Studio offers 26 rule types; a results evaluation writes to S3 (optionally Iceberg/Data Catalog tables you can query with Athena), and anomaly detection "requires a minimum of three data points".

4.2 What happens when a rule set fails

4.3 Row-level outcomes, and the eleven rules that cannot produce them

rowLevelOutcomes adds four columns and a per-row Passed/Failed status. Eleven rule types cannot be attributed to rows, AggregateMatch, ColumnCount, ColumnExists, ColumnNamesMatchPattern, CustomSql, RowCount, RowCountMatch, StandardDeviation, Mean, ColumnCorrelation (Glue tutorial, accessed Oct 2026). So "which rows failed?" is answered by row-scoped rules in an ETL job, never by a Catalog evaluation, and never by an aggregate rule.

4.4 Worked example E3: score, and why you route on state, not score

A rule set of 6 rules runs: 4 pass, 2 fail.

A skipped rule does not count as a pass: the score is rules that passed over the evaluated rules, so never assume state and score agree.

---

5. AWS Glue DataBrew: rules, rule sets, findings

5.1 Rule anatomy

Example checks from AWS's own material: Number of rows = 5000000, Duplicate rows, Unique values = 100% rows, Value is not missing >= 100% rows, Value in column APY between 0 and 100, Number of missing values in column group_name doesn't exceed 5%.

5.2 Rule sets run in profile jobs, not recipe jobs

CreateRuleset is documented as "Creates a new ruleset that can be used in a profile job to validate the data quality of a dataset". Recipe jobs apply recipe steps; that family populates missing values, removes invalid data and removes duplicates, which is remediation, not validation. The canonical false statement on this exam is "attach the rule set to the recipe job".

Quotas that questions quote directly (as of Oct 2026): 100 rules per rule set, 10 rule sets per dataset, 100 rule sets per account, 100 steps per recipe, 300 nodes per account.

5.3 Findings, validation report and events

• Profile outputs: Summary, Correlations, Value distribution, Column Summary, plus a Column statistics tab.
• Validation results: on the Data quality rules / Data quality tab of the job run.
• Builder help: a Dataset preview tab and a Recommendations tab while you author.
• Report: the profile job "produces a validation report in addition to the data profile … at the same location as your profile data".
• Events: one EventBridge event per run and per rule set:

One rule with that pattern matches every failed validation in the account: the examinable payload is detail-type + validationState, not a per-dataset ARN.

5.4 Sampling (Task 3.4.4)

AWS's own wording: "A smaller sample size allows DataBrew to perform transformations faster … A larger sample size more accurately reflects the makeup of the underlying source data."

• 📚 Did you know? DataBrew schedules are six-field cron expressions with a hard floor: "Cron expressions that lead to rates faster than 5 minutes aren't supported", and day-of-month versus day-of-week is exclusive (use ? for one of them). So the "every 2 minutes, validate the stream" option is wrong for two independent reasons: DataBrew is not a stream processor, and its clock cannot tick that fast (DataBrew jobs docs, accessed Oct 2026).

5.5 Worked example E4: an HR rule set that fits the quotas

Five rules over a nightly HR extract of 5,000,000 rows:

5.6 Skew (Task 3.4.5): the timeliness dimension's twin

Skew does not corrupt rows; it destroys the timeliness of the run. Detect it with the driver metrics glue.driver.skewness.job and glue.driver.skewness.stage, the Spark UI, or partition-size statistics. Mitigations AWS documents: hot-key filtering, incremental aggregation, salting, broadcast joins, repartition/coalesce, adaptive query execution (default on Glue 4.0+), input partitioning and vertical scaling: AWS's own conclusion is that "there is no single universal solution". On the exam, "one task runs 40 minutes while the other 39 finish in 30 seconds" ⇒ skew, and the fix named in the option is salt the hot key or broadcast the small table.

---

6. Glue job quality patterns: assertions, error tables, quarantine

6.1 In-script assertions: the strongest in-flight gate

Scala form from the AWS tutorial: assert(DQ_Results.filter(_.getField("Outcome").contains("Failed")).count == 0, "…caused the job to fail."). An assertion turns a score into a failed job, which is what "run checks while processing" looks like when the option says stop before bad data lands.

6.2 Error tables, and the name that does not exist

6.3 The quarantine prefix pattern

"AWS Glue Data Quality helps you identify the exact records that caused your quality scores to go down. Easily identify them, quarantine and fix them.": AWS Glue Data Quality product page, accessed Oct 2026

AWS's InsuranceLake reference sample states the contract precisely: "The quarantine action removes individual rows that fail the quality check … stores them in a separate quarantine storage location and Data Catalog table", with rules enforced immediately after schema mapping … before any transforms are run, at stages before_transforms / after_transforms / after_sparksql, and the <table>_quarantine_* S3 folder and catalog table are created regardless of whether any rows are quarantined, so downstream monitoring never has to guess whether the path exists.

6.4 Streaming: the dead-letter side of quality

For Lambda consumers of Kinesis/Kinesis Data Firehose, the quality failure path is the on-failure destination, not a console alarm: DestinationConfig.OnFailure → SQS, SNS, S3 or Amazon MSK, with BisectBatchOnFunctionError to isolate the poison record, MaximumRetryAttempts 0 to 10,000, MaximumRecordAgeInSeconds 60 to 604,800, and ReportBatchItemFailures returning batchItemFailures for partial-batch handling. A function-level DLQ is the older mechanism; the on-failure destination is the current one, and the two are not interchangeable in an option pair. Glue streaming jobs checkpoint rather than use job bookmarks.

6.5 Worked example E5: the quarantine split

One million rows land; the job carries 6 rules, 4 of which are row-scoped:

Read the two halves together: the data path quarantined 4,000 rows and published 996,000; the control path published state = FAILED and a score of 0.67; while the job still reported success because the action was None. Three different signals, three different answers to "did it work?".

---

7. Schema drift detection: what actually sees a changed shape

7.1 Bookmarks are not drift detection

Pair the crawler with the Glue Data Catalog Table State Change EventBridge event (UpdateTable, CreatePartition, UpdatePartition, DeletePartition plus batch variants) for drift alerts; LOG behaviour means the catalog is never mutated by an alerting-only crawler.

7.2 resolveChoice: the repair that costs data

The crawler samples only a 2 MB prefix of the prefix; a source where 2 rows in 160,000 hold strings in a numeric column becomes choice<long,string>:

Order matters: resolveChoice → apply_mapping → toDF(). Choosing cast without a completeness rule converts a schema problem into a data problem you never see.

7.3 Worked example E6: the Redshift constraint illusion

Same check written as a DQDL rule, IsPrimaryKey "id" = "not NULL and unique", does reject the duplicate. CHECK and exclusion constraints are unsupported in Redshift. So "the load failed because a primary key duplicate was rejected" is wrong, and "add a uniqueness rule in the job" is right.

---

8. Orchestration: gate the pipeline on the quality result

8.1 The EventBridge payloads you must recognise

DataBrew's counterpart is source: aws.databrew, detail-type: DataBrew Ruleset Validation Result, detail.validationState: FAILED. Glue crawler drift arrives as Glue Data Catalog Table State Change. Three services, three detail-type strings: the exam tests whether you can pick the right one.

8.2 Step Functions: the Choice state is the gate

Amazon States Language semantics worth memorising: Choices[] are evaluated in order, and "If no Choices evaluate to true … and no Default is provided, the state machine will throw an error"; which is why AWS recommends a Default.

8.3 Dashboards and metrics

The documented QuickSight path is EventBridge → Lambda → S3 → crawler → Athena → QuickSight, giving a score time series rather than a per-run verdict. CloudWatch metrics live under dataQualityEvaluationContext (default off) and logs under /aws-glue/data-quality/output and /aws-glue/data-quality/error. A question asking "how do analysts see the quality trend over 90 days?" is a dashboard question, not an alert question: the answer needs stored results plus a BI layer, not an SNS topic.

8.4 The whole pipeline, one picture

Every arrow above is a dependency: nothing downstream can exist until the artefact upstream of it has been produced. Order the whole gate the same way:

---

9. Quarantine versus fail-the-job, and the tradeoff the exam actually asks

9.1 The decision in one line each

• In-flight beats at-the-end for 3.4.1, but an at-rest Catalog evaluation is right for monitoring a table that already lands.

• DataBrew for no-code, analyst-owned, column/table rules run as a profile job; Glue Data Quality for in-job checks, row-level outcomes and Flex/auto scaling; custom code for anything with cross-row logic no rule type covers.

• Route on state, score on score; they are different signals (E3).

• Bookmarks scope, crawlers and SchemaMatch detect drift: never the other way round.

• Sampling is for previews; scores need the full scan.

• 📚 Did you know? The replay half of the quarantine loop now has an in-lake mechanism: since 2026-04-23 Amazon Redshift can UPDATE / DELETE / MERGE partitioned and unpartitioned Apache Iceberg tables, including Amazon S3 Tables, and since AWS's Patch 202 an Iceberg DELETE on a Lake Formation table needs the DELETE permission, UPDATE/MERGE need INSERT + DELETE, and every Iceberg DML statement also needs ALTER. So "fix the quarantined rows and merge them back" is an ordinary operation, while a lab that granted only INSERT now fails halfway through the remediation (AWS Redshift behaviour-changes documentation, accessed Oct 2026).

---

2026 Updates (as of October 2026)

• 📚 Did you know? The service names around this topic fossilise fast: Kinesis Data Firehose became Amazon Data Firehose (2024-02-09) yet the DEA in-scope list still prints the old name, and Amazon QuickSight became the Quick Suite / Quick Sight family (2025-10-09). For data quality specifically the rule is simpler and more durable: Glue Data Quality has been GA since 2023-06-06 and is still DeeQu-based: the placement rules (in-flight vs at-rest, row-level vs aggregate) have not changed, so study those rather than the release notes (AWS docs, accessed Oct 2026).

---

Real-World Case Studies

AWS publishes what these abstractions look like in production. Every figure below is customer- or AWS-claimed and unaudited, with the source named so you can check it: the examinable point is the pattern (which quality dimension was the binding constraint, which service did the work, which number moved), not the marketing.

Case A, Nasdaq: timeliness as the binding quality dimension

"We were able to easily support the jump from 30 billion records to 70 billion records a day because of the flexibility and scalability of Amazon S3 and Amazon Redshift.": Robert Hunt, VP Software Engineering, Nasdaq

Read the exchange case as a timeliness question in disguise: the deadline is external and non-negotiable, so the quality gate must run in flight, a RowCount between X and Y rule plus a freshness/watermark check at the pre-write stage, because there is no window to profile afterwards and reload.

Case B, PayU: consistency and freshness across 40 databases

"In 1 month, we cut down queries by 77 percent, which would have been a 6-month exercise in the previous environment.": PayU (AWS case study, accessed Oct 2026)

The examinable pattern: freshness is a measurable dimension, not an adjective. Moving from a daily batch to a sub-30-minute feed changes what "timeliness" means operationally, and every step in that path (Glue → S3 → Redshift) needs a boundary check, because the speed-up multiplies the number of opportunities for a bad row to reach a consumer.

Case C, FINRA: volume and completeness under a regulator's clock

"FINRA processes approximately 6 terabytes of data and 37 billion records on an average day … On busy days, the stock markets can generate 75 billion+ records.": John Brady, VP Cyber Security/CISO, FINRA (AWS blog, accessed Oct 2026)

Read FINRA as a volume-rule design problem: a busy day carries roughly twice an average day, so RowCount = 37,000,000,000 is wrong the first time the market moves. The examinable shape is a range, RowCount between <quiet-day floor> and <busy-day ceiling>, evaluated in flight at the pre-write boundary and paired with completeness on the columns surveillance depends on, so a truncated feed is caught before the downstream aggregate publishes a plausible-looking number.

Case D, Hearst: 30 TB of clickstream a day, gated per batch

"I don't know how we could have made our clickstream data pipeline work without Amazon Kinesis services. It would have involved many weeks of engineering.": Peter Jaffe, Data Scientist, Hearst (AWS, accessed Oct 2026)

Read Hearst as the streaming half of Task 3.4.1: with 30 TB arriving continuously there is no end-of-load window to profile afterwards, so quality becomes a per-batch gate, ReportBatchItemFailures plus an on-failure destination to isolate poison records (Section 6.4), checkpoints instead of job bookmarks, and a volume/freshness rule on each micro-batch before it is written. The name in the story also carries a trap: Kinesis Data Firehose became Amazon Data Firehose on 2024-02-09 with no API change, and the DEA-C01 in-scope list still prints the old name, both are correct.

• 📚 Did you know? None of these AWS case studies mentions a quality tool at all: AWS publishes the outcomes and leaves the wiring to you. That is deliberate exam design: DEA-C01 never asks you to recall a customer's rule set, it asks you to recognise that a market-open deadline implies a pre-write gate, and that forty source databases imply cross-dataset consistency checks (AWS case studies, accessed Oct 2026).
• 📚 Did you know? Date-stamp every case number you quote: FINRA's 6 TB / 37 billion records a day comes from an AWS Public Sector blog of 2017-10-03, while the "over 100 billion events a day from 22 exchanges and 1,500 broker-dealers" figure belongs to the Consolidated Audit Trail and an AWS press release of 2019-12-04: two different systems, two different dates, and AWS has published no newer average-day total. An exam option that blends them ("FINRA processes 100 billion records daily") is testing your source discipline, not your arithmetic (AWS, accessed Oct 2026).

---

Practice Questions

Comparative Verdict: how this topic compares on exam day

• Versus another cloud: DEA-C01 tests AWS services only: nothing asks you to compare Glue Data Quality with a competitor's quality product. Answer with an AWS service, an AWS-documented rule type or an AWS payload; any option pivoting to an unverified third-party benchmark (including tools absent from the in-scope list) is out of scope by construction.
• Versus self-managed / on-premises tooling: the AWS answer is always managed and declarative over bespoke and manual: a DQDL rule set or DataBrew rule beats a hand-rolled validator, because it is versioned, auditable, scheduled and alertable. An option describing a custom framework you must maintain is the distractor, not the bonus.
• Versus another AWS service: pick by placement, not prestige, DataBrew for no-code profile validation and findings, Glue Data Quality for in-job and row-level outcomes (and Catalog evaluations for at-rest monitoring), custom code in Lambda/EMR only for logic no rule type covers. Crawlers detect schema change, EventBridge delivers results, Step Functions gates promotion, none of them evaluate row quality.
• Versus a manual, human process: Nasdaq's market-open deadline and PayU's move from daily to sub-30-minute freshness (both customer-claimed, accessed Oct 2026) show why the exam prefers an automated gate: humans triage the quarantine, they do not discover the failure. Treat every published percentage as "the customer achieved", never as "AWS guarantees".