Data Quality and Observability: DQDL, Quarantine, Lineage, and SLOs

Lessons 01 to 14 built something that works: a bronze prefix, Iceberg tables, Glue jobs, a catalog, Athena and QuickSight, orchestration, Lake Formation grants, PII controls and a hardened perimeter. None of that answers the question your consumers will actually ask: can you prove today's gold numbers are right, and can you tell me what changed when they are not? A lakehouse without quality and observability is a very fast way to move wrong data.

This lesson builds the proof layer in the order AWS's own guidance lays it out: engines → rules → runtimes → results and signals → quarantine → SLOs → per-layer observability → audit → lineage → remediation. Two ideas carry the whole thing. First, quality is a pipeline step with an output, not a dashboard: it either gates a write, splits rows, or raises a signal someone is paged for. Second, observability has three distinct planes that teams constantly confuse: the Glue catalog holds technical metadata and permissions, DataZone holds business meaning and lineage, CloudTrail holds who called what. Every number below is dated to October 2026.

In this lesson you will:

• pick a quality engine from AWS's own Prescriptive Guidance mapping, Glue Data Quality, DataBrew, Deequ or custom code, and defend the default;
• author DQDL: rulesets bound to one table, the four modifiers (expressions, where, dynamic rules, labels), the 32 rule types, and which of them can ever produce a failing row;
• run evaluation on both runtimes, the Data Catalog task and EvaluateDataQuality inside a Glue job, and know exactly what each one can and cannot do;
• land results in three sinks: S3 JSON, the Glue service store, and Iceberg tables in the Data Catalog, then read them in Athena;
• wire the signals: EventBridge events, CloudWatch metrics in the Glue Data Quality namespace, alarms, dashboards: including the checkbox that silently kills alerts;
• build the quarantine pattern end to end: split rows, write the quarantine prefix and table, emit the metric, alert, and run a backfill that stays idempotent;
• compute a freshness SLO across three clocks and defend a P95 number with real arithmetic;
• scope CloudTrail data events per sensitive prefix so the audit trail does not become the biggest line on the bill;
• separate the three planes, Glue catalog, DataZone business catalog and lineage, CloudTrail audit, and name only the lineage sources AWS actually documents;
• wire a working EventBridge → SNS → Lambda remediation path with IAM that fits;
• study two AWS customer cases, a sourced October 2026 update box, eleven exam-style questions and four interactive checks.

---

1. Four quality engines, one decision

1.1 AWS's own mapping

AWS Prescriptive Guidance publishes a solution-mapping table for data-quality checks. It is the fastest correct answer to "which tool":

(AWS Prescriptive Guidance: Data quality checks, accessed 2026-10-11.)

For this course's lakehouse the answer is Glue Data Quality by default, DataBrew kept for business-analyst profiling of CSV/JSON datasets, and PyDeequ only where Deequ-level customization is genuinely needed. That is not a popularity contest: Glue DQ evaluates "tables and supported Amazon Athena views that are stored in the AWS Glue Data Catalog" over "Amazon S3, Amazon Redshift, JDBC sources compatible with the Data Catalog, and transactional data lake formats such as Apache Iceberg, Apache Hudi, and Delta Lake"; that is exactly the substrate you built in Lessons 02 to 08 (Glue Developer Guide, accessed 2026-10-11).

• 📚 Did you know? Glue Data Quality is not a re-implementation: AWS states it is "built on top of the open-source DeeQu framework" and that there is "No lock-in… allowing you to keep the rules you are authoring in an open language" (Glue Developer Guide, accessed 2026-10-11). That is why a DQDL rule reads like Deequ's constraint vocabulary, and why the open-source library is still a legitimate escape hatch for analyzers the product does not expose.

1.2 What "managed" buys, in one table

The last row matters. Great Expectations appears in this lesson only as a bring-your-own framework: AWS publishes no managed GX offering, GX's own legacy docs pin the AWS Glue integration to "the most recent version of GX that supports this integration is GX 0.15.50", and the Glue Catalog connector walkthrough is a partner-authored post. Treat GX as your runner to version, patch and support.

---

2. DQDL: the language, the modifiers, the ruleset contract

2.1 The contract: Rules = [ ... ] bound to one table

A ruleset is a bracketed, comma-separated list bound to exactly one catalog table:

That is the shape of create_data_quality_ruleset (Glue Developer Guide: Using APIs to measure and manage data quality, accessed 2026-10-11). Two limits define how big a single ruleset can get: 2,000 rules per ruleset ("If your rulesets are larger, we recommend splitting into multiple rulesets") and a 65 KB ruleset size. Statistics are a separate budget: 100,000 statistics per account, stored free for 2 years (Glue pricing, accessed 2026-10-11).

One language-level exclusion to memorize: "DQDL doesn't currently support nested or list-type column data." Nested structs and arrays are out of scope for direct rule targets: flatten or use CustomSql against a pre-processed view.

2.2 The four modifiers that change semantics

(Glue DQ release notes and AWS What's New, 2025-11-25, accessed 2026-10-11.)

Three supporting features round it out: constants (2025-11-21) let you define a value once and reference it throughout a script, "preventing issues related to query size limits"; pre-processing queries (GA 2025-11-25) run SparkSQL against the table before evaluation via the PreProcessingQuery field of DataQualityGlueTable, so you can derive columns, filter or validate cross-column relationships; and complex composite rules (2024-11-22) let you nest AND/OR groups instead of stacking flat rules.

Note the trap in the with threshold column: that clause works only with ColumnValues, ColumnDataType and CustomSql. Put it on Completeness and you are writing a rule DQDL will not evaluate the way you think.

2.3 The 32 rule types, and the ones that can name a row

The DQDL rule-type reference lists exactly 32 rule types as of October 2026: AggregateMatch, AllStatistics, ColumnCorrelation, ColumnCount, ColumnDataType, ColumnExists, ColumnLength, ColumnNamesMatchPattern, ColumnValues, Completeness, CustomSql, DataFreshness, DatasetMatch, DistinctValuesCount, Distribution, Entropy, IsComplete, IsPrimaryKey, IsUnique, Mean, ReferentialIntegrity, RowCount, RowCountMatch, SchemaMatch, StandardDeviation, Sum, Uniqueness, UniqueValueRatio, DetectAnomalies, FileFreshness, FileMatch, FileUniqueness and FileSize.

They sort into four families:

AWS's own Studio documentation is blunt about the split: dataset-level rules "cannot identify error records because they're applied at the dataset level" (Glue Developer Guide: Evaluating data quality with AWS Glue Studio, accessed 2026-10-11). Build a quarantine on RowCount and you will quarantine nothing.

2.4 Worked example E1: the silver orders ruleset

Target: lakehouse_silver.orders, an Iceberg table partitioned by dt, evaluated inside the bronze→silver job and re-checked daily as a catalog task.

Why each line earns its place: IsPrimaryKey collapses two rules into one; the threshold forms on ColumnValues tolerate sub-1% dirt without failing a whole run; RowCount <= avg(last(10)) * 1.5 is a dynamic spike guard that needs no hand-tuned constant; the row-level CustomSql is the only rule here that can hand you a list of order IDs to quarantine; and DataFreshness … where "dt = current_date - 1" scopes staleness to the partition that is supposed to be fresh instead of screaming about the whole history.

AWS's getting-started walkthrough shows the same shape on its sample tickets table, IsComplete "SET_FINE_AMOUNT", Uniqueness "TICKET_NUMBER" > 0.95, ColumnValues "PROVINCE" in [...], ColumnLength "INFRACTION_DESCRIPTION" between 15 and 31, with the important caveat in the docs: recommendations "make it easy to start… without writing code", but the rules are generated from current statistics, so treat them as a draft to tighten (accessed 2026-10-11).

2.5 Rule recommendations: BASIC, and ADVANCED via Bedrock

start_data_quality_rule_recommendation_run(DataSource={'GlueTable': {...}}, Role=role_arn) returns a RunId; fetch ResultIds from get_data_quality_ruleset_evaluation_run, then each get_data_quality_result(ResultId=…). Recommendation runs are automatically deleted after 90 days. Since 2026-09-18 / 2026-09-25, an ADVANCED mode uses Amazon Bedrock with table metadata plus sampled rows to generate DQDL: a drafting assistant, not an approval.

---

3. Two runtimes: the Data Catalog task and EvaluateDataQuality in the job

They share DQDL, EventBridge, CloudWatch metrics, S3 results and IAM actions. They are not interchangeable.

(Glue Developer Guide: capability matrix, accessed 2026-10-11.)

3.1 Inside the job

The transform's output option is the decision that matters: "Original data" versus "Add new columns to indicate data quality errors". The second appends four columns: DataQualityRulesPass, DataQualityRulesFail, DataQualityRulesSkip (string arrays) and DataQualityEvaluationResult ("Passed" or "Failed" at the row level). AWS's own note is the one to memorize: "your overall results can be FAIL, but a certain record might pass." Rules in Studio go "within the brackets and… separated by commas": Rules = [ Completeness "year" > 0.8, Completeness "month" > 0.8 ].

Incremental quality comes from job bookmarks: for S3 inputs the watermark is object last-modified time, with a maxBand default of 900,000 ms (15 minutes) during which the listing "is likely to be consistent". Bookmarks require job.init at the top and job.commit at the end and key off a unique transformation_ctx: omit it and the operator is not tracked at all (which is exactly how you deliberately re-read a dimension table every run).

3.2 Three result sinks, and since July 2026 the catalog is one of them

1. S3 JSON results (opt-in via resultsS3Prefix or the task's results location) → register with a crawler, CREATE EXTERNAL TABLE with a JSON SerDe, MSCK REPAIR TABLE, then Athena. AWS ships the full DDL, partitioned by year/month/day.
2. The Glue service store → readable through GetDataQualityResult / ListDataQualityResults and rendered in the console's Data quality tab; the Data quality snapshot shows the last 10 runs across rulesets by default.
3. Iceberg tables in the Data Catalog (GA 2026-07-27) → four opt-in result types, rule results, profiling results, row-level results, observation results, configured through --additional-run-options on StartDataQualityRulesetEvaluationRun or additional_options in an ETL job. Nothing is written by default: "You must explicitly enable each result type."

This is the single biggest change to this lesson's subject in 2026: your quality history becomes an ordinary Iceberg table with the same Athena, Lake Formation, time-travel and QuickSight tooling as the rest of the lakehouse.

3.3 The alerting contract has a hidden checkbox

Every completed evaluation emits an EventBridge event:

The ETL variant uses contextType: "GLUE_JOB" with jobId, jobName and evaluationContext. The parallel CloudWatch path is equally simple: "Each AWS Glue Data Quality evaluation emits a pair of metrics named glue.data.quality.rules.passed… and glue.data.quality.rules.failed" under the Glue Data Quality namespace, published per table and per ruleset.

AWS's stated preference: "Our recommended approach is to set up data quality alerts using Amazon EventBridge, because Amazon EventBridge requires a one-time setup to alert customers." The CloudWatch alternative is a standard alarm: glue.data.quality.rules.failed, statistic Sum, period 1 minute, threshold ≥ 1, action = SNS topic.

• 📚 Did you know? The score is not a quality grade you pass around: AWS defines it as "the percentage of data quality rules that passed for the input." A ruleset of one weak rule scores 1.0 while 40% of rows are junk. That is why the row-level columns and the results tables exist: score for the alert, rows for the fix (Glue Developer Guide, accessed 2026-10-11).

---

4. Deequ, PyDeequ and DataBrew: the verified side paths

4.1 Deequ: the engine, and where it still earns its keep

AWS's PyDeequ announcement describes the original: Deequ "is used internally at Amazon to verify the quality of many large production datasets… The system computes data quality metrics on a regular basis (with every new version of a dataset), verifies constraints defined by dataset producers, and publishes datasets to consumers in case of success. In error cases, dataset publication can be stopped, and producers are notified to take action." Compute → verify → publish-or-stop is the loop Glue DQ productized in 2023 (AWS Big Data Blog, 2020-12-30, updated June 2024).

Remaining legitimate reasons to run Deequ yourself, per AWS's own posts:

PyDeequ "supports Spark up to version 3.3.0" as of the PyDeequ 1.1.x notes AWS links, and "can function within Amazon EMR and SageMaker." The honest 2026 summary: most of Deequ's value is already inside Glue Data Quality; reach for the library when the runtime is exotic or the constraint is not expressible.

4.2 DataBrew: rules, but not DQDL

DataBrew's quality story is a ruleset of CheckExpressions evaluated inside a profile job: "If any of a rule's criteria isn't met, the ruleset as a whole fails validation", and the run "produces a validation report in addition to the data profile" at the same location. API shape: CheckExpression, ColumnSelectors (Name or Regex), Disabled, SubstitutionMap, Threshold (Type/Unit/Value). Constraints to respect: only columns of simple types (string, numbers, boolean) participate, and rules live on the Recommendations tab of the ruleset screen.

On completion, "DataBrew sends an Amazon CloudWatch event with the details of that job run… for each validated ruleset… and the event contains its result (SUCCEEDED, FAILED, or ERROR)" plus a link to the report. That event is your EventBridge pattern for analyst-owned checks. Metrics live in the AWS/DataBrew namespace: the documented example is SessionCount.

4.3 Worked example E2: what a quality gate costs, and where to put it

AWS's own benchmark: 104 columns, 1 million rows, 10 DPUs (AWS Big Data Blog, 2024-03-12):

Two conclusions survive the price ambiguity: budget by table size × frequency, not by rule count, and for a quarantine design the in-job evaluation dominates because it is cheaper and it is the only path that can name the rows.

---

5. Quarantine: a pattern, not a product

5.1 Three actions, two checkpoints

The AWS Solutions Library's InsuranceLake sample formalizes what every production pipeline needs: three actions at two pipeline checkpoints:

Checkpoints: before_transform ("immediately after schema mapping… before any transforms are run") and after_transform ("just before saving the data to the Cleanse bucket"). The rule that matters for your ruleset design: "The data quality expression must be written using row level outcomes to evaluate and quarantine individual rows" and "Rules that rely on ratios or thresholds are not supported as quarantine rules."

InsuranceLake also shows what a halt on freshness looks like when you want the pipeline to stop rather than split:

5.2 The split, in code

AWS's Studio guidance also offers a first-party variant: in Data quality actions, choose Fail job after loading target data, which surfaces as AssertionError: The job failed due to failing DQ rules for node: …: halt without quarantine.

5.3 Worked example E3: the quarantine loop with real numbers

The last two lines are the point: quarantine without a backfill path is just data loss with extra steps. And because the results are Iceberg tables now, the "what did gold look like before the bad partition" question is answered with time travel instead of a restore.

5.4 Write-Audit-Publish: the Iceberg-native form

AWS's WAP pattern replaces the physical quarantine copy with branches: write everything to a stg branch → run Glue DQ against it → filter passing rows into an audit branch → fast_forward to main with Spark's fast_forward procedure. The blog's demo rule set checked sensor temperature "between −10 and 50 °C", and it names the trade-off honestly: "The primary challenge of the WAP pattern is the increased latency it introduces." The dead-letter-queue approach (the S3 quarantine prefix) is the other documented option: pick by whether you can afford the latency or the duplicate storage.

• 📚 Did you know? Bookmarks track sources, not targets. AWS's docs state plainly that "when you rewind or reset a bookmark, AWS Glue does not clean the target files because there could be multiple targets and targets are not tracked with job bookmarks." That is why the backfill step above writes to a new prefix or MERGEs, replaying into the same path appends duplicates that your IsPrimaryKey rule will then dutifully quarantine next run (Glue Developer Guide, job bookmarks, accessed 2026-10-11).

---

6. Freshness SLOs: three clocks and the arithmetic

6.1 Separate the clocks before you set a target

Freshness SLO = P95(availability − event) ≤ target. The pipeline contributes two spans: landing − event (ingestion lag) and availability − landing (processing + DQ + backfill lag). DQDL gives you two encodings: DataFreshness against a timestamp column of the table ("Checks if data is fresh", metric Column.[Column].DataFreshness.Compliance, row-level results: Yes) and FileFreshness against S3 file timestamps (file/folder path + threshold, row-level: No).

6.2 Worked example E4: a 3-hour SLO, P95 arithmetic

Add the component P95s rather than averaging means, P95 of a sum is bounded by the sum of the P95s, so 162 minutes is the defensible number to put in front of a stakeholder, and the 18 minutes of headroom tells you which lever to pull before you breach. The failure ladder is deliberately redundant: a freshness rule catches stale data, a zero-RowCount alarm catches a dead pipeline, and a job-FAILED event catches the crash, three different problems that all look like "the dashboard is old".

6.3 Encoding volume and schema drift as alarms

DetectAnomalies deserves a special mention: it "checks for anomalies in another rule type's reported metrics", which turns yesterday's threshold rules into a self-tuning baseline: at 1 DPU per statistic, per run, which is why AWS recommends enabling it selectively.

---

7. Operational observability: one dashboard per layer

7.1 Logs by layer

Multi-account setups centralise with cross-account subscription filters, a Firehose/Kinesis log destination, or Organizations-level CloudWatch Logs data centralization (destination log-group names support ${source.accountId}, ${source.region}, ${source.logGroup}, ${source.org.id}).

7.2 The dashboard map

AWS Prescriptive Guidance's recommendation is one dashboard per concern: "a CloudWatch dashboard can show users the status of loads… or what percentage of their data was dropped due to low quality or which sources have the maximum failures." For this lakehouse that becomes one dashboard per medallion layer plus an account-level data-operations board:

Free tier as of Oct 2026: 10 custom metrics, 3 dashboards referencing up to 50 metrics each, 10 alarm metrics, 1 million API requests and 5 GB log ingestion per month; custom metrics are $0.30 per metric-month for the first 10,000, then $0.10 and $0.05 tiers. Dashboards and alarms are prorated by the hour from creation: an unused alarm is not free, and anomaly-detection alarms "incur costs for each metric that you list in the alarm metric expression, plus two additional alarm-metric costs."

---

8. CloudTrail data events: scoped prefixes and the bill

CloudTrail is the data-plane audit trail, and, unscoped, the surprise bill. "By default, trails and event data stores do not log data events. Additional charges apply for data events." Enabling them requires advanced event selectors with eventCategory Equals Data and resources.type Equals AWS:S3:Object, scoped with resources.ARN StartsWith, note the trailing slash is part of the ARN: arn:aws:s3:lakehouse-bronze/.

Rules of the road:

• "Selectors don't support the use of wildcards like *"; you enumerate prefixes, you do not glob.
• Basic selectors: up to 5 event selectors and 250 data resources per trail; advanced selectors allow up to 500 specified values across conditions, and the two styles cannot be mixed.
• Exclusions are free filters: {"Field": "eventSource", "NotEquals": ["s3lifecycle.amazonaws.com"]} drops lifecycle noise; readOnly = false gives write-only audit for a landing zone.
• Prices: data events $0.10 per 100,000 delivered to S3 (aggregations $0.03 per 100,000 analyzed); management events first copy free, additional copies $2.00 per 100,000. Management events answer who ran StartJobRun / GrantPermissions / StartQueryExecution; data events answer who touched which object.

8.1 Worked example E5: three selectors, one bill

---

9. Three planes: DataZone, the Glue catalog, CloudTrail

9.1 What each plane is for

Lake Formation registration authorizes access to a location, it is a permission scope, not a lineage edge. CloudTrail records activity, it is not a derivation graph. Presenting either as "lineage" is the single most common conflation in this space.

9.2 Lineage: the sources you can actually name

DataZone sits on top of the business catalog: you create data sources for Glue and Redshift, import technical metadata into a project inventory, curate it with business names/glossary terms/metadata forms, and publish: "Only the latest version of the inventory asset can be published to the catalog." A consumer then sees "a view of that asset's lineage with 1-level upstream and downstream." It is a governance and discovery layer, not a replacement for the Glue Data Catalog that Athena and Redshift Spectrum query.

• 📚 Did you know? DataZone lineage reached general availability on 2024-12-03, and the announcement is explicit that capture is automatic for Glue and Redshift and programmatic for everything else: producers "hydrate the lineage information programmatically by creating custom lineage nodes using APIs that accept OpenLineage compatible events from existing pipeline components, such as schedulers, warehouses, analysis tools, and SQL engines." So the integration test for your lakehouse is simple: publish a Glue asset, run its data source, and see whether the upstream node appears, before you promise anyone end-to-end lineage (AWS News Blog, 2024-12-03).

Caption to memorise: catalog = technical metadata and permissions (Glue + Lake Formation); DataZone = business catalog and lineage; CloudTrail = access audit. Three planes, three tools.

---

10. Worked example E6: EventBridge → SNS → Lambda remediation, wired

10.1 The rule

10.2 The targets, in order

Target 1, SNS topic dq-alerts (email / Chatbot for the human channel). SNS delivery to Lambda and SQS is free of an SNS delivery charge, "No charge for deliveries to SQS Queues. Standard SQS pricing applies", and likewise for Lambda.

Target 2: Lambda dq-remediate:

10.3 The IAM that fits, and what it costs

The punchline is the economics: the alerting path is almost free; the data path is what costs money. Which is the correct shape: a lakehouse that pages someone for $1.80 a month and spends $21.90 a month actually measuring its data.

---

2026 Updates (as of October 2026)

---

Real-World Case Studies

AWS publishes what this looks like at scale. Every figure below is customer- or AWS-claimed and unaudited, with source and date named: the examinable point is the pattern.

Case A, Siemens: centralized logs that turned days into 10 minutes

The lesson for this course is the shape of the answer: Siemens's 10-minutes-versus-several-days is what per-layer log groups, scoped selectors and one dashboard per layer buy you; you are not diagnosing faster because the tool is smarter, you are diagnosing faster because the evidence was already indexed and scoped. Note what the case does not claim: Security Lake is a security-log aggregation service, not AWS Lake Formation, and it produces audit evidence, not column lineage.

Case B, Vanguard: a "data swamp" turned into owned data products

Read the 600 jobs and 500,000 monthly queries as a quality-budget statement: at that volume a silent bad partition is not an inconvenience, it is a month of wrong decisions. The "data swamp" phrase is the industry's own shorthand for what happens when a lake accumulates files without curation, ownership and checks; which is precisely the failure mode this lesson's rules, quarantine and SLOs exist to prevent. (Scale numbers are Vanguard/AWS-reported as of the March 2026 post; they are snapshots, not guarantees.)

---

Practice Questions