Capstone Reference Architecture: Observability, ML Activation and the 90-Day Plan for Databricks on AWS

Sixteen lessons built the machine. Lesson 01 gave you the lakehouse concept and the control/compute plane split on AWS; lesson 02 the VPC, root bucket, KMS and PrivateLink substrate; lesson 03 the Unity Catalog metastore and external locations; lesson 04 the Delta transaction log and liquid clustering; lesson 05 Auto Loader; lesson 06 DBR, Photon and cluster economics; lesson 07 Lakeflow declarative pipelines with expectations; lesson 08 Databricks SQL warehouses and Lakeview; lesson 09 Lakeflow Jobs; lesson 10 data quality gates and quarantine; lesson 11 grants, dynamic views and audit; lesson 12 Delta Sharing and Marketplace; lesson 13 Structured Streaming; lesson 14 performance; lesson 15 DBU cost; lesson 16 Declarative Automation Bundles and Terraform. This capstone does one thing: it assembles those sixteen layers into a single reference architecture, then adds the two layers the earlier lessons only sketched, the observability stack and a governed ML slice, and hands you a sequenced plan to actually build it.

The observability layer deserves special care, because it is where production lakehouses quietly fail. "System tables provide a Databricks-hosted analytical store of your account's operational data found in the system catalog", and "System tables are free to use. You are only charged for the compute used to query them" (docs.databricks.com/aws/en/admin/system-tables, footer Oct 8, 2026). Free is not the same as automatic: retention accrues from enablement, history is never backfilled, and the docs are blunt that there is "No support for real-time monitoring. Data is updated throughout the day." Every fact in this lesson carries its Databricks or AWS source and an October 2026 date; anything the docs do not confirm is left out or explicitly flagged as course-authored.

In this lesson you will:

• model observability as five distinct planes that all sit on Unity Catalog, plus a sixth model-awareness plane, and learn exactly which latency each plane carries;
• inventory the current system-table names (Oct 8, 2026 reference), their free retentions and their per-table latencies, and run a six-query audit dashboard against them;
• use data quality monitoring, anomaly detection plus data profiling, the product family that absorbed "Lakehouse Monitoring", with its health indicators, incidents and alert rules;
• close the loop with an alert → incident → replay runbook, wired through job webhooks, DQM alert rules and SQL Alerts-as-code;
• draw the full end-to-end capstone diagram: AWS VPC/S3/KMS, both Databricks planes, Unity Catalog, Lakeflow pipelines, DBSQL serving, Delta Sharing and the observability stack on one page;
• sequence a course-authored 30/60/90-day roadmap against Databricks' 10 production-planning phases, with a milestone scorecard that names the system table proving each milestone;
• place your team on a four-stage maturity model (ad hoc → governed → self-service → ML-ready) with exit criteria;
• activate a governed ML slice, MLflow models in Unity Catalog, signatures, aliases, serving and inference-table observability, while respecting the boundary between this course and a dedicated ML course;
• study Rivian, Block and NAB on AWS, read the October 2026 update box, map the course onto the Data Engineer Associate blueprint, and answer twelve exam-style questions plus four interactive checks.

---

1. Five observability planes, one governance plane

The most common architectural mistake in this layer is treating observability as one product you "turn on". It is not: it is five planes with different owners, different privileges and, critically, different latencies. All five sit on Unity Catalog, and system tables themselves are UC-governed securables: "Users with both the account admin and metastore admin roles have access to system tables by default… USE CATALOG on the system catalog, USE SCHEMA on the system schemas, and SELECT on the system schemas" (system tables reference, 2026-10-08). Everyone else needs those grants explicitly, or a dynamic view. Observability is granted, not ambient.

Three consequences follow, and each one has bitten production teams:

1. The planes do not substitute for each other. Expectations and Delta constraints prevent bad writes at write time (lesson 10); anomaly detection detects drift across a schema; profiling measures and stores drift metrics; SQL and job alerts notify. Different privileges too: MANAGE on a schema to enable anomaly detection, SELECT on metric tables to build alerts, CAN MONITOR on a warehouse to open someone else's query profile. Mixing them into "we have DQ" hides which layer is missing.
2. Minutes-level reaction must be in-band. System tables are updated "throughout the day": anything that must fire in minutes is an expectation fail policy, a job webhook, a DQM alert rule or a SQL Alert inside a job, not a scan of system.access.audit.
3. Latency is a design input, not trivia. Query history records are "typically available within one hour" (query-history reference, 2026-10-06); billable usage "typically available within 12 hours" for record_type = 'ORIGINAL' (billing reference, 2026-09-15). An "hourly cost anomaly" dashboard on system.billing.usage is a category error: the table cannot support it.

📚 Did you know? The older schemas system.operational_data and system.lineage are "deprecated and will contain empty tables" (system tables reference, 2026-10-08). Any runbook, notebook or vendor dashboard still pointing at them is silently reading nothing. The current homes are system.lakeflow.* for job/pipeline timelines and system.access.table_lineage / system.access.column_lineage for lineage.

---

2. Plane 1: system tables as the account plane

2.1 The inventory that matters (verified Oct 8, 2026)

You do not need every table in the system catalog on day one; you need these, with their retentions, so your dashboards and retention policies match reality:

Sources: system tables reference (2026-10-08); audit-logs reference (2026-07-01); query-history reference (2026-10-06); billing reference (2026-09-15); lineage reference (accessed 2026-10). Configurable retention exists in Beta, opt-in, default 395 days, configurable 30 to 3,650 days, but it "does not apply to system.data_classification, system.data_quality_monitoring, system.lakebase" (2026-10-08). And note the streaming caveats if you want to stream system tables into your own Delta tables: DBR 16.4+ to stream, 17.3+ for CDF, Trigger.AvailableNow on 18.0+, and the system schemas themselves use "the default 7 days retention for VACUUM".

2.2 Worked example E1: the six-query audit dashboard

This is the lesson's centerpiece: one Lakeview dashboard, six queries, all valid against the schemas above, deployed as code via a bundle (lesson 16). Run it weekly; publish it to a platform-ops group granted USE CATALOG / USE SCHEMA / SELECT, or better, a dynamic view per group (query-history page's own recommendation).

Teaching notes, each one earned from a documented trap. Q1 is your compliance-evidence query: always event_date-predicated, because the audit page says "To improve query performance, filter on event_date rather than event_time". Q3 is the bridge from the fleet view to the one-query view: click any statement_id in the Query History UI and you land on that statement's query profile. Q5 must be paired with an AWS Cost Explorer tile, because system.billing.usage contains zero AWS spend: no EC2, no S3, no NAT (lesson 15). Unfiltered scans of any of these tables can hit the documented guard: "System Table query returned too much data. Please repeat query with more selective predicates." And Databricks "highly discourages" exporting system tables off-platform: the data "can expose sensitive data and put your deployment at risk".

⚠️ WARNING, observability bolted on last is observability that starts empty. Retention accrues from enablement, and history is never backfilled: a dashboard built on day 200 has 200 days of data; built on day 400 it still has 200 days. The system-tables page's own best practice (carried into the deployment guide's Phase 9, 2026-09-11) is to "Enable system tables for all metastores to capture comprehensive usage data" during foundation, which is exactly why the 30/60/90 roadmap in section 8 puts enablement in days 1 to 30, not day 89.

---

3. Plane 2: data quality monitoring (the product that ate "Lakehouse Monitoring")

3.1 The names, once and for all

If a 2024 tutorial, a vendor deck or even a September 2026 Databricks blog post says Lakehouse Monitoring, the October 2026 answer is: that capability is now data profiling, one of two capabilities under data quality monitoring in Unity Catalog, the other being anomaly detection (data-quality-monitoring page, 2026-09-11: "Data profiling was formerly known as Lakehouse Monitoring"; product page, accessed 2026-10). Teach the new name, recognize the old one, and know that even current Databricks material drifts: the deployment guide still heads a section "Lakehouse Monitoring capabilities" (2026-09-11).

3.2 Anomaly detection: schema-scoped, learned, and patient

Anomaly detection is enabled on a schema or catalog: "up to 50 schemas at a time" via the catalog UI, requiring MANAGE, and "Databricks creates a background job that monitors tables for freshness and completeness" (anomaly-detection page, 2026-10-09, Public Preview). Freshness: "If a commit is unusually late, the table is marked as stale." Completeness: "If the number of rows committed over the last 24 hours is less than the lower bound of this range, a table is marked as incomplete." Results land in system.data_quality_monitoring.table_results; health indicators (Healthy / Unhealthy / Training / Error / Excluded / Not enabled) appear on table pages and lineage nodes; and "Intelligent scanning prioritizes high-impact tables as determined by popularity and downstream usage": leaning on UC lineage and certification, so an org that never certifies anything gets weaker monitoring. Expect "Smart scanning might delay the population of health indicators for some tables by up to two weeks", incidents that can be assigned ("Assign to me" persists 7 days) or dismissed ("Not an issue"), and the irreversible foot-gun: "When you disable anomaly detection, the anomaly detection job and all anomaly detection tables and information are deleted. This action cannot be undone."

3.3 Data profiling: two metric tables, three profile types

Profiling is per-table and produces exactly what a drift dashboard needs: "Profiling creates two metric tables and a dashboard… The profile metric table contains summary statistics… The drift metrics table contains statistics related to the data's drift over time" (data-profiling page, 2026-09-11). The metric tables are Delta tables in a UC schema you choose. Three profile types: time series (metrics across time-based windows), inference (model request logs) and snapshot ("The complete table is processed with every refresh. The maximum table size for a snapshot profile is 4 TB"). Time-series and inference profiles "only compute metrics over the last 30 days". Profiling runs on serverless compute "but does not require that your account be enabled for serverless compute", bills as serverless DBUs under billing_origin_product = 'DATA_QUALITY_MONITORING' (results from February 2026 onward), and, a comfort for regulated estates, "Data quality monitoring does not modify any tables it monitors, nor does it add overhead to any jobs that populate these tables."

3.4 Worked example E2: wiring the alerting layer end to end

Three entry planes, one destination vocabulary. (i) Job webhook: an admin configures a system destination once (email, Slack, Microsoft Teams, PagerDuty or any webhook); each job then sets notifications.on_failure, capped at "a maximum of three system destinations for each notification event type" per job or task (jobs notifications page, 2026-10-09). The payload is stable JSON:

(ii) DQM alert rule (Beta), scoped to catalog.acme or catalog.acme.sales, requiring MANAGE, delivering to the same destination vocabulary when a table goes Unhealthy. (iii) SQL Alert as code, "Databricks SQL Alerts is Generally Available (GA), with more than 4,000 customers already using Alerts in production" (Databricks blog, 2026-05-19); an alert bundles "a SQL query, an evaluation condition, a schedule, and a set of notification destinations", is versionable in Git and deployable through bundles/Terraform, and can run as a Lakeflow Job SQL Alert task (Public Preview) whose output OK / TRIGGERED / ERROR gates the next task via run_if. A monitoring cost check you will actually run:

• 📚 Did you know? Databricks' own internal reliability stack, RADAR, is the four planes of this lesson running in production: reliability metrics → SPOT anomaly detection (an unsupervised streaming model learning "normal" from the past 14 days) → alerting → root-cause dashboards. Published outcome: "95% reduction in incident-discovery time, at over 90% precision" (Databricks blog, 2026-09-19). The build map in that post, metrics in Delta, an MLflow-trained detector served on a Model Serving endpoint, Workflows recurrence, SQL Alerts, Genie for root cause, all deployed as one DAB, is this capstone, internally.

---

4. Planes 3 to 5: pipelines, workloads and query performance

4.1 The pipeline event log, and the deletion that breaks Auto Loader

Lakeflow pipelines expose three nested observability surfaces: the UI (monitoring page, graph, dataset metrics, Data quality tab); the per-pipeline event log, a hidden Delta table readable via SELECT * FROM event_log(<pipelineId>) and publishable to a named UC table; and the region-wide system.lakeflow_pipeline_events_preview.pipeline_events (Beta), which the docs now call "the recommended way to query events", with a ready-to-import Lakeview dashboard JSON on the monitoring page (dated 2026-10-09). Expectation outcomes ride the flow_progress events: details:flow_progress.data_quality.expectations is an array<struct<name: string, dataset: string, passed_records: int, failed_records: int>>. Cost attribution for one pipeline: SELECT sku_name, sum(usage_quantity) FROM system.billing.usage WHERE usage_metadata.dlt_pipeline_id = :pipeline_id GROUP BY 1.

⚠️ IMPORTANT, never delete the pipeline event log. "The event log location also serves as the schema location for any Auto Loader queries in the pipeline… Do not delete the event log or the parent catalog or schema" (pipeline event log page, 2026-10-09). Deleting it does not merely lose history, it breaks future updates, because Auto Loader's exactly-once state lives there (lesson 05).

One operational nuance worth carrying into your runbooks: UI/API-started triggered updates use "fast-start, debugging-focused behavior", while "Updates started from Jobs or the Pipelines API, and all updates for continuous pipelines, use automatic retry and restart behavior"; which is precisely why production pipelines must be launched from a job (lesson 09), not the UI.

4.2 Job notifications: the paging layer, with two documented traps

Job notifications cover start, successful completion, failure, duration-threshold, streaming-backlog-threshold and continuous-maintenance events, delivered to email or admin-configured system destinations (Slack, Teams, PagerDuty, webhook), maximum three destinations per event type per job or task. Webhook event_type codes are stable: jobs.on_start, jobs.on_success, jobs.on_failure, jobs.on_duration_warning_threshold_exceeded. Streaming-backlog alerts evaluate "average backlog over a 10-minute period" and re-notify after 30 minutes. The traps: "Job-level notifications aren't sent when failed tasks are retried" (mute-until-last-retry, or use task-level notifications), and "Succeeded with failures" counts as success. For context on scale: a workspace is limited to 2,000 concurrent task runs, can hold up to 12,000 saved jobs, create 10,000 jobs per hour, and a single job can contain up to 1,000 tasks (Lakeflow Jobs overview, 2026-10-09).

4.3 Query profiles: the per-statement plane

For SQL warehouses and serverless compute, the query profile is the per-statement diagnostic: operator DAG with Time spent / Memory peak / Rows, a Top operators panel, Performance insights ranked "by their projected effect on total task duration", and Genie Code-assisted rewrites presented for your approval (query-profile page, 2026-09-11). Access is a permissions fact with real org consequences: "you must either be the owner of the query or you must have at least CAN MONITOR permission on the SQL warehouse that ran the query". Profiles do not exist for cache-hit queries, and profile JSON export/import is not persisted. Classic-cluster workloads use the Spark UI instead: the deployment guide's Phase 9 states the split plainly: "For serverless compute and SQL warehouses, use query profile… For classic compute clusters, use the Spark UI". E1's Q3 is the fleet view in system.query.history; the profile is the one-query view; statement_id joins them.

The runbook body, each step naming its tool: 1. Detect via any of the three planes. 2. Classify freshness vs completeness vs expectation failed_records spike vs compute failure. 3. Scope with system.access.table_lineage / column_lineage. 4. Root-cause in pipeline_events (error.exceptions[0].error_class) and the expectations array. 5. Repair in dev via a DAB PR (lesson 16). 6. Replay idempotently: job repair re-runs only failed/skipped subsets; Auto Loader replays from the landing prefix exactly once. 7. Validate and close, allowing for up to two weeks of health-indicator scanning lag, and write the new threshold back into the runbook (Phase 9: "Document runbooks for common operational scenarios", 2026-09-11).

---

5. Plane 6: the ML slice: governance first, algorithms elsewhere

5.1 The boundary line, stated once

This course owns the lakehouse substrate ML runs on: UC as the governance plane for models and features (three-level names, grants, aliases, lineage), inference-table and serving-endpoint observability as the same observability pattern applied to models, and where ML artifacts sit in the reference architecture. A dedicated ML course owns algorithms, training and hyperparameter methodology, AutoML, experiment-tracking discipline, feature-engineering design, LLM/agent development, AI Gateway policy authoring and deployment performance tuning. The clean split: we teach governance and observability of models, not how to build them.

5.2 Models in Unity Catalog: the verified mechanics

"Models in Unity Catalog, which Databricks recommends" (model-lifecycle page, 2026-10-09) is "a hosted version of MLflow Model Registry in Unity Catalog… including centralized access control, auditing, lineage, and model discovery across workspaces". Mechanics to know cold:

• the default registry URI in MLflow 3 is databricks-uc; models get three-level names <catalog>.<schema>.<model>;
• models are a subtype of the FUNCTION securable: grant with GRANT ON FUNCTION, create with GRANT CREATE MODEL ON SCHEMA, use with EXECUTE;
• "New ML model versions in UC must have a model signature": without one you lose input enforcement, AI-function usage and auto-generated serving input examples;
• aliases replace stages: "Stages are not supported for models in Unity Catalog": a mutable named reference like Champion is the promotion mechanism;
• promotion across environments is done "by deploying ML pipelines as code" (preferred) or copy_model_version() (MLflow 2.8.0+);
• table→model lineage via mlflow.log_input (MLflow 2.11.0+) renders on the model version's Lineage tab in Catalog Explorer;
• unsupported on UC models: webhooks, email notifications, comment threads, the activity log: use audit logs and CI gates instead.

MLflow version pairing by runtime (release notes, 2026-09-11): DBR 19 → MLflow 3.12.0; 18 LTS → 3.8.1; 17.3 LTS → 3.0.1; 16.4 LTS → 2.21.3.

5.3 Worked example E3: the governance-first ML activation slice

Twenty lines that satisfy every UC rule above, then observe the model like data:

Grants: GRANT CREATE MODEL ON SCHEMA acme.ml TO the training principal; GRANT EXECUTE ON FUNCTION acme.ml.churn_model TO consumers. Observability: an inference table logs predictions for drift analysis; system.serving.endpoint_usage (90-day retention, usage tracking must be enabled) and system.serving.served_entities give the fleet view; system.mlflow.runs_latest / run_metrics_history (180 days) carry training telemetry. Model Serving itself is a serverless REST surface that "can support over 25K queries per second with an overhead latency of less than 50 ms" (model-serving page, 2026-10-06): with two caveats you must design around: metrics data is retained only 14 days (container build logs 30 days), and "Model Serving does not provide security patches to existing model images because of the risk of destabilization to production deployments". Feature Store is current and UC-first: Feature Views (declarative, Public Preview, recommended) and feature tables both publish to the Online Feature Store with point-in-time joins that kill training/serving skew; Workspace Feature Store is deprecated.

📚 Did you know? Databricks announced Genie ZeroOps on 2026-06-16: "an autonomous background agent that monitors your data and AI assets… and takes action before or when things go wrong", with shallow-clone validation of proposed fixes against real data without touching production. Its stated availability at announcement was "entering private preview in the coming weeks"; this lesson cites it only as an announced direction of travel for the runbook layer, never as a capability you can depend on today.

---

6. The full capstone: AWS + Databricks end to end

Here is the whole course on one page. Read it top to bottom: sources on the left of your mind, storage in the customer AWS account, governance in Unity Catalog, medallion pipelines in the middle, serving and sharing on the right, observability wrapping everything, and the two billing meters (AWS invoice vs DBU) that lesson 15 taught you to never conflate.

6.1 Worked example E4: the 17-component checklist

Every component below was built in an earlier lesson; the "proof" column names the system table or artifact that demonstrates it exists. This is the table you review at day 90.

---

7. 30/60/90 days: a course-authored roadmap over Databricks' 10 phases

📌 Provenance note. No Databricks-published "30/60/90-day" adoption schedule exists; the schedule below is a course synthesis anchored to the official 10-phase production-planning sequence (deployment-guide page, 2026-09-11; whose own intro page elsewhere says 9 phases; we teach the 10-phase table, the page that actually lists the phases). The phase names are Databricks'; the days are ours.

Days 1 to 30: Foundation (phases 1 to 5: account, workspace strategy, Unity Catalog, network, storage). Account + metastore live; SSO/SCIM; one metastore per region; root + external S3 buckets with KMS; storage credential + external location for the landing prefix; dev/prod workspaces; a Terraform module for account-level resources; system tables enabled; USE CATALOG / USE SCHEMA / SELECT granted to a platform-ops group. Milestone: a first Auto Loader bronze table lands on S3 through an external location, is queryable in UC, and system.access.audit returns rows for it.

Days 31 to 60: Pipeline + quality (phases 6 to 8: Delta Lake, IaC, compute). Lakeflow pipeline bronze→silver→gold with expectations and a quarantine table; Delta constraints on silver; multi-task job with run_if gates, retries, service-principal run_as; DABs deploying dev/prod targets; cluster policies + instance pools + auto-stop; a DBSQL serverless warehouse with auto-stop; anomaly detection enabled on the silver schema and one time-series profile on the gold KPI table; a jobs.on_failure webhook to the team channel. Milestone: a deliberately corrupted file triggers expectation metrics → quarantine → DQM health flips → webhook fires, end to end, in a game-day drill.

Days 61 to 90: Operate + extend (phases 9 to 10: observability, HA/DR). A Lakeview "Ops" dashboard on system.lakeflow.job_run_timeline, system.query.history, system.billing.usage and pipeline_events; SQL Alerts for freshness/row-count KPIs with Git/DAB versioning; the audit dashboard from E1 reviewed weekly; runbooks published (section 4.3); backup/time-travel + VACUUM retention policy documented; cost attribution by tag with <5% untagged spend. Milestone: on-call resolves a page using only the ops dashboard and a runbook.

7.1 Worked example E5: the milestone scorecard

7.2 Worked example E6: a monthly observability cost sketch

System tables are free; the queries and dashboards are not. Assume an ops warehouse (serverless, $0.70/DBU list, SQL pricing page accessed 2026-10) running the E1 dashboard plus ad-hoc triage 2 hours/day × 20 days/month at ~4 DBU/hour: 160 DBU × $0.70 = $112/month Databricks-side, plus zero storage for the tables themselves. Add one snapshot profile on a 500 GB gold table (under the 4 TB cap) and anomaly detection on two schemas, both bill as serverless DBUs under billing_origin_product = 'DATA_QUALITY_MONITORING'; the rate is not published, so measure quantities in system.billing.usage and price them against your enterprise agreement rather than assuming a list rate. On the AWS side, add nothing beyond the S3 storage already counted in lesson 15 ($0.023/GB-month) and the NAT traffic of your VPC. The design lesson: the observability plane is a rounding error next to the pipeline plane, provided auto-stop is on and the dashboard warehouse is not left at 45-minute defaults with nobody watching.

---

8. The maturity model: ad hoc → governed → self-service → ML-ready

📌 Provenance note. No official Databricks "maturity model" page exists; the four stages are the course's construct mapped onto verified features, with Phase 9's recommendations and the September 2026 data-products post as the closest official supporting language.

Each exit criterion is observable in a system table or a UC object, which is the point. Stage 2→3 is blocked by org design (catalog ownership, delegated MANAGE, data contracts), not by missing features; teams that install DQM without owners get alerts nobody answers, which is worse than no alerts. Stage 3→4 requires lineage spanning table → feature → model, which only exists if training used mlflow.log_input and features are UC-registered. The 2026-09-03 Databricks data-products post prescribes "data contracts upfront that include quality metrics, schema definitions, usage policies and security parameters" plus an "Operate and Govern" phase, the vocabulary of Stage 3, in Databricks' own words.

---

9. Course synthesis: lessons 01 to 16 mapped to the Data Engineer Associate blueprint

The capstone's last academic act is connecting this course to the credential it most directly prepares you for. The Databricks Certified Data Engineer Associate exam guide of record is dated May 4, 2026: 45 scored multiple-choice questions, 90 minutes, $200, valid 2 years, with the seven-domain blueprint below (cert page and exam-guide PDF, accessed 2026-10-12). Note the rebrand the 2026 guide absorbed: the older "Databricks Workflows" audience wording is now "Lakeflow Jobs".

9.1 Worked example E7: course → DEA domain mapping

Where the course stops and the Data Engineer Professional (59 items, 120 min, same $200, ten domains) begins: DEP's heaviest domain, "Developing Code for Data Processing" at 22%, plus systematic failure triage and modelling theory, are awareness-level here, this course demonstrates production patterns; DEP assumes production ownership (on-call, backfills, schema-break incidents, cost overruns). The free AWS Platform Architect accreditation (20 questions, unlimited time, 80% pass, valid 1 year) is the substrate-side counterpart and covers exactly lesson 02's territory: "Customer-managed VPCs · Customer-managed keys · Cloud service integrations". Budget line as of Oct 2026: one certification attempt = $200, no free retakes, 14-day wait between attempts; badges (Fundamentals, accreditations) are free, 10 to 25 questions, 80% pass, via Accredible, not Credly.

---

Comparative Verdict: Databricks-on-AWS vs AWS-native vs Snowflake (final, as of Oct 2026)

• Databricks on AWS, convergence. One storage layer (S3 + Delta, lesson 04), one governance layer (UC, lessons 03/11), one observability layer (system tables + DQM + event logs, this lesson), one orchestration layer (Lakeflow Jobs, lesson 09), one delivery mechanism (DABs/Terraform, lesson 16), one model registry (models in UC). The widest single-pane inventory in this comparison, and the tables are free, you pay only query compute. The honest cost: a second vendor, DBU metering on top of your AWS invoice, and an observability inventory that is only as good as your grants, your tags and your willingness to run the dashboard every week.
• AWS-native (S3 + Glue + Athena + Lake Formation + QuickSight + CloudWatch/CloudTrail), assembly. Every capability above is a separate service you wire together: Glue Catalog and crawlers ($0.44/DPU-hour), Athena serverless SQL ($5.00/TB scanned, 10 MB minimum per query), Lake Formation for grants, CloudTrail as the audit plane. Suits SQL-only, mostly-batch shops that want no platform vendor and per-service billing control. Weaknesses this course kept meeting: no ACID MERGE/CDF story as rich as Delta's, batch and streaming are different toolchains, ML lives in a different product (SageMaker), and the audit plane is CloudTrail, not SQL-queryable governance tables inside the same catalog as your data. AWS itself documents Delta interoperability (Athena reads Glue-registered Delta, read-only, engine v3, no time travel), so the exit door is real.
• Snowflake, the warehouse abstraction. Proprietary storage format, strong SQL/BI concurrency, minimal ops, observability largely through ACCOUNT_USAGE plus its native data-quality suite inside one warehouse abstraction. Choose it when the workload is pure SQL/BI with predictable concurrency and you want warehouse simplicity. It is weaker when ML/AI, Python/Spark, streaming ingestion and open-format portability are first-class requirements, and both vendors publish mutually contradictory comparison-page numbers (treat all of them as date-stamped marketing claims, never as facts).
• Verdict in one line: Databricks-on-AWS is the right call when one governed copy of the data must serve BI, streaming and ML with SQL-queryable observability; AWS-native is the right call when you want per-service billing and will accept assembling the planes yourself; Snowflake is the right call when the workload is SQL-only and you want the simplest possible operational surface, and the case studies below show real AWS customers landing on the first option for exactly the convergence reason.

---

2026 Updates (as of October 2026)

---

Real-World Case Studies

📋 Case A: Rivian (electric vehicles, manufacturing, Cloud: AWS) Challenge. Terabytes of vehicle and field data per day across 70,000+ vehicles; legacy cloud tooling that "struggled to scale pipelines cost-effectively and spent significant resources on maintenance"; siloed teams blocking predictive-maintenance and battery-efficiency ML. Stack on AWS. Databricks SQL, Delta Lake, Unity Catalog, Agent Bricks / Mosaic AI, Liquid Clustering: Cloud: AWS on the customer-story page. Reported outcome. "1,000+ platform users, a 75% increase from a year ago" and "Unity Catalog gives us a truly centralized data catalog across all of our different teams" (https://www.databricks.com/customers/rivian, undated, accessed 2026-10); "50% better runtime performance"; "More than 70,000 Rivians on the road, generating terabytes of data every day" (https://www.databricks.com/your-ai/customers/rivian, 2024-05-30); "Unity Catalog for data governance, Delta Tables for storage, and Liquid Clustering for data layout" for real-time charge-prediction ML (https://www.databricks.com/blog/data-intelligence-action-100-data-and-ai-use-cases-databricks-customers, 2025-07-09). Maps to this lesson. Sections 5 to 6: Rivian is the capstone's named AWS example: one governed lakehouse feeding dashboards and production ML, with liquid clustering (lessons 04/14) under the feature tables and UC (lesson 11) as the single catalog across teams. ⏳ Temporal note: story-page figures accessed Oct 2026; the 2024 and 2025 blog metrics are reported outcomes at the time, not 2026 guarantees.

📋 Case B, Block (Square, Cash App, TIDAL, financial services, Cloud: AWS) Challenge. 12PB of graph + analytics data fragmented across business units; painful two-step IAM flow (policies to roles, then to S3 buckets); GDPR/CCPA PII governance. Stack on AWS. Databricks, Delta Lake, Unity Catalog, Spark Declarative Pipelines, Delta Sharing (planned clean rooms); S3 as co-located storage: Cloud: AWS. Reported outcome. "12X Reduction in compute cost · 20% Reduction in data egress cost · 12PB Of data managed and governed"; data-sharing permission time cut "from days to seconds"; "the adoption of Unity Catalog promoted the collection of data into S3 buckets, resulting in improved latency and the co-location of compute and storage" (https://www.databricks.com/customers/block/unity-catalog, undated, accessed 2026-10). Maps to this lesson. Sections 2 and 6: Block is the governance-and-cost proof point: UC replaced a governance tool and chunk of IAM/S3 plumbing, and the E1 audit dashboard is the operational surface that keeps a 12PB estate answerable. ⏳ Temporal note: undated page, accessed Oct 2026: reported customer outcomes, not platform benchmarks.

📋 Case C: National Australia Bank (banking, Cloud: AWS) Challenge. A 26-year-old Teradata warehouse with ~1,000 use cases and 140 reporting suites; hand-written ETL with up to 64 union-all per pipeline across 300 to 400 engineers. Stack on AWS. Databricks SQL, Auto Loader, Spark Declarative Pipelines on Lakeflow, Unity Catalog (lineage, masking, access control), asset bundles for CI/CD: Cloud: AWS. Reported outcome. "26-year-old Teradata warehouse decommissioned · 456 Use cases migrated to Databricks SQL · 1,200+ Users enabled" (https://www.databricks.com/customers/nab/dbsql, undated, accessed 2026-10); "1,800 Spark Declarative Pipelines running on Databricks Lakeflow · 99.6% Job success rate · 80% Less transformation complexity", latency "under 15 minutes", data quality "+38%" (https://www.databricks.com/customers/national-australian-bank-nab, undated, accessed 2026-10). Maps to this lesson. Sections 6 to 7: a 99.6% job success rate across 1,800 pipelines is what the workload plane (lesson 09) plus the alert→replay runbook (section 4.3) look like at national-bank scale; it is the day-90 milestone of this roadmap, achieved by a real team. ⏳ Temporal note: undated customer-story pages, accessed Oct 2026: reported outcomes at the time of the story.

---

---

Practice Questions

Read the bars as a study allocation, not trivia: Transformation (22%) plus Ingestion (21%) plus Lakeflow Jobs (16%) is 59% of the exam, lessons 04, 05, 07, 09, 13 and 14 are your majority path, while Governance (15%) is where this course's AWS-specific depth (storage credentials vs instance profiles, KMS, PrivateLink, UC audit tables) differentiates you from a generic Spark candidate.