Lakehouse Architecture on Databricks: Planes, Pillars, and the AWS Substrate

Every other lesson in this course assumes three things you have not yet proven to yourself: that a lakehouse is a real architectural category and not a marketing word, that Databricks software runs in two cloud accounts at once, and that the money arrives on two separate bills. Get any of those three wrong and you will design a workspace whose data lake sits in the wrong bucket, whose threat model points at the wrong account, and whose cost model undercounts by roughly half.

This lesson proves all three with primary sources. Every architectural fact below comes from docs.databricks.com pages dated between 2026-07-28 and 2026-09-25, from Databricks' own AWS deployment blogs (2022-09-30, 2023-01-27), and from AWS pricing pages accessed October 2026. Every price is a list price in USD, stamped with the date it was read, because DBU rates and AWS rates both move.

In this lesson you will:

• define the lakehouse against the data lake and the data warehouse, using the three properties Databricks published in 2020 and the medallion (bronze → silver → gold) pattern;
• separate the two pillars, Delta Lake for reliability and format, Unity Catalog for governance, and map each to the AWS object it touches (your S3 bucket, your IAM role);
• draw the control plane / compute plane split, including secure cluster connectivity, the cross-account IAM role and the serverless plane's region rule;
• work out where the data actually lives, and why the workspace storage bucket is emphatically not your data lake;
• build the workload map (ingest → curate → serve → ML) and match each workload to the compute that runs it;
• read the DBU rate card as of Oct 2026 and hand-compute six worked examples with real numbers;
• compare the platform against the AWS-native stack, Snowflake and on-prem Hadoop without repeating vendor marketing as fact;
• study three named AWS case studies, read the 2026 update box, and answer twelve exam-style questions.

---

1. What a lakehouse is: one architecture, three properties

Databricks' current definition (databricks.com/blog/what-is-data-lakehouse-article, 2026-06-23) is the one to memorise:

"A data lakehouse is a new, open data management architecture that combines the flexibility, cost-efficiency, and scale of data lakes with the data management and ACID transactions of data warehouses, enabling business intelligence (BI) and machine learning (ML) on all data."

The canonical 2020 post that named the pattern lists exactly three defining properties (databricks.com/blog/2020/01/30/what-is-a-data-lakehouse.html, 2020-01-30):

1. Transaction support: ACID transactions on object storage, so concurrent writers and failed jobs cannot corrupt a table.
2. Schema enforcement and governance: the storage layer rejects data that violates the declared schema instead of silently accepting it.
3. Openness: the storage formats are "open and standardized, such as Parquet", so engines other than the one that wrote the data can read it.

Read those three properties against the two architectures the lakehouse sits between:

The phrase "enabling BI and ML on all data" is the whole business case: warehouses force a choice between governed SQL and usable ML, lakes force a choice between cheap storage and trustworthy data. The lakehouse removes both choices by putting warehouse-grade table management directly on lake storage.

1.1 The medallion pattern

Databricks documents the standard logical layout as the medallion architecture, "a data design pattern used to logically organize data in a lakehouse ... (from Bronze ⇒ Silver ⇒ Gold layer tables)" (databricks.com/blog/what-is-medallion-architecture, 2022-03-09):

• Bronze: raw, append-only, everything that arrived, with the source metadata intact.
• Silver: cleansed, deduplicated, conformed, one row per real-world entity.
• Gold: business-level aggregates and wide tables ready for BI, dashboards and feature stores.

The layers are logical. Physically they are ordinary Delta tables in ordinary S3 buckets, separated by catalog or schema names; which is why lesson 03 (Unity Catalog) and lesson 04 (Delta transactions) both start from this diagram.

• 📚 Did you know? "Lakehouse" is an architecture, not a Databricks product. Databricks' own 2020 naming post concedes: "Other managed services such as BigQuery and Redshift Spectrum have some of the lakehouse features listed above" (2020-01-30). Snowflake, Amazon Redshift, Microsoft Fabric and others all now claim lakehouse features. The discriminator in this course is never the word; it is the combination of an open table format on your storage plus a governance layer that spans every engine.

---

2. The two pillars: Delta Lake and Unity Catalog

Everything Databricks sells on AWS rests on two open-source projects. Databricks' own 2024 framing pairs them explicitly: "Data reliability ... Databricks provides Delta Lake" and "Eased Governance ... With the help of Unity Catalog" (databricks.com/blog/lakehouse-common-language-across-enterprise, 2024-06-21).

2.1 Pillar one, Delta Lake: the reliability and format layer

Delta Lake is the open-source storage layer that adds ACID transactions, schema enforcement, time travel and scalable metadata management to object storage, and it is the default table format in Databricks: "the foundation of the lakehouse" (databricks.com/blog/what-is-data-lakehouse, 2020-02-11). Physically it is simple: a write-ahead transaction log stored alongside Parquet data files in your S3 bucket (databricks.com/blog/2020/09/10/diving-deep-into-the-inner-workings-of-the-lakehouse-and-delta-lake.html, page updated 2024-06-20).

What that buys you, all covered in lesson 04: serialisable commits, MERGE upserts, change data feed, time travel by version or timestamp, check constraints, and schema evolution under explicit rules.

2.2 Pillar two, Unity Catalog: the governance layer

Unity Catalog models "Every asset you govern ... as a securable object", covering "tables, views, volumes, functions, models, and services", all following "a three-level namespace (catalog.schema.object)" (docs.databricks.com/aws/en/data-governance/unity-catalog, accessed 2026-10). Three facts shape every AWS design you will ever build:

• The metastore is the top-level securable and is scoped to one cloud region: "The metastore is the top-level securable object ... A metastore contains all securable objects registered in Unity Catalog in a single cloud region" (docs .../unity-catalog/securable-objects).
• You need "one metastore for each region in which your organization operates", and "users must be on a workspace that is attached to a metastore in their region" (docs .../unity-catalog/create-metastore).
• Grants travel across workspaces: "a privilege granted in one workspace is effective in all other workspaces sharing that metastore" (same source).
• On AWS, a storage credential is an IAM role; that is the seam where governance meets your cloud account (docs.../securable-objects).

Tables and volumes are either managed (Unity Catalog owns both governance and the file lifecycle) or external (Unity Catalog governs; you own the files) (docs .../unity-catalog).

2.3 Why the pairing matters on AWS

Delta answers "can I trust these bytes?" without moving them off S3. Unity Catalog answers "who may read which bytes?" with one grant model that survives workspace moves, region scoping and multiple AWS accounts. Neither replaces the other: a perfectly ACID table with no grants is an ungoverned leak, and a perfectly governed catalog over sloppy files is an audit waiting to fail.

---

3. Workspace anatomy: the control plane and the compute plane

"Databricks operates out of a control plane and a compute plane" (docs.databricks.com/aws/en/getting-started/high-level-architecture, page last updated 2026-09-11). This is the single most testable sentence in lesson 01, and the one most often mangled by engineers who have only read Databricks' older engineering blogs.

3.1 The three planes

• Control plane (Databricks' account). "The control plane includes the backend services that Databricks manages in your Databricks account. The control plane is located in the Databricks account, not your cloud account" (same page, 2026-09-11). It holds the web UI, notebook commands and metadata, cluster manager, job orchestration and Unity Catalog metadata: encrypted at rest (databricks.com/blog/best-practices-and-guidance-cloud-engineers-deploy-databricks-aws-part-1, 2022-09-30).
• Classic compute plane (your account). Clusters and pro/classic SQL warehouses run as EC2 instances "within each workspace's virtual network in the customer's AWS account", which gives the plane "natural isolation because it runs in each customer's own AWS account" (docs .../high-level-architecture, 2026-09-11).
• Serverless compute plane (Databricks' account). "Databricks creates a serverless compute plane in the same AWS region as your workspace's classic compute plane" and isolates it "within a network boundary for the workspace" (same page). Same region as your classic plane: not an arbitrary Databricks region.

Older material, including Databricks' own 2022 to 2023 deployment blogs, says data plane where today's docs say compute plane. Same concept, current vocabulary. If a tutorial says "data plane", it means the classic/serverless compute plane described here.

3.2 Secure cluster connectivity: no open ports, no public IPs

"All new workspaces are created with secure cluster connectivity by default. Secure cluster connectivity means that customer VPCs have no open ports and classic compute plane resources have no public IP addresses" (docs.databricks.com/aws/en/security/network/classic, updated 2026-09-25). Clusters reach the control plane through a secure tunnel over HTTPS port 443 to an SCC relay, and traffic is always initiated from the compute plane outward (databricks.com/blog/...part-1, 2022-09-30). There is no inbound path for Databricks to open, because there is no inbound path at all.

The consequence students miss: SCC is not "no egress". Clusters still need an outbound route, NAT gateway, internet gateway or PrivateLink, to reach the control plane and AWS APIs, and an S3 gateway endpoint keeps S3 traffic on the AWS backbone so it never bills through the NAT (databricks.com/blog/optimizing-aws-s3-access-databricks, 2022-08-11). Lesson 02 sizes this properly.

3.3 The cross-account IAM role: how Databricks may launch your EC2

Workspace creation needs three configurations, credential (a cross-account IAM role), network (VPC/subnets/security groups) and storage (the workspace storage bucket), and can be driven from the account console, the Account API or Terraform (docs.databricks.com/aws/en/admin/workspace, 2026-07-28). The credential is a role you create in your account whose trust policy names Databricks' AWS account: "Create a new cross-account IAM role with an access policy... In the Account ID field, enter the Databricks account ID 414351767826" (docs.../create-uc-workspace; docs.../administration-guide/account-api/iam-role.html, 2026-01-24), with your Databricks account ID as the external ID. Prerequisites on that page: "Have an available VPC and NAT gateway in your AWS account in the workspace's region."

---

4. Where the data actually lives (and where it does not)

A classic workspace has three storage surfaces in your AWS account, and only one of them is the lake:

Two distinctions from the official docs prevent the classic mistakes:

1. Workspace system data vs DBFS root. The docs separate them explicitly: "DBFS root is a user-accessible file system, while workspace system data is used internally by Databricks features": even when both sit in the same bucket. DBFS root and DBFS mounts are legacy and not recommended for new designs (docs.../high-level-architecture, 2026-09-11).
2. Managed vs external tables. A managed Delta table's files live where Unity Catalog put them (still your S3, under a path UC manages); an external table's files live at a prefix you already own, and UC governs only the metadata. Both are "your data": neither is in the workspace storage bucket.

Serverless workspaces change the workspace storage story (workspace data goes to Databricks-managed default storage), but they do not change the lake story: analytical tables still live in your S3 buckets, reached through Unity Catalog. The data-plane question and the storage question are separate questions.

• 📚 Did you know? The terminology has drifted twice. Databricks' 2023 deployment blog says S3 root bucket; current docs say workspace storage bucket; and the "data plane" of the same blogs is now the compute plane. When you read an older tutorial or an exam-prep site and it uses the older words, translate: root bucket → workspace storage bucket, data plane → classic/serverless compute plane. Teaching materials that still present "data plane" as current Databricks vocabulary are quoting 2022 to 2023 engineering posts.

---

5. The workload map: ingest → curate → serve → ML

The reference flow for this course, assembled from the Auto Loader, medallion, Databricks SQL and MLflow docs:

1. Sources: Kafka/Amazon MSK, Kinesis, JDBC, SaaS APIs and files land raw payloads in an S3 landing bucket.
2. Ingest: Auto Loader (cloudFiles) incrementally discovers new files and writes bronze Delta tables. It "incrementally and efficiently processes new data files as they arrive in cloud storage" and its key-value state store "ensures that data is processed exactly once" (docs.../auto-loader, page dated 2026-04-01).
3. Curate: Lakeflow Spark Declarative Pipelines (the renamed Delta Live Tables) or ordinary jobs cleanse, dedupe and conform to silver, then aggregate to gold.
4. Serve: Databricks SQL on a serverless/pro/classic warehouse powers dashboards, the SQL editor, alerts and external BI over JDBC/ODBC. "Databricks SQL is a cloud data warehouse built on lakehouse architecture. It runs directly on your data lake, supports ANSI SQL with Delta Lake extensions" (docs.../aws/en/sql/, updated 2026-09-11).
5. Learn: MLflow tracks experiments, and the Model Registry in Unity Catalog provides "centralized access control, auditing, lineage, and model discovery across workspaces" (docs.../manage-model-lifecycle).
6. Govern: Unity Catalog covers every layer, with lineage and audit available from system tables.

5.1 Workload → compute mapping

This table is the practical core of the lesson; every later cost decision starts here.

• 📚 Did you know? The same PySpark code can cost materially less as a job than as an interactive cluster. Databricks' cost-management guidance states the mechanism directly: "Better performance along with the lower cost rates per DBU that job clusters offer lead to dramatic cost savings" (databricks.com/blog/best-practices-cost-management-databricks, 2022-10-18). Job clusters are ephemeral and single-purpose; all-purpose clusters stay warm for humans. The exact rate gap between SKUs varies by region and plan tier: Databricks publishes only "starting at" figures on its public pricing pages (accessed 2026-10), so always re-check your own contract rather than memorising a grid from a third-party site.

---

6. The two meters: DBU rates and the AWS bill

"A Databricks Unit (DBU) is a normalized unit of processing power on the Databricks Lakehouse Platform used for measurement and pricing purposes" (databricks.com/product/pricing, accessed 2026-10), and the glossary puts it as "units of processing capability per hour based on VM instance type" (docs .../resources/glossary, updated 2026-08-15). You pay DBUs for processing; AWS bills you separately for the metal and the plumbing.

6.1 The rate card, as of October 2026

List prices, USD, accessed 2026-10; they vary by cloud, region and plan tier (AWS: Standard/Premium/Enterprise) and by contract. Databricks publishes "starting at" figures for jobs and interactive SKUs: treat any full grid you find on a third-party cost site as unverified.

6.2 Worked example E1: a four-node job cluster, both meters

As of Oct 2026. A nightly job runs on a 4-node cluster whose instance type is billed at 3 DBU/hour on the jobs SKU.

Reading: the DBU side of an 8-hour nightly job is $432/month at the published "starting at" jobs rate; the AWS side is a second invoice you must add yourself. Same code, different SKU: if that workload were driven interactively at the "starting at" interactive rate of $0.40/DBU, the identical 2,880 DBU would bill $1,152: a 2.67× difference for compute that does the same arithmetic. That gap is why production workloads belong on job clusters and BI belongs on warehouses.

6.3 Worked example E2: a serverless warehouse month, and the idle tax

A team runs one SQL serverless warehouse 8 hours/day, 20 days/month, averaging 4 DBU/hour (serverless auto-scales within the hour; 4 DBU/h is a plausible small-team average: adjust to your own system.billing.usage numbers when you get there).

Reading: Databricks' own doc warns that "Idle SQL warehouses continue to accumulate DBU and cloud instance charges until they are stopped" (docs .../compute/sql-warehouse/create). Auto-stop defaults as of Oct 2026: pro/classic 45 minutes (minimum 10), serverless 10 minutes (UI minimum 5; the API can set 1 minute). Turning auto-stop from 45 minutes down to 10 on a pro warehouse used in 30-minute sessions is a free saving; leaving anything on 24/7 is the single largest avoidable line item in a BI estate.

6.4 Worked example E3: the storage bill under the same tables

The lake's own storage, for scale: 1 TB (1,024 GB) held one month in S3 Standard at $0.023/GB-month (aws.amazon.com/s3/pricing, accessed 2026-10) = $23.55/month, plus PUT/GET requests (S3 Standard GET is $0.0004 per 1,000 requests). A 10 TiB curated gold estate therefore costs about $235/month to hold: roughly half what one idle serverless warehouse bills in the same window (E2). Storage is the cheap half of a lakehouse; compute discipline is where the money is.

---

7. The same query, three meters: Databricks versus the AWS-native stack

The AWS-native alternative is a composition, not a product: S3 for storage, Glue Data Catalog for metadata/crawlers/ETL, Athena for serverless SQL, Redshift for a provisioned or serverless warehouse, Lake Formation for grants, QuickSight for BI. Prices, all accessed 2026-10:

7.1 Worked example E4: Athena versus Databricks SQL for a dashboard workload

A dashboard scans 20 GB/day for 22 days/month.

Reading: for a light, scan-heavy BI dashboard, Athena's per-TB meter is dramatically cheaper: ~$2.15 vs ~$168. That is an honest result, not a Databricks failure: per-byte pricing punishes repeated heavy scans, ETL, joins, MERGE, streaming and ML, and rewards rare, wide, ad-hoc reads. The design question is never "which engine is cheaper" but "which meter matches the workload shape". This is exactly why lesson 15 teaches cost attribution through system.billing.usage instead of vendor benchmarks.

7.2 Worked example E5: sizing the landing zone

A pipeline lands 500 GB/day of raw files into S3 Standard.

Reading: $345/month to hold a month of landing data. Two levers dominate: retention (clean source files after Auto Loader commits; the docs' cleanSource options exist for exactly this) and file size (thousands of tiny files inflate both request charges and Delta planning). Storage is predictable; the request and compute sides are where designs quietly fail.

7.3 The two stacks, side by side

• 📚 Did you know? AWS itself documents Delta interoperability rather than treating it as a competitor: Athena "can read Delta Lake tables stored in Amazon S3 directly without having to generate manifest files or run the MSCK REPAIR statement" (docs.aws.amazon.com/athena/latest/ug/delta-lake-tables.html, accessed 2026-10): with the honest limits of read-only, engine v3 and no time travel. Glue, EMR and Redshift Spectrum can also process Delta tables on S3 (databricks.com/blog/databricks-aws-guide-data-ai-summit-2023-featuring-labcorp-conde-nast-grammarly-vizio-ntt-data, 2023-06-23). Open formats cut both ways: your lake is portable, and your competitors' engines can read it.

---

8. The AWS substrate you inherit (worked sizing)

Lesson 02 goes deep; here is the frame. A standard deployment's core requirements are "a customer-managed VPC, data plane to control plane connectivity, cross-account IAM role, and S3 root bucket" (databricks.com/blog/...part-2, 2023-01-27), and "For each workspace there must be a minimum of two dedicated private subnets that each have a netmask of between /17 and /26 and are in separate availability zones" (same source). The well-architected guidance adds: "Create a VPC with minimum /18 CIDR block for workspace deployments" (docs.databricks.com/aws/en/lakehouse-architecture/security-compliance-and-privacy/best-practices, 2026-07-16).

8.1 Worked example E6: subnet sizing for one workspace

Reading: two /24s in separate AZs satisfy the published rule with room to spare. The constraint that actually bites is "dedicated per workspace": subnets are not pooled across workspaces, so each workspace you create consumes its own two subnets from your VPC CIDR. Plan the VPC as /18 (per the well-architected guidance) and you can host several workspaces before you touch routing.

Three Databricks network reference architectures exist as of Oct 2026, Managed security, Hardened connectivity and Isolated environment, differing in PrivateLink depth, IP access lists, serverless egress controls and inbound PrivateLink, while secure cluster connectivity plus a customer-managed VPC is present in all three (docs.databricks.com/aws/en/security/network/deployment-architecture, accessed 2026-10). Also note the plan gates: customer-managed VPC, UC metastore creation and serverless SQL all require the Premium plan or above (docs.../permissions, updated 2026-09-11; docs.../create-metastore; docs.../admin/sql/serverless).

---

9. Choosing the platform: Snowflake, and the Hadoop you are leaving

9.1 Versus Snowflake

Neutral framing, because both vendors publish comparison pages that contradict each other:

Both pages make opposing performance and price/performance claims. This course treats every one of them as a vendor claim with a date, never as a fact, and never quotes a head-to-head multiplier as something a student should remember. Choose Snowflake when the workload is pure SQL/BI with predictable concurrency and you want warehouse simplicity; choose Databricks when ML/AI, streaming, Python/Spark and open-format portability are first-class requirements; increasingly, co-exist: federation and Delta Sharing mean the two are not mutually exclusive (see lessons 12 and 17).

9.2 Versus on-prem Hadoop (HDFS + YARN + Hive)

Databricks' migration guidance is direct: "Even if you migrate legacy on-premises Hadoop workloads to cloud-based managed services like EMR ... you'll still run into the same reliability and scalability issues" (databricks.com/solutions/migration/hadoop, 2023-09-03). The mapping is mechanical:

Skills and BI investments carry over precisely because the formats stay open: the SQL does not have to be rewritten to move from Hive-style tables to Delta-backed gold tables, and lesson 03 covers that migration path (hive_metastore appears as a legacy top-level catalog).

9.3 The decision, drawn

The diagram encodes three rules of thumb from this lesson: open-format portability and mixed workloads point at the lakehouse; SQL-only plus one-vendor preference points at AWS-native; and a Hadoop estate is not "lifted and shifted"; it is redesigned (which is why lesson 03 treats hive_metastore migration as a first-class path).

---

10. Why openness is a design requirement, not a slogan

Three concrete interoperability facts, each official and each dated:

1. Your data is Parquet plus a Delta transaction log in your S3 bucket: any engine that understands the format can read it without exporting from a proprietary store (databricks.com/blog/2020/01/30/what-is-a-data-lakehouse.html).
2. AWS-side readers: Athena reads Delta tables registered in the AWS Glue catalog (read-only, engine v3, no time travel: docs.aws.amazon.com/athena/latest/ug/delta-lake-tables.html, accessed 2026-10).
3. Third-party readers via Unity Catalog: "Through open APIs and credential vending, Unity Catalog enables external engines such as Trino, DuckDB, Apache Spark, Daft, and Iceberg REST catalog-integrated engines, such as Dremio, to access managed tables" (docs .../tables/managed, 2026-08-18), through the Unity REST API and the Iceberg REST Catalog (IRC) (docs .../external-access/unity-rest, 2026-08-26).

The strategic point is Databricks' own: a proprietary lakehouse stores data "in formats only one vendor can read", whereas open lakehouse data lives in "Delta Lake and Apache Iceberg ... on open file formats like Apache Parquet" (databricks.com/blog/what-open-lakehouse-open-data-standards-explained, 2026-06-16). On AWS that matters in a second way: you keep the data in your account, so an engine change, a vendor dispute or an acquisition does not become a data-extraction project.

• 📚 Did you know? Sharing is also open: Delta Sharing is an open protocol (covered in lesson 12), while Databricks Marketplace is the commercial catalogue built on top of it. Because the underlying files are open formats, a share can be consumed by engines that are not Databricks; which is the practical answer to "what happens to my lake if we ever leave?".

---

11. The traps: eleven confusions this course will not let you keep

1. "Data plane" vs "compute plane." Same thing; docs since 2026 say compute plane, split into classic and serverless. Older blogs say data plane.
2. Control plane ≠ your account. It is in the Databricks AWS account, but the workspace storage bucket in a classic workspace is in your account. Confusing the two produces a wrong threat model.
3. Workspace storage bucket ≠ data lake. System data only; undeletable; never shared across workspaces; never used for customer data.
4. DBFS root is legacy. It may sit in the same bucket as workspace system data, but "DBFS root is a user-accessible file system, while workspace system data is used internally by Databricks features" (docs.../high-level-architecture, 2026-09-11), and DBFS root/mounts are not recommended for new designs.
5. Renames are real; verify before teaching or answering. Delta Live Tables → Lakeflow Spark Declarative Pipelines (docs .../ldp/concepts/where-is-dlt, 2026-01-23; release note 2025-11-12) with no code migration required and classic SKUs still beginning DLT; SQL endpoints → SQL warehouses (2023); workspace Model Registry → Models in Unity Catalog.
6. Two bills. DBUs plus your AWS invoice; serverless rates bundle instances, classic rates do not.
7. Jobs vs all-purpose SKU. The same code can cost materially less as a job, official guidance says so (2022-10-18), but the exact gap is region/contract dependent, so re-check rather than memorise.
8. Athena reading Delta ≠ full Delta support. Glue-catalog only, engine v3, read-only, no time travel.
9. Metastore scope is regional, not per workspace. One metastore per region; hive_metastore is a workspace-level legacy catalog.
10. Serverless is conditional. Premium plan required, and serverless compute runs in the Databricks account in the same AWS region as your classic plane.
11. Vendor benchmarks are marketing in both directions. Databricks-vs-Snowflake and Snowflake-vs-Databricks pages make mutually contradictory claims; cite them only as "vendor claim, date-stamped".

2026 Updates (as of October 2026)

---

Real-World Case Studies

📋 Case A: Biogen (pharmaceuticals, Cloud: AWS) Challenge. On-premises informatics could not scale to petabytes of genomics: "a pipeline that previously required 2 weeks to process 700,000 variants". Stack on AWS. Databricks for Genomics runtime with Delta Lake, alongside the DNAnexus platform on AWS: the customer story page carries Databricks' own Cloud: AWS tag. Reported outcome. "2 million Genomic variants analyzed in 15 minutes · 2 drug targets Discovered" (https://www.databricks.com/customers/biogen, undated page, accessed 2026-10; story origin 2020-11). Maps to this lesson. L01: migration to a cloud lakehouse measured in weeks of compute removed, not only dollars: the lakehouse concept and the open-format pillar made an on-prem estate portable. ⏳ Temporal note: a 2020-origin figure; treat it as what Biogen reported then, not a 2026 platform guarantee.

📋 Case B: Punchh / retail loyalty (Cloud: AWS) Challenge. Outgrown AWS MySQL, a warehouse too costly, tools without ACID, and Tableau reports paying a 12-table join on every refresh. Stack on AWS. Databricks SQL over Delta Lake bronze/silver/gold, notebooks, Tableau on top: Cloud: AWS on the story page. Reported outcome. "10x Faster time-to-insight · 30% Lower operational costs · 12% Increase in customer satisfaction" (https://www.databricks.com/customers/punchh, undated page, accessed 2026-10; story origin 2021-09). Quote. "We are able to store raw data as well as refined data in the same lakehouse.": Jagan Mangalampalli, Punchh (same URL). Maps to this lesson. L01 secondary: the medallion layout of section 5 as a cost control: pre-joined gold tables beat 12-way joins at report time. ⏳ Temporal note: reported in 2021; the ratios are Punchh's, not a platform SLA.

📋 Case C: Rivian (electric vehicles, Cloud: AWS) Challenge. Terabytes of vehicle data per day from 70,000+ vehicles, with legacy tooling that "struggled to scale pipelines cost-effectively" and siloed teams blocking predictive-maintenance ML. Stack on AWS. Databricks SQL, Delta Lake, Unity Catalog and liquid clustering: Cloud: AWS on the story page; the 100-use-case roundup names "Unity Catalog for data governance, Delta Tables for storage, and Liquid Clustering for data layout" for real-time charge-prediction ML. Reported outcome. "1,000+ platform users, a 75% increase from a year ago" (https://www.databricks.com/customers/rivian, undated page, accessed 2026-10); "50% better runtime performance" (https://www.databricks.com/your-ai/customers/rivian, 2024-05-30); stack detail from https://www.databricks.com/blog/data-intelligence-action-100-data-and-ai-use-cases-databricks-customers (2025-07-09). Maps to this lesson. L01: one governed AWS lakehouse feeding dashboards and production ML from the same tables, which is the "single copy of data" claim of section 1 made concrete. ⏳ Temporal note: metrics are from 2024 to 2025 pages; date-stamp them when you quote them.

Comparative Verdict: lakehouse vs AWS-native vs Snowflake vs on-prem Hadoop

• Versus a pure data lake (S3 + Parquet, no table layer): you keep the cost and elasticity but lose ACID, schema enforcement and time travel. If nothing ever fails, overwrites or runs concurrently, a raw lake is cheaper; in production, Delta's transaction log is the cheapest insurance in the stack.
• Versus the AWS-native stack (S3 + Glue + Athena + Redshift + Lake Formation): AWS-native wins for SQL-only, mostly-batch, AWS-standardised shops that want one vendor and per-usage meters ($5.00/TB Athena, $0.44/DPU-hour Glue). Databricks wins when you need one governed copy for SQL and Python/Spark and streaming and ML: at the price of a second vendor, a DBU meter and a Premium plan for the full feature set. Both sit on the same S3, and AWS documents Delta interop rather than walling it off.
• Versus Snowflake: Snowflake wins on pure SQL/BI simplicity and zero VPC work; Databricks wins on ML/AI, streaming and open-format portability. Publish no head-to-head multiplier in a design doc: both vendors' comparison pages are date-stamped marketing that contradicts the other side's.
• Versus on-prem Hadoop (HDFS + YARN + Hive): not a close contest in 2026, S3 replaces HDFS, elastic/serverless compute replaces YARN, and Unity Catalog replaces the Hive metastore, while open formats keep your SQL and BI assets. The honest caveat: "Even if you migrate legacy on-premises Hadoop workloads to cloud-based managed services like EMR... you'll still run into the same reliability and scalability issues" (2023-09-03), the architecture, not the hosting, is what you are changing.

---

Practice Questions