Lakehouse Architecture Fundamentals: Lakes, Warehouses, and the Medallion Pattern

AWS defines the term this way: "A data lakehouse is a unified data architecture that combines data warehouses and data lakes, providing analytics capabilities such as structuring, governance, and reporting. The data lakehouse can store raw data in a range of formats from thousands to hundreds of thousands of sources in a central location" (AWS What is a Data Lakehouse?, accessed 2026-10-11). The same page adds the sentence that governs every design decision in this course: "A data lakehouse's compute is separate from storage. With separate storage and compute, you can scale these functions independently to maximize cost savings."

That is a definition, not yet an architecture. Between the sentence and a running platform sit three questions this lesson answers: which store you are actually building (lake, warehouse, or lakehouse), how data is layered inside it (the medallion pattern), and how it is processed (batch, Lambda, or Kappa): plus the map of AWS services that implements each position. Everything from lesson 02 to lesson 17 is an implementation detail of the skeleton you build here.

Most teams meet these three words in the wrong order: they pick services first, then discover they have neither a single source of truth nor a cheap archive. We will do it the other way round: pattern first, services second.

In this lesson you will:

• separate a data lake, a data warehouse and a lakehouse using AWS's published definitions, then apply a decision table to concrete requirements;
• read the six AWS-published differences (schema timing, data quality, users, analytics fit, price/performance, cost posture) that decide the store before any service is named;
• understand why Parquet on Amazon S3 is still not a lakehouse, and which capabilities an open table format adds;
• assign responsibilities to the bronze, silver and gold medallion layers and map each one onto real AWS primitives (buckets, prefixes, Glue databases, Iceberg tables);
• choose between batch-only, Lambda and Kappa processing patterns, including the one naming trap AWS's own whitepaper warns about;
• place ten core analytics services, S3, Glue, Athena, Redshift, QuickSight, Kinesis, MSK, EMR, Lake Formation, Macie, on a single layer map;
• walk the end-to-end reference architecture twice: as a flowchart and as a sequence diagram, plus the multi-account container view;
• work seven priced worked examples with real list prices (as of Oct 2026), including a 10 TB/day pipeline sizing sketch;
• study three AWS customer cases, FINRA, Easygo and BMW Group, and read their numbers the way a practitioner should;
• check the October 2026 update box (S3 Tables, S3 Vectors, Glue 6.0, Quick Suite) and the course roadmap for the sixteen lessons that follow.

---

1. Lake, warehouse, lakehouse: the three definitions AWS actually publishes

Three terms, three different promises. The fastest way to misuse them is to treat them as sizes ("we are too big for a warehouse") rather than as contracts about schema, curation and access.

1.1 The data warehouse

AWS's compare page: "A data warehouse stores data in a structured format. It is a central repository of preprocessed data for analytics and business intelligence." The data-lake concepts page adds the mechanism: "A data warehouse is a database optimized to analyze relational data coming from transactional systems and line of business applications. The data structure, and schema are defined in advance… Data is cleaned, enriched, and transformed so it can act as the 'single source of truth' that users can trust."

The warehouse's defining move is schema on write: the shape and the quality bar are settled before the data lands, which is why the same page can assign it "Batch reporting, BI, and visualizations" and "Highly curated data that serves as the central version of the truth" (AWS compare page, accessed 2026-10-11). On AWS the managed expression of this pattern is Amazon Redshift (RA3 provisioned and Serverless, with Spectrum for lake tables).

1.2 The data lake

AWS: "A data lake is a centralized repository that allows you to store all your structured and unstructured data at any scale" (AWS What is a Data Lake?, page footer "Updated 2025", accessed 2026-10-11). The compare page supplies the counterpart to schema-on-write: schema is "Written at the time of analysis (schema-on-read)", data quality is "Any data that may or may not be curated", and the analytics list is "Machine learning, exploratory analytics, data discovery, streaming, operational analytics, big data, and profiling".

Loading discipline follows from that: "Organizations typically use Extract, Load, Transform (ELT) tools. They load the data in the lake first and transform it only when required." The cost argument is explicit: "data lakes offer a comparatively lower cost for more volume, especially for large numbers of images and videos." The failure mode is equally explicit, and you will meet it in every post-mortem: "an unmanaged data lake can lead to data swamps. Data swamps are a state of disorganized data that makes it harder to extract meaningful insights" (AWS What is a Data Lakehouse?, accessed 2026-10-11).

1.3 The lakehouse

The lakehouse is the combination, and AWS is careful to call it an architecture rather than a product. The legacy AWS whitepaper phrases it as a pattern: "Lake house: A lake house is an architectural pattern that combines the best elements of data warehouses and data lakes. Lake houses enable you to query data across your data warehouse, data lake, and operational databases… With a lake house architecture, you can store data in open file formats in your data lake and query it in place while joining with data warehouse data" (AWS Data Warehousing on AWS, page banner: "This whitepaper is for historical reference only", accessed 2026-10-11).

AWS also publishes the logical layers of the pattern, which become the spine of this whole course:

1.4 The decision table

This is the table to keep open during every design review. Every row comes from the AWS compare page's warehouse-versus-lake comparison plus the lakehouse definition page (all accessed 2026-10-11).

How to use the table. Start at the bottom row and read upward. If the requirement sentence contains single source of truth and tight BI SLAs, you are in column 1. If it contains images, videos, model training, unknown schema next year, you are in column 2. If it contains both, and I refuse to copy the data twice, you are in column 3, and the SageMaker Lakehouse FAQ states the economic reason plainly: "you don't have to migrate your data… access and query your data in-place" (accessed 2026-10-11).

Note that AWS does not present this as an either/or contest: "Most large organizations use a combination of data lakes, warehouses, and marts in their storage infrastructure" (AWS compare page, accessed 2026-10-11). The lakehouse is the architecture that lets those pieces share one governed copy rather than three.

• 📚 Did you know? AWS's compare page treats lake, warehouse and mart as complementary inventory, not rivals: "Most large organizations use a combination of data lakes, warehouses, and marts in their storage infrastructure" (accessed 2026-10-11). The trap runs both ways: any design doc claiming you must pick exactly one of the three is arguing with AWS's own page.

---

2. Why Parquet on Amazon S3 is still not a lakehouse

This is the single most expensive confusion in the field, so it earns its own section. A bucket full of well-partitioned, well-compressed Parquet files is a data lake. It becomes a lakehouse only when a transactional metadata layer sits on top of it.

AWS Prescriptive Guidance is unusually blunt about what plain lakes lack. Traditional/Hive-format object lakes "don't provide support for atomicity, consistency, isolation, and durability (ACID) processing semantics", and they cannot natively:

• "Performing efficient record-level updates and deletions as data changes in your business";
• "Managing query performance as tables grow to millions of files and hundreds of thousands of partitions";
• "Ensuring data consistency across multiple concurrent writers and readers";
• "Preventing data corruption when write operations fail partway through the operation";
• "Evolving table schemas over time without (partially) rewriting datasets".

(Using Apache Iceberg on AWS, Modern data lakes, accessed 2026-10-11.) The same source notes these gaps are "particularly prevalent" for change-data-capture, privacy/deletion and streaming ingestion, three workloads this course covers in lessons 05, 06 and 13.

The open table format answers with a feature list AWS publishes verbatim:

• "Delete, update, and merge"
• "Fast scan planning and advanced filtering"
• "Full schema evolution"
• "Partition evolution"
• "Hidden partitioning"
• "Version rollback"
• "Time travel"
• "Serializable isolation"
• "Concurrent writers"
• "Open file formats" (Parquet, Avro, ORC)

Two supporting facts you will reuse all course: AWS lists Apache Iceberg as "supported by AWS services such as Amazon EMR, Amazon Athena, Amazon Redshift, AWS Glue, and Amazon SageMaker", and "Athena… uses the AWS Glue Data Catalog for the Iceberg metastore" (Prescriptive Guidance, accessed 2026-10-11). The catalog and the table format are different components that must be designed together: a distinction lesson 07 makes operational.

---

3. The medallion pattern: bronze, silver, gold

Once the store decision is made, the next question is what shape does data take as it moves from raw to trusted? The AWS answer is the medallion architecture, defined in AWS's own Builder Center workshop as "a data design pattern that promotes data quality, consistency, and governance through a layered approach" (accessed 2026-10-11).

3.1 The three layers and their responsibilities

Three properties of this table are worth memorising. First, the layers are progressive: each hop raises the guarantee and narrows the audience. Second, the layer is a logical boundary, not an AWS product: bronze/silver/gold map to S3 prefixes or buckets, Glue Data Catalog databases and Iceberg tables; AWS presents the pattern in a workshop and a blog, not as a managed service. Third, the quarantine path is part of silver, not an afterthought: a record that fails a rule must land somewhere observable rather than silently disappearing.

3.2 Medallion on AWS, end to end

3.3 What each layer is not

• Bronze is not a dump-and-forget zone. It still needs retention rules, encryption and a catalog entry; Well-Architected Data Analytics Lens best practice BP 3.7 is "Implement data retention policies for each class of data in the analytics workload", and BP 3.5 is "Record data classifications into the Data Catalog so that analytics workloads can understand" (AWS Well-Architected Data Analytics Lens, publication date 2023-12-22, announced as a lens on 26 March 2024; accessed 2026-10-11).
• Silver is not optional if you want a lakehouse rather than a swamp: it is where "an unmanaged data lake can lead to data swamps" gets prevented, one validated table at a time. Lens BP 1.1 is "Validate the data quality of source systems before transferring data for analytics", and BP 1.2 is "Monitor operational metrics of data processing jobs and the availability of source data".
• Gold is not "the reporting extract". It is still an open table that Athena, Redshift, SageMaker and third-party engines can query in place; that is what stops it from becoming a proprietary silo two years later.

AWS's prescriptive starting point for a nightly batch hop, if you want numbers instead of opinions: "File size: 128 MB, because Apache Spark tasks process data in 128 MB chunks"; "Write type: copy-on-write"; "Partition variables: year/month/day"; "Sort order: timestamp"; "Data catalog: AWS Glue Data Catalog" (AWS Prescriptive Guidance, reference architectures for Apache Iceberg on AWS, accessed 2026-10-11).

Worked example E1: how many files does one bronze day actually create? (as of Oct 2026)

A source delivers 10 TB per day into bronze, and you follow the Prescriptive Guidance target file size of 128 MB so that Spark tasks map one-to-one onto files.

• 1 TB = 1,024 GB = 1,048,576 MB, so 10 TB = 10,485,760 MB
• 10,485,760 MB ÷ 128 MB = 81,920 files per day
• With year/month/day partitioning (the AWS recommendation), a single partition day therefore receives about 82,000 objects; which is exactly why "tables grow to millions of files" appears in AWS's list of plain-lake problems, and why compaction (lesson 04) and partition planning (lesson 03) are not optional extras
• One month of bronze at this rate: 81,920 × 30 = 2,457,600 files. Manage that number with Iceberg metadata, not with ls on a bucket

Takeaway: medallion layers are cheap to describe and expensive to skip. The bronze→silver hop is where data quality lives, the silver→gold hop is where business meaning lives, and the catalog plus Lake Formation band is what makes both readable by the right people.

• 📚 Did you know? AWS publishes a three-statement version of this pattern: with Apache Iceberg materialized views in Amazon SageMaker, "you can build a Bronze, Silver, and Gold medallion architecture as three SQL statements", and when you configure scheduled refresh, "the AWS Glue Data Catalog uses managed Spark compute to incrementally update the materialized view" (AWS Big Data Blog, accessed 2026-10-11). The pattern is portable: the mechanism (who runs the refresh, on what compute) is what you are actually choosing in lessons 08 and 11.

---

4. Batch, Lambda, or Kappa: choosing the processing pattern

The medallion layers tell you what shape data takes. The processing pattern tells you how many pipelines you must build, test and operate. There are three documented answers, and AWS publishes all three.

4.1 Batch-only

Land files or streams to S3, process on a schedule (Glue or EMR), serve with Athena or Redshift. The Lambda whitepaper's "batch layer" produces "batch views" that are "stored in a persistent storage layer". Simple, cheapest to run, and latency equals your schedule: a nightly job means dashboards are at best one night stale, and every late-arriving file silently becomes next week's incident.

4.2 Lambda architecture (the pattern)

The AWS whitepaper defines it as "a data-processing design pattern to handle massive quantities of data and integrate batch and real-time processing within a single framework." Two paths run in parallel:

• a batch layer computing authoritative batch views: exact, complete, recomputed over the full history;
• a speed layer producing incremental views, "Kinesis Data Firehose, As data is ingested in real-time, customers can use Kinesis Data Firehose to easily batch and compress the data to generate incremental views";
• a serving layer where "the merged query is aimed at joining and analyzing the data from both the batch view from the batch layer and the incremental stream view from the speed layer."

The operational cost of this correctness is that you now own two codepaths that must agree, plus a merge step that must reconcile late and out-of-order data.

4.3 Kappa architecture

One pipeline, one codebase, reprocessing by replay. AWS's Builder Center walkthrough (18 May 2026, accessed 2026-10-11): "It's Kappa: one unified streaming pipeline that handles everything. On AWS, that means Kinesis Data Streams, Spark Structured Streaming on EMR…": with a table format on S3 as the durable layer and a "Replay window of 24 hours: Kinesis retains events" as presented in that article.

Kappa trades the batch layer's exactness for operational simplicity: correctness comes from being able to replay the same code over the same log, which is only possible inside the stream's retention window: everything older must come from the lake (see §8.2).

4.4 The choice, in one table

Worked example E2: one requirement, three architectures, priced (as of Oct 2026)

Requirement: e-commerce clickstream + orders + nightly ERP dumps; leadership wants dashboards within 15 minutes of data landing, plus a data-science sandbox. Two hundred dashboard readers.

• Shared serving cost: 200 readers × $3 per user/month = $600.00 per month (QuickSight Reader, pricing page accessed 2026-10-11).
• Shared batch transform cost: one nightly bronze→silver job at 10 DPU × 20 minutes = 3.333 DPU-hours × $0.44 = $1.47 per night, i.e. $44.00 per month.
• Shared query cost: if the BI layer scans 1 TB per day in aggregate, that is 30 TB/month × $5.00 per TB = $150.00 per month on Athena.
• (a) Batch-only: those three lines only, about $794.00 per month of directly attributable serverless spend, but dashboard freshness equals job cadence, so a 15-minute SLA would force 96 runs per day instead of one, multiplying the Glue line by ~96 and still failing to model live events.
• (b) Lambda: the same batch path plus a Kinesis → streaming-job speed path writing incremental views into the same gold tables. Freshness drops to seconds for recent events; you now own two codepaths plus the merge. The streaming path adds Kinesis Data Streams and streaming-compute charges: budget those separately, because this lesson only anchors prices AWS publishes on the Glue, Athena and QuickSight pages.
• (c) Kappa: one Kinesis → Spark Structured Streaming path into the table format on S3, with backfills replayed from the stream inside the presented 24-hour replay window and from the lake beyond it.

Verdict for this scenario: the 15-minute SLA eliminates batch-only at any reasonable run count, and a team of five cannot comfortably own two production codepaths, so Kappa, with the lake (not the stream) as the system of record for history. That is a judgement supported by the table in §4.4, not a universal rule: if a regulatory requirement demands exact batch recomputation of every historical view, Lambda remains the correct pattern.

• 📚 Did you know? The Lambda whitepaper closes on a fact that explains why this whole course can be serverless: "Amazon Glue, Amazon S3, Amazon Athena and Amazon Kinesis are serverless services." You can run the batch, speed and serving layers of any of the three patterns without provisioning a cluster; which is why the cost lines in E2 are quoted per-hour and per-TB rather than per-instance (AWS Lambda Architecture whitepaper, accessed 2026-10-11).

---

5. The AWS analytics service landscape: where each service sits

AWS's own strategy guide summarises the map in one sentence: "you can perform real-time streaming with Kinesis or Amazon MSK, data processing with Amazon EMR or AWS Glue, search with Amazon OpenSearch, ad-hoc queries with Athena, and data warehousing with Amazon Redshift" (AWS Prescriptive Guidance, AWS modern data architecture, accessed 2026-10-11). Governance sits across the whole sentence: "AWS Lake Formation helps govern data and share it across the company. In addition, you can create a unique data catalog on AWS Glue and a business glossary by using Amazon DataZone… to enable your employees to find the data they need."

5.1 The cast, one line each

5.2 Placement rules that outlive the table

1. S3 is the floor. Every pattern in lesson 01 lands data on S3 or S3 Tables first; nothing above the storage layer is allowed to become the system of record.
2. Glue is the spine, not just a job runner. It owns the catalog (metadata layer) and the serverless transform (bronze→silver→gold). Lesson 07 is entirely this service.
3. Athena and Redshift are alternatives over the same copy, not rivals. Both query S3/Iceberg; Redshift adds managed concurrency, materialized views, data sharing and a warehouse experience at node/RPU cost, while Athena is serverless and pay-per-TB-scanned: ideal for ad-hoc and federated use (worked example E6).
4. Lake Formation is a band, not a box. It wraps the catalog and the tables, and its grants apply across engines rather than to one tool.
5. Macie classifies; Lake Formation authorizes; Glue catalogs. Three services, three jobs. Confusing the first two is a security-review failure, not a naming quibble.
6. EMR is the power tool. Petabyte-scale Spark, Flink, Trino and Hive when serverless Glue is not enough, and EMR Serverless when you want that power without a cluster fleet.

• 📚 Did you know? Amazon Macie never appears alone in AWS's modern-data-architecture security list; it is named with "AWS Key Management Service (AWS KMS), AWS Identity and Access Management (IAM), AWS Secrets Manager, AWS Audit Manager, and Amazon Macie" (AWS Prescriptive Guidance, accessed 2026-10-11). Encryption, identity, secret storage, audit evidence and data discovery are five different jobs; lesson 14 assigns each one.

Worked example E3: the Athena bill is a layout decision, not a query decision (as of Oct 2026)

AWS publishes this exact ladder on its Athena pricing page (accessed 2026-10-11). Table: 3 TB of uncompressed text on S3, four equal columns, query reads one column.

AWS's own page states the arithmetic: "Price for 3 TB scanned is 3 * $5/TB = $15", ending at "$1.25" for the optimised path: a 12× difference for the same answer. Scale it: 20 such queries per day × 30 days = 600 queries.

• Naive: 600 × 3 TB = 1,800 TB × $5.00 = $9,000.00 per month
• Optimised: 600 × 0.25 TB = 150 TB × $5.00 = $750.00 per month
• Monthly difference: $8,250.00: bought entirely by format and column pruning, before a single cluster is resized

Worked example E4: pricing a bronze-to-silver Glue job (as of Oct 2026)

The Glue pricing page (accessed 2026-10-11): "The price of 1 DPU-hour is $0.44"; "A single DPU provides 4 vCPU and 16 GB of memory"; billing is "in increments of 1 second, rounded up to the nearest second, with a 1-minute minimum duration per run."

• Small job, 6 DPU × 15 minutes: 6 × 0.25 h = 1.5 DPU-hours × $0.44 = $0.66 per run; 30 nightly runs = $19.80 per month
• Heavier hop, 10 DPU × 20 minutes: 3.333 DPU-hours × $0.44 = $1.47 per run; × 30 = $44.00 per month
• Interactive Session default is 5 DPU (minimum 2 DPU); a Glue Studio data-preview session is 2 DPUs for up to 30 minutes, auto-stopping: exploration is billed by the session, not the month
• Table-optimization work (compaction, statistics, materialized-view auto-refresh, crawlers) is listed at $0.44 per DPU-hour as well, so maintenance is a metered line item, not free overhead
• Version matters: AWS announced on 21 August 2026 that "AWS Glue 6.0 is now generally available, delivering a 30% price reduction and introducing full support for Apache Iceberg v3". Budget with the published $0.44 meter as the conservative anchor and re-read the pricing page for your Region and Glue version before committing a forecast

Worked example E5: what the serving layer costs in seats (as of Oct 2026)

QuickSight pricing page (accessed 2026-10-11): Author $24 per user/month, Reader $3 per user/month, Reader Capacity "From $250 For 500 sessions/month", plus "A $250/month per account infrastructure fee applies if your account has any of the following: At least one Pro user; Q&A enabled via topics; Dashboard Q&A enabled."

• 10 authors + 200 readers: (10 × $24) + (200 ×$3) = $240.00 + $600.00 = $840.00 per month, with no $250 infrastructure fee (no Pro user, no Q&A topics)
• Add one Pro user and the $250 per month account fee applies: $1,090.00 per month for the same 210 people: a 30% jump for a single role change, which is why role design is a cost control in lesson 17
• Capacity alternative for occasional readers: from $250 for 500 sessions/month, which beats 84 casual readers at $3 each (84 × $3 = $252.00) and loses beyond that: the seat model wins once readers are daily users

Read the chart in order: storage dominates raw ingestion (the tallest bar), so compression and lifecycle policy are the first cost levers (lessons 02 and 16); query and BI costs are an order of magnitude smaller and are controlled by layout (E3) and seat model (E5); compute for scheduled transformation is the smallest line of all (E4); which is why "we will save money by writing fewer Glue jobs" is almost never the right optimisation.

---

6. End-to-end reference architecture

6.1 The flow: source to dashboard

This is the spine every later lesson plugs into. Read it left to right, then note the two dotted control planes, catalog and governance, which touch every stage rather than one.

The numbered subgraphs are the AWS layer order from §1.3 with the medallion zones from §3 drawn inside the storage layer: one picture, both vocabularies.

6.2 The same architecture as a conversation

Flowcharts hide ordering and triggers. This sequence diagram shows what actually happens for one day of data, and where the bill is incurred.

Three details the flowchart cannot show. (1) The ACID commit boundary sits at each medallion hop, so readers never see half-written data. (2) Lake Formation's decision happens before bytes are scanned, which is why governance is cheap to enforce early and expensive to bolt on later. (3) The Athena billing event is the scan, not the query count: the quantity you control is bytes read, which is why lessons 03, 04 and 09 all restate E3's ladder.

6.3 The container view: producers, central catalog, consumers

AWS Prescriptive Guidance's growth-and-scale reference architecture splits accounts deliberately: "A data producer layer in different AWS accounts"; "A data consumer layer in different AWS accounts"; "A centralized catalog in an AWS account"; and "The centralized catalog is the interface used by data producers and data consumers to share and consume data": with the stated intent to "standardize data sharing and consumption, and independently scale data producers and consumers without growing your management overhead."

The dotted lines are the point: metadata and permissions travel through the central catalog account, while data and compute stay where they are economical. This is the decoupling discipline that FINRA's case makes concrete in §10.

Worked example E6: Athena or Redshift Serverless for the same dashboard? (as of Oct 2026)

A dashboard reads 50 GB of Parquet per day. Prices from the Athena and Redshift pricing pages (accessed 2026-10-11):

• Athena: 50 GB × 30 = 1,500 GB = 1.5 TB per month scanned × $5.00 per TB = $7.50 per month, plus zero idle cost, because there is no cluster
• Redshift Serverless: 64 RPU for ~2 hours per day = 128 RPU-hours × $0.375 per RPU-hour = $48.00 per month, plus Redshift Managed Storage at $0.024 per GB-month: the pricing page's own worked example reads "40 TB x 1,024 GB per TB x $0.024"
• Ratio: about 6.4× more expensive for the Redshift path on this workload: before accounting for what the extra money buys: concurrency scaling (the pricing page notes "up to one hour of free Concurrency Scaling credits per day" per cluster), materialized views, data sharing, and no per-query scan anxiety
• Rule of thumb: sparse, ad-hoc, federated or bursty access → Athena; dense, concurrent, always-on BI with predictable concurrency → Redshift Serverless. Both read the same S3/Iceberg copy; you are choosing a serving engine, not a second data store

---

7. Why open table formats are non-negotiable

An open table format adds a transactional metadata layer over open file formats (Parquet, ORC, Avro) so the lake behaves like a database while staying vendor-neutral. AWS states the practical consequence, "Iceberg supports standard SQL commands for data warehousing for use with data lake tables", and the business consequence as a genuine either/or: a "fully integrated but proprietary platform" versus "a vendor-neutral but resource-intensive, self-built data lake" (AWS Prescriptive Guidance, accessed 2026-10-11).

What the format buys you, from AWS's published Iceberg feature list: delete, update and merge; fast scan planning and advanced filtering; full schema evolution; partition evolution; hidden partitioning; version rollback; time travel; serializable isolation; concurrent writers; and open file formats.

Why "open" is the operative word: the S3 Tables team records the interoperability consequence: table-management APIs "compatible with the Apache Iceberg REST Catalog standard, enabling you to use any Iceberg-compatible query engine (e.g., Spark, Trino, PyIceberg, or DuckDB) to access tabular data directly from S3 Tables" (AWS Storage Blog, 2025-05-27). Engine choice becomes a deployment decision instead of a migration project.

7.1 Three positions on AWS today

The managed-Iceberg numbers AWS publishes: S3 Tables are "specifically optimized for analytics workloads, resulting in up to 3x faster query throughput and up to 10x higher transactions per second compared to self-managed tables" (AWS What's New, 2024-12-03). Quotas as reported five months after launch: "supporting the creation of up to 10,000 tables within each table bucket… up to 100,000 tables across 10 table buckets within a single AWS Region and AWS account" (AWS Storage Blog, 2025-05-27).

• 📚 Did you know? S3 Tables' Iceberg REST Catalog support makes the "which engine?" question testable: if a tool speaks the Iceberg REST Catalog, it can read your tables directly. AWS's own example list is "Spark, Trino, PyIceberg, or DuckDB" (AWS Storage Blog, 2025-05-27): a laptop running DuckDB included. That is what "open" buys you that a warehouse export never will.

---

8. Worked mini-scenarios: sizing the thing you just designed

8.1 The 10 TB/day pipeline sizing sketch

Worked example E7: 10 TB/day, storage, query and transform (as of Oct 2026)

Anchor: BMW Group "uses AWS to process 10 TB of data daily from 1.2 million vehicles" (AWS case study, accessed 2026-10-11). We will size storage, query and transform for the same daily rate, using list prices read on 2026-10-11. Retention assumption: 30 days of hot data.

Step 1: raw storage.

• 10 TB/day × 30 days = 300 TB per month = 307,200 GB (1,024 GB per TB)
• At S3 Standard's first-50-TB tier of $0.023 per GB-month: 307,200 × $0.023 = $7,065.60 per month
• AWS's published volume tiers (51 to 100 TB at $0.022, 500 TB+ at $0.021) and any lifecycle policy reduce this: treat $7,065.60 as the conservative ceiling, not the invoice

Step 2: the same data after Parquet at 3:1 (the ratio from AWS's own Athena example).

• 300 TB ÷ 3 = 100 TB = 102,400 GB × $0.023 = $2,355.20 per month
• Saving: $4,710.40 per month (−66.7%): from format alone, before a single query is optimised

Step 3: query.

• If analysts and dashboards scan 1 TB per day in aggregate: 30 TB × $5.00 per TB = $150.00 per month on Athena
• If they instead scan the whole 300 TB (full-table, unpartitioned): 300 × $5.00 = $1,500.00 per month: ten times more, for the same answers, purely because partition pruning and columnar layout were skipped

Step 4: transform.

• One nightly bronze→silver job at 10 DPU × 20 minutes (from E4): $1.47 per run → $44.00 per month
• A silver→gold hop of the same size doubles it to $88.00 per month

Step 5: the sketch.

Conclusion: at 10 TB/day the format decision (row 1) moves roughly $4,710 per month, while the compute decision (row 3) moves about $88. Order your optimisation backlog accordingly; this is AWS's own argument in miniature: fix the data layout before touching infrastructure size. And notice what the sketch does not include: governance, monitoring, and the people who run the platform. Those are real costs, which is why lesson 16 returns to this table with lifecycle and rightsizing levers.

8.2 The Kappa replay check

Scenario 8.2: six months of history, 24 hours of replay (as of Oct 2026)

A fraud team adopts the stack from AWS's Builder Center Kappa walkthrough: Kinesis Data Streams → Spark Structured Streaming on EMR → table format on S3. The article presents a 24-hour replay window for the stream (2026-05-18).

• A compliance request arrives for six months of reprocessing. 24 hours ÷ 180 days ≈ 0.6% of the required window is available from the stream.
• The correct move is to re-read the lake: the durable copy on S3 behind the table format, where retention is governed by your own lifecycle rules (lessons 02 and 16), not by stream retention.
• Design rule that falls out: streams are transport, lakes are record. Never let the stream be your only copy of anything you might need to reprocess, and never size a stream for archival storage, because you would pay streaming prices for archive duty.
• Cross-check with the medallion: a six-month reprocess re-runs the silver and gold hops over bronze, which is exactly why bronze is retained raw and untransformed, and why E1's 2.4 million monthly bronze files are a feature (recoverability) rather than only a cost.

8.3 What the scenarios share

Both mini-scenarios make the same move: take a published AWS number, apply one assumption stated in the open, and let the arithmetic decide the backlog. Storage ratio first, then query layout, then compute size, then seats. If you invert that order, sizing clusters before looking at file format and partitioning, you will optimise the cheapest line item on the E7 table while the storage bill, which is 95% of the sketch before optimisation, keeps running.

---

9. Course roadmap: where this lesson sits

Lessons 02 to 17 are the same architecture, layer by layer. Nothing in later lessons contradicts the skeleton above; they fill in the boxes.

Three habits to carry forward: quote a source and a date for every number (this lesson used only AWS pages read 2026-10-11); name the layer before you name the service; and read customer metrics as claims against their own baseline, never as guarantees.

2026 Updates (as of October 2026)

---

Real-World Case Studies

AWS publishes what this architecture looks like under load. Every figure below is customer-reported within an AWS-owned page: unaudited, dated, and never a guarantee for your workload. The transferable part is the pattern: which layer did the work, which services held it, which number moved. All sources were accessed 2026-10-11 unless a publication date is given.

Case A, FINRA: decoupling storage, compute and catalog (home lesson 01)

• Context. The US securities regulator "processes approximately 6 terabytes of data and 37 billion records on an average day… On busy days, the stock markets can generate 75 billion+ records" (AWS Public Sector Blog, published 2017-10-03).
• Problem. Its on-premises capacity could not flex with market spikes; the legacy HDFS design "required over 2 PB of storage on the cluster with 60 hs1.8xlarge instances" (AWS Big Data Blog, published 2016-11-21).
• Architecture. An S3 data lake holding raw, curated and archive zones; Amazon EMR for Hive, Spark and HBase compute; and FINRA's own open-source catalog: "Enter herd, our open source data catalog and orchestration tool… capture audit and data lineage information for our highly regulated environment" (2017-10-03).
• Outcomes, as AWS reported them in 2016 to 2017. "We extended the data lake pattern with EMR, Hbase, and Amazon S3 to allow interactive random access queries across trillions of records spanning 600+ terabytes of data" (2017-10-03); "cost savings of over 60%, drastically reduced the time for recovery or upgrades"; a new HBase cluster was "able to launch… and be ready to accept queries in less than 30 minutes because the data (HFiles) stays on S3. This would have taken almost two days on our old cluster", with "most queries returning in less than 3 seconds" (2016-11-21).
• Takeaway for lesson 01. This is §6.3 with the boxes renamed: storage, compute and catalog were split, so any engine could read the same S3 truth and clusters became disposable. A lakehouse is a decoupling discipline, not a product you can purchase. FINRA's phrase about lowering "the cost of curiosity" is the business case for keeping bronze cheap and open.

Case B, Easygo: a modern lakehouse with a streaming-first pattern (home lessons 09 and 01)

• Context. A gaming company running near-real-time analytics across global operations (AWS case study, accessed 2026-10-11).
• Architecture. "The company implemented a streaming-first pattern using Amazon Kinesis to capture and process events in real time, Amazon Simple Storage Service (Amazon S3) as the central data lake, and Amazon Athena for faster querying on partitioned datasets. Easygo also incorporated Amazon Redshift Serverless", plus "Amazon Aurora zero-ETL Integration with Amazon Redshift" so data "moves from Aurora PostgreSQL into Amazon Redshift automatically" (AWS case study, accessed 2026-10-11).
• Outcomes (customer-reported). "Query execution times improved by 95 percent, with workloads that previously required up to 1 hour now completed in about 10 minutes"; "reduced data incidents by 80 percent"; 9 hours of maintenance saved weekly: previously "at least 3 engineers from different departments to spend roughly 3 hours each week addressing pipeline interruptions"; fraud-threshold freshness improved from 15 to 30 minutes late to near real-time.
• Quote. "Changing our architecture to a more modern lakehouse on AWS helped us improve performance and set up a foundation we can reuse as we grow into other markets" (Carine Oliveira, Engineering Manager).
• Takeaway for lesson 01. Easygo is §4's Kappa-leaning column plus §5's service map in production: streaming ingestion → S3 lake → serverless query → warehouse for the relational side, with zero-ETL removing a copy. Note what moved: freshness (1 hour → about 10 minutes) and operational load (−80% ingestion incidents, −9 engineer-hours per week): both are layer-and-pattern outcomes, not hardware outcomes.

Case C, BMW Group: layering at 10 TB per day (home lesson 07, reused here)

• Context. "BMW Group uses AWS to process 10 TB of data daily from 1.2 million vehicles" (AWS case study, accessed 2026-10-11): the same daily rate we sized in E7.
• Architecture. "The individual data layers are stored in Amazon S3 buckets, and their schemas are registered in the AWS Glue Data Catalog." Providers ingest and transform "with AWS services such as Amazon Kinesis Data Firehose, AWS Lambda, AWS Glue, and Amazon EMR"; consumers then use "services such as Amazon Athena, Amazon SageMaker, AWS Glue, and Amazon EMR".
• The catalog as a product surface. A Data Portal offers a "'popularity index' based on data usage patterns for more than 500 users across the organization": §6.3's centralized catalog interface, made visible to users rather than buried in an infrastructure account.
• Takeaway for lesson 01. Two points to bank: (1) a real lakehouse at this rate keeps layers in S3 with schemas in Glue: exactly the §3 mapping of medallion to AWS primitives; (2) the catalog is the sharing interface, which is why lesson 07 earns a whole lesson rather than a paragraph.

Same pattern, different industry (reuse this frame whenever you need to argue the architecture to a non-engineer):

---

Practice Questions

Twelve scenario items. Each has exactly four options with one correct answer, and the explanation names the most tempting wrong option so you learn why it is wrong, not merely which letter to pick.