S3 as the Data Lake Foundation: Storage Classes, Glacier, and Durability

Every lakehouse in this course is built on one assumption: Amazon S3 is the substrate. Glue crawls it, Athena scans it, Redshift Spectrum mounts it, Lake Formation grants on it, Macie inspects it, QuickSight reads it. Nothing downstream can rescue a lake whose storage layer was designed by accident, and in S3, "by accident" has a specific price. The wrong storage class bills you a retrieval fee on every byte an engine reads; the wrong prefix layout throttles ingest at 3,500 writes per second; the wrong object size adds 40 KB of archive metadata per file; the wrong retention design turns a lifecycle rule into a one-way door with a 180-day minimum bill.

The good news is that S3's cost model is knowable. Every rate in this lesson comes from the AWS Price List offer for Amazon S3 in us-east-1 (publication date 2026-09-28, effective 2026-09-01, accessed October 2026) plus the AWS S3 user guide, pricing page and FAQ (accessed October 2026). Do the arithmetic yourself; it is the only way to design a lake that is still affordable in year seven.

In this lesson you will:

• read the full storage-class table: all eight classes plus every Glacier tier, with retrieval time, retrieval fee and minimum storage duration;
• walk the storage-class decision tree and the one-way lifecycle waterfall as mermaid diagrams, then order the waterfall yourself in an interactive drag-and-drop;
• model versioning, replication, Object Lock (WORM) and legal holds, including exactly which mutations replication does not mirror;
• choose between SSE-S3, SSE-KMS, DSSE-KMS and SSE-C, and use an S3 Bucket Key to cut KMS request cost "by up to 99 percent";
• design buckets, prefixes and Hive-style partition keys against the ≥3,500 write / ≥5,500 read requests per second per prefix model;
• apply Block Public Access (four settings) and the eight bucket-policy patterns a private lake needs;
• build the governance plumbing, S3 Inventory and S3 Batch Operations, that lifecycle rules and live replication cannot do alone;
• hand-compute five cost models with real numbers: the 1 TiB class ladder, a 50 TiB Deep Archive restore, a hot-prefix throughput design, a 7-year lifecycle-to-Deep-Archive TCO, an Intelligent-Tiering break-even, and the 40 KB archive-metadata trap;
• study two named AWS customer cases (Nasdaq and Siemens), read the 2026 update box, and practise with 12 exam-style questions plus four interactive checks.

---

1. The eight storage classes: one durability, eight economics

1.1 The comparison table (as of Oct 2026)

Amazon S3 offers eight current storage classes plus one deprecated class. AWS documents them side by side in the S3 user guide's storage-class comparison (accessed Oct 2026). All of them share 99.999999999% designed durability, eleven nines, so durability is never the discriminator. Availability design, AZ footprint, minimum storage duration, minimum billable object size and retrieval fees are.

Three details in that table do more damage than any others:

1. S3 Express One Zone is a directory-bucket class. It lives in a different bucket type (--bucket-prefix directory buckets), is single-AZ, and is priced at $0.11/GB-month, about 4.8× S3 Standard storage, in exchange for consistent single-digit-millisecond latency. It is a serving/scratch tier, never the lake of record.
2. Intelligent-Tiering has no retrieval fee at all. It replaces that fee with a per-object monitoring charge: $0.0025 per 1,000 objects per month (AWS Price List offer, 2026-09-28). Objects smaller than 128 KB are not monitored and pay no fee; which is both a trap and a hint (section 10).
3. The three Glacier classes are not "slower S3". Two of them (Flexible Retrieval, Deep Archive) cannot be read with GetObject at all until you run an asynchronous restore. That single fact drives every archive design in this course.

1.2 Worked example E1: the 1 TiB monthly ladder

As of Oct 2026 (us-east-1 list). How much does it cost to simply hold 1 TiB = 1,024 GB for one month in each class?

Two readings of that ladder matter. First, Deep Archive is 23× cheaper than Standard to hold ($1.014 vs$23.552), which is exactly why the archive tier is a retention decision, not a performance one. Second, Express One Zone is 4.8× more expensive than Standard to hold, which is why it hosts gold serving tables and ML feature stores, never bronze.

The S3 FAQ puts it plainly: "From just $0.00099 per GB-month … S3 Glacier Deep Archive offers the lowest cost storage in the cloud" (accessed Oct 2026). One caveat worth carrying: the us-east-1 Price List offer of 2026-09-28 contains the Deep Archive staging and Intelligent-Tiering Deep Archive Access SKUs but no standalone Deep Archive storage SKU, so treat $0.00099 as AWS-published rather than price-list-confirmed, and re-check before you commit a seven-figure archive to it.

• 📚 Did you know? Reduced Redundancy (RRS) still appears in the S3 price list at $0.0240→$0.0220/GB-month, but the S3 user guide says only: "We recommend not using this storage class" (accessed Oct 2026). It is more expensive than S3 Standard with less redundancy. Any exam-style option, vendor deck or legacy runbook that still routes data to RRS is wrong as of Oct 2026.

---

2. The Glacier family: every tier, every fee, every clock

"Glacier" is three storage classes and five restore tiers, and the difference between them is the difference between a query that returns in milliseconds and a ticket that closes tomorrow.

2.1 The archive table

Sources: S3 Glacier storage-classes comparison and tier bullets (user guide, accessed Oct 2026); retrieval and restore-request prices from the 2026-09-28 us-east-1 offer; "S3 Glacier Flexible Retrieval Bulk data retrievals and requests are free of charge" and "DELETE and CANCEL requests are free" (S3 pricing page, accessed Oct 2026).

Read the table vertically and three rules fall out:

• Glacier Instant Retrieval is the only archive class a query engine can read directly: milliseconds, but every byte read costs $0.03/GB, seven-and-a-half times its own storage rate.
• Glacier Flexible Retrieval's Bulk tier is free but slow: $0.00/GB, typically 5 to 12 hours. The FAQ phrases it as "Bulk retrievals typically complete within 5 to 12 hours, and are free of charge."
• Deep Archive has no Expedited tier through S3. The fastest published path is Standard at $0.02/GB "within 12 hours"; Bulk at $0.0025/GB takes "within 48 hours". The comparison table elsewhere in the same guide quotes the range as "9 to 48 hours": plan against the tier bullets, not the best case.

2.2 A restore is not a GET

This is the single most misunderstood mechanic in the entire S3 pricing model, so it gets its own diagram.

AWS's own pricing page states the consequence: "When you restore an archive, you are paying for both the archive… and a copy… restored temporarily (charged at the S3 Standard storage rate for a duration of time you choose)" (accessed Oct 2026). So the full restore bill has four components: archive storage (still accruing) + the tier's per-GB retrieval fee + the per-1,000 restore-request fee + Standard storage on the temporary copy for the whole chosen duration. The archived original never changes class: HeadObject keeps reporting GLACIER or DEEP_ARCHIVE after a successful restore.

2.3 Worked example E2: restoring 50 TiB from Deep Archive

A media house must regurgitate a 50 TiB (51,200 GB) broadcast library inside a five-day window. Hold the restored copy for 5 days.

Verdict: Bulk for planned recoveries; Standard only when the business clock is shorter than 48 hours. The $896 delta is pure retrieval fee: the temporary-copy charge is identical either way, which is why the temp copy, not the retrieval fee, dominates any long hold. Need the data in under 15 minutes? Deep Archive cannot help you: there is no Expedited tier via S3, so a same-hour SLA means the data should never have left Standard, Standard-IA or Glacier Instant Retrieval.

• 📚 Did you know? The standalone Amazon Glacier vault service, a separate product with its own vault APIs and ARNs, "is no longer accepting new customers as of December 15, 2025" (S3 Glacier pricing page, accessed Oct 2026). AWS now recommends: "We recommend using the S3 Glacier storage classes within the Amazon S3 service for all of your long-term data." In 2026, "Glacier" in a design conversation means three storage classes on a normal S3 bucket, not a vault.

---

3. Choosing a class: the decision tree and the waterfall

3.1 The storage-class decision tree

Access frequency, required latency and whether the object is recreatable decide the class: in that order. The leaf labels carry the price and the clock you are signing up for.

Two branches deserve the commentary:

• "Unknown access" does not automatically mean Intelligent-Tiering. It means Intelligent-Tiering once objects are large enough to be monitored (≥128 KB) and few enough that the monitoring fee is not the whole bill. For a known-quarterly archive, Glacier Instant Retrieval is usually cheaper: section 10.3 does the break-even.
• One Zone-IA is a durability trade, not just a discount. 99.5% availability in a single AZ means a genuine question of whether the data can be recreated. Lakes use it for reproducible derived data, never for the only copy of raw source data.

3.2 The lifecycle waterfall is one-way

Amazon S3 "supports a waterfall model for transitioning between storage classes" (S3 lifecycle user guide, accessed Oct 2026). Objects flow down; they do not flow back up. The back-arrows in the diagram below are not lifecycle edges; they are restore-and-copy operations you run by hand or through S3 Batch Operations, and each one restarts the destination class's minimum-duration clock and re-bills at the destination rate.

Rules the S3 lifecycle docs put in writing (accessed Oct 2026):

• "You can't create a single Lifecycle rule that transitions objects from one storage class to another before the minimum storage duration period has passed."
• "The transition of objects to the S3 Glacier Deep Archive storage class can go only one way."
• "An object is eligible for only one S3 Lifecycle action per day. When multiple rules match, Amazon S3 applies the least costly action."
• Precedence: permanent deletion > transition > creation of delete markers. When both a Glacier Flexible Retrieval and a Standard-IA transition are eligible, "Amazon S3 chooses the S3 Glacier Flexible Retrieval transition."
• A configuration can hold up to 1,000 rules: "This limit is not adjustable."
• Since September 2024, "Objects smaller than 128 KB will not transition by default to any storage class."

---

4. Durability, versioning and replication

4.1 Versioning: three states, full-object billing

S3 Versioning is bucket-level and all-or-nothing, with exactly three states: unversioned (the default), enabled and suspended. The AWS user guide is blunt about the billing: "Each version of an object is the entire object; it is not just a diff… if you have three versions of an object stored, you are charged for three objects" (accessed Oct 2026).

Versioning is a prerequisite for two things you will need later: S3 Object Lock (which works only in versioned buckets) and S3 replication (which requires versioning on both source and destination).

4.2 Replication: CRR, SRR, and the archived-class wall

Cross-Region Replication (CRR) and Same-Region Replication (SRR) are the same mechanism with different destinations. Versioning must be enabled on both sides. What replicates by default (S3 replication docs, accessed Oct 2026):

• Replicated: new objects created after the configuration, object metadata, tags, annotations, ACL updates, Object Lock retention, and objects encrypted with SSE-S3 / SSE-KMS / DSSE-KMS / SSE-C.
• Not replicated by default: objects that are themselves replicas (no chaining), delete markers when the configuration uses a Filter element, permanent version deletes, bucket-level subresource changes, and lifecycle-created delete markers.
• Not replicable at all until restored: objects in Glacier Flexible Retrieval, Glacier Deep Archive, Intelligent-Tiering Archive Access and Intelligent-Tiering Deep Archive Access. AWS's exact words: "You cannot replicate these objects until you restore them and copy them to a different storage class."

Three operational consequences for a lake:

1. CRR is not a live backup of every mutation. With the current configuration version (a Filter), delete markers are not replicated unless you explicitly enable delete marker replication, and version-ID deletes are never mirrored: deliberately, so a compromised principal cannot erase the DR copy by deleting versions in the source.
2. Retroactive backfill needs S3 Batch Replication. Live replication only covers objects created after the configuration; everything older is a Batch Operations job.
3. Monitoring costs $0.015 per 1 million objects per month in the source bucket (2026-09-28 offer). There is no separate "$0.02 per GB replicated" SKU in that offer; you pay destination-bucket storage and ordinary inter-region data transfer on top of the monitoring fee. Do not teach the per-GB figure without re-checking the Replication pricing tab.

• 📚 Did you know? S3 replication carries Object Lock retention information to the replicas: "it applies those same retention controls to your destination buckets" (S3 replication docs, accessed Oct 2026). That is what makes CRR into a second compliance copy rather than merely a second copy: the retain-until date survives the Region hop.

---

5. Object Lock: WORM for regulated lake zones

S3 Object Lock "uses a write-once-read-many (WORM) model" and "works only in buckets that have S3 Versioning enabled" (S3 user guide, accessed Oct 2026). It protects an object version, not a key, and that distinction is where most designs fail.

AWS has had Object Lock "assessed by Cohasset Associates for use in environments that are subject to SEC 17a-4, CFTC, and FINRA regulations" (accessed Oct 2026); which is exactly why the regulated-data stories in this course (FINRA-style surveillance archives, Nasdaq's compliance tier) reach for it.

• 📚 Did you know? A single rule limit most people miss: an S3 Lifecycle configuration "can have up to 1,000 rules" and each rule's filter may use "an object key name prefix, one or more object tags, object size, or any combination of these" (S3 API docs, accessed Oct 2026), and you cannot use a bucket policy to prevent deletions or transitions by a lifecycle rule. Object Lock stops principals; lifecycle expiry is a service action, so audit-zone designs pair Object Lock with lifecycle rules that deliberately stop short of the retention boundary.

---

6. Encryption: SSE-S3, SSE-KMS, DSSE-KMS, SSE-C and Bucket Keys

Since 5 January 2023, "all new object uploads to Amazon S3 are automatically encrypted at no additional cost and with no impact on performance"; that default is SSE-S3 (S3 user guide, accessed Oct 2026). Everything else is an upgrade with a price and a constraint.

Three upgrades a lake typically needs:

1. An S3 Bucket Key. "Using a bucket-level key for SSE-KMS can reduce your AWS KMS request cost by up to 99 percent" (S3 user guide, accessed Oct 2026). At lake write volumes, hundreds of thousands of PutObject calls per day, the KMS request bill is the difference between SSE-KMS being affordable and being a line item.
2. A customer managed CMK for anything cross-account. "Objects encrypted using SSE-KMS with AWS managed keys can't be shared cross-account. If you need to share SSE-KMS data cross-account, you must use a customer managed key." Grant kms:Decrypt to the consumer account and enable the Bucket Key.
3. Encryption survives class transitions, but note the billing footnote: "Objects in S3 Glacier Flexible Retrieval and S3 Glacier Deep Archive are billed at S3 Standard storage class rates when you upload them using multipart uploads." Always land in Standard (or complete the multipart upload promptly) and let lifecycle do the transition.

Two request-shape traps worth memorising: SSE-KMS requests must arrive over TLS with SigV4, and you must not send encryption headers on GET/HEAD: S3 answers HTTP 400.

---

7. Buckets, prefixes and partition keys: the performance model

7.1 S3 scales per prefix, not per bucket

The S3 performance documentation states the model exactly: "your application can achieve at least 3,500 PUT/COPY/POST/DELETE or 5,500 GET/HEAD requests per second per partitioned Amazon S3 prefix. There are no limits to the number of prefixes in a bucket… if you create 10 prefixes… you could scale your read performance to 55,000 read requests per second" (accessed Oct 2026). Scaling is gradual, and clients see 503 Slow Down errors while it ramps.

The composite incident AWS's Well-Architected lab material describes matches this shape: a streaming ingest job wrote 800 small objects/second into s3://lake-prod/raw/events/2026-10-11/events.json.gz, one prefix, throughput collapsed at ~3,500 writes/s with sustained 503 Slow Down, and re-sharding the producer key to 16 shard-{000..015} prefixes restored headroom to ~56,000 writes/s with no other change. Partition spread is a performance control independent of partition semantics.

7.2 Partition keys are the physical layout

For Hive-style tables, the key layout is the physical partition layout: s3://lake/events/dt=2026-10-11/hour=07/part-…parquet. Design rules that hold across Glue, Athena, EMR and Redshift Spectrum:

• 📚 Did you know? LIST requests are not free and are not priced like GET: "LIST requests for any storage class are charged at the same rate as S3 Standard PUT, COPY, and POST requests": $0.005 per 1,000 (S3 pricing page and 2026-09-28 offer). DELETE and CANCEL requests, by contrast, are free. That is a second, quieter argument for moderate prefix fan-out: more prefixes mean more LIST/GET fan-out on full-bucket scans and more Glue partition metadata, so shard to your real time partitions rather than synthetically to the limit.

---

8. Locking the lake down: Block Public Access and bucket policies

8.1 Block Public Access is the kill switch

"By default, new buckets, access points, and objects don't allow public access." On top of that default, S3 Block Public Access provides four settings, BlockPublicAcls, IgnorePublicAcls, BlockPublicPolicy and RestrictPublicBuckets, and "if the block public access settings… differ, then Amazon S3 applies the most restrictive combination" (S3 user guide, accessed Oct 2026).

AWS's recommendation for a lake account is unambiguous: "we recommend that you turn on all four settings… for your account." Remember the layering trap: disabling BPA on one bucket does nothing if the account or organization still blocks, and S3 always enforces the most restrictive combination. "Public" means an ACL or policy granting to AllUsers/AuthenticatedUsers, or a bucket policy with a wildcard principal not constrained by a fixed aws:PrincipalArn, aws:PrincipalOrgID, aws:SourceAccount, aws:SourceArn, aws:SourceVpc, aws:SourceVpce, aws:SourceIp or s3:DataAccessPointArn.

8.2 Eight bucket-policy patterns for a private lake

Block Public Access stops the bucket going public; the policy below stops everything else a data lake actually needs to refuse. A Deny statement with "Principal": "*" restricts access rather than granting it, so it composes cleanly with all four BPA settings left on.

Notes on the JSON, because each statement encodes a rule that is easy to get subtly wrong:

• One condition per statement for the SSE-KMS checks. A single statement with two StringNotEquals keys only applies when both conditions match, which would let a request through with the right encryption header but the wrong key. Two statements = two independent denials.
• Statement 2 alone is incomplete without statement 3: pinning the algorithm does not pin the key, and an unpinned key is an unpinned blast radius.
• The full pattern list for a lake (S3 bucket-policy and condition-key docs, accessed Oct 2026): (1) deny non-TLS; (2) deny PutObject unless SSE-KMS; (3) pin the CMK ARN; (4) deny DeleteObjectVersion outside the retention role; (5) prefix-scoped grants such as arn:aws:s3:lakehouse-prod/raw/* for engineering roles; (6) Region pinning with aws:RequestedRegion; (7) organization pinning with aws:PrincipalOrgID; (8) service-principal source restrictions (aws:SourceArn + aws:SourceAccount) for Lake Formation, Glue, Athena and Macie: plus retention caps via s3:object-lock-remaining-retention-days.

---

9. The governance plumbing: S3 Inventory and S3 Batch Operations

Lifecycle rules and live replication cover forward motion. Two services cover everything else, and both are cheap enough that skipping them is a false economy.

9.1 S3 Inventory: the scheduled metadata snapshot

S3 Inventory produces "comma-separated values (CSV), Apache optimized row columnar (ORC) or Apache Parquet output files that list your objects and their corresponding metadata on a daily or weekly basis" (S3 user guide, accessed Oct 2026). The properties that matter for a lake:

That field list is an audit pipeline: who is locked, until when, in which tier, replicated or not: queryable in Athena, without a single ListObjectsV2 call against the live bucket.

9.2 S3 Batch Operations: retroactive governance

S3 Batch Operations is manifest-driven: build the manifest from an S3 Inventory report, choose an operation, pay per job and per object. Us-east-1 list, as of Oct 2026: $0.25 per job and $1.00 per million objects processed (2026-09-28 offer).

The operations lifecycle rules and live replication cannot do:

• Retro-apply Object Lock retention or legal holds to millions of pre-existing objects;
• Restore millions of Glacier objects (the only sane way to run a bulk recovery like E2 at scale);
• Batch Replication to backfill objects that pre-date the replication configuration;
• Bulk-apply SSE-KMS after a key policy change;
• Copy, tag, ACL updates, or invoke a Lambda function per object.

The pattern to internalise: Inventory is the read side (what do we have, in which state), Batch Operations is the write side (change all of it, once), and lifecycle is the standing order (keep doing this as objects age). S3 Select, by the way, is not part of this toolkit for new designs: AWS closed it to new customers on 25 July 2024 and documents Athena as the recommended approach for data-lake querying (AWS Storage Blog, 2024-07-25).

---

10. Lifecycle cost math: Deep Archive TCO and the Intelligent-Tiering trap

10.1 Worked example E5, the headline: 7-year TCO with a lifecycle waterfall

A compliance lake holds 100 TiB (102,400 GB) of Parquet averaging 256 MB per object (≈409,600 objects), written once, queried for 90 days, then archived for 7 years. All prices us-east-1 list, as of Oct 2026.

The ~$185,000 saving is real, and it is conditional on exactly three things: the data genuinely stays ≥180 days in Deep Archive (otherwise the minimum duration bills you anyway), a 12-hour / 48-hour restore SLA is acceptable, and the objects are large enough that the 40 KB metadata overhead stays a rounding error (it is: ~$0.08/month on 409,600 objects). Miss any one of the three and the model breaks; which is why section 10.4 exists.

10.2 Worked example E6: Intelligent-Tiering break-even

Same question every lake asks: 10 TiB (10,240 GB), access unknown, 200 million objects. Should this be Intelligent-Tiering, or an explicit class?

10.3 Worked example E7: the 40 KB archive-metadata trap

Every object moved into Glacier Flexible Retrieval or Glacier Deep Archive carries 40 KB of additional metadata: "32 KB… charged at the default rate for that storage class" plus "an additional 8 KB… charged at the S3 Standard rate" (S3 archival-storage docs, accessed Oct 2026). On 256 MB Parquet files that is invisible. On a lake of tiny JSON files it is the bill.

10.4 The Intelligent-Tiering monitoring-fee trap

A second, quieter trap from the same family: multipart uploads into Glacier classes are billed at S3 Standard storage rates until every part completes and S3 applies the real class. Land in Standard (or finish the upload promptly) and let lifecycle transition the completed object.

2026 Updates (as of October 2026)

---

Real-World Case Studies

Case A, Nasdaq: query-in-place, with a compliance off-ramp

Nasdaq (exchange operator, financial services). As AWS's case study describes it (page year 2020, accessed Oct 2026), Nasdaq moved to Amazon Redshift in 2014, grew a cluster to 70 nodes by 2018, and was ingesting 30 to 55 billion records, more than 4 TB, every night from thousands of sources. Load-versus-query contention on the warehouse was the constraint. The fix was the pattern this course calls a lakehouse: an Amazon S3 data lake of 15 TB queryable in place via Redshift Spectrum, which Nasdaq itself calls a "lake house architecture". AWS reports Redshift queries "32 percent faster" and 90% data-load completion 5 hours sooner, and a January 2019 AWS Data Lab (four days) supported the jump "from 30 billion records to 70 billion records a day."

The reason Nasdaq appears in this lesson is the off-ramp, not the speedup: AWS positions the S3 lake with Glacier and Object Lock as the compliance/archive tier beside the queryable hot data. That is the storage-class decision tree made real: recent windows stay in S3 Standard for Spectrum and Redshift to scan, and the multi-year regulatory tail is an Object Locked, lifecycle-managed archive. The lesson angle: archive is a retention decision with a restore SLA attached, exactly as in worked example E5.

Customer-reported outcomes from an undated AWS page; not an AWS commitment. Date-stamp them: as AWS reported in 2020.

Case B, Siemens: 5 to 7 TB a day, and S3 server access logs as the audit trail

Siemens (industrial manufacturing and technology; Cloud Security Operations). Per AWS's case study (undated page, accessed Oct 2026), Siemens governs 800+ AWS accounts and onboards 5 to 7 TB/day of security data into Amazon Security Lake, forwarding only 600 GB onward to Splunk, a 90% volume reduction, for a reported $1.3 million annual savings. New-log onboarding fell from "several weeks" to 1 to 2 days, and root-cause identification time dropped 75%. One investigation identified the impacted S3 data in 10 minutes versus several days.

The lesson-relevant detail is in the onboarded source list: VPC Flow Logs, S3 server access logs, and AWS WAF logs. S3 server access logging is the lake's own black box, every request recorded, in the lake, at Standard rates, and it is the S3-native complement to CloudTrail S3 data events. Note the name trap AWS itself warns about: Amazon Security Lake is a security-log aggregation service on S3; it is not AWS Lake Formation, which governs permissions (lesson 12).

---

Practice Questions