PII Protection on the Lakehouse: Amazon Macie, Masking, and Privacy Workflows

Lessons 01 to 12 built a lakehouse that is fast, cheap and governable. That combination creates one new obligation: sooner or later somebody puts a customer's email address, phone number or national ID into s3://lakehouse/bronze/, and you have to be able to say where it is, who can see it, what you did about it, and when you would have to delete it. This lesson is the answer, and the answer is a pipeline, not a product.

The teaching rule for this lesson: classify, route, protect, control, respond: five layers, each with a different service, none of them optional. Amazon Macie only ever does the first one. Everything after a finding is your EventBridge rule, your Lambda, your Glue job and your Lake Formation grant.

In this lesson you will:

• run the five-layer pipeline end to end and name the service that owns each layer, and name the layer that has no AWS service at all (routing glue);
• go deep on Amazon Macie: inventory and monitoring, automated discovery versus targeted jobs, sampling depth, schedules, storage-class support and the hard scope limits;
• separate the three numbers students constantly confuse: finding severity (1 to 3), bucket sensitivity score (−1 to 100) and occurrence count;
• choose between managed data identifiers and custom data identifiers, write a real regex, and know exactly which identifier you must build yourself (email);
• read the Finding schema and route on it with an EventBridge rule into a Lambda that quarantines a prefix and stamps PII=true;
• price a classification programme to the dollar, using the three billing dimensions and the sampling-depth dial;
• choose a masking technique by join-fidelity requirement, redact, hash, format-mask or tokenize, and avoid the mask() runtime trap on Glue 4.0 and Athena;
• tie classification back to Lake Formation column and row security so a tag becomes an enforcement, not a note;
• build the audit trail from CloudTrail data events, Macie findings and JSONL exports;
• handle GDPR, CCPA and LGPD at AWS depth, including why S3 versioning quietly breaks erasure.

---

1. The five-layer pipeline: classify, route, protect, control, respond

1.1 The layers and who owns them

Three properties of this table are worth memorising:

1. Layers 1 and 5 are detective; layers 3 and 4 are preventive; layer 2 is the only glue. Macie will never mask a column or revoke a grant: the wiring between a finding and a control is code you write.
2. Layer 1 splits by engine. Macie inspects S3 general purpose buckets; Glue's Detect PII transform inspects DataFrames inside a job. Parquet in the lake and columns in the Glue Data Catalog are Glue's territory, not Macie's.
3. Layer 4 must exist even if layer 3 is perfect. A beautifully masked silver table does nothing for the analyst who bypasses the lakehouse and reads bronze/ with an IAM policy.

1.2 Worked example E1: pricing a classification programme (us-east-1, as of Oct 2026)

Take a bronze landing zone of 1 TB holding 2,000,000 objects, plus 14 smaller buckets (15 buckets total). Macie bills on three dimensions (Amazon Macie Pricing, accessed 2026-10-11):

The lesson inside the example: automated discovery is the cheap continuous layer; targeted jobs are the expensive precision layer. The AWS cost-reduction playbook (Security Blog, 2023-03-20) is exactly this arithmetic: use automated discovery to find the risky buckets, sample them at 10% (about $102 on 1,024 GB) to validate, and reserve the full $1/GB scan for the handful of buckets that actually matter.

• 📚 Did you know? The 30-day free trial covers up to 150 GB inspected per account and the bucket inventory dimension, but the pricing page is explicit that "Any sensitive data discovery jobs you create will not be included in the 30-day free trial" (Amazon Macie Pricing, accessed 2026-10-11). A demo team that runs a 200 GB, 100%-sampling job on day two of the trial has just billed itself about $200. The free trial covers discovery; it does not cover your jobs.

---

2. Amazon Macie in depth

2.1 Inventory and monitoring: the layer that is always on

Macie "automatically and continually evaluates your S3 buckets for the presence of sensitive data" using "bucket inventory and monitoring, automated sensitive data discovery, and machine learning (ML) to profile buckets and sample objects" (Macie User Guide, Automated sensitive data discovery: how it works, accessed 2026-10-11).

Three facts make this layer behave differently from a scan:

• Buckets are clustered "by attributes such as bucket name, file types, and prefixes to cost-efficiently sample data broadly across an organization and minimize the data scanning needed". Clustering is why a representative sample can be statistically useful without touching every object.
• Coverage is capped at 10,000 buckets per account and Region (quota raised 2024-12-06), and billing is $0.10 per bucket per month, prorated daily.
• Results are not instant. Analysis results "can begin to appear within 48 hours of enabling automated sensitive data discovery", and scores sit at 50 — Not yet analyzed until they do. An empty bucket scores 1 — Not sensitive, which means no data, not no PII.

2.2 Automated discovery versus targeted jobs

These are different mechanisms with different bills, and conflating them is the most common Macie misunderstanding.

Job operational limits worth quoting in a design doc (Macie User Guide, Quotas, accessed 2026-10-11): 50 concurrent jobs, 10,000 jobs per account, job-creation API burst 1 per 20 seconds, findings require objects of at least 15 bytes, and the pattern-match window is 4 KB of text per object (1 MB for S3 Select). Sensitive-data findings are trimmed at 15,000 findings per bucket per day, and policy findings are deduplicated on a 15-minute window.

• 📚 Did you know? Macie's supported storage classes list is a gotcha generator: Standard, Standard-IA, One Zone-IA, Intelligent-Tiering, Reduced Redundancy and Glacier Instant Retrieval are inspected, but Glacier Deep Archive and Glacier Flexible Retrieval are not, and objects in S3 Express directory buckets are not inspected at all. Encrypted objects are readable only "if the objects are encrypted with SSE-KMS or DSSE-KMS keys" (Macie User Guide, accessed 2026-10-11). An archive tier full of legacy exports is not "covered"; it is invisible until restored.

2.3 Managed versus custom data identifiers

Managed data identifiers (MDIs) are AWS-maintained detectors. The quick reference lists 40, grouped into Credentials (ACCESS_KEYS, AWS_ACCESS_KEYS, AWS_ACCOUNT_ID, DB_CREDENTIALS, ENCODED_PRIVATE_KEY, PRIVATE_KEY, SECRET_KEYS, …), Financial information (BANK_ACCOUNT_NUMBER, CREDIT_CARD_NUMBER, CVC_NUMBER, …), Personal information: PHI (DOB, DRIVERS_LICENSE, HEALTHCARE_NUMBER, MEDICAL_RECORD_NUMBER, PASSPORT_NUMBER, SSN, …) and PII (ADDRESS, GENDER, NAME, PHONE, POSTAL_CODE, …).

Custom data identifiers (CDIs) are yours: "A custom data identifier helps you detect sensitive data in S3 objects… based on custom criteria that you specify": a regular expression, a dictionary of words and phrases (keywords), and "an optional list of regular expressions for text to ignore (ignore words)" (Macie User Guide, accessed 2026-10-11).

2.4 Worked example E2: a custom data identifier, regex by regex

Requirement: detect Northwind Health member numbers of the form NWH followed by ten digits, in a bronze prefix of migrated policy documents, while not firing on order numbers or on test fixtures.

Two supporting facts: custom identifiers "are unique to each customer and are not shared with other customers" (Macie FAQ, page dated 2025-09-30), there is no marketplace of community detectors, and false-positive control belongs to allow lists (up to 500 regexes, 100 characters each), not to forking the identifier.

2.5 The Finding: schema, severity, score, count

A Macie finding is a JSON object. The API Finding model exposes id, createdAt, updatedAt, partition, region, resourceArn, title, severity, type, either SensitiveData or Policy, and count, described as "The total number of occurrences that were observed" and "always 1 for sensitive data findings" at the top level. Per-identifier occurrence detail lives in the occurrences sub-objects, which carry their own count and up to 15 location entries. Findings are retained for 90 days (raised from 30 on 2021-10-01).

Three numbers, three different objects: never conflate them:

AWS is explicit that the score "neither… implies or otherwise indicates the criticality or importance that a bucket or a bucket's objects might have for you" (Macie User Guide, Sensitivity scoring for S3 buckets, accessed 2026-10-11). A score of 100 is always a manual override; Macie never assigns it. If discovery is disabled at the organization level, existing scores and labels persist for 30 days and are then reset.

• 📚 Did you know? Macie's own findings are sensitive artifacts. Occurrence location entries describe where a match was found in the object, and the S3 results repository writes JSON Lines files (*.jsonl, gzip-compressed, encrypted with SSE-KMS) that each carry a keyed-hash message authentication code (HMAC) signature (added 2023-11-08). Treat exports with the same care as the data: encrypt them with a restricted key, limit GetFinding to the security team, and expire the export prefix with a lifecycle rule.

2.6 EventBridge: the routing contract

"When Macie detects new sensitive data findings or policy findings… it publishes corresponding events to EventBridge. EventBridge then routes the events to targets that you specify. Targets can include AWS Lambda functions, Amazon SNS topics, and Amazon SQS queues.": Macie User Guide, Findings, events, and EventBridge (accessed 2026-10-11)

The two fields you filter on:

Documented event flavours include new sensitive data findings, new occurrences of existing sensitive data findings, new policy findings and subsequent occurrences of existing policy findings; which means your rule must be idempotent, because an existing finding can fire again when a new occurrence shows up. Security Hub ingests the same findings, exposes the CSPM control Macie.1 ("Amazon Macie should be enabled"), and offers custom actions that turn a human button press into a second EventBridge rule into a Lambda (AWS Security Blog, 2022-07-18; sample repo aws-samples/security-hub-macie-remediation).

2.7 Worked example E3: sampling depth as a cost and coverage dial

A 200 GB bucket, one targeted job, $1.00 per GB inspected (us-east-1, as of Oct 2026):

The x-axis is linear in money and non-linear in what you learn: going from 10% to 100% costs 10× and buys the only result you can defend in an audit. Depth is a cost/coverage dial, not a quality setting; there is no depth at which Macie's regexes get smarter.

---

3. The classify → route pipeline: finding, Lambda, quarantine, tag

3.1 The routing diagram

The top half is detection; the bottom half is prevention. Both must exist, because Macie is periodic (up to a 48-hour warm-up, then your chosen cadence) while Glue masking in Section 4 runs per load. A pipeline with only the top half reports on damage; a pipeline with only the bottom half never learns about the data that arrived before the masking job existed.

3.2 Worked example E4: the rule and the handler

The EventBridge rule, filtered to high-severity sensitive-data findings:

The Lambda handler's five side effects, in the order they must run:

Cost of the routing layer itself, as of Oct 2026: the first copy of CloudTrail management events is free; CloudTrail S3 data events are $0.10 per 100,000 events and are opt-in: without them you cannot answer "who read the object" after an incident (AWS CloudTrail Pricing, accessed 2026-10-11). Lambda, EventBridge rules and SNS are metered separately, but on this volume they are cents.

---

4. Protection in the lake: detect bronze → tag → mask/tokenize silver

Classification without transformation just tells you where the problem is. The medallion answer is a write-time transformation: detect at bronze, tag at the catalog boundary, mask or tokenize on the way into silver, and let Lake Formation guard whatever residual PII survives.

4.1 Choose the technique by the question "must it join? must it come back?"

Only these four are verified AWS patterns. Format-preserving encryption (AES-FF1 style FPE) is not an AWS managed lakehouse feature: if you see it, it is a third-party or general cryptographic technique, not something Glue or Athena ships.

4.2 Pattern A: static redaction, and the Glue Detect PII transform

The Detect PII transform detects "personally identifiable information (PII)… columns in your input data that might contain PII data", either on a per-cell basis (default) or a per-column basis where you supply a sampling percentage and a matching threshold. Its four actions: Enrich, Redact (default, replaces sensitive data with ********), Partial redact, and SHA-256 hash. Sensitivity levels are High and Low; categories are Universal, HIPAA, Networking, US, UK, CA, ES, IT; and "You can also apply different transformations to different columns by using fine-grained overrides". In Glue 3.0/4.0 job output the reported actionUsed values are DETECT, REDACT, PARTIAL_REDACT, SHA256_HASH.

4.3 Pattern B: deterministic tokenization that still joins

Deterministic means the same input always produces the same token, so GROUP BY hashed_email still clusters a customer across tables. It also means the mapping is reversible in principle for low-entropy inputs: a 10-digit phone number space is enumerable, so a plain SHA-256 of a phone number is a dictionary attack away from being plaintext. The fixes, in order of preference:

1. Keyed HMAC with a secret salt stored in Secrets Manager: same determinism, no rainbow table. Use this for phone, SSN, postcode-grade values.
2. Reserve plain SHA256_HASH for high-entropy identifiers: UUIDs, account numbers with check digits.
3. Full tokenization vault when you need reversibility and to remove PII from the analytical path: the AWS Guidance Tokenization to Improve Data Security and Reduce Audit Scope on AWS (reviewed 2024-07-23) architecture is API Gateway → Lambda → DynamoDB token vault encrypted with KMS, with tokens carrying an HMAC so lookups are consistent. Downstream analytics consume tokens only; the vault is the only component that can ever see the original.

4.4 Worked example E5: the same email, four times, and why they differ

4.5 Pattern C: format masking with Spark's mask()

Spark's SQL mask() function ("New in version 3.5.0") "Masks a column with default characters for letters, numbers, and 'other' characters: 'X' (uppercase), 'x' (lowercase), '0' (digit), '' (other)"*, and accepts replacement characters as arguments: mask(col, upperChar, lowerChar, digitChar, otherChar).

4.6 Worked example E6: one Glue job, three different protections

A DataFrame of 100,000 customer rows leaving bronze for silver (Glue Detect PII API shape, accessed 2026-10-11):

The point of the worked example: one job produces three different outcomes, destroyed, joinable, and merely governed, chosen per column by what the consumer actually needs. Redaction is irreversible; hashing preserves joins; tag-plus-grant preserves the value for entitled roles only.

4.7 Pattern D: the encryption backstop

Since 2023-01-05 every new S3 upload is encrypted with SSE-S3 by default at no additional cost. That is not the posture you want for PII: SSE-S3 uses AWS-managed keys, gives you no per-key audit granularity and no deny-by-key-policy control. PII buckets should therefore require SSE-KMS, set bucket default encryption to aws:kms and add a bucket policy condition denying uploads that are not aws:kms, with S3 Bucket Keys switched on.

4.8 Worked example E7: what the backstop costs (us-east-1, as of Oct 2026)

• 📚 Did you know? Determinism and reversibility pull in opposite directions, and AWS's own tokenization guidance exists precisely because of that. A deterministic hash is join-friendly but attackable for low-entropy fields; a token vault is reversible but requires a lookup service; a redaction is safe but final. The AWS Guidance tokenization paper (reviewed 2024-07-23) puts the vault behind API Gateway with a KMS-encrypted DynamoDB table, which also shrinks your audit scope: the analytic path never holds the plaintext, so the systems that need scope reduction stop being in scope for it.

---

5. Lake Formation tie-back: from tag to column and row enforcement

Classification only becomes security when a permission consumes the classification. Lesson 12 owns the grant mechanics; this section is the PII-specific shape.

5.1 The two mechanisms

• LF-Tags (tag-based access control, LF-TBAC): attach pii = true (or data_class = phi) to databases, tables and columns. Key and value strings are ≤ 50 characters, stored lowercase, with ≤ 50 tags per Lake Formation resource, ≤ 10,000 tag keys and ≤ 1,000 values per key. Grants then name attributes, so every table a crawler creates inherits the policy the moment someone tags it.
• Data filters: "Data filters in Lake Formation enable you to implement column-level, row-level and cell-level security on Data Catalog tables", expressed with a PartiQL row-filter expression, inclusion columns (column-level), exclusion columns (cell-level) and both together (cell-level). They can be shared with accounts, AWS Organizations, IAM principals and IAM roles.

5.2 Worked example E8: one table, two roles, one PII column

Cell-level filters are also how you "show the column but null out the values" without materialising a masked copy; which is the cheapest possible answer to "we need the schema for the dashboard but not the data".

• 📚 Did you know? Lake Formation grants are cheap to over-grant financially and expensive to over-grant politically: permission checks cost nothing, while the integrated engines bill for what they scan. That asymmetry is why the PII pattern is always over-restrict by default, then grant exceptions: the failure mode of the opposite approach is not a bigger bill, it is an incident.

---

6. Audit: CloudTrail data events, Macie history, and the evidence store

6.1 What each artifact can and cannot answer

6.2 The evidence flow, drawn

Two properties of this diagram are worth stating out loud: the export bucket is the long-term evidence store (Macie itself keeps detailed information for up to 90 days), and CloudTrail is the only leg that answers "who": Macie answers "what", never "who read it". That split is the whole reason a privacy programme needs both services.

6.3 Worked example E9: the evidence pack for one quarter

6.4 Worked example fill-in: the routing contract

---

7. Compliance framing: GDPR, CCPA and LGPD at AWS depth

Compliance is a mapping exercise, not a feature flag. Each regulation maps onto specific layers of Section 1, and each has one AWS-specific pitfall.

7.1 The three regulations, in scope

AWS also maintains a LGPD workbook (AWS Security Blog, 2022-04-25) mapping controls to CIS v8.0, NIST CSF, the NIST Privacy Framework and AWS CAF 3.0, useful when a regulator asks for a control framework rather than a list of services. And note the shared-responsibility boundary: the HIPAA Eligible Services Reference (last updated September 3, 2026) lists Amazon Macie, AWS Lake Formation, AWS Glue, Amazon Athena, AWS Lambda, Amazon EventBridge, AWS CloudTrail, AWS KMS, Amazon S3, AWS Step Functions, Amazon GuardDuty and more, the whole toolchain in this lesson is eligible in the right configuration, and eligibility never transfers your own compliance obligation.

7.2 Worked example E10: a versioning-aware erasure

Request: delete customer C-123 from bronze and silver, under GDPR Article 17 (erasure) or LGPD/CCPA equivalents.

• 📚 Did you know? Evidence and erasure work against each other by design. Macie's findings contain occurrence locations and are retained 90 days; your JSONL exports contain the same detail and are retained until your lifecycle rule says otherwise. A defensible privacy programme therefore declares in advance which store is the system of record for evidence, how long it lives, and what the deletion procedure is for that store too: otherwise you will satisfy an erasure request in the lakehouse while the audit trail still holds the value.

---

8. The worked end-to-end flow

Everything in this lesson, in one pass: bronze to serve, detection to evidence:

Read it left to right once more: every box in Layer 1 costs money only when data is inspected; every box in Layers 3 to 4 costs nothing until someone queries; Layer 5 is the only layer whose output you can hand to a regulator.

8.1 The decision checklist before you ship

2026 Updates (as of October 2026)

---

Real-World Case Studies

AWS publishes what these designs look like at production scale. Every figure below is customer- or AWS-claimed and unaudited, with source and date named so you can check it: the examinable point is the pattern.

Case A, FanDuel: two roles, one column grant, the security angle

The security angle. FanDuel's design is a default-deny structure wearing a performance costume. The dangerous property of a dual-file design is not the storage cost, it is that the PII copy exists as a separately addressable object, so every new source creates a second artifact that needs its own bucket policy, its own tag hygiene and its own review, and every one of those is a place where the PII-bearing copy can be left wide open. Collapsing it to one dataset plus a column grant removes the second artifact entirely: there is no "PII file" to mis-tag, no shadow copy drifting out of compliance, and no IAM policy that can grant the PII columns without also going through Lake Formation. The default assignment matters just as much, NonPIIAnalyst is what analysts get unless somebody actively promotes them, which is the only defensible default when the asset is a customer's personal data.

That is the same lesson as Section 5.2, arrived at from the incident side: the cheapest PII control is the one that makes the unauthorised copy unnecessary.

• 📚 Did you know? The Oportun and Expedia figures above are AWS customer-story card titles, quoted verbatim from aws.amazon.com/macie (accessed 2026-10-11). AWS does not publish the methodology behind "95%" on the card itself, so present it as a customer claim with its date: never as an AWS guarantee or a benchmark you can plan against. The durable lesson from both stories is architectural: tuned identifiers plus allow lists, at scale, instead of one regex team.

---

Practice Questions