Security, Encryption and IAM for Data Pipelines

Every pipeline is a chain of authorised calls: a Glue job assumes a role, decrypts a CMK-wrapped data key, reads s3://raw/, writes s3://curated/, and hands Redshift a manifest. Break any link: a key policy that never enabled IAM, a grant nobody revoked, a bucket policy that denies aws:SecureTransport, and the pipeline fails with an error message that names the last hop, not the real cause. Domain 4 of the DEA-C01 exam guide (18% of scored content) tests exactly this chain: authentication (4.1), authorization (4.2), encryption and masking (4.3), audit logs (4.4) and privacy/governance (4.5) (DEA-C01 exam guide, accessed Oct 2026). This lesson teaches the chain the way the exam asks it: as decisions, defaults and numbers, each sourced.

In this lesson you will:

• separate customer managed, AWS managed and AWS owned KMS keys; who owns, who pays, who can share;
• evaluate access the way KMS does: key policy first, then IAM and grants, explicit Deny always winning;
• apply envelope encryption, its 4,096-byte direct-API cap, and what rotation does not re-encrypt;
• price KMS per use, key fee, request fee, free tier, Bucket Keys, with October 2026 numbers;
• build the at-rest matrix for S3, Glue, Redshift, Kinesis, DynamoDB, EMR and Athena;
• enforce TLS in transit, including the aws:SecureTransport redaction trap that breaks replication;
• write least-privilege pipeline roles: prefix-scoped S3, KMS actions on key ARNs, no AmazonS3FullAccess;
• wire cross-account access through the four documents that must all agree;
• control S3 access with bucket policies, retired ACLs, Block Public Access, VPC endpoints and presigned URLs;
• choose between Secrets Manager and Parameter Store SecureString on features and price;
• place security groups and PrivateLink so pipeline traffic never hairpins the public internet;
• practise with 14 exam-style questions plus three interactive checks.

---

1. The Domain 4 task map: where this lesson lands

The exam guide's Domain 4 carries five tasks; this lesson is the technical core of 4.1 to 4.3, and it supplies the identity machinery that 4.4 (audit) and 4.5 (governance) consume.

---

2. AWS KMS key ownership: customer managed, AWS managed, AWS owned

AWS KMS offers three tiers of key, and the DEA-C01 exam distinguishes them on five axes: where the key lives, what you can configure, what you pay, whether it can rotate on a schedule you control, and whether it can be shared cross-account.

The rule of thumb AWS's own KMS documentation supports: control → CMK, convenience → AWS owned. When a question asks you to "use a key you control so another account can decrypt", the answer is always a CMK: AWS managed keys cannot be shared cross-account, and AWS owned keys are not addressable at all (KMS concepts, accessed Oct 2026).

• 📚 Did you know? The AWS managed tier is a legacy pattern: AWS's KMS documentation notes that AWS managed keys are not created for new service integrations since 2021: new integrations default to AWS owned keys (free) or ask you to bring a CMK. That is why DynamoDB's default is AWS owned (rotated ~365 days, AES-256, cannot be disabled) while S3's managed option is still the aws/s3 AWS managed key (KMS concepts; DynamoDB encryption docs, accessed Oct 2026).

---

3. Key policies versus IAM policies versus grants

This is the single most examined mechanic in Domain 4. Three mechanisms can allow a KMS action, and they do not stack the way beginners assume.

3.1 The three rules, verbatim from the KMS docs

1. "A key policy is a resource policy for an AWS KMS key. Key policies are the primary way to control access to KMS keys. Every KMS key must have exactly one key policy."
2. "No AWS principal, including the account root user or key creator, has any permissions to a KMS key unless they are explicitly allowed, and never denied, in a key policy, IAM policy, or grant."
3. "Unless the key policy explicitly allows it, you cannot use IAM policies to allow access to a KMS key. … (You can use an IAM policy to deny permission to a KMS key without permission from a key policy.)" (KMS key-policies and control-access docs, accessed Oct 2026)

Two further structural facts: key policies are Regional, IAM policies are global; and a grant can allow but never deny, covers exactly one key, has no automatic expiry, and is capped at 50,000 grants per key (KMS grants docs, accessed Oct 2026).

3.2 Evaluation order

3.3 Anatomy of the default key-policy statement

The default statement every new CMK carries names the account, not the root user:

Read arn:aws:iam:<account-id>:root as "this account, via its IAM policies"; it delegates to IAM so that identity policies in the account can grant key actions. It does not grant the root user special standing beyond what IAM already allows, and replacing it with "Principal": "*" in an Allow statement is the canonical hole (KMS key-policy overview, accessed Oct 2026).

3.4 Worked example E1: an identity policy that looks right and does nothing

An analyst's IAM policy in account B:

If account A's CMK key policy never named account B or arn:aws:iam:B:role/AnalystRole, this policy is inert. The analyst receives AccessDenied with the misleading hint "not authorized to perform kms:Decrypt", the fix is never in the IAM policy; it is in the key policy (and, for the data itself, the bucket policy). Exam phrasing: "Which change grants the cross-account analyst access?", answer the key policy first.

• 📚 Did you know? The three mechanisms have different verbs on purpose: policies are declarative JSON evaluated on every call; roles are assumable identities whose trust policy decides who may create a session; grants are KMS-specific, allow-only, machine-friendly tickets that integrated services create on your behalf: RetireGrant and RevokeGrant are the cleanup APIs, and a forgotten duplicate grant is a standing privilege escalation (KMS grants best-practices doc, accessed Oct 2026).

---

4. Envelope encryption: how bulk data actually gets encrypted

KMS keys never leave FIPS 140-3 Level 3 HSMs unencrypted, so encrypting terabytes through the key itself would be both slow and impossible: the direct Encrypt/Decrypt APIs accept a maximum of 4,096 bytes of plaintext. Every AWS at-rest integration therefore uses envelope encryption: generate a cheap per-object data key, encrypt the payload locally, and keep only the data key's ciphertext under the KMS key.

What this means for exam scenarios:

• The CMK wraps the data key, not your data. kms:Decrypt on the key is necessary but not sufficient: the caller also needs the ciphertext data key stored with the object (S3 does this for you under SSE-KMS).
• ReEncrypt* is two permissions, not one. Re-wrapping a data key under a new key requires both kms:ReEncryptFrom and kms:ReEncryptTo (KMS API permissions reference, accessed Oct 2026).
• Rotation never rewrites ciphertext. New key material encrypts future data keys; old ciphertext decrypts with the retained old backing key forever. Nothing about "enabling rotation" migrates your objects.
• Client-side encryption (CSE-KMS) is the same maths run entirely on your side, AWS never sees plaintext, which is why CSE-KMS is the "strongest" Athena result tier and an EMRFS option.

---

5. Rotation and grants: what is automatic, what is optional, what is dangerous

5.1 Rotation rules

Rotation is transparent: existing ciphertext is never re-encrypted, and the old HSM backing keys are retained so old data remains decryptable. Events you can audit: KMS CMK Rotation to EventBridge, RotateKey in CloudTrail (KMS rotate-keys docs, accessed Oct 2026).

5.2 Grant mechanics

• A grant can allow access, but not deny; each grant allows access to exactly one KMS key; grants do not automatically expire (KMS grants docs, accessed Oct 2026).
• One grantee per grant: an IAM identity or a service principal (the way Glue, DMS and Redshift obtain least-privilege, time-boxed access).
• Limits: 50,000 grants per key; CreateGrant is roughly as powerful as PutKeyPolicy, so constrain it with kms:GranteePrincipal, kms:GrantOperations, kms:GrantConstraintType and kms:GrantIsForAWSResource.
• A GrantToken returned by CreateGrant sidesteps post-creation eventual consistency in the same workflow.

5.3 Worked example E2: what "enable rotation" costs and does not do

A team enables automatic rotation on one symmetric CMK on 2026-01-15, rotates it five times over three years, and stores 40 TB of S3 objects under SSE-KMS with that key.

The exam trap: "enabling rotation re-encrypts existing objects with the new key material" is false. Re-encryption is a separate, deliberate job (S3 Batch Operations, ReEncrypt* calls, or rewrite the data).

---

6. KMS pricing: pay for keys and for use

Every number in this section is from the AWS KMS pricing page, as of Oct 2026; re-verify before dating a design decision later.

6.1 Worked example E3: the monthly KMS bill for an SSE-KMS lake

One CMK, and a busy month of 2,010,000 KMS calls (GETs and PUTs each triggering data-key operations):

6.2 Worked example E4: the same lake with an S3 Bucket Key

S3 Bucket Keys reduce the number of KMS requests that reach your key by up to 99%, because S3 forwards one data key instead of one per object operation. Unsupported for DSSE-KMS; supported for SSE-KMS.

Exam framing (and the fix in Case A below): "Which single change most reduces the KMS cost of an SSE-KMS bucket without weakening the encryption posture?": enable S3 Bucket Keys.

---

7. Encryption at rest: the per-service matrix

At-rest encryption on this exam is a defaults game. Note the pattern: managed services increasingly encrypt always, with your control limited to which key tier you upgrade to.

7.1 Worked example E5: the migration that did not re-encrypt

A compliance team flips an existing S3 bucket's default encryption from SSE-S3 to SSE-KMS with a CMK on 2026-10-01 and declares the lake compliant.

---

8. Encryption in transit: TLS everywhere, and the two enforcement patterns

AWS's baseline: S3 accepts TLS 1.2 and 1.3 at its endpoints, and all AWS service API endpoints have required a minimum of TLS 1.2 since 2024-02-27. Kinesis requires TLS 1.2 and recommends 1.3. Redshift, however, accepts non-SSL connections by default; you must set require_SSL=true in the cluster's parameter group. The exam loves that asymmetry.

8.1 Server-side versus client-side, in one line each

• Server-side (SSE-*): the service encrypts on receipt and decrypts on serve; you never hold plaintext.
• Client-side (CSE-KMS, S3 encryption client, Redshift COPY of client-encrypted files, EMRFS CSE): you encrypt before upload; AWS cannot read the plaintext; which is why CSE is simultaneously the strongest posture and the hardest to operate.

8.2 Worked example E6, an aws:SecureTransport deny that broke replication

Team policy: deny any request that is not HTTPS.

The condition polarity is right, aws:SecureTransport is true for HTTPS, but S3 replication then fails to write its configuration. Reason, from the S3 docs: "When AWS services make calls to other AWS services on your behalf (service-to-service calls), certain network-specific authorization context is redacted, including s3:TlsVersion, aws:SecureTransport, aws:SourceIp, and aws:VpcSourceIp." The replication service principal presents no aws:SecureTransport value, so the Deny matches. Fix:

…or equivalently add "aws:PrincipalIsAWSService": "false" to the Deny's condition so only human/role callers are policed. Same trap applies to s3:TlsVersion minimums and to aws:SourceIp conditions used by backup or replication jobs.

8.3 Redshift SSL and the other per-service transit rules

Config automation exists for the two most-tested levers: AWS Config managed rules s3-bucket-ssl-requests-only and redshift-require-tls-ssl (plus kinesis-stream-encrypted) continuously assert the posture you configured once.

---

9. IAM for data services: roles, least privilege, and the anti-pattern

9.1 The building blocks

9.2 The AWSGlueServiceRole pattern and the AmazonS3FullAccess anti-pattern

AWS's console historically attaches AmazonS3FullAccess to demo Glue jobs. The Glue documentation itself tempers this: "You might want to provide your own policy for access to specific Amazon S3 resources." The managed AWSGlueServiceRole grants Glue plus limited S3 (its own aws-glue-* script and scratch buckets), CloudWatch and ENI plumbing; it is not a data-access grant. Production pattern:

Three things this snippet gets right that AmazonS3FullAccess gets wrong: prefix-scoped Resource ARNs, s3:ListBucket on the bucket ARN paired with s3:GetObject/PutObject on the object ARN (the two-level S3 permission model), and KMS actions bound to the specific key ARN; which the key policy must also allow (Section 3).

AWS's own minimums for a Glue job match the split: "Data sources require s3:ListBucket and s3:GetObject permissions. Data targets require s3:ListBucket, s3:PutObject, and s3:DeleteObject." And if you pass a role through the console, its name must be prefixed AWSGlueServiceRole (Glue IAM docs, accessed Oct 2026).

9.3 KMS permissions inside pipeline policies

The minimum action set by workload shape (KMS API permissions reference, accessed Oct 2026):

• 📚 Did you know? S3's two-level permission model is a frequent silent failure: s3:ListBucket is authorised against the bucket ARN (arn:aws:s3:::lake-raw), while s3:GetObject/s3:PutObject are authorised against object ARNs (arn:aws:s3:::lake-raw/raw/*). Granting only the object ARNs leaves ListBucket failing; the AWS Glue docs' source/target split above exists precisely because crawlers and partition projection call ListBucket constantly (Glue minimum-privileges doc, accessed Oct 2026).

---

10. Cross-account access: the four documents that must agree

Cross-account is Domain 4.3.3's "encryption across account boundaries", and it fails with an error that names one mechanism while the real gap is another. Account A owns a curated bucket and CMK; account B's analysts need s3:GetObject.

Two structural constraints worth memorising: AWS managed keys cannot be shared cross-account (so the design forces a CMK), and for Kinesis resource-policy sharing the key must likewise be a CMK. On the billing side, "the AWS account that makes the API request is charged" for KMS usage (AWS KMS pricing, accessed Oct 2026).

---

11. S3 access: bucket policies, ACLs, endpoints, presigned URLs

11.1 Bucket policies and the ACL retirement

A bucket policy is a resource policy: principal, action, resource, condition. It is also the only place you can express aws:SecureTransport, aws:SourceVpce or aws:PrincipalIsAWSService constraints.

ACLs are legacy. Since AWS enabled Object Ownership = Bucket owner enforced by default, all ACLs are disabled, the bucket owner owns every object, and ACL writes fail with AccessControlListNotSupported. AWS's guidance: "we recommend that you keep ACLs disabled" and migrate any surviving ACL grants into bucket policies (S3 Object Ownership docs, accessed Oct 2026). Block Public Access defaults to all four switches ON for new buckets (account, bucket, access point, organization), with the most restrictive winning.

11.2 VPC endpoints: gateway versus interface

11.3 Worked example E7: presigned URL lifetimes

A presigned URL delegates the presigner's permissions for one operation; it never widens them, and validity is re-checked on every request (bucket-policy Denies and Block Public Access still apply).

Prescriptive Guidance's framing: you cannot rate-limit URL generation; your levers are least privilege + short expiry + monitoring (S3 presigned-URL docs; Prescriptive Guidance, accessed Oct 2026).

---

12. Secrets Manager versus Parameter Store SecureString

Both store encrypted text under KMS; they diverge on rotation, cross-account, size and price. AWS's own selection guidance: Secrets Manager for secrets, Parameter Store for simple key-value pairs (AWS Security Blog, 2025-10-22, accessed Oct 2026).

12.1 Worked example E8: 50 database credentials, twelve months

A nightly Glue job and three EMR steps rotate through 50 warehouse/service credentials.

• 📚 Did you know? The 30-day Secrets Manager trial applies per secret: AWS charges from day 31, and rotation versions created during rotation are never charged, so the cost of "test it and revert" is literally zero for a month. Parameter Store's standard tier, by contrast, is free forever but capped at 10,000 parameters of 4 KB per account per Region, which is why large estates quietly outgrow it into the advanced tier (AWS pricing pages, accessed Oct 2026).

---

13. Security groups and PrivateLink: keeping pipeline traffic off the internet

Network controls are Domain 4.1.1's first skill ("update VPC security groups") and 4.1.5's "policies on roles/endpoints/services (S3 Access Points, PrivateLink)". The pattern for a private pipeline: place Glue/EMR/Redshift/DMS in private subnets, reach S3 and KMS through VPC endpoints, and let security groups express service-to-service reachability rather than raw CIDRs.

PrivateLink (interface endpoints) keeps traffic on the AWS network, layers an endpoint policy on top of your bucket/key policies, and, with private DNS enabled, makes SDK calls resolve privately so they never touch the internet. It is the correct answer whenever on-premises clients over Direct Connect/VPN must reach pipeline services; the free gateway endpoint is the correct answer for pure S3/DynamoDB egress from a VPC with no on-premises requirement.

---

2026 Updates (as of October 2026)

---

Real-World Case Studies

AWS publishes what these abstractions look like in production. Every figure below is customer- or AWS-claimed and unaudited, with the source named so you can check it: the examinable point is the pattern (which control was the binding constraint, which service did the work, which number moved), not the marketing.

Case A, Integral Ad Science: hundreds of permission rules down to two

Read IAS as the anti-pattern-removal story: every "hundreds of rules" an estate accumulates is a candidate for a tag (classification=PII) plus a role (analyst), which is exactly the ABAC pattern Task 4.2.5 names. The encryption side is implied: S3 objects under SSE-KMS decrypt only through Lake Formation-governed roles that also hold kms:Decrypt on the bucket's key: the four-document set from Section 10, automated.

Case B, FINRA: a regulated surveillance stack on KMS, GuardDuty and CloudTrail

"FINRA processes approximately 6 terabytes of data and 37 billion records on an average day … On busy days, the stock markets can generate 75 billion+ records.": John Brady, VP Cyber Security/CISO, FINRA (AWS blog, accessed Oct 2026)

The examinable pattern is which control does which job: KMS encrypts the warehouse and lake objects (4.3), CloudTrail records who decrypted what (4.4), GuardDuty watches for anomalous API behaviour inside the account (4.1/4.4), and security groups keep the EMR/HBase cluster off the public internet (4.1). No single service "does security": the chain does.

Case C, Oportun: finding the PII before the regulator does

Read Oportun as the discovery-before-authorisation story: Macie tells you what is classified as sensitive (Domain 4.5), and only then can Section 7's encryption choices and Section 9's least-privilege policies name the right prefixes, tags and keys. Note the scope discipline the exam rewards: Macie discovers inside Amazon S3, it does not natively scan RDS or Redshift, and it feeds the governance loop rather than replacing it (AWS Macie docs and case material, accessed Oct 2026).

• 📚 Did you know? Oportun's speed-up is a tooling result, not a policy result: the reported −80% comes from replacing manual sampling with automated, bucket-inventoried discovery that scores each bucket for sensitivity, while the +95% is S3 discovery accuracy: not a claim about general-purpose data-loss prevention across every AWS store. The dedicated Oportun case-study link now redirects to the general case-study index, so AWS's Macie product card and the re:Invent 2022 deck remain the sources of record for both numbers (Amazon Macie case material, accessed Oct 2026).

Case D, GoDaddy: central governance over a multi-petabyte data mesh

GoDaddy is the scale answer to Case A: where IAS collapsed hundreds of rules into two tags, GoDaddy collapses permission sprawl into one governance account whose Lake Formation grants and RAM shares are the only road into the lake. Consumers never touch the raw bucket policy; they receive a resource link, so Section 10's "four documents that must agree" are maintained once, centrally, instead of once per consumer account.

• 📚 Did you know? The mesh's headline counts, 2,000+ data products, multiple petabytes, hundreds of accounts, are exactly why hand-written per-resource ACLs fail: at that width, authorisation stops being a list of principals and becomes a catalogue of grants over shares (AWS RAM share → resource link → Lake Formation permission), which is the same shape as the exam's cross-account KMS question but applied to tables rather than keys. AWS's 2026 answer for that shape is Lake Formation sharing v5, where one RAM share carries unlimited tables (AWS Big Data Blog, accessed Oct 2026; AWS What's New, 2026-02-11, accessed Oct 2026).

• 📚 Did you know? Not one of these four customer stories quotes a KMS key policy or an IAM policy: AWS publishes the outcomes and leaves the wiring to you. That is deliberate exam design: DEA-C01 never asks you to reproduce FINRA's policy JSON, it asks you to recognise that a regulator's audit requirement implies CloudTrail on every KMS action, that "hundreds of rules" implies tag-based access control, and that "2,000+ data products across hundreds of accounts" implies one governance account plus RAM shares (AWS-published case material, accessed Oct 2026).

---

Practice Questions

Comparative Verdict: how this topic compares on exam day

• Versus another cloud: DEA-C01 tests AWS services only: nothing asks you to compare KMS with a competitor's KMS or IAM with another provider's IAM. Answer with an AWS service, an AWS-documented default or an AWS price as of Oct 2026; any option pivoting to an unverified third-party benchmark or a non-AWS key system is out of scope by construction.
• Versus self-managed / on-premises key management: the AWS answer is managed, auditable, per-use over bespoke HSM ops: a CMK with CloudTrail-per-key events beats a hand-rolled key server, because rotation, grants and cross-account sharing are API-shaped and reviewable. An option describing keys you patch yourself is the distractor, not the bonus.
• Versus another AWS service: pick by control surface, not prestige: CMK when you need key-policy control, per-key audit or cross-account sharing; AWS owned (SSE-S3, DynamoDB default) when zero key overhead is acceptable; Secrets Manager when rotation or cross-account is required, Parameter Store for cheap static config; gateway endpoint for VPC-only S3/DynamoDB egress, PrivateLink when on-premises or cross-Region clients must connect. Encryption at rest never implies encryption in transit (Redshift proves it), and result encryption never implies table encryption (Athena proves it).
• Versus a manual, human process: IAS's "hundreds of rules → two" and FINRA's KMS+CloudTrail audit chain (both customer-claimed, accessed Oct 2026) show why the exam prefers tag-based access control and automated encryption over human review: humans triage findings, they do not decrypt or authorise at pipeline speed. Treat every published number as "the customer achieved", never as "AWS guarantees".