Data Engineering for AI on AWS

Models do not fail in training. They fail because the table they were trained on had a column that leaked the target, because a 3 TB CSV was scanned instead of 0.25 TB of Parquet, because a labeling job went out to Mechanical Turk without declaring that the images were free of personally identifiable information, or because a feature group served millisecond reads online while the offline copy was still buffering into Amazon S3. Data engineering is the stage where all four of those mistakes are cheap to fix, and the AIF-C01 exam knows it: Domain 1 (AI/ML Foundations) is weighted 20%, Domain 2 (Generative AI) 24%, Domain 3 (Foundation Model Applications) 28%, Domain 4 (Responsible AI) 14% and Domain 5 (Security, Compliance and Governance) 14%, and this lesson sits across Task 1.3 (ML lifecycle), Task 3.3 (fine-tuning) and Domain 5's secure-data-engineering statements.

By the end of this lesson you will be able to:

• draw the end-to-end data pipeline for an AI project and name the AWS service at each hop;
• choose between CSV, Parquet and JSONL and predict the cost consequence of that choice in Athena;
• pick the right processing engine, Glue vs. EMR vs. Athena vs. a manual script, and defend it;
• apply Lake Formation's two-layer authorization model and name the permission that underlying S3 access additionally requires;
• configure a Ground Truth labeling job end to end, including the PII declaration that makes or breaks it;
• size an automated labeling run (1,250 floor, 5,000 recommendation, 20%/10% validation split);
• explain the Feature Store online and offline stores, their partitioning rules and the 15-minute write buffer;
• use Macie to discover PII in S3 and predict the severity of a finding;
• prepare a .jsonl fine-tuning dataset for Amazon Bedrock and check it against the published per-model caps;
• answer ten exam-style questions in the official practice-question voice.

---

1. Where this lesson sits in the exam

1.1 The task statements you are studying for

The AIF-C01 Exam Guide is the contract. Three statements make this lesson examinable:

1.2 The tool-per-task matrix

This table is the single most reusable artifact of the lesson. Learn the left column (the task) and the middle column (the primary tool); the right column is where exam distractors live.

• 📚 Did you know? The Glue Data Catalog is Hive-compatible, which is why Athena, EMR and the Feature Store's Glue table format can all read the same table definition. AWS's own Big Data blog (2024) documents that Glue crawlers detect schemas across Apache Iceberg, Apache Hudi and Apache Delta Lake, so "which table format" is now a governance decision, not a cataloging limitation.

---

2. The data pipeline end to end

2.1 One flowchart, eleven hops

2.2 Why the order matters

Each hop exists to remove one class of failure:

---

3. Storage and data formats: CSV vs. Parquet vs. JSONL

3.1 The comparison that decides your Athena bill

3.2 The numbers behind the defaults

Example 1, the format decision, with AWS's own numbers. Take a 3 TB uncompressed four-column text table in S3. A query that reads one column scans all 3 TB: 3 TB × $5/TB = $15.00. Apply GZIP at 3:1 and the same table is 1 TB → $5.00. Convert to Parquet and the query reads only the referenced column: 1 TB ÷ 4 = 0.25 TB × $5/TB = $1.25. That is 12× cheaper than the baseline, 3× from compression and 4× from column pruning. AWS publishes this exact worked example on its Athena pricing page.

• 📚 Did you know? Glue's groupFiles grouping only switches itself on automatically when the input exceeds 50,000 files. Below that threshold you must set "groupFiles": "inPartition" yourself: otherwise a directory of small files becomes a directory of many tiny Parquet files, which is exactly the small-file problem Athena charges you to read.

---

4. Catalog and governance: Glue Data Catalog and Lake Formation

4.1 Two permission families, not one

AWS Lake Formation documents two families of permission: permissions on the Data Catalog (databases and tables) and permissions on the underlying data and data locations (S3 prefixes). Three rules follow, and all three are examinable:

1. A principal needs both Lake Formation grants and IAM permissions;
2. underlying read/write access additionally requires lakeformation:GetDataAccess;
3. an IAM-only setup exists solely by granting Super to IAMAllowedPrincipals.

4.2 Table permissions

4.3 What replaced Governed Tables

Example 2, column and row filtering in practice. An HR analytics table has columns employee_id, department, salary_band and ssn. A Lake Formation SELECT grant with a column filter can exclude ssn entirely for the analyst role, while a row filter (DataCellsFilter) can restrict rows to department = 'Ops'. Neither is possible with a plain S3 prefix grant, which is precisely the "data access control" and "data integrity" language of Domain 5.

• 📚 Did you know? Lake Formation's SELECT permission is the only table permission documented to support row- and cell-level data filters. Everything else in the table-permission list (INSERT, DELETE, ALTER, DROP, DESCRIBE) is all-or-nothing at the table or column level.

---

5. Processing engines: Glue, EMR and Athena

5.1 Billing and scale facts

5.2 Glue worker sizes

Example 3: right-sizing Glue. AWS's Prescriptive Guidance Parquet-conversion pattern runs 2 workers: the documented minimum: instead of the console default 10 × G.1X, which is 5× fewer allocated DPUs for the same conversion job. A Python shell job configured at 0.0625 DPU (1 GB) instead of 1 DPU (16 GB) uses 16× less memory and bills proportionally. The exam rewards the pattern "start at the minimum, scale on evidence", not the console default.

5.3 Athena pricing rules

Example 4: the CTAS conversion loop. A team keeps a raw CSV table, runs CTAS (CREATE TABLE … AS SELECT) to write Parquet with Snappy compression partitioned by year, lands at 1.2 GB, and points the downstream dashboard at the new table. The query scans only the referenced columns of one partition: cheaper and faster, with no application change. AWS documents this as the standard ETL pattern (CTAS/INSERT INTO).

5.4 Choosing the engine

---

6. Data quality and preparation: Data Wrangler and DataBrew

6.1 SageMaker Data Wrangler (now inside SageMaker Canvas)

Example 5, the leakage catch. A classifier scores 99% on training data and 71% on held-out validation. The Data Wrangler Insights report flags a column derived from the target variable. The diagnosis is target leakage plus overfitting: remove or derive-safe the leaked column, then reduce overfitting with regularization or more representative data. Training longer makes it worse, and deleting the validation set only hides it, the exam's distractors propose exactly those two wrong moves.

6.2 AWS Glue DataBrew rulesets and profile jobs

The distinction the exam tests: DataBrew is the rules and profiling answer (a ruleset, a profile job, a JSON report, an event), while Data Wrangler is the visual flow and transform answer (300+ transforms, an Insights Report that flags leakage, exports to the Feature Store). Both are managed: neither asks you to provision a Spark cluster yourself.

• 📚 Did you know? DataBrew column rules reject complex types: you can write a rule for a string, number or boolean column, but not for a nested struct. That is why deep JSON validation belongs in a Glue job or a Data Wrangler flow, and why the ruleset document is deliberately shallow by design.

---

7. Human labeling: Ground Truth and A2I

7.1 The twelve-step labeling workflow

7.2 The numbers

Example 6: active-learning validation split math. 4,000 objects (< 5,000) → validation = 20% = 800 objects, leaving 3,200 in the loop. 20,000 objects → validation = 10% = 2,000. A job with 800 objects is rejected outright (below the 1,250 floor); 5,000 is the strongly recommended floor. Note what the arithmetic does not say: 4,000 objects is legal but leaves only 3,200 for learning, while 5,000 costs 500 validation objects and still clears the floor with margin.

Example 7, the bounding-box budget. In AWS's official object-detection example notebook, one human bounding-box label costs $0.08 + 5 ×$0.036 = $0.26, and one auto-label costs $0.08. So 100 images labeled by humans run about $26, while 1,000 images with auto-labeling land near $200 end to end in roughly 6 hours of total runtime (about 167 images/hour, a derived figure from a demo that also includes training and inference overhead, not an AWS SLA). Treat the per-object price tiers you may have seen as historical: AWS now markets Ground Truth+ without a public per-object table.

7.3 Workflow diagram

The A2I distinction: Amazon A2I is for human review of model predictions, created with CreateFlowDefinition and StartHumanLoop. It has no annotation consolidation, so it cannot be used to merge multiple worker votes into a consensus label; that capability belongs to Ground Truth's AnnotationConsolidationLambdaArn.

---

8. SageMaker Feature Store: online and offline

8.1 The two stores

8.2 Write semantics and capacity modes

• PutRecord data is buffered, batched and written into Amazon S3 within 15 minutes;
• online tiers: Standard (default) and InMemory ("powered by Amazon ElastiCache (Redis OSS)"), both online-only: no online→offline replication and no customer-managed KMS key for InMemory;
• capacity modes: ON_DEMAND and PROVISIONED (exceeding provisioned capacity throttles); a group may be switched to on-demand once per 24 hours, and PROVISIONED is allowed only for offline-only or Standard groups;
• InMemory takes roughly 10 to 15 minutes to provision.

Example 8, Feature Store ingestion math. AWS's example scale is "large batches (1 million rows of data or more)". Because PutRecord output flushes to S3 within 15 minutes, a training job started immediately after ingestion may wait up to 15 minutes for the offline copy, while GetRecord serves the online copy sooner. Over one day, a Glue-format offline store lands 24 hourly partitions, while an Iceberg-format store lands 1 daily partition, neither is configurable.

---

9. Privacy, PII and secure data engineering with Amazon Macie

9.1 How Macie finds sensitive data

9.2 Severity: the table you must be able to predict

Multi-type findings take the highest severity among the types detected.

Example 9: the Macie triage queue. An object with 1 full name → Low; 50 names → Medium; 150 names → High. A separate object with 1 SSN → High immediately, because SSN-like identifiers are High at any count. One object containing 10 names (Medium) and 5 passport numbers (High) → the finding severity is High, because multi-type findings take the highest severity. Queues are therefore prioritized by identifier sensitivity, not by raw volume.

The Domain 5 pairing: Amazon Macie is the AWS-named service for PII/PHI/credential discovery in S3 and appears in the AIF-C01 Domain 5 service list. Pair it with Amazon Comprehend for PII detected inside text at inference time, and with Lake Formation column and row filters (section 4) for access control. Macie discovers what is there; Lake Formation decides who may see it; KMS decides how it is encrypted.

• 📚 Did you know? Macie's automated discovery does not scan every object continuously; it evaluates the S3 inventory daily with sampling, and sensitive-data findings are retained for 90 days. If your compliance story needs a full forensic history, export findings to S3 or Security Hub; do not assume they live in Macie forever.

---

10. Preparing training data for foundation-model fine-tuning

10.1 The format is not negotiable

Amazon Bedrock states it directly: "You create .jsonl files, where each line is a JSON object corresponding to a record." Non-conversational records carry prompt and completion; conversational records carry a messages array with alternating user and assistant turns. AWS also publishes an estimation rule: "Use approximately 6 characters per token to estimate … dataset size." The files live in S3 and are referenced by trainingDataConfig.s3Uri and validationDataConfig.s3Uri in CreateModelCustomizationJob.

10.2 Published dataset limits

A dash means not published in that table: never guess a value AWS has not published.

10.3 Amazon Nova specifics

• Understanding fine-tuning: 8 to 20k samples, 32k context, up to 10 images per sample (each ≤ 10 MB), up to 1 video per sample ≤ 90 seconds and ≤ 50 MB.
• Reinforcement fine-tuning: JSONL in OpenAI chat-completion format with a messages array plus reference_answer, minimum 100 records, one prompt per line, and Bedrock automatically validates the format.

Example 10: sizing a Claude 3 Haiku dataset. The floor is 32 records, the ceiling is 10,000 training records (plus 1,000 validation), the token cap is 32,000, and the training file cap is 10 GB. Using AWS's ~6 characters per token rule, a 4,096-token sample is roughly 24,576 characters (derived), so the ~1 GB file cap binds far later than the record cap for short prompts. Llama 3.1/3.3 binds earlier: 100 to 10,000 records and 16,000 in+out tokens. Tokenization varies, so treat the 6-chars-per-token figure as an estimator, not a guarantee.

Example 11: curation, governance and representativeness. The exam guide's wording for Task 3.3 is specific: datasets should show inclusivity, diversity, curated data sources and balanced datasets, and preparation includes data curation, governance, size, labeling, representativeness and RLHF. Concretely: deduplicate before you count records (a 10,000-line file with 3,000 duplicates is 7,000 effective records, not 10,000); balance classes before you split; hold out a validation file inside the 1,000-record ceilings above; and remove PII with Macie before the file reaches trainingDataConfig.s3Uri.

---

11. Conflicting, unverified and non-examinable facts

Keep these out of your memorization set. Nothing here should appear as a confident exam answer.

Primary sources for this lesson: the AIF-C01 Exam Guide; the SageMaker developer guide pages feature-store, feature-store-offline, feature-store-storage-configurations-online-store, sms, sms-create-labeling-job-api, sms-automated-labeling, data-wrangler, data-wrangler-transform; the SageMaker API reference API_CreateFeatureGroup, API_HumanTaskConfig; the Glue developer guide Parquet and S3-connections pages; the Athena user guide columnar-storage, ctas-insert-into-etl; the Lake Formation permissions overview and reference; the Macie user guide data-classification, managed-data-identifiers, findings-severity; the Bedrock user guide model-customization-prepare, model-customization-submit, rft-prepare-data; the Nova user guide fine-tune-prepare-data-understanding; and the AWS pricing pages for Athena plus the EMR vs. EMR Serverless comparison.

---

12. Comparative verdict and summary tables

12.1 Quality, labeling and governance at a glance

12.2 The five examinable "defaults"

• 📚 Did you know? The Athena billing floor is 10 MB per query, rounded to the nearest MB. DDL statements and failed queries are free, but canceled queries are billed, so "run it, watch it, and hit cancel" is not a free strategy. The cheapest safe habit is LIMIT plus a partition filter before you commit to a full scan.

12.3 2025 to 2026 Updates

The verified 2025 to 2026 change set touches this lesson in exactly three places: the SageMaker quality and labeling services moved into maintenance, enterprise search moved to Bedrock, and the exam guide itself was revised. Nothing here is speculation: every row carries the AWS document it came from.

Lifecycle vocabulary (AWS General Reference, "Service lifecycle", 24 Sep 2026): Maintenance = no new customers, no new features, still supported · Sunset = planned end of operations, typically a 12-month horizon · Full Shutdown = removed from the portfolio. Most exam distractors get this wrong by turning "maintenance" into "discontinued": the service still runs for existing customers.

What did not change: nothing in the verified 2025 to 2026 update set alters the Glue, Athena, Lake Formation, Feature Store or Macie figures memorized in this lesson: Glue's 10 workers × G.1X default (documented minimum 2), Athena's $5 per TB with a 10 MB floor, the Feature Store 15-minute write buffer, Macie's 90-day finding retention and Lake Formation's two-layer model all stand as written. The one genuinely new storage primitive worth knowing is Amazon S3 Vectors, launched at re:Invent 2025: Sun Finance stores its fraud-similarity embeddings there (section 13), and Bedrock Knowledge Bases now accepts it as a vector store.

• 📚 Did you know? AWS states that exam-guide updates appear on the exam about one month after publication. Exam guide v1.0 shipped 26 Mar 2026 and v1.1 shipped 30 Apr 2026, so the seven new objectives (agentic AI and MCP, context engineering, token-based pricing, prompt versioning, hallucination detection and grounding, business-alignment metrics, traditional ML vs. FMs) were fair game from roughly late May 2026, and the exam is still called AIF-C01.

---

13. Real-World Case Studies

Sections 1 to 12 take their numbers from AWS documentation. This section adds the other half of the evidence: what AWS customers published about running those same services in production. Every figure below is an AWS-published customer claim (unaudited, and "up to" figures are ceilings), retrieved 6 October 2026 from aws.amazon.com/solutions/case-studies/* and aws.amazon.com/blogs/machine-learning/*.

13.1 Five production pipelines: services, numbers, sources

Case file 1: Anthem: prebuilt OCR first, humans in the tail. Claim forms arrived as scans and images, and keying them by hand took 20 minutes per claim at a cost AWS describes in the millions. Anthem chose the prebuilt tier that section 1.2 calls the default: Amazon Textract performs the OCR plus table and form extraction, and a classification step routes each result. The published outcome is 80 % of the workflow automated with 90 %+ as the target across thousands of claims per day. The exam-relevant consequence is the tail: the remaining 10 to 20 % still needs a human exception path, which is exactly the problem Amazon A2I (section 7) exists to solve.

Case file 2, RareJob: the one customer running this lesson's catalog stack. The PROGOS speaking-test scorer first trained on a developer's local PC (one model per developer) and then on EC2 and ECS, and both bottlenecked on queueing rather than on model quality. The shipped architecture puts AWS Glue and Amazon Athena in front of SageMaker managed spot training: data preparation and exploratory SQL run in the managed services while parallel spot jobs replace the queue. The published result is −25 % training time, >10× developer efficiency, 100 hours per month recovered and scores returned in 2 to 3 minutes. This is the section 5 Comparative Verdict observed in production, Glue and Athena are the feeding layer, never the model.

Case file 3: Sun Finance: the documented failed prototype. 60 % of microloan applications needed manual review, taking anywhere from 10 minutes to 20 hours each. The first attempt: sending ID photos straight to Claude Sonnet 4 for JSON extraction: scored only 61.8 % overall and 43 % on ID numbers on a 585-image evaluation set, and AWS rejected it because the model's privacy protections block direct PII extraction. The shipped pipeline separates the concerns this lesson keeps separate: Textract for OCR, Rekognition as fallback and face handling, the LLM only for structuring the already-extracted text, explicit validation rules, and Titan Multimodal Embeddings in Amazon S3 Vectors for fraud-similarity search. Accuracy rose 79.73 % → 90.80 % (ID number 74.32 % → 89.40 %, document type 78.43 % → 96.40 %), cost per document fell 91 %, and processing went from 20 hours to under 5 seconds.

13.2 Before and after, side by side

• 📚 Did you know? AWS's Generative AI Innovation Center publishes the only project-outcome rate in its whole case corpus: 65 % of its generative AI projects reached production in 2025 (some in as little as 45 days), out of more than 1,000 implementations, assessed with AWS's Five V's framework: Value → Visualize → Validate → Verify → Venture. The other 35 % did not ship, so plan for iteration rather than for a straight line from pilot to production.

---

14. Practice Questions