ML Algorithms and Model Evaluation on AWS

An algorithm is easy to name and hard to justify. The AIF-C01 exam almost never asks "which algorithm exists"; it asks "the business needs X, the labels look like Y, and AWS gives you Z; justify the pair and defend the metric." Task 2.1 asks you to select appropriate ML techniques for specific use cases (regression, classification, clustering), Task 2.2 asks you to describe supervised, unsupervised and reinforcement learning, and Task 1.4 asks you to describe model performance metrics (accuracy, AUC, F1) and business metrics (cost per user, development cost, customer feedback, ROI). This lesson turns those three sentences into an operable decision system.

By the end of this lesson you will be able to:

• choose between supervised, unsupervised and reinforcement learning from the shape of the data alone;
• map sixteen algorithm families to their use case, their AWS route and their key hyperparameter;
• state which algorithms are built-in to SageMaker, and which one famously is not;
• compute accuracy, precision, recall, F1, MAE, MSE, RMSE and R² from a confusion matrix or a residual set without a calculator;
• pick the right split (70/30, 60/20/20, holdout 20 to 30%, k-fold 5 to 10) and defend it;
• diagnose underfitting vs. overfitting and apply the AWS-prescribed remedy in the right direction;
• attack class imbalance with the three levers AWS documents (reweighting, resampling, metric swap);
• answer ten exam-style questions written in the official voice.

---

1. The three learning paradigms and how AWS implements them

1.1 Selection guide: read the data, not the buzzword

The exam decides the paradigm for you with one clue: what does the training data contain? Labeled input/output pairs point to supervised learning; features with no answers point to unsupervised; a reward signal coming back from an environment points to reinforcement learning.

The distinction that exam items lean on hardest: unsupervised learning never sees a validation set with labels. SageMaker's own k-means documentation says it "doesn't use a validation dataset" because there is nothing to validate against; it takes a test channel and reports distance-based metrics only. A question that offers validation:f1 for a clustering job is testing whether you noticed.

1.2 What each paradigm looks like as a workflow

1.3 Reinforcement learning on SageMaker: the specific facts

Reinforcement learning (RL) is the paradigm the exam tests by name recognition rather than by computation. The verified AWS statements are:

• SageMaker AI supports RL in TensorFlow and Apache MXNet;
• it supports the Intel Coach and Ray RLlib toolkits;
• the SageMaker FAQ lists DQN, PPO, A3C and many more algorithms;
• the open RL container repository ships Vowpal Wabbit (VW) images;
• jobs were launched with the RLEstimator.

If a question describes an agent learning a policy to maximize cumulative reward in a simulation, the answer is SageMaker RL with Coach or Ray RLlib: never k-means, never Autopilot, never PCA.

• 📚 Did you know? SageMaker's RL support is deliberately narrow: TensorFlow and MXNet only for frameworks. If an option offers "SageMaker RL with PyTorch", it is a distractor: the documented framework list does not include PyTorch, even though PyTorch is fully supported for supervised deep learning through the pre-built deep learning containers.

Check your paradigm selection:

---

2. The algorithm → use case → AWS route matrix

This is the single most examinable table in Domain 2. Read it as three questions answered at once: what family, what business case, where does it run on AWS.

2.1 The full matrix

2.2 From task type to algorithm family

2.3 The built-in lists you must be able to reproduce

AWS organizes its built-in algorithms into supervised, unsupervised and reinforcement groups. The tabular supervised set is the one the exam reuses most:

Built-in tabular (classification or regression): AutoGluon-Tabular, CatBoost, Factorization Machines, k-NN, LightGBM, Linear Learner, TabTransformer, XGBoost.

The Linear Learner documentation is explicit about what it is: it "learns a linear function for regression or a linear threshold function for classification"; which makes it the built-in answer whenever a question says logistic regression or linear regression.

Built-in unsupervised: PCA (dimensionality reduction), K-Means (clustering), Random Cut Forest (anomaly detection), IP Insights (IP-address anomaly), LDA and Neural Topic Model (topic modeling).

• 📚 Did you know? AWS publishes algorithm lists, never a count. Any option that begins "SageMaker has exactly N built-in algorithms" is an invented number: the lists change as algorithms are added (AutoGluon-Tabular and TabTransformer are recent additions to the tabular set) and removed. Learn the membership of the lists, not a total.

---

3. The supervised families: linear, trees, boosting and neural nets

3.1 Linear Learner: the regression and classification workhorse

Linear Learner is the answer to "which built-in does linear or logistic regression?" and it carries three examinable properties:

That tuning rule is a free answer whenever an option says "tune against the training metric". The documented remedy is the opposite: tune on the validation metric.

3.2 Gradient boosting: three built-ins plus an AutoML wrapper

Gradient boosting is represented on SageMaker by three built-in algorithms plus a stack-ensembling option:

XGBoost details the exam reuses:

• the container covers XGBoost 1.0, 1.2, 1.3, 1.5, 1.7 and 3.0;
• num_round is required (plus num_class for multi:softmax and multi:softprob);
• the default objective is reg:squarederror, so an unconfigured XGBoost job is a regression job;
• GPU training is enabled with tree_method=gpu_hist;
• the highest-impact hyperparameters for tuning are alpha, min_child_weight, subsample, eta and num_round.

3.3 The XGBoost metric menu: twelve objectives with directions

Automatic Model Tuning needs an objective and a direction. XGBoost exposes twelve validation metrics, and the direction is part of the answer:

Two traps live in this table. First, ranking metrics (map, ndcg) are Maximize while every error metric is Minimize: selecting validation:rmse with direction Maximize would tune the model toward the worst result. Second, merror and mlogloss are the multiclass siblings of error and logloss; confusing them in an option is a common distractor pattern.

3.4 Trees, forests and neural networks: the routes that need code

Object2Vec deserves a specific memory hook: it uses cross-entropy for classification and mean squared error (MSE) for regression: the same algorithm changes its loss function with the task, which is exactly the kind of detail an option can flip to make a wrong answer.

---

4. The unsupervised families: clustering, reduction, anomaly and topics

4.1 K-Means: no validation set exists

K-Means is the exam's favourite way to test whether you understand that unsupervised learning has no labeled validation set. The verified documentation points:

• "Because it's unsupervised, it doesn't use a validation dataset… But it does take a test dataset."
• Objective metrics: test:msd (mean squared distances) and test:ssd: both Minimize;
• inference returns closest_cluster and distance_to_cluster;
• hyperparameters: k, init_method (random or k-means++), mini_batch_size.

Note the direction: distance metrics are minimized, because a smaller squared distance means tighter clusters. Every distance-based clustering objective in SageMaker points the same way.

4.2 Silhouette: a real metric that is not a SageMaker metric

The silhouette coefficient ranges from −1 to 1; a value close to 1 means the points are tight inside their own cluster and far from the other clusters. AWS's guidance is to pair silhouette with inertia, and to be clear that SageMaker does not emit it: you compute it yourself outside the training job. A question that offers "select validation:silhouette as the tuning objective" is testing precisely that boundary.

4.3 PCA: three required hyperparameters

PCA is built in, and its configuration is examinable because it is short:

PCA outputs the mean, the eigenvector matrix v and the singular values s; which is how you later reconstruct or project data with the components you kept.

4.4 Anomaly detection and topic modeling

• 📚 Did you know? test:msd and test:ssd are not interchangeable synonyms: msd is the mean squared distance and ssd is the sum squared distance to the centroid. On a test set of n rows, ssd = msd × n, so ssd grows with dataset size while msd does not. Both are minimized, and both are valid clustering objectives, but only msd is comparable across test sets of different sizes.

---

5. Four routes to an algorithm

AWS documents exactly four ways to get an algorithm running, and the exam tests whether you can name them in order of increasing effort.

5.1 The decision flow

5.2 BYOC: the three non-negotiable container rules

If a question pushes you to Route 4, three documented facts decide the answer:

1. the sagemaker-training toolkit package is mandatory;
2. ENV SAGEMAKER_PROGRAM train.py is the only required environment variable; it names the entry point SageMaker invokes;
3. /opt/ml and all of its subdirectories are reserved by SageMaker training, so inputs, model artifacts and output must follow that layout.

Two more container rules appear as distractors:

• SageMaker overrides the default CMD of your image by appending the train argument;
• GPU images must be nvidia-docker compatible: bundle the CUDA toolkit, but do not bundle NVIDIA drivers: the host supplies them.

5.3 Container support windows are examinable dates

AWS publishes an explicit GA → end-of-patch schedule for its algorithm containers. The pattern worth memorising is the shape: roughly one year of patch support after general availability, which is why an exam item dated in 2026 can legitimately reference a 2025 release reaching end of patch in the same year.

---

6. Evaluation metrics: formulas, directions and when to use them

6.1 The formula sheet

AWS's own summary of the precision/recall relationship is worth memorising verbatim: precision cuts false positives, recall cuts false negatives, and F1 is the harmonic mean of the two.

6.2 Choosing a metric from the confusion matrix

6.3 Worked confusion matrix #1: balanced churn data

A churn model is evaluated on 200 customers: TP = 80, FP = 20, FN = 10, TN = 90.

$$\text{Accuracy} = \frac{TP+TN}{N} = \frac{80+90}{200} = \frac{170}{200} = 0.85$$ $$\text{Precision} = \frac{TP}{TP+FP} = \frac{80}{100} = 0.80$$ $$\text{Recall} = \frac{TP}{TP+FN} = \frac{80}{90} = 0.889$$ $$F1 = \frac{2TP}{2TP+FP+FN} = \frac{160}{190} = 0.842$$

Reading: the model catches about 89% of churners, but 20 of its 100 alerts are wrong. Accuracy of 0.85 is respectable here because the classes are reasonably balanced (90 positives, 110 negatives).

6.4 Worked confusion matrix #2: the 1% fraud trap

A fraud model is evaluated on 10,000 transactions of which only 100 are fraudulent: TP = 85, FP = 40, FN = 15, TN = 9,860.

$$\text{Accuracy} = \frac{85+9860}{10000} = \frac{9945}{10000} = 99.45\%$$

Yet a model that always predicts "not fraud" already scores:

$$\frac{9900}{10000} = 99.0\%$$

So 99.45% accuracy buys 0.45 percentage points over a useless baseline. The metrics that matter:

$$\text{Precision} = \frac{85}{85+40} = \frac{85}{125} = 0.68$$ $$\text{Recall} = \frac{85}{85+15} = \frac{85}{100} = 0.85$$ $$F1 = \frac{2 \times 85}{170+40+15} = \frac{170}{225} = 0.756$$

Reading: 32% of the fraud alerts are false alarms, the model misses 15 of 100 frauds, and the headline accuracy is close to meaningless. This is the single most repeated evaluation pattern in AWS's own fraud-detection material.

---

7. Data splits, cross-validation and A/B testing

7.1 The published split standards

AWS's stated reason for holding data out is a sentence worth reproducing in an exam answer: evaluating on training data "rewards models that can 'remember' the training data, as opposed to generalizing from it."

7.2 Split and cross-validation arithmetic on 100,000 rows

7.3 Autopilot cross-validation: the 50,000-instance threshold

With 40,000 training instances (≤ 50,000, so HPO mode applies automatic k-fold with k = 5):

1. the automatic 80/20 split gives 32,000 train / 8,000 validation;
2. each of the 5 folds trains on 4/5 × 32,000 = 25,600 instances;
3. each fold validates on 6,400 instances;
4. the five validation scores are averaged into the reported objective.

A 60,000-instance dataset exceeds the ≤ 50,000 threshold, so automatic k-fold in HPO mode does not trigger; while ensemble mode runs cross-validation regardless of size alongside its automatic 80/20 split.

7.4 How the loop fits together

Interactive check: the formulas you must not have to look up:

---

8. Underfitting, overfitting and the tuning controls that fix them

8.1 The diagnosis

AWS's official signature of overfitting is short: high training accuracy plus low testing accuracy. The remedy direction matters as much as the remedy: overfitting gets more regularization, underfitting gets less. Reversing them is a classic distractor.

8.2 The tuning controls that prevent overfitting mechanically

Four documented controls do the work that a human would otherwise do by eye:

1. Tune against a validation metric, not a training metric. Linear Learner's documentation states this as a recommendation, not a preference.
2. Early stopping. Each epoch, the objective metric is compared with the median of running averages of prior jobs; jobs performing worse stop early: AWS says this "helps you avoid overfitting your model". The setting is early_stopping_type='Auto'.
3. Hyperband also terminates under-performing jobs early, allocating the budget to promising configurations.
4. Regularization direction: increase it for overfitting, decrease it for underfitting.

8.3 Order the diagnostic sequence

• 📚 Did you know? Early stopping in SageMaker Automatic Model Tuning does not compare your job to a fixed target; it compares each epoch's objective value with the median of the running averages of all previous jobs. That makes the threshold adaptive: an unusually strong early job raises the bar for everything that follows, and a weak early job lowers it. The support list for early stopping carries a "current as of December 13, 2018" stamp, so treat any claim about which algorithms support it as historical.

---

9. Imbalanced data: three levers and the numbers behind them

AWS documents three independent levers for imbalance. A strong answer uses at least two of them together.

9.1 The three levers

9.2 Class-weight arithmetic

Case A: a 98% / 2% binary split. To make the minority class count as much as the majority:

$$0.02 \times w = 0.98 \implies w = \frac{0.98}{0.02} = 49$$

So Linear Learner's positive_example_weight_mult='balanced' on a 98/2 dataset produces a positive weight of 49.

Case B: AWS's own fraud dataset. With 284,807 examples of which 492 are fraudulent (0.173%), AWS's XGBoost variant uses the square-root rule rather than the raw ratio:

$$\text{scale\_pos\_weight} = \sqrt{\frac{284{,}315}{492}} = \sqrt{577.9} \approx 24$$

The raw ratio would be 577.9; the square-root damping keeps training stable while still telling the model that fraud matters. AWS additionally reports that raising Cohen's Kappa above 0.8 would be "generally very favorable" on that problem.

9.3 What weighting costs you: the documented trade-off

Weighting is not free. AWS's own multiclass example shows both sides:

The lesson the exam wants: you traded 13 points of accuracy for a model that actually recognizes the rare class. On an imbalanced problem, that trade is usually correct, but it must be a decision, not a side effect.

9.4 The metric swap, side by side

---

10. Evaluation by task type: the lookup table

10.1 Foundation-model metrics: what Clarify actually computes

Two sampling numbers are examinable: Clarify evaluates 100 records by default for most tasks and 300 for factual knowledge. And one bias metric name is unique to Clarify: CI (Class Imbalance).

• 📚 Did you know? Balanced classification accuracy has an official AWS gloss: "the sum of recall and the true negative rate divided by 2." For a multiclass problem it degrades gracefully into the mean per-class recall; which is exactly why it survives class imbalance while plain accuracy does not: every class contributes equally regardless of how many rows it owns.

---

11. Putting the three axes together

The exam almost always forces a choice across three axes at once: the algorithm family, the task type, and the AWS tool that implements the pair. Getting two of the three right is not enough.

---

12. Worked examples with numbers

Worked example 1: regression metrics from scratch. True values $y = [100, 120, 140, 160, 180]$, predictions $\hat{y} = [110, 115, 150, 155, 190]$, so the errors are $[10, -5, 10, -5, 10]$.

• $\text{MAE} = (10+5+10+5+10)/5 = 40/5 = \mathbf{8.0}$
• $\text{MSE} = (100+25+100+25+100)/5 = 350/5 = \mathbf{70}$
• $\text{RMSE} = \sqrt{70} \approx \mathbf{8.37}$: back in the target's own units
• $\bar{y} = 140$, so $SS_{tot} = 1600+400+0+400+1600 = 4000$, and $R^2 = 1 - 350/4000 = \mathbf{0.9125}$

The pattern: MSE punishes the two 10-point misses four times as hard as the 5-point misses (100 vs. 25), and RMSE undoes that squaring so the number can be quoted in dollars or days.

Worked example 2: split arithmetic on 100,000 rows. The 70/30 default gives 70,000 / 30,000; the recommended 60/20/20 gives 60,000 / 20,000 / 20,000; SageMaker's 20 to 30% holdout reserves 20,000 to 30,000 rows for test; k = 10 gives 90,000 train / 10,000 validation per fold across 10 models whose metrics you average.

Worked example 3: Autopilot cross-validation math. Dataset of 40,000 instances (≤ 50,000 → automatic k-fold, k = 5, in HPO mode): 80/20 gives 32,000 train / 8,000 validation; each of the 5 folds trains on 4/5 × 32,000 = 25,600 and validates on 6,400; the five scores are averaged. The same recipe on 60,000 instances would exceed the threshold and lose automatic k-fold in HPO mode.

Worked example 4: class-weight math, both variants. 98% negative / 2% positive ⇒ $0.02 \times w = 0.98 \Rightarrow w = \mathbf{49}$. The AWS fraud variant instead uses $\text{scale\_pos\_weight} = \sqrt{284{,}315/492} = \sqrt{577.9} \approx \mathbf{24}$. Both are legitimate; the raw ratio is harsher, the square root is the documented damping for extreme skew.

Worked example 5: the imbalanced confusion matrix in full. 10,000 transactions, 100 frauds, TP = 85, FP = 40, FN = 15, TN = 9,860 ⇒ accuracy 99.45%, majority baseline 99.0%, precision 0.68, recall 0.85, F1 0.756. The decision that follows: report F1 or AUC, not accuracy, and state that 40 of 125 alerts (32%) are false alarms.

Worked example 6, balancing a 600-prediction evaluation set. A model produces TP = 480, FP = 60, FN = 20, TN = 40 on 1,000 rows. Accuracy = $880/1000 = \mathbf{0.88}$; precision = $480/540 = \mathbf{0.889}$; recall = $480/500 = \mathbf{0.96}$; F1 = $2(0.889)(0.96)/(0.889+0.96) = 1.707/1.849 = \mathbf{0.923}$. True negative rate = $40/100 = 0.40$, so balanced accuracy = $(0.96 + 0.40)/2 = \mathbf{0.68}$, far below the 0.88 accuracy, because the negative class is where the model fails. That gap between 0.88 and 0.68 is exactly what balanced accuracy exists to expose.

---

13. Traps, conflicts and unverified claims

13.1 Conflicting, unverified and non-examinable claims

Primary sources for this lesson: the AIF-C01 Exam Guide; the SageMaker developer guide pages algorithms, algorithms-tabular, xgboost, xgboost-tuning, linear-learner-tuning, k-means, PCA-reference, object2vec, how-it-works-model-validation, autopilot-metrics-validation, automatic-model-tuning-early-stopping, your-algorithms-training-algo-dockerfile, train-model, pre-built-containers-frameworks-deep-learning, pre-built-containers-support-policy, clarify-foundation-model-evaluate-overview and reinforcement-learning; the machine-learning developer guide pages model-fit-underfitting-vs-overfitting and evaluating_models; and the AWS Machine Learning blog posts on fraud detection, Data Wrangler balancing, Autopilot for financial services and Linear Learner multiclass classification.

---

Real-World Case Studies

Every rule in sections 1 to 13 has a customer version. The four cases below are drawn from the course case-study digest (sources restricted to aws.amazon.com/solutions/case-studies/* and aws.amazon.com/blogs/machine-learning/*), and each was chosen because it is an algorithm-selection or evaluation story rather than a marketing anecdote: somebody picked a route from section 5, picked a metric from section 6, and then measured the result on data the model had not been trained on.

Case study 1, Chronomics: the threshold is a decision, not a detail

After four months of in-house computer-vision modelling never reached its target, Chronomics moved the same COVID-test-reading task to Amazon Rekognition Custom Labels (AutoML) and shipped in 3 to 4 weeks at 96.5% accuracy and 97.9% F1, scored with DetectCustomLabels. The examinable part is what happened next: raising the confidence threshold bought precision and paid for it in coverage.

Source: AWS Machine Learning Blog, blogs/machine-learning/chronomics-detects-covid-19-test-results-with-amazon-rekognition-custom-labels (13 Dec 2022).

The lesson mirrors section 6: a single accuracy number never tells you what a threshold change costs. Chronomics did not retrain between the two rows; it changed only which predictions it was willing to sign off on, which is why "what happens to the discarded rows?" must be answered before deployment.

Case study 2, Sun Finance: evaluate every attempt, and never let one model do two jobs

Sun Finance automated KYC document extraction across nine countries. The documented attempt ladder is the cleanest evaluation narrative in the digest:

Source: AWS Machine Learning Blog, blogs/machine-learning/sun-finance-automates-id-extraction-and-fraud-detection-with-generative-ai-on-aws (30 Apr 2026); evaluated on 585 images.

Two exam habits live here. First, attempt 1 was measured and rejected: the team did not ship on vibes, it shipped on a held-out evaluation set. Second, OCR and reasoning are separate jobs: the prebuilt extraction service (Route 1) did the reading, the foundation model only structured what was already read, and validation rules caught the remainder.

Case study 3, HAYAT HOLDING: automatic tuning replaces the manual sweep

HAYAT HOLDING streams 194 sensors through OPC-UA into an AWS IoT Greengrass SiteWise Edge Gateway, then trains on SageMaker Model Training with SageMaker Automatic Model Tuning, deploys with SageMaker Model Deployment, and runs the models on-device through SageMaker Edge Manager. The documented result is $300,000 per year saved plus higher panel quality.

Source: AWS Machine Learning Blog, blogs/machine-learning/hayat-holding-uses-amazon-sagemaker-to-increase-product-quality-and-optimize-manufacturing-output-saving-300000-annually (2023).

This is section 8.2 running unattended: instead of a human sweeping hyperparameters by eye, the tuner searches the configuration space against a validation metric and early-stops weak jobs. When a question asks how a team scales tuning across many models or inputs, the documented answer is Automatic Model Tuning: not a larger spreadsheet.

Case study 4, Adobe: chunking and embeddings are hyperparameters too

Adobe improved developer documentation search with Amazon Bedrock Knowledge Bases, testing four chunking strategies (400-token with 20% overlap, 1,000-token, hierarchical, semantic) paired with Amazon Titan Text Embeddings V2 into OpenSearch Service, retrieved through the Retrieve API. The result on Adobe's own test set was +20% retrieval accuracy, and the simplest strategy, 400-token chunks with 20% overlap, was also the most accurate.

Source: AWS Machine Learning Blog, blogs/machine-learning/adobe-enhances-developer-productivity-using-amazon-bedrock-knowledge-bases (11 Jun 2025).

The evaluation discipline is identical to section 7: candidates were compared on a fixed, held-out test set, not on the queries used to build them. Retrieval quality is a model-selection problem: chunk size, overlap and embedding choice are the hyperparameters, and the metric decides.

• 📚 Did you know? AWS publishes exactly one project-outcome rate: 65% of Generative AI Innovation Center projects reached production in 2025, drawn from more than 1,000 implementations, with some shipping in as little as 45 days. The method behind it is the Five V's framework, Value → Visualize → Validate → Verify → Venture, which starts from baseline metrics (accuracy and business outcome) before a single model is trained. Source: AWS Machine Learning Blog, blogs/machine-learning/beyond-pilots-a-proven-framework-for-scaling-ai-to-production (2025). The corollary examiners like: 35% did not reach production, so evaluation is a phase, not a ceremony.

• 📚 Did you know? The human path for Chronomics' discarded 5% already has a product name: Amazon Augmented AI (A2I), generally available since 2020 with 60+ workflows, exists because "customers still say there are critical use cases where human judgment is required." A2I is how a high-threshold vision or document pipeline converts abstentions into reviewed outcomes instead of dropped rows: the same design principle as Anthem's human exception path for the claims Textract did not automate.

---

14. Practice Questions