MLOps and Scaling on AWS

A model that works in a notebook is not a system. The moment a second engineer must reproduce the training run, the moment traffic doubles at 09:00, and the moment the training data drifts six months later, you stop doing machine learning and start doing operations, and AWS gives you two distinct families of tooling for that: a pipeline / registry / CI-CD layer that decides how a model reaches production, and a scaling layer that decides how much compute serves it. The AIF-C01 exam tests whether you can name each tool precisely, quote its limits, and refuse the options that sound plausible but are documented for a different service.

By the end of this lesson you will be able to:

• place an ML practice on the AWS three-level maturity model (0, 1, 2) and separate it from the Google, Azure and AWS-blog variants of the same framework;
• read a SageMaker Pipeline as a JSON DAG and say which edges carry data (properties) versus ordering only (DependsOn);
• select the right step type for a task, including Condition, Fail, Callback, Lambda, ClarifyCheck and QualityCheck, and quote how many types exist;
• configure ParallelismConfiguration and step caching (ExpireAfter in ISO 8601, successful runs only) correctly;
• drive deployment through the Model Registry: version numbering, ModelApprovalStatus, and the four approval transitions that do or do not start CI/CD;
• build or adopt a SageMaker Project with CodePipeline / Jenkins, explain why CodeCommit templates disappeared on 28 October 2024, and name the CodeConnections tag and IAM actions required for GitHub;
• choose between Experiments (Studio Classic) and MLflow on SageMaker AI, and say where tracking data and artifacts physically live;
• configure target tracking, step scaling and scheduled actions with the correct ServiceNamespace, ResourceId and ScalableDimension, and do the ceiling-division capacity math;
• wake an async endpoint from zero with a step policy plus a HasBacklogWithoutCapacity alarm;
• pick the correct inference option from payload, latency, duration and endpoint requirements, and defend the choice in cost terms;
• wire the MLOps flywheel: Model Monitor → CloudWatch → EventBridge → StartPipelineExecution;
• defend the Comparative Verdict: managed MLOps vs DIY scripts vs manual operations.

---

1. MLOps maturity: level 0, level 1, level 2

1.1 The AWS three-level standard

AWS frames MLOps as a three-level maturity model, and the exam expects you to recognize the level from a description and to name the tooling that gets you there. The distinguishing question is never "how much ML do you do"; it is what is the deployed artifact.

1.2 Where this sits on the exam

MLOps belongs to Domain 1, Task 1.3 of the AIF-C01 guide. The exam itself is 65 questions = 50 scored + 15 unscored, with no penalty for guessing and published domain weights of 20 / 24 / 28 / 14 / 14 %. AWS publishes only the domain weights: the per-task question count is not published, so treat any per-task sizing you read elsewhere as orientation, not as a budget.

• 📚 Did you know? The 15 unscored questions on an AIF-C01 exam are indistinguishable to you while you take the test: they are interspersed with the 50 scored ones and are used to gather statistics before an item is ever scored. The practical consequence is a strategy point, not a trivia point: never skip a question thinking "this one might be unscored." Every question consumes the same time, and only the scored 50 decide your result, so the rational move is to answer all 65 to the best of your ability.

---

2. SageMaker Pipelines: the DAG contract

2.1 What a pipeline is

A SageMaker Pipeline is a directed acyclic graph expressed in JSON. Each node is a step, each edge is a dependency, and cycles are rejected, a DAG that loops is not a slower pipeline, it is an invalid definition. The definition follows the SageMaker AI Pipeline Definition JSON Schema, and it can be produced by four equivalent producers: the visual editor in Studio, the SageMaker SDK, boto3, and AWS CloudFormation. Pipeline executions are billed as the underlying jobs (processing, training, transform) plus Studio usage, the orchestration layer itself is not a separate heavyweight SKU.

Two kinds of edge exist, and confusing them is the most common architectural error in this area:

The rule of thumb: if the value flows, it is a properties reference; if only the timing matters, it is DependsOn. ParallelismConfiguration is not an ordering mechanism at all; it caps how many already-eligible steps run at once.

2.2 The pipeline DAG

This is the canonical shape the exam describes: preprocess → train → evaluate → register only if the metric passes, otherwise fail with a message.

2.3 Worked example 1: the MSE gate, end to end

A regression pipeline for an abalone-style tabular model, built with the SageMaker SDK:

1. Parameters are declared once and reused everywhere: processing_instance_count, instance_type, model_approval_status (default PendingManualApproval), input_data, batch_data, mse_threshold.
2. Process produces train_data and test_data outputs. The training step consumes them with step_process.properties.ProcessingOutputConfig.Outputs["train_data"].S3Output.S3Uri; that string is the DAG edge, not a comment.
3. Eval is a Processing step with property_files, so its output evaluation.json (containing regression_metrics.mse) is exposed to later steps as a structured property rather than as an S3 path you have to parse yourself.
4. MSECond is a Condition step: if_steps = [RegisterModel], else_steps = [Fail]. The Fail step carries a composed message such as "MSE above threshold " + mse_threshold and terminates the execution with status Failed: the run does not "finish with a warning".
5. .start() executes it. Passing ModelApprovalStatus="Approved" as a parameter value makes the registration deployable immediately (a development/test shortcut), and CacheConfig(enable_caching=True, expire_after="PT1H") lets unchanged steps reuse their previous outputs.

---

3. The 16 step types, and the two that decide outcomes

3.1 The complete step map

AWS documents 16 step types. Learn the table by the "exam hook" column: the exam rarely asks "what does a Training step do", it asks which step should be used for X.

3.2 Condition and Fail as an evaluation gate

The documented pattern is mechanical and worth memorizing:

1. The evaluation Processing step declares property_files=["evaluation.json"].
2. The Condition step reads evaluation.json and compares a scalar: for example regression_metrics.mse against the pipeline parameter mse_threshold.
3. True branch → Register model (optionally preceded by Create model, and followed by a Transform step for a batch sanity pass).
4. False branch → Fail with a Join(...)-composed error_message, producing an execution status of Failed.

A second, equally common variant puts a QualityCheck or ClarifyCheck step in front of registration: the baseline is produced with register_new_baseline=true and skip_check=true on the first run, and subsequent runs compare against that baseline. Same shape: a gate whose false branch is a Fail step.

• 📚 Did you know? The Fail step is not a crash; it is a controlled termination. The step itself succeeds in doing its job, and the pipeline execution is what ends as Failed, with your message attached. That distinction matters when you build an EventBridge rule: you react to the execution status, not to a thrown exception in your Python code, so a Fail step gives you a clean, schema-stable signal that a shell script's exit 1 does not.

---

4. Parallelism and caching: the two knobs that change runtime cost

4.1 Parallelism is per execution, not per pipeline

Independent steps run in parallel by default: whatever is eligible runs. You cap that with ParallelismConfiguration(MaxParallelExecutionSteps). Two properties of this setting matter for the exam:

• it applies per execution: the documented example of 50 means one execution runs at most 50 steps concurrently, and two concurrent executions can therefore run 100 steps at once;
• it can be overridden at start, retry and update time, so it is a policy default, not an immutable property of the pipeline.

Parallelism is not a dependency mechanism. Setting MaxParallelExecutionSteps = 2 does not make step B wait for step A; it only limits how many ready steps execute simultaneously.

4.2 Caching is opt-in, ISO 8601, and only for successful runs

The practical use is a dev loop: you re-run a five-step pipeline after touching only the report, and PT1H lets Process, Train and Eval return their previous artifacts in seconds while the changed step executes for real.

---

5. Model Registry: versions and the approval that drives deployment

5.1 Registry structure

The registry organizes models into model groups, and a group holds packages/versions. Versions are simple and strictly ordered: they start at 1 and increment by 1 for each package added to the group. A versioned package must belong to a group, and its ARN carries the coordinates: …:model-package-group/name/version.

5.2 ModelApprovalStatus

Every model package carries a status drawn from exactly three values:

A versioned model must be Approved to be deployed. The status can be set through update_model_package, through Studio, through a Condition step inside a pipeline, or passed as a pipeline parameter whose default is PendingManualApproval.

5.3 The four approval transitions

This is the highest-yield table in the lesson, because the exam asks "which action starts the deployment?":

5.4 Worked example 2: approval to production

1. A SageMaker Project is created from the template "Model building, training, and deployment with third-party Git repositories using CodePipeline", with two GitHub repositories and a CodeConnections ARN tagged sagemaker=true.
2. A commit to the model-build repository triggers the source action (DetectChanges), which runs source → build: CodeBuild creates or updates the SageMaker Pipeline and starts it.
3. The pipeline registers version 7 with ModelApprovalStatus = PendingManualApproval.
4. A reviewer flips v7 to Approved (Studio, CLI, or a Condition step fed by an approval parameter).
5. The template's EventBridge rule starts the deploy pipeline: source → build → deploy-staging (CloudFormation stack) → manual approval → deploy-prod.
6. If staging surfaces a defect and v7 is flipped Approved → Rejected, the latest still-Approved package is redeployed; that is your documented rollback. In a pure dev/test loop you short-circuit steps 4 to 5 by passing ModelApprovalStatus="Approved" as a pipeline parameter.

---

6. SageMaker Projects, Git and CI/CD

6.1 What a Project actually is

SageMaker Projects are AWS Service Catalog products backed by CloudFormation. Creating one provisions, as a single unit: Git repositories, a SageMaker Pipeline, CodePipeline or Jenkins, a model group, an S3 bucket, an ECR registry and EventBridge rules. Two canonical shapes exist:

Provided template families cover build/train/deploy with third-party Git (CodePipeline or Jenkins), Model Monitor, and image building (ECR + CodeBuild + EventBridge).

6.2 The CodeCommit change of 28 October 2024

The GitHub source action mechanics are worth quoting exactly:

• the CodePipeline action is CodeStarSourceConnection;

• the connection must be a CodeConnections connection tagged sagemaker=true;

• the pipeline role needs both codestar-connections:UseConnection and codeconnections:UseConnection;

• change detection is DetectChanges on commit, so the build starts without a manual trigger;

• a custom template's CloudFormation must declare SageMakerProjectName and SageMakerProjectId.

• 📚 Did you know? "CodeStar" survives in the action name (CodeStarSourceConnection) long after the product was replaced by CodeConnections. That is why the IAM policy needs two nearly identical actions: one for the legacy namespace and one for the current service. Candidates who memorize only one of the two consistently fail this item, and AWS's own walkthroughs list both.

---

7. Experiments and MLflow: where run tracking lives now

7.1 The migration

The SageMaker Experiments API, CreateExperiment, Trials as run groups, trial components as individual runs, is documented as available in Studio Classic only. For the new Studio experience, AWS recommends MLflow on SageMaker AI. In exam terms: an option describing Trials and trial components as the current, recommended way to track runs is describing a legacy surface.

The split is the exam point: compute and backend store run in the SageMaker service account, but the artifacts live in S3 that you own and pay for; which is exactly what a compliance reviewer wants to hear about reproducibility.

---

8. Endpoint auto scaling: target tracking, step scaling, scheduled actions

8.1 The identifiers

Endpoint scaling runs through Application Auto Scaling (RegisterScalableTarget plus PutScalingPolicy) or through the console wizard. Quote the coordinates exactly:

AWS offers three strategies, target tracking (recommended), step scaling and scheduled actions (CLI/API), and, for target tracking, AWS manages the alarms for you.

8.2 The metrics

8.3 Worked example 3: target-tracking capacity math

Capacity always rounds up; that is the documented adjustment behavior, and it is examinable arithmetic:

1. Load test shows 210 invocations/min, target 70 → 210 / 70 = 3 instances.
2. Traffic rises to 351/min → 351 / 70 = 5.01, rounded up → 6 instances (never 5).
3. Register the target: register_scalable_target(ServiceNamespace="sagemaker", ResourceId="endpoint/prod/variant/AllTraffic", ScalableDimension="sagemaker:variant:DesiredInstanceCount", MinCapacity=1, MaxCapacity=10).
4. Attach the policy: put_scaling_policy(PolicyType="TargetTrackingScaling", {"TargetValue": 70.0, "PredefinedMetricSpecification": {"PredefinedMetricType": "SageMakerVariantInvocationsPerInstance"}, "ScaleInCooldown": 600, "ScaleOutCooldown": 300}).
5. Effect: scale-out within about one minute; the 600 s scale-in cooldown suppresses flapping after a burst, and the 300 s scale-out cooldown prevents over-provisioning during a spike train.

8.4 Worked example 4: high-resolution concurrency math

1. With ConcurrentRequestsPerModelHighResolution at target 5, a load of 18 concurrent requests → 18 / 5 = 3.6, rounded up → 4 instances.
2. The metric emits every 10 seconds instead of every 1 minute: roughly 6× faster detection for scale-out; scale-in timing is unchanged.
3. Using inference components? The metric becomes InferenceComponentConcurrentRequestsPerCopyHighResolution.
4. Multiple target-tracking policies may share one scalable target: add a second policy on CPUUtilization with TargetValue = 50, and both hold their own targets against the same DesiredInstanceCount.

8.5 The scaling strategy comparison

---

9. Async inference: scaling from zero

9.1 Why async is special

Async endpoints have a queue in front of them, which means the backlog, not the request rate, is the signal that matters. AWS documents these queue metrics:

9.2 The scale-from-zero design

9.3 Worked example 5: queue math and the wake-up

1. Configure MinCapacity=0, MaxCapacity=5 and a target-tracking policy on ApproximateBacklogSizePerInstance with TargetValue = 5.
2. A burst queues 27 requests on 1 running instance → 27 / 5 = 5.4, rounded up → 6 → capped at MaxCapacity = 5.
3. Traffic stops; scale-in returns the endpoint to 0 instances and idle spend goes to zero. Requests that arrive meanwhile are queued, not lost: ApproximateAgeOfOldestRequest reports how long the oldest one has waited.
4. The next request after idle: HasBacklogWithoutCapacity flips to 1, holds for 2 evaluation periods of 60 s, fires the alarm, the step policy adds +1 instance after the 300 s cooldown, and target tracking resumes.

---

10. Choosing the inference option

10.1 The four options and their hard limits

Batch transform specifics: MaxPayloadInMB at most 100, and MaxConcurrentTransforms × MaxPayloadInMB at most 100 MB.

Serverless exclusions (memorize the list): no GPUs; no Marketplace or private registry images; no multi-model endpoints; no VPC configuration; no data capture; no Model Monitor; no multiple variants; no inference pipelines; container image at most 10 GB; converting an existing real-time endpoint to serverless is rejected; Provisioned Concurrency is optional.

10.2 The decision

10.3 Worked example 6: 40 GB, nightly, no users

A fraud team asks for "a real-time endpoint for our nightly 40 GB scoring job." The correct answer is batch transform:

1. No interactive traffic → real-time's only advantage (latency) has no value, and its cost shape, paying for provisioned instances including idle hours, is pure waste for a job that runs once a day.
2. 40 GB blows past the serverless 4 MB cap and the async 1 GB cap; it is squarely "GB-scale data over days" territory.
3. Batch transform runs instances only for the job: MaxPayloadInMB ≤ 100 and MaxConcurrentTransforms × MaxPayloadInMB ≤ 100 MB keep each request within limits, S3Output collects the scored records, and no endpoint exists between runs, so nothing bills at 03:00.
4. Monitoring uses BatchTransformInput rather than Model Monitor; if the team later wants drift detection, that is a separate decision about a real-time or async endpoint.

• 📚 Did you know? AWS's cost-optimization guidance for inference reduces to one sentence: match the option to the traffic shape. Async is sized for an optimal processing rate plus scale to zero; serverless is per request; real-time is provisioned capacity; and across all of them SageMaker Savings Plans can be bought for 1 or 3 years and apply over Studio, processing, training, real-time inference and batch transform. Buying a Savings Plan for a job that runs 20 minutes a night is the classic mismatch: the plan discounts committed usage, and a nightly batch has almost none.

---

11. EventBridge and the MLOps flywheel

11.1 What AWS sends, and to what

SageMaker emits events to EventBridge in near real-time for jobs, endpoints, model packages, pipeline executions and pipeline steps, and image changes. Targets include Lambda, Step Functions, Kinesis and SNS, and, critically, SageMaker Pipelines itself is a target:

11.2 The flywheel

11.3 Worked example 7: monitor, alarm, retrain, approve, deploy

1. Enable data capture on the production endpoint so requests and responses land in S3, and schedule Model Monitor hourly (CronExpressionGenerator.hourly()).
2. Produce the baseline with a QualityCheck step using register_new_baseline=true and skip_check=true on the first run; later runs compare against it.
3. When drift exceeds the constraint, the CloudWatch alarm enters ALARM → the EventBridge rule fires → target = SageMaker Pipeline, invoking StartPipelineExecution with a mapped SageMakerPipelineParameter.
4. The pipeline retrains, evaluates, passes the Condition step, registers a new version as PendingManualApproval → approval flips it to Approved → the deploy template's rule updates the endpoint.
5. Run the calendar trigger alongside the drift trigger: cron(0 12 1 * ? *) retrains on the 1st of every month at 12:00 UTC, or use PipelineSchedule(name, cron="15 10 ? * 6L 2022-2023") for a bounded schedule. Remember at most 5 targets per rule and no schedule on custom or partner event buses.

11.4 Infrastructure as code for the whole stack

Everything in this lesson is describable as code, and AWS's guidance points at CloudFormation either directly or through the AWS CDK: cdk synth emits a CloudFormation template per stack, and the L1 Cfn* constructs cover the surface: sagemaker.CfnEndpointConfig (including AsyncInferenceConfig), sagemaker.CfnPipeline, plus native AWS:SageMaker:Domain, AWS:SageMaker:UserProfile and AWS:SageMaker:Pipeline.

Codify, at minimum: VPC, KMS, the S3 bucket, the pipeline execution role, the Model Package Group, CodePipeline + CodeBuild + CodeConnections, EventBridge rules, and the scalable target with its policies. For project templates AWS prefers S3-hosted templates via CfnTemplateProvider over Service Catalog portfolios, and a custom template must expose SageMakerProjectName and SageMakerProjectId. Splitting the stack into DevStack and ProdStack lets you unit-test the infrastructure before it touches production.

11.5 2025 to 2026 Updates

Three verified 2025 to 2026 changes touch the material in sections 8 through 13, and AWS now announces them through lifecycle vocabulary rather than through silent retirements. All three are documented, dated and examinable.

1. The product name changed. AWS documents that "the current Amazon SageMaker has been renamed to Amazon SageMaker AI" (3 December 2024), alongside a separate next-generation SageMaker generation (Unified Studio, Catalog, Lakehouse, zero-ETL). An exam option saying SageMaker AI is current, not exotic, and the identifiers you quote from section 8 did not move with the rename: ServiceNamespace = sagemaker, ResourceId = endpoint/…/variant/…, SageMakerVariantInvocationsPerInstance and the role action SageMaker:StartPipelineExecution are all still spelled the old way.

2. Model Monitor and Clarify are in maintenance. The SageMaker AI End of Support Notice lists Model Monitor and Clarify: together with Ground Truth, A2I, Studio Lab, Debugger, Role Manager, Geospatial and Mechanical Turk: as "no longer open to new customers starting 30 July 2026". Read that sentence with precision: the status is maintenance = no new customers, no new features, existing customers keep using them. It is not a full shutdown, and it is not a rebrand into Bedrock Model Evaluations. The drift flywheel of 11.2 therefore still runs for existing customers; for new work AWS points to CloudWatch metrics and anomaly detection, CloudWatch Evidently, EventBridge + Lambda, SHAP, and, on the generative side, Bedrock Model Evaluations; which is exactly why the exam guide's example 4.2.2 now reads "Clarify plus Bedrock Model Evaluations".

3. The exam guide moved. AWS's change history lists v1.0 published 26 March 2026 and v1.1 published 30 April 2026, and states that updates appear on the exam about one month after publication (treat v1.1 as the reliable anchor; the v1.0 date is a documentation re-publication). Changed examples that touch this lesson: 1.1.3 (async and serverless inference), 1.3.4 (Bedrock, Amazon Quick, Kiro, SageMaker AI) and 4.2.2 (Clarify plus Bedrock Model Evaluations). Unchanged, and therefore safe to keep quoting: 65 questions (50 scored + 15 unscored), 90 minutes, pass 700 / 1000, 3-year validity, domains 20 / 24 / 28 / 14 / 14 %.

Lifecycle vocabulary (AWS General Reference): Maintenance = no new customers, no new features, still supported · Sunset = planned end of operations, typically about 12 months · Full Shutdown = removed from the portfolio.

• 📚 Did you know? AWS states that exam-guide changes surface on the live exam about one month after publication, so v1.1 (30 April 2026) material was examinable from roughly late May 2026. The corollary runs in both directions: a feature announced after your guide version is not automatically testable, while a changed example such as 4.2.2 → Clarify plus Bedrock Model Evaluations is testable long before any retirement date matters. Read the change history, not the release blog.

---

12. Comparative verdict: managed MLOps vs DIY scripts vs manual

---

13. Exam traps and the numbers worth memorizing

Flagged as not verified in this lesson (do not assert them in the exam): the phrase "requests per target" as it appears in some study dumps, it maps to the generic ALBRequestCountPerTarget, which is not documented for SageMaker endpoints, so treat it as InvocationsPerInstance unless an ALB is in the architecture; the exact count of provided project templates (AWS enumerates families, not a numbered list); the multi-model endpoint roadmap (docs live and un-deprecated in October 2026, but no public roadmap page); the question weight of MLOps specifically (only domain weights are published); any end-of-support date for Experiments in Studio Classic; MLflow server pricing; the memory sizes and maximum concurrency of serverless endpoints; endpoint quotas (instances per variant, variants per endpoint); the numeric default of MaxParallelExecutionSteps (docs show how to set it, example 50, but publish no default); and GitHub Actions as an exam answer, it appears only in a blog and an aws-samples custom template, while the safe answers are CodePipeline / CodeBuild and Jenkins.

• 📚 Did you know? The SageMaker Lambda step defaults to a 2-minute timeout and permits up to 10 minutes: a deliberately small ceiling, because the step exists for lightweight orchestration work (flip an approval, call an API, move a file), not for running a training job. If your "step" needs an hour, it is a Processing step wearing a costume.

---

14. Real-World Case Studies

AWS publishes customer stories because each one is a documented version of a decision this lesson taught in the abstract. The five below are the MLOps-, pipeline- and inference-relevant cases, quoted with the exact AWS services, the exact printed numbers, and the source page: a percentage with no source is marketing, and a percentage with no "before" is worse.

14.1 Forethought: inference is a pipeline decision with a bill attached

• Services: SageMaker multi-model endpoints, Serverless Inference and Model Deployment, replacing self-managed Amazon EKS.
• Numbers: −66 % inference cost with multi-model endpoints and better latency; ≈ −80 % on Serverless Inference (headline up to −80 %); more than 80 % of GPU inference now runs on SageMaker; 30 M interactions/year; a 3-person team that could no longer run the models and Kubernetes (memory exceptions, outages).
• Source: aws.amazon.com/solutions/case-studies/forethought-technologies-case-study (the page carries no year chip: treat the date as unstated).
• Exam angle: this is section 10 with an invoice attached. Shared GPU capacity → multi-model endpoints; spiky small classifiers → serverless. The cost they deleted was the DIY serving tax: the "DIY scripts" column of the Comparative Verdict, applied to inference instead of to CI/CD.

14.2 HAYAT HOLDING: managed training and tuning instead of hand-built environments

• Services: AWS IoT Greengrass (SiteWise Edge Gateway) → SageMaker Model Training + Automatic Model Tuning + Model Deployment, with SageMaker Edge Manager serving the model on-device.
• Numbers: 194 sensors streamed over OPC-UA; self-built ML environments were "time-consuming and cumbersome"; documented outcome $300,000 per year saved plus higher panel quality.
• Source: aws.amazon.com/blogs/machine-learning/hayat-holding-uses-amazon-sagemaker-to-increase-product-quality-and-optimize-manufacturing-output-saving-300000-annually (2023).
• Exam angle: this is Level 1 maturity in a factory: one automated pipeline from sensor to tuned model, with Automatic Model Tuning replacing the manual hyperparameter sweep at 194 inputs and Edge Manager owning the on-device deployment path.

14.3 RareJob: spot training fixed a queueing problem, not a model problem

• Services: SageMaker training on managed spot, fed by AWS Glue and Amazon Athena, replacing a local PC and then a bottlenecked EC2/ECS setup.
• Numbers: −25 % training time, more than 10× development efficiency, 100 hours per month saved, scoring results in 2 to 3 minutes; one model per developer was the ceiling before the move.
• Source: aws.amazon.com/solutions/case-studies/rare-job-case-study (2020).
• Exam angle: the bottleneck was throughput of experiments, not accuracy. Parallel spot jobs are the documented cost lever for training, and the same "pay for interrupted capacity at a discount" logic that makes a Savings Plan or spot strategy a Domain 4 (cost) answer.

14.4 Chronomics: four months of DIY versus three weeks of managed AutoML

• Services: Amazon Rekognition Custom Labels (AutoML), scored with DetectCustomLabels.
• Numbers: 4 months of in-house custom computer vision never reached target; Custom Labels shipped in 3 to 4 weeks at 96.5 % accuracy / 97.9 % F1. Raising the confidence threshold to 0.99 gives 99.6 % precision while discarding 5 % of predictions; 0.999 gives 99.87 % while discarding 27 %.
• Source: aws.amazon.com/blogs/machine-learning/chronomics-detects-covid-19-test-results-with-amazon-rekognition-custom-labels (2022).
• Exam angle: two lessons. First, for a narrow vision task, the managed tier beats an in-house build: the documented failure pattern is DIY CV. Second, every threshold you raise creates a discarded tail that needs a human path (the reason A2I exists), so a precision question is always also a coverage question.

14.5 Sun Finance: a multi-step pipeline where the first prototype was rejected

• Services: Amazon Textract (OCR) → Amazon Rekognition (fallback and face checks) → Claude Sonnet 4 (structuring only) → validation rules → Amazon Titan Multimodal Embeddings in S3 Vectors for fraud similarity; built with the AWS Generative AI Innovation Center, evaluated on 585 images.
• Numbers: accuracy 79.73 % → 90.80 % (ID number 74.32 → 89.40 %, document type 78.43 → 96.40 %), −91 % cost per document, 20 hours → under 5 seconds, fraud detection 81 %. Attempt 1, Claude Sonnet 4 alone, scored 61.8 % overall and 43 % on ID number and was rejected.
• Source: aws.amazon.com/blogs/machine-learning/sun-finance-automates-id-extraction-and-fraud-detection-with-generative-ai-on-aws (2026).
• Exam angle: this is a Condition-step mindset in production form: build the baseline, measure, reject the failing attempt, re-architect (separate OCR from reasoning), re-measure. Evaluation is a phase, not a checkbox, and LLM-only extraction of PII is the documented failed prototype.

Table 14-A: Before / after, exactly as AWS printed them

Table 14-B: Choosing the tier: prebuilt → managed custom → generative

• 📚 Did you know? AWS publishes exactly one production-rate figure for its own generative AI work: 65 % of Generative AI Innovation Center projects reached production in 2025, drawn from more than 1,000 implementations, with the fastest in 45 days, and AWS frames the method as the Five V's: Value → Visualize → Validate → Verify → Venture. The number worth internalizing is the other 35 %: AWS expects you to plan for projects that stall, which is why baseline before build and evaluation as a phase appear in every case above.

---

Practice Questions

---