Troubleshooting, Performance and Cost Optimization

Domain 3 of DEA-C01 is Data Operations and Support, 22% of the exam, and its task statements say the same thing four different ways: troubleshoot performance issues (3.3.4), troubleshoot and maintain pipelines such as AWS Glue and Amazon EMR (3.3.6), analyze logs with AWS services (3.3.8), and implement data skew mechanisms (3.4.5). Cost sits beside them in 2.1.1, pick storage services for specific cost and performance requirements, and in 3.2.5, describe tradeoffs between provisioned and serverless services. This lesson is the union of those skills: a repeatable triage loop, one symptom→cause→fix map per service, and the arithmetic that turns "make it cheaper" into a defensible answer.

In this lesson you will:

• run a four-step triage loop and a failure taxonomy covering schema drift, corrupt files, OOM, timeout, IAM, throttling, small files and encoding with symptom→cause→fix per service;
• size AWS Glue: DPU and worker types, partition math, auto scaling, flexible execution, bookmark failure modes, repartition versus coalesce, and merging small files;
• diagnose Amazon Redshift: distribution and sort key mistakes, SVV_TABLE_INFO skew triage, automatic VACUUM/ANALYZE and ATO, WLM and query monitoring rules, Spectrum pruning, and reading an EXPLAIN plan;
• right-size Amazon EMR: primary/core/task roles, node-count arithmetic, Spot task nodes, HDFS versus S3, and when EMR Serverless is the answer;
• fix Amazon Kinesis Data Streams: hot shard versus hot key, partition key redesign, iterator age alarms, and the three capacity modes;
• operate AWS Lambda: concurrency pools, reserved versus provisioned concurrency, timeouts, asynchronous retries and dead-letter queues;
• apply a cost-lever table per service, a cost-versus-performance tradeoff table, and the MOST operationally efficient / LEAST costly stem heuristics;
• work seven worked examples: Glue partition and cost math, Redshift skew triage, an EXPLAIN drill, EMR node sizing, Kinesis shard cost, Lambda concurrency and provisioned-concurrency cost, and S3 request cost;
• read the October 2026 update box, four AWS customer case studies, and finish with fourteen exam-style questions plus three interactive checks.

---

1. The triage loop and the failure taxonomy

1.1 Four steps, in order

AWS's own troubleshooting guidance puts it bluntly: start in the job run's Error log, then move to the Executors tab / Spark UI to see which executor died and when (AWS Glue troubleshooting and Builder guidance, accessed Oct 2026).

1.2 Symptom → cause → fix, per service

This table is the examinable core of Skill 3.3.4. Every row is a documented AWS failure mode with the fix AWS documents for it (AWS Glue troubleshooting, Redshift diagnostic and EMR/Kinesis/Lambda troubleshooting documentation, accessed Oct 2026).

1.3 The failure decision tree

1.4 Interactive check: match the symptom to the fix

• 📚 Did you know? A Glue job can report SUCCEEDED while writing nothing: a bookmark that has already recorded the source files will skip them on the next run, and a wrong prefix or a dead catalog path looks identical to a healthy empty partition. AWS's own troubleshooting list therefore puts "success but no data" next to real errors: the fix is a timestamp check, not a stack trace (AWS Glue troubleshooting guide, accessed Oct 2026).

---

2. Data-shape failures: drift, corrupt files, encoding

2.1 Schema drift is a version problem, not a parse problem

When an upstream team adds, drops or retypes a column, three things happen in sequence: the source changes, the Glue table gains a new version in the Data Catalog, and every downstream consumer (crawler, Athena query, Redshift external schema, QuickSight dataset) silently inherits the change. AWS's pattern for it is: diff the table versions → publish to an SNS topic → evolve the consumer before the data arrives (AWS Big Data Blog, "Identify source schema changes using AWS Glue", 2022-09-14).

Inside the job itself, a DynamicFrame represents "some rows have this column, some do not" as a choice type. The documented repair is resolveChoice, which either casts (you name the type: uncastable values become NULL) or make_cols (adds a nullable column). Closely related: the crawler reports "Partitions were not updated" when the incoming DynamicFrame schema is not identical to, or a subset of, the catalog schema and updateBehavior is not UPDATE_IN_DATABASE.

2.2 Corrupt records: reject loudly or quarantine

Malformed CSV rows that "fit" the inferred schema are dropped silently. The documented remedy is errorsAsDynamicFrame(), which splits the frame into a good frame and a reject frame, plus stageThreshold / totalThreshold counts that fail the job when rejects exceed a budget. The operational pattern from Lesson 11 completes it: write rejects to a quarantine prefix with _dq_rule, _dq_reason and _run_id, and alert on the count. The vectorized CSV reader is a specific trap; it cannot handle multiLine or multibyte content, so "parse errors that only appear on some rows" usually means "use the row-based reader".

2.3 Encoding

Charset mismatches present as corrupted characters, truncated fields or rows that split at the wrong comma. Set encoding in the Glue format options, avoid the vectorized reader for multibyte data, and put a DataBrew encoding rule before the load rather than cleaning mojibake in the warehouse. Encoding is a source-to-target contract, and like schema drift it is invisible in row counts; which is exactly why both belong in the taxonomy rather than in a "data quality" side chapter.

---

3. AWS Glue performance: DPU, workers, bookmarks, small files

3.1 What you are actually paying for

One DPU = 4 vCPUs of compute capacity and 16 GB of memory (AWS Glue worker types documentation, accessed Oct 2026). Glue 2.0+ jobs are sized with WorkerType + NumberOfWorkers, there is no MaxCapacity on modern versions, and one Spark executor runs per worker node, all G and R types.

Sizing rule from the Prescriptive Guidance tuning guide: scale out (more workers) only "until you observe idle workers: beyond that, adding more workers increases costs without improving results", and scale up (bigger type) for memory-heavy transforms, skewed aggregations and huge driver plans.

3.2 Partition math: the arithmetic the exam can ask you to redo

For Glue 3.0+:

Initial partitions from S3: for objects ≤ 64 MB, count the objects; for > 64 MB, use size / 64 MB; unsplittable gzip counts as one task per object: a single 10 GB .csv.gz is 1 task, no matter how many workers you buy. Too many partitions explode the driver and the task scheduler, so fix them with groupFiles, coalesce(N) (no full shuffle, so cheaper than repartition) or Spark 3 AQE coalescing.

3.3 Auto scaling, flexible execution and AQE

• Auto scaling (--enable-auto-scaling=true, Glue 3.0+, all G/R types): Glue adds and removes workers per stage up to NumberOfWorkers as the maximum, and it also sets spark.sql.shuffle.partitions and spark.default.parallelism from that maximum: override with --conf if your workload is uneven.
• Flexible execution (Glue 3.0+, G.1X and G.2X only, non-urgent jobs): a discounted DPU-hour in exchange for a variable start time. Pre-prod and overnight backfills are the documented use case.
• AQE is on by default in Glue 4.0; in 3.0 set spark.sql.adaptive.enabled=true. It coalesces shuffle partitions, converts sort-merge joins to broadcast, and handles skew with skewedPartitionFactor / skewedPartitionThresholdInBytes.
• Scale-out is not linear. AWS's own worked metrics example (10 → 55 DPU halves runtime, 55 → 100 DPU adds nothing because executors sit idle) comes from a page only applicable to Glue 0.9/1.0: take the method ("add workers only until they idle"), never the absolute minutes.

3.4 Worked example E1: Glue partition math and cost

3.5 Bookmarks: the most-missed failure mode on this exam

AWS documents seven ways a bookmarked job reprocesses data: MaxConcurrency > 1, missing job.init() / job.commit(), missing or renamed transformation_ctx, a non-sequential JDBC primary key, a changed last-modified time, Spark DataFrame input instead of DynamicFrame, and a changed source path under the same context. Two sentences cover most exam items:

• "Currently AWS Glue bookmarks don't support concurrent job runs and commits will fail.": keep Max concurrent runs = 1.
• "If you don't pass in the transformation_ctx parameter, then job bookmarks are not enabled for a dynamic frame or a table used in the method.": bookmarks track DynamicFrame operations, and they track sources, not targets: a rewind or reset never deletes what was already written, so replay into a new target prefix.

3.6 repartition versus coalesce, and merging small files

---

4. Amazon Redshift: distribution, skew, maintenance, plans

4.1 Three diseases, three treatments

Redshift performance questions almost always resolve to one of three column-level problems, and each has a different cure (AWS Redshift Identifying tables with data skew or unsorted rows, accessed Oct 2026).

Automatic maintenance is on by default: automatic analyze runs in light-load windows with analyze_threshold_percent = 10 %, auto vacuum sort/delete pause under high load or exclusive DDL locks, and automatic table optimization (ATO) applies sort and distribution keys to AUTO tables within hours of seeing enough queries. Audit it with SVV_ALTER_TABLE_RECOMMENDATIONS and SVL_AUTO_WORKER_ACTION. Key facts the exam reuses: explicit sort/distribution keys take a table out of automation, and VACUUM skips the sort phase by default when more than 95 % of rows are already sorted.

4.2 Worked example E2: skew triage in four queries

Then re-EXPLAIN and look for the redistribution tag you want: DS_DIST_NONE / DS_DIST_ALL_NONE.

4.3 EXPLAIN basics: read the plan before you resize the cluster

Redistribution tags: DS_DIST_NONE and DS_DIST_ALL_NONE are good; DS_BCAST_INNER is fine for a small dimension and is not automatically bad; DS_DIST_INNER / DS_DIST_OUTER are costly; DS_DIST_ALL_INNER serializes execution onto one slice; DS_DIST_BOTH is the worst. If the planner picks a join direction that looks wrong, the documented cause is stale statistics: run ANALYZE.

4.4 Worked example E3: the plan AWS itself uses in training

Two examinable lessons: the operator name tells you the strategy, the cost estimate tells you whether it mattered, and neither requires a bigger cluster (AWS Prescriptive Guidance, "Evaluating the query plan", accessed Oct 2026).

4.5 Workload management, concurrency scaling and Spectrum

• WLM: default is 1 queue with 5 concurrent queries. Automatic WLM creates up to 8 queues (service classes 100 to 107) each with a priority (NORMAL default; CRITICAL is superuser-only). Manual WLM allows concurrency 1 to 50 per queue, 50 slots total, with a memory percentage per queue.

• Query monitoring rules run every 10 s. Action severity: log < hop (manual WLM only, and only for CTAS/read-only queries) < abort (never stops COPY, ALTER, ANALYZE or VACUUM) < change priority (automatic WLM only). Templates to remember: spill query_temp_blocks_to_disk > 100000 blocks (100 GB) and I/O skew io_skew > 1.30.

• WLM timeouts cover only the running phase: queue, lock, compile and hop waits sit outside them.

• Concurrency scaling routes overflow queries to transient clusters instead of queueing, per queue (auto/off), with 1 free hour per 24 hours and max_concurrency_scaling_clusters of 1 to 10. It relieves queueing, not per-query memory pressure.

• Spectrum bills $5.00 per TB scanned (10 MB minimum per query, as of Oct 2026), so pruning is a cost lever as much as a speed lever: partition on predicates you actually filter, use Parquet/ORC for column elimination, keep files ≥ 64 MB, and set TABLE PROPERTIES (numRows = n) because Redshift never analyzes external tables and otherwise assumes the external table is the largest table in the plan. Verify with SVL_S3PARTITION (assigned_partitions vs qualified_partitions), SVL_S3QUERY_SUMMARY (s3_scanned_*) and EXPLAIN markers (S3 Seq Scan, Partition Loop). DISTINCT and ORDER BY are never pushed down.

• 📚 Did you know? A Redshift table can be 86 % unsorted and only 5 % improvable: vacuum_sort_benefit exists precisely so you do not pay for a VACUUM that changes nothing, and by default VACUUM skips the sort phase entirely above 95 % sorted rows. That is why "run VACUUM after every load" is a distractor, not a best practice (AWS Redshift vacuuming documentation, accessed Oct 2026).

---

5. Amazon EMR: sizing, Spot task nodes, HDFS versus S3

5.1 Three node roles, three purchase decisions

AWS's own rule: "Use Spot Instances on task nodes. The task nodes process data but do not hold persistent data in Hadoop Distributed File System (HDFS). If task nodes shut down… no data is lost." All-Spot is only for transient, loss-tolerant jobs. Note the asymmetric Spot behaviour: task-Spot instances launch as capacity is available, while core-Spot waits for all requested instances before launching, and Spot reclaim allows roughly 20 s of graceful decommission (spark.decommissioning.timeout.threshold, EMR ≥ 5.11.0).

5.2 Worked example E4: node count and HDFS capacity

HDFS versus S3 for exam purposes: S3/EMRFS is durable, decoupled, survives cluster termination and costs less per GB → it is the pipeline default; HDFS is faster for intermediate shuffle but consumes the replication budget and dies with the cluster. On EMR, adaptive query execution is on by default (5.30+) with shuffle-partition coalescing (advisory size 64 MB, minPartitionNum 25, initialPartitionNum 1000), dynamic partition pruning (5.26+), and spark.default.parallelism = 2 × YARN cores.

5.3 EMR Serverless: the no-sizing option

EMR Serverless bills per vCPU-hour and GB-hour plus storage GB-hour, $0.052624 / vCPU-hour, $0.0057785 / GB-hour, $0.000111 / storage GB-hour (as of Oct 2026), and workers auto-scale between the job's minimum and maximum, so the sizing decision disappears. It is the natural pick for intermittent Spark/Hive jobs where a cluster would idle.

---

6. Amazon Kinesis Data Streams: hot shards, iterator age, capacity modes

6.1 Hot shard versus hot key: the single most tested distinction

Per-shard limits are 1 MB/s write and 1,000 records/s, 2 MB/s and 2,000 records/s read, and 5 read TPS; a single record may be 10 MiB and PutRecords accepts up to 500 records / 10 MiB per call (AWS Kinesis limits, accessed Oct 2026).

• Hot shard = many keys landing unevenly → split the shard (provisioned UpdateShardCount / SplitShard) or let on-demand split it.
• Hot key = one key exceeding a shard's limit → redesign the partition key (UUID or composite suffix) or, if you need per-key ordering/locality, keep the key and accept the ceiling.
• Adding shards does not fix a hot key: every record for one key hashes to one shard, forever. And on-demand mode "does not detect and isolate hash keys that are driving a higher portion of incoming traffic to a particular shard": documented explicitly, which is why it is a favourite wrong answer.

6.2 Iterator age is a data-loss warning, not just lag

GetRecords.IteratorAgeMilliseconds (stream) / IteratorAgeMilliseconds (shard) measures consumer lag; 0 means fully caught up. AWS's warning: "If an iterator's age passes 50% of the retention period (by default, 24 hours, configurable up to 7 days), there is risk for data loss due to record expiration." So alarm on Maximum at 50 % of retention, and fix with: more shards, Lambda ParallelizationFactor (1 to 10 per shard; ordering is still preserved per partition key), a batching window up to 5 min / 6 MB, enhanced fan-out (a dedicated 2 MB/s per consumer per shard), or simply a faster consumer.

6.3 Three capacity modes, not two

Mode switches are limited to 2 per rolling 24 hours. EFO costs $0.015 per consumer-shard-hour plus per-GB, and supports 20 registered consumers per stream (Standard) or 50 (Advantage); with fewer than about 3 consumers, shared reads are the cheaper choice.

• 📚 Did you know? Iterator age is one of the few AWS metrics where "higher" can mean permanent data loss rather than "slower": once a record passes the retention window it is gone, and no amount of scaling brings it back. That is why the AWS recommendation is an alarm at half the retention period, not at 100 % (AWS Kinesis monitoring documentation, accessed Oct 2026).

6.4 Worked example E5: shard cost and the payload-unit bill

---

7. AWS Lambda: concurrency, timeouts, cold starts, retries

7.1 The concurrency pool

The account default is 1,000 concurrent executions per Region; a function scales up to 1,000 environments per 10 seconds, and the invocations-per-second ceiling is about 10 × the concurrency quota. Beyond the quota, invocations are throttled with 429.

Reserved and provisioned concurrency both count against the account pool, and provisioned ≤ reserved for the same function. Application Auto Scaling targets 70 % utilization (scale out above 70, in below 63), and AWS's own guidance is that asynchronous data-processing pipelines usually do not need provisioned concurrency.

7.2 Timeouts and retries

Raise memory and timeout together: CPU is proportional to memory, so a "timeout" is frequently an under-sized function. Asynchronous retries do not fix mis-sizing; they only hide it until the DLQ fills:

• Synchronous: the caller sees the error; retries follow the caller's policy.
• Asynchronous: Lambda queues, then retries 2 times (after 1 minute, then 2 minutes) for function errors; for throttling (429) and system errors (500-series) the event returns to the queue and is retried for up to 6 hours, with backoff from 1 s up to 5 min. Then the dead-letter queue (SQS standard or SNS standard only: never FIFO) or the on-failure destination receives it; an on-failure destination additionally carries the response payload.
• Stream (event source mapping): retries until bisect_batch_on_error, maximum_record_age or retry_attempts fires, then the on-failure destination. For Kinesis, ParallelizationFactor 1 to 10 raises per-shard parallelism while keeping order per partition key.

7.3 Worked example E6: concurrency, reserved vs provisioned

---

8. Cost levers per service

8.1 The billable dimension, then the lever

Never name a lever without naming the dimension it reduces (AWS Glue, Redshift, EMR, Kinesis, Lambda and S3 pricing pages, as of Oct 2026).

8.2 Worked example E7: small files cost twice

• 📚 Did you know? The storage half of the bill has been falling for twenty years while the request half has not: AWS's S3 retrospective (2026-03-13) puts the price reduction since the 2006 launch at about 85 % and claims customers have saved more than $6 billion by using S3 Intelligent-Tiering instead of S3 Standard. That is why this lesson keeps splitting S3 into storage-class and requests levers: automating the class moves a big number, merging objects moves the other one (AWS "Twenty years of Amazon S3" post, accessed Oct 2026).

---

9. Cost versus performance: the tradeoff table and the stem heuristics

9.1 The tradeoff the exam actually asks

9.2 Where each service sits

9.3 The two stem heuristics (derived from AWS's documented behaviour, not an AWS document)

• "MOST operationally efficient" ⇒ pick the option with no capacity planning and no cluster to size: AWS Glue (managed workers, auto scaling), EMR Serverless, Redshift Serverless, Kinesis on-demand, Lambda, then re-read the stem for a constraint that overrides it (HDFS, hot-key control, steady 24/7 duty cycle).
• "LEAST costly" (steady 24/7) ⇒ pick the provisioned or committed option: provisioned shards, a Redshift cluster over Serverless, On-Demand/committed compute, fewer DPU-hours, lifecycle-tiered S3. A serverless option that idles 20 hours a day is rarely the cheapest.
• "FASTEST to implement" ⇒ a managed configuration change (WLM, auto scaling, an alarm, a sort key) beats a code refactor: unless the stem's root cause is code (a missing join predicate, a missing transformation_ctx).
• Always read the constraints first: steady vs spiky, SLA vs throughput, isolation vs latency. The heuristic is the tie-break, not the premise.

9.4 Interactive check: fill in the numbers the exam reuses

9.5 Interactive check: order the cost decision

---

2026 Updates (as of October 2026)

• 📚 Did you know? Price cuts move exam answers: Glue 6.0 (−30 %), S3 Express One Zone (storage −31 %, PUT −55 %, GET −85 %, 2025-04-10) and Kinesis on-demand Advantage all changed the "cheapest" column, but the dimensions (DPU-hour, shard-hour, GB-s, bytes scanned) have not changed. Learn the dimension and re-verify the rate (AWS pricing pages, accessed Oct 2026).

---

Real-World Case Studies

AWS publishes what this lesson looks like in production. Every figure below is customer- or AWS-claimed and unaudited, with the source named so you can check it: the examinable point is the pattern (which lever moved, and in which direction), not the marketing percentage.

Case A, Amazon Customer Service: right-sizing a warehouse

Read it as Skill 3.3.4 plus 2.1.1 in one story: performance and cost moved together, because RA3 separates managed storage from compute, so you stop paying for local disk you were only filling with cold data. The distractor version of this story claims cost fell because a discount was purchased: no discount appears in the case study.

Case B, Paytm: a phased cost lever on EMR

"Migrating to AWS Graviton-based instances is a long-term investment and will be beneficial in terms of total cost of ownership.": Parakh Aggarwal, Director of Cloud Excellence and FinOps (AWS case study, accessed Oct 2026)

The examinable pattern is the gap between list price and realised saving: a lever is only worth its points once you have measured the workload, and the timeline (2022 → 2024) shows why "change everything tonight" is never the answer on this exam.

Case C, GE Aerospace: a legacy ODS rebuilt on Redshift

"We were seeing queries that used to run for an hour and a half running in 7 minutes using Amazon Redshift.": Bejoy John, Senior Director of Data Analytics (AWS case study, accessed Oct 2026)

Read it as Skill 3.3.4 (troubleshoot performance) plus 2.1.1 (cost/performance sizing): the win came from layout, plan and node generation, not from a purchased discount. Two exam habits to rehearse here: the >$500,000 is GE's own estimate, so never generalise it to your workload; and a 9-month migration with a PoC first is the opposite of the "fix it tonight" distractor.

Case D, Edmunds.com: three levers, one bill

"The main motivation there is that I am no longer bound to a particular instance type or Region.": Emil Ndreu, Executive Director of Cloud and Security Operations (AWS case study, accessed Oct 2026)

Work the arithmetic AWS does not publish: $100,000 over 11 months ≈ $9,091 per month, but AWS never splits that figure between Spot, Savings Plans and Fargate Spot, so any option that claims "Savings Plans alone saved X %" is unsupported by the source. The examinable distinction is the lever type: a Savings Plans is a commitment discount (Compute SPs are not tied to one instance type or Region), Spot is interruptible capacity for fault-tolerant work (AWS lists up to 90 % off On-Demand), and neither is the Graviton silicon switch from Case B.

• 📚 Did you know? All four stories avoid the trap this lesson keeps repeating: none of these customers "made it cheaper" by buying a commitment on an oversized system. One resized, one re-platformed, one rebuilt the plan and the layout first, one stacked two capacity levers, and every one of them measured first. AWS's published percentage is always their workload, never a guarantee for yours (AWS case studies and Big Data Blog, accessed Oct 2026).
• 📚 Did you know? Three levers look alike on a bill and are different exam answers: Savings Plans = commitment discount (Compute SPs are not bound to an instance type or Region), Spot = interruptible capacity you must be able to lose (AWS quotes up to 90 % off On-Demand), and Graviton = different silicon with a lower list price. Distractors on DEA-C01 routinely swap two of the three: name the dimension each one changes before you pick (AWS Pricing and case studies, accessed Oct 2026).

---

Practice Questions

Comparative Verdict: how this topic compares on exam day

• Versus another cloud: DEA-C01 tests AWS services only: nothing asks you to compare Glue DPU-hours with a competitor's cluster pricing. Answer with an AWS service, an AWS-published threshold (4.00 skew, 95 % VACUUM skip, 1 MB/s per shard, 1,000 Lambda concurrent) or an AWS-published rate as of Oct 2026; any option pivoting to an unverified third-party benchmark is out of scope by construction.
• Versus self-managed / on-premises: the AWS answer is always managed and measured over bespoke and manual: automatic VACUUM/ANALYZE and ATO beat a cron-job VACUUM, Glue auto scaling beat a hand-tuned cluster, Lambda concurrency quotas beat a self-hosted queue worker. An option describing you maintaining the machinery is the distractor, not the bonus.
• Versus another AWS service: pick by the billable dimension, not prestige: Glue for managed Spark ETL (DPU-hour), Redshift for SQL at scale (node/RPU-hour + bytes scanned), EMR when you need cluster-level control (EC2 + EMR fee), EMR Serverless/Glue when sizing is the risk, Kinesis for streaming (shard-hour or GB), Lambda for event glue (requests + GB-s). "Serverless" wins the operational stem; "provisioned" wins the steady-state cost stem.
• Versus a manual, human process: Amazon Customer Service's ~55 % cost cut with 47 % faster dashboards and Paytm's phased 30 to 35 % EMR saving (both customer-claimed, accessed Oct 2026) show why the exam prefers an automated, measured lever: humans do not find the skew, the plan or the idle worker: a threshold, a metric or an alarm does. Treat every published percentage as "the customer achieved", never as "AWS guarantees".