Monitoring and Observability for Data Pipelines

Two exam tasks meet in this lesson. Task 3.3, maintain and monitor data pipelines, asks you to extract logs for audits, deploy logging and monitoring solutions that enable auditing and traceability, send alert notifications, troubleshoot performance issues, track API calls with AWS CloudTrail, maintain Glue and EMR pipelines, log application data with Amazon CloudWatch Logs, and analyze logs with Athena, EMR, OpenSearch and CloudWatch Logs Insights. Task 4.4, prepare logs for audit, asks the same material from the compliance side: CloudTrail, CloudWatch Logs storage, CloudTrail Lake, log analysis, and integrating services such as EMR for large log volumes. The exam treats them as one discipline with two questions: what happened (audit evidence) and what is happening right now (operational signal).

In this lesson you will:

• separate the audit plane from the operations plane, and place management events, data events and Insights events correctly;
• choose between service metrics, custom metrics and Embedded Metric Format, with the cardinality and permission traps;
• design alarms: states, state-change actions, missing data, composite and log alarms, dashboards and metric math;
• use the three CloudWatch Logs mechanisms, metric filter, subscription filter, Logs Insights, without mixing them;
• write Logs Insights queries (query language, field indexes, bytes-scanned billing);
• monitor each pipeline stage natively: Glue, Kinesis, Redshift, EMR, Lambda, DMS, Step Functions, S3, EventBridge;
• apply logging best practices: structured JSON, correlation IDs, PII data-protection policies, retention and cross-account routing;
• work through nine worked examples: six alarm configurations and three Logs Insights queries;
• read the October 2026 update box, four AWS customer case studies, and finish with fourteen exam-style questions plus three interactive checks.

---

1. Two planes of evidence: audit versus operations

1.1 The split the exam tests

1.2 CloudTrail: management, data, Insights

• Management events (default): Describe*, Get* reads and write calls such as RunInstances, plus non-API events such as ConsoleLogin.
• Data events (opt-in, charged): high-volume resource-level operations, AWS:S3:Object, AWS:Lambda:Function (Invoke), AWS:DynamoDB:Table, AWS:Kinesis:Stream, AWS:SQS:Queue, AWS:StepFunctions:StateMachine, AWS:Glue:Table. A question about s3:GetObject is unsolvable unless data events are enabled, they are off by default.
• Insights events: management-event anomaly detection, charged and off by default.
• Integrity: SHA-256 hashing plus SHA-256 with RSA digital signing produces an hourly digest file, and validation is not retroactive.
• Organization trails (management account or delegated admin): one bucket for every account, member accounts can see but not modify or delete, propagation ≤24 hours, log files delivered approximately every 5 minutes.
• Glue control-plane actions are all logged with eventSource: glue.amazonaws.com; many Get*/BatchGet* responses log as null.

1.3 Worked example E1: "who deleted the partitions?" in four moves

• 📚 Did you know? CloudTrail Event history gives you 90 days of management events for the signed-in account only: no setup, no trail. It is the fast answer to "who did this in the last 90 days?" but it cannot answer a cross-account or data-event question; that needs an organization trail or CloudTrail Lake (AWS CloudTrail documentation, accessed Oct 2026).

---

2. CloudWatch metrics: service, custom, embedded

2.1 Three ways a number reaches CloudWatch

Statistics are retained for 15 months. After you publish a new custom metric it becomes visible to GetMetricStatistics within 2 minutes and to ListMetrics within 15 minutes: an exam trap when you "just created it and see nothing".

2.2 Worked example E2: the cardinality bill

An EMF log line dimensions a metric by job (50 names) × status (4 values) × region (6 values):

EMF is attractive precisely because it needs no extra IAM permission, but one custom metric exists per unique dimension value: high-cardinality fields (request IDs, user IDs, SQL strings) must stay in the log message, not in the dimensions.

• 📚 Did you know? Because EMF metrics come from log events, they are delivered at-least-once: a retried PutLogEvents can produce a duplicate metric data point. Design alarms on statistics that tolerate that (for example Sum over a period rather than exact SampleCount), and never treat an EMF-derived count as an exactly-once ledger (AWS EMF documentation, accessed Oct 2026).

---

3. CloudWatch alarms: states, actions and combining them

3.1 The state machine

Key rules (all AWS-verified, accessed Oct 2026):

• An alarm invokes actions only when the alarm changes state: the exception is alarms with Auto Scaling actions. A pipeline that stays broken does not re-page you every minute.
• States are OK / ALARM / INSUFFICIENT_DATA; idle resources frequently stop publishing and sit in INSUFFICIENT_DATA.
• Alarm types: metric alarms (including metric-math expressions you can alarm on), composite alarms (Boolean AND/OR/NOT over other alarms), log alarms (a scheduled Logs Insights query with an M-out-of-N condition: no metric filter required), and PromQL alarms. There is no quota on the number of alarms.
• Composite alarms: same account and Region only, start in INSUFFICIENT_DATA (the only time a new alarm does), cannot perform EC2 actions or Auto Scaling actions, children carry no actions so the composite pages once, and limits are 100 children / 500 elements / 150 composites per child.
• Evaluation windows: up to 7 days when the period is ≥1 hour, or 1 day for shorter periods.

3.2 Missing-data behaviour is a design decision

TreatMissingData has three settings, and the right one depends on the metric: breaching (missing = bad; correct for iterator age, where silence means no reader), notBreaching (missing = fine; correct for DLQ depth, where no data points means an empty queue), and ignore (keep the current state). Choosing the default (missing) is how teams get INSUFFICIENT_DATA alarms nobody trusts.

3.3 Anomaly detection instead of a static threshold

CloudWatch anomaly detection trains an ML band per metric and per statistic, so you alarm when the value leaves the band rather than crossing a hand-tuned number. It replaces fixed thresholds where the workload is seasonal (daily ETL windows) and complements them where a compliance limit exists (a 300-second CDC latency).

---

4. Dashboards and metric math

• Dashboards are unlimited per account, and all dashboards are global, not Region-specific: a monitoring dashboard pulls Region-qualified metrics per widget.
• Quotas: ≤500 widgets per dashboard and ≤500 metrics in a widget's metrics array (as of Oct 2026).
• Cross-account widgets: from a monitoring account, each widget names its accountId and region.
• A metric that has been silent for 14 days disappears from the dashboard metric picker: a quiet pipeline looks like a deleted one.
• Metric math lets one widget show 100 * (Errors / Invocations) as an error rate, and alarms can watch the expression, not just the raw metric.

---

5. CloudWatch Logs: groups, retention, metric filters, subscription filters

5.1 The container model and its quotas

A log group holds log streams that share retention and access control; there is no limit on the number of streams per group. As of Oct 2026: 1,000,000 log groups per Region, metric filters 100 per group, subscription filters 5 per group, log event 1,024 KB, PutLogEvents batches of 1 MB / 5,000 per second, StartQuery 10/s, and regex limits of ≤5 regex patterns per group and ≤2 regexes per pattern.

5.2 One pattern, three jobs

5.3 Worked example E3: metric filter plus alarm on JSON job logs

Because filters are not retroactive, this alarm only sees errors from the moment the filter exists: retrospective questions ("did it error last Tuesday?") are Logs Insights questions.

5.4 Worked example E4: subscription filter delivery health

Throttled deliverables are retried for up to 24 hours, after which the failed deliverables are dropped, so a DeliveryThrottling alarm is a data-loss alarm wearing a performance costume.

---

6. CloudWatch Logs Insights: querying the evidence

6.1 Three languages, one billing model

Insights offers the query language (fields, filter, parse, stats, sort, limit, join, lookup, filterIndex, estimate), OpenSearch PPL, and OpenSearch SQL: SQL is the only one that can JOIN across log groups. You are billed by uncompressed log data scanned, queries time out after 60 minutes, results live for 7 days, and field indexes (plus the filterIndex command) cut the scanned volume dramatically. AWS service logs are auto-discovered as @-prefixed fields; your own structured JSON fields appear by name.

Guardrails: narrow the time range first, cancel queries before you close the console, and prefer metric filters or EMF over a dashboard widget that re-runs Insights.

6.2 Worked example E5: triage a failed nightly job (30-second window)

6.3 Worked example E6: error rate by job from structured JSON

Both queries are investigations. To alert on the same signal you would add a metric filter (Example E3) or publish via EMF, and, because metric filters are not retroactive, you would also keep the Insights query for the retrospective half of the question.

• 📚 Did you know? Logs Insights auto-discovers the @-fields of AWS service logs (for example @message, @timestamp, @logStream), but field indexes you define are what make filter cheap: without an index every candidate line is scanned, so the bill tracks bytes scanned, not rows returned. Index the fields you filter on, and keep wide @message LIKE scans for short time ranges only (AWS CloudWatch Logs Insights documentation, accessed Oct 2026).

---

7. Streaming: Amazon Kinesis Data Streams

7.1 The metric that defines stream health

Namespace AWS/Kinesis. Stream-level metrics are free and published every minute; enhanced (shard-level) metrics are an opt-in, additional-cost feature (EnableEnhancedMonitoring).

7.2 Worked example E7: the iterator-age alarm

AWS's own guidance: if an iterator's age passes 50% of the retention period you risk records expiring before they are read, and you should use CloudWatch alarms on the Maximum statistic. Threshold arithmetic on a 24-hour default retention is exactly the kind of calculation DEA-C01 expects you to perform, not recall.

---

8. AWS Glue: DPU, bookmarks, observability and job insights

8.1 How Glue metrics behave

AWS Glue reports metrics to CloudWatch every 30 seconds, as delta values from the previously reported values, so a dashboard graphed at 1-minute granularity is an aggregation, and the right statistic per minute is Sum, not Average. The CLI/API metric namespace is Glue.

8.2 Failure alerting is usually EventBridge, not CloudWatch

There is no default "job failed" metric. The canonical alert (skill 3.3.3) is an EventBridge rule on the Glue Job State Change event.

8.3 Worked example E8: alert when the nightly job fails

Bookmarks are the quiet failure mode: a job can succeed while reading nothing because the bookmark watermark says it already did. Alarm on glue.driver.aggregate.recordsRead falling to 0 against an expected volume, and pair it with the state-change rule so silence and success are distinguishable.

---

9. Amazon Redshift: three planes of evidence

Spectrum and usage limits: UsageLimitConsumed carries dimension FeatureType, including SPECTRUM: the bytes scanned by external queries against S3, which is how you alarm a runaway Spectrum query before the bill arrives.

Worked example (WLM congestion): alarm WLMQueueLength with dimension QueueName=etl > 5 for 3 periods, then confirm with QueryRuntimeBreakdown at stage=QueryWaiting: if QueryWaiting is high, queries are queuing (add WLM queues or concurrency); if it is low but QueryDuration is high, the SQL itself is slow. The wrong conclusion from the wrong plane is the trap.

---

10. Amazon EMR: YARN and the 5-minute default

• Default metrics arrive every 5 minutes and are free; on EMR 7.0 and later the CloudWatch agent adds 34 extra metrics at 1-minute granularity (charged), and 7.1 adds YARN and HBase classifications.
• IsIdle is sampled every 5 minutes: AWS's own alarm guidance is to alarm "when IsIdle is 1 for more than one consecutive 5-minute check".
• Capacity and scaling: AppsRunning, AppsPending, ContainerPendingRatio (the scaling signal), HDFSUtilization, and YARNMemoryAvailablePercentage: the driver metric for managed scaling.
• Managed scaling also emits YarnContainersUsedMemoryGBSeconds, which AWS describes as critical for managed scaling.
• Logs: the agent ships /mnt/var/log/hadoop-yarn/... into CloudWatch log groups, which is what makes Skills 3.3.6/4.4.5 (analyze EMR logs, use EMR for large log volumes) answerable with Logs Insights or a subscription filter to S3.

Exam reading: an EMR question that demands 1-minute metrics is really asking which component publishes them: the answer is the CloudWatch agent on EMR 7.0+, not a console setting.

---

11. AWS Lambda: throttles, duration and errors

Metrics arrive at 1-minute granularity.

Event source mapping metrics are opt-in: PolledEventCount, InvokedEventCount, FailedInvokeEventCount, DroppedEventCount, MaxOffsetLag. ESM delivery is at-least-once, so counts can double: never read them as an exactly-once count of business events.

Alarm design: Errors / (Invocations + Throttles) as metric math is misleading because throttles are excluded from both; alarm Throttles > 0 and Errors > 0 separately, then investigate concurrency limits versus code.

---

12. AWS DMS: CDC latency is two numbers, not one

Metrics live in AWS/DMS, are expressed in seconds, and are dimensioned by instance and task.

• CDCLatencySource: the last event captured at the source versus the replication-instance clock: the source-read side.
• CDCLatencyTarget: the oldest unconfirmed event waiting to commit on the target versus the clock: the target-apply side.
• CDCLatencyTarget is always greater than or equal to CDCLatencySource.

Worked example E9: read the two latencies before you touch anything

That ordering, source first when both are high, is the single most testable DMS monitoring fact, and it follows from the inequality: target latency includes everything source latency does.

---

13. AWS Step Functions: execution history and backlog

Namespace AWS/States, dimension StateMachineArn; without a dimension the metrics are account-level.

• Execution metrics: ExecutionsStarted, ExecutionsSucceeded, ExecutionsFailed, ExecutionsTimedOut, ExecutionThrottled, ExecutionTime (milliseconds). Use Sum to count executions and Average to characterise durations.
• Gotcha: Step Functions emits two ExecutionsStarted metrics for every state machine execution, so SampleCount is 2: never read SampleCount as a count of executions.
• Metrics are emitted best-effort; a non-ASCII state machine name prevents CloudWatch logging.
• Activity metrics (dimension ActivityArn): ActivitiesSucceeded, ActivitiesFailed, ActivitiesTimedOut, ActivitiesHeartbeatTimedOut.
• Backlog: OpenExecutionCount against the limit (default 1,000,000): AWS recommends a Maximum alarm at 100,000+; for Distributed Map, ApproximateOpenMapRunCount against a limit of 1,000, alarmed at 900+.
• The execution history in the console (and the GetExecutionHistory API) is the per-run evidence: CloudWatch gives you the aggregate, execution history gives you the state-by-state story.

---

14. Amazon S3: request metrics versus storage metrics

Also: CloudWatch delivery of S3 metrics is best-effort, so treat a missing data point as ambiguity, not as zero traffic.

Worked example (prefix error rate): to alarm 4xx on s3://curated/, first create a metrics configuration filtered by that prefix, then alarm 4xxErrors on dimension FilterId=curated. There is no way to get a per-prefix error rate from the free daily storage metrics: the exam reuses this pattern constantly.

---

15. Amazon EventBridge: invocations, failures and the dead-letter queue

Namespace AWS/Events.

• Invocations: successful plus failed first attempts; it excludes retries until they become permanent failures, and is emitted only when non-zero.
• FailedInvocations: permanent failures only, also only when non-zero.
• Delivery detail: InvocationAttempts, SuccessfulInvocationAttempts, RetryInvocationAttempts; a useful metric-math alarm is a SuccessfulInvocationRate = SuccessfulInvocationAttempts / InvocationAttempts.
• Troubleshooting order: TriggeredRules = 0 ⇒ your pattern never matched; Invocations = 0 but the rule fired ⇒ target configuration (permissions, ARN); failures with invocations ⇒ the problem is in the target.
• Retries run for up to 24 hours / 185 attempts, after which the event can be sent to an optional dead-letter queue: at which point DLQ depth is an SQS metric, not an EventBridge one.

Worked example E10: DLQ depth plus permanent failures

The pairing matters: FailedInvocations says "gave up", depth says "how many are waiting", and retry attempts say "still trying": three different urgencies from three different metrics.

---

16. Logging best practices for pipelines

16.1 Six rules that earn marks (Skills 3.3.2, 3.3.7, 4.4.2)

1. Structured JSON logs. Field indexes, parse, filter and metric filters all assume structure; free-text lines force @message LIKE scans billed by bytes.
2. Correlation IDs everywhere. Step Functions $$.Execution.Id, Glue JobName + JobRunId, Lambda request ID, DMS task ID. One ID lets a single Insights query join across log groups and stages.
3. Never log PII when you can avoid it; if you must, attach a Logs data protection policy: detection and masking happen when the data is ingested, so events written before the policy existed are not masked, reading masked data requires logs:Unmask, and detections publish the free (vended) metric LogEventsWithFindings.
4. Set retention per log group: the default is indefinite, deletion takes up to 72 hours.
5. Centralize across accounts: a CloudWatch monitoring account, Logs Centralization, an organization CloudTrail in one bucket, or a subscription filter to Firehose landing in a central S3 bucket.
6. Keep the two planes apart: audit evidence goes to CloudTrail (then CloudTrail Lake / Athena), operational logs go to CloudWatch Logs.

16.2 Worked example E11: PII guardrail alarm

16.3 The alerting chain, end to end

Skill 3.3.3 ("use notifications during monitoring to send alerts") is satisfied only when a signal becomes a notification: metric (service, custom or EMF) or log alarm → alarm state change → Amazon SNS topic (email, SMS, Lambda, Chatbot) or an EventBridge target. CloudWatch alarms do not email by themselves, and because actions fire only on state change, escalation policies must live downstream of the first notification.

• 📚 Did you know? The three cross-account log patterns are genuinely different tools: a monitoring account (CloudWatch cross-account observability) lets one Logs Insights query span linked log groups: /aws/glue/* across six accounts in one query; Logs Centralization moves log groups' delivery to a central account; an organization CloudTrail puts audit files from every account into one S3 bucket. Choosing between them is a Skills 3.3.1/4.4.5 question, and the wrong option usually mixes two of the three (AWS CloudWatch and CloudTrail documentation, accessed Oct 2026).

---

2026 Updates (as of October 2026)

---

Real-World Case Studies

Every figure below is customer- or AWS-claimed and unaudited, with the source named so you can check it. The examinable point is the pattern, which signals had to be alarmed, and why the operation survived, not the marketing.

Case A, AGCO: one person running a telemetry pipeline

Read it as a monitoring story: a pipeline that one person runs can only work if the alarms are automatic. The two signals that matter here are exactly Section 7's, GetRecords.IteratorAgeMilliseconds on Maximum for readers falling behind, and WriteProvisionedThroughputExceeded for producers being throttled, plus Firehose delivery health downstream. −78% is a customer claim (accessed Oct 2026), not a guarantee.

Case B, FINRA: regulated scale and the audit plane

Derived arithmetic (label it as such): 37,000,000,000 ÷ 86,400 s ≈ 428,000 records/second; a busy 75-billion day ≈ 868,000 records/second. At that rate the audit trail cannot be a by-product: CloudTrail management events plus an organization trail into one bucket, and EMR/YARN metrics at Section 10's granularity, are what make "prove what happened" answerable.

• 📚 Did you know? FINRA's two published stories split along the two planes of this lesson: the surveillance architecture is an operations story (EMR, HBase, YARNMemoryAvailablePercentage, HDFS utilization), while the Consolidated Audit Trail is an audit story (CloudTrail, KMS, GuardDuty, Redshift). Both are AWS-published and unaudited (accessed Oct 2026): the examinable pattern is which plane a question names, not which customer achieved which percentage.

Case C, Nasdaq: two paths, two sets of alarms

Read it as a monitoring story: because storage and compute are decoupled, a slow query cannot block ingestion, and the two paths therefore alarm on different namespaces. The write path is AWS/S3, 4xxErrors, 5xxErrors, FirstByteLatency on a metrics configuration (Section 14), while the read path is AWS/Redshift, HealthStatus, WLMQueueLength, QueryRuntimeBreakdown at stage=QueryWaiting (Section 9). Object Lock turns the archive tier into WORM evidence, which is the Domain 4 half of the same architecture. Derived (label it): 70,000,000,000 ÷ 86,400 s ≈ 810,000 records/second sustained, a rate no nightly human review can keep up with.

• 📚 Did you know? AWS's Nasdaq case study quotes Robert Hunt, VP Software Engineering: "We were able to easily support the jump from 30 billion records to 70 billion records a day because of the flexibility and scalability of Amazon S3 and Amazon Redshift." The scaling headroom came from the architecture, not from a monitoring feature; which is why the exam rewards you for separating write-path alarms (S3 request metrics, opt-in and billed) from read-path alarms (Redshift cluster and WLM metrics, free) rather than for knowing any single threshold (AWS Nasdaq case study, accessed Oct 2026).

Case D, Oportun: discovering PII that nobody indexed

Read it as a monitoring story: Macie is inventory + classification + score, so the signals you alarm are the bucket inventory, the sensitivity score trend, and findings routed through Amazon EventBridge to SNS or a Step Functions remediation: the same state-change-to-notification chain as Section 16.3. Keep the boundary sharp: Macie discovers PII in Amazon S3; it does not natively scan Amazon RDS or Amazon Redshift, and PII inside log lines is the job of a CloudWatch Logs data protection policy (LogEventsWithFindings, Section 16.2), not Macie. The +95% / −80% figures are AWS-published customer claims (accessed Oct 2026), not a benchmark you can generalise.

• 📚 Did you know? Amazon Macie's own limits are monitoring facts: an account's bucket inventory covers up to 10,000 buckets, custom identifiers must be 1 to 300 characters (default 50), and detections default to MEDIUM severity, so "we scanned everything" is a claim about an inventory limit, not about the absence of PII, and a sensitivity score that stops moving deserves the same scrutiny as a metric that stops publishing (Amazon Macie documentation and document history, accessed Oct 2026).

---

Practice Questions

Comparative Verdict: how this topic compares on exam day

• Versus other clouds: the DEA-C01 guide tests AWS services only: CloudWatch, CloudWatch Logs, CloudTrail, CloudTrail Lake, and the native metrics of Glue, Kinesis, Redshift, EMR, Lambda, DMS, Step Functions, S3 and EventBridge. Any option pivoting to a competitor's observability stack, or to a third-party APM statistic, is out of scope by construction; answer with an in-scope AWS service or an AWS-published metric name.
• Versus self-managed / on-premises (Prometheus, ELK, syslog you run): you can build the same pipeline yourself, but then you own collectors, retention storage, query capacity and cross-account routing. The exam's preference is the managed, integrated path, service metrics → alarm → SNS, logs → CloudWatch Logs → Insights, with self-run tooling appearing as the concept AWS implements. Note that AWS X-Ray is explicitly out of scope, so "just add distributed tracing" is a distractor even where it would be sound engineering.
• Versus another AWS service in the same layer: CloudTrail is the audit plane (API calls), CloudWatch Logs is the operations plane (application output), CloudWatch metrics/alarms are the alerting layer, AWS Health is the "us or AWS?" signal, and CloudTrail Lake / Athena / Logs Insights are the analysis layer. A question asking "who called the API" never answers "CloudWatch Logs", and one asking "job error rate" never answers "CloudTrail".
• Versus a manual, human process: AGCO's one-operator pipeline and FINRA's regulated scale (customer claims, accessed Oct 2026) are outcomes of mechanisms, iterator-age alarms, state-change rules, YARN metrics, organization trails, not of someone watching a screen. On the exam, prefer the option that produces a notification or a queryable record over the option that requires someone to remember to look.