Orchestrating Data Pipelines: Step Functions, EventBridge and Airflow

A data pipeline that transforms correctly but starts late, runs twice, silently skips a failed step or cannot be replayed has still failed. Ingestion and transformation (Lessons 02 and 03) decide what moves and how it is reshaped; orchestration decides when it runs, in what order, what happens when it fails, and how you recover, and DEA-C01 tests exactly that judgment in Domain 1, Task 1.3 "Orchestrate data pipelines", through four skills: 1.3.1 orchestration services (AWS Lambda, Amazon EventBridge, Amazon MWAA, AWS Step Functions, AWS Glue workflows), 1.3.2 resilient and scalable pipelines, 1.3.3 serverless workflows and 1.3.4 alerts via SNS and SQS (DEA-C01 exam guide, accessed Oct 2026).

In this lesson you will:

• separate Standard and Express workflows on duration, delivery guarantee, billing and supported integration patterns;
• read and write Amazon States Language: the eight state types, Retry, Catch, built-in error names and terminal error states;
• choose between inline and Distributed Map and size ItemReader, ItemBatcher and ResultWriter;
• build EventBridge rules on schedules and event patterns, understand buses, targets, retry policies and DLQs;
• use EventBridge Pipes for code-free stream/queue fan-out and EventBridge Scheduler for time-zone-aware schedules;
• describe an Amazon MWAA environment, its DAG storage, operators, worker classes and Airflow versions;
• wire AWS Glue Workflows and triggers (schedule, on-demand, EventBridge, conditional) into one graph;
• apply the orchestration selection matrix: cron → EventBridge, multi-step retries → Step Functions, complex DAGs → MWAA, event-reactive flows → Pipes;
• apply the retry, DLQ, idempotency and backfill patterns that make a pipeline resilient;
• study four AWS customer case studies, read the October 2026 update box, and practise with 14 exam-style questions plus interactive checks.

---

1. The orchestration surface: five primitives, one job

1.1 What Task 1.3 is really asking

Every orchestration question on this exam reduces to four prompts: What starts the run? (schedule, event, API call, upstream success) What order do steps run in? (serial, parallel, fan-out, conditional) What happens on failure? (retry, catch, alert, park in a DLQ) How do you run it again without corrupting data? (backfill, redrive, idempotency). Different services answer those four prompts differently, and the exam's distractors are always a correct service used for the wrong prompt.

The Application Integration category on the DEA-C01 in-scope list is exactly this family: Amazon EventBridge, Amazon MWAA, Amazon SNS, Amazon SQS, AWS Step Functions, with AWS Lambda as the compute that most of them call (DEA-C01 in-scope services page, accessed Oct 2026).

1.2 Resilient and scalable is a checklist, not an adjective

Skill 1.3.2 expects you to name the mechanisms, not admire them:

1. Retry with backoff and jitter: transient failures re-run automatically before anyone is paged.
2. Catch / error state: unrecoverable failures route to a defined place instead of vanishing.
3. Dead-letter queue (DLQ): events or messages that exhausted retries are preserved for inspection and replay.
4. Idempotency: the same input processed twice produces the same output once (conditional writes, idempotency keys, job bookmarks).
5. Backfill and redrive: re-run a window of history or a set of failed executions deliberately.
6. Alerts: SNS for humans, SQS for machines, CloudWatch alarms on the metrics (ExecutionsFailed, ExecutionThrottled, FailedInvocations, InvocationThrottleCount).

• 📚 Did you know? The four CloudWatch metric names above are the exam's cheapest troubleshooting shortcut: ExecutionsFailed and ExecutionThrottled belong to Step Functions, while FailedInvocations and InvocationThrottleCount belong to EventBridge targets and Pipes. A question that pairs the wrong metric with the wrong service is testing whether you know which layer failed: the state machine, or the delivery of the event to its target (AWS CloudWatch documentation, accessed Oct 2026).

---

2. AWS Step Functions: Standard versus Express

2.1 The core exam axis

AWS offers two workflow types, and the difference is not speed; it is duration, delivery guarantee, meter and available integration patterns (AWS Step Functions developer guide, Standard vs Express, accessed Oct 2026):

The one-line rule: anything that must wait (.sync, .waitForTaskToken, a human approval lasting days, a one-year run) or must be exactly-once is a Standard workflow; anything that is short, high-volume and idempotent is an Express workflow.

2.2 Worked example E1: pricing a Standard pipeline

A nightly pipeline has 9 state transitions per run; one state retries 3 times before succeeding, adding 3 transitions per run (retries are billable state transitions). It runs 10,000 times a day for a 30-day month (Step Functions pricing, us-east-1, as of Oct 2026):

Two lessons fall out: retries cost money (they are transitions), and a Wait state that loops is a meter you run on purpose.

2.3 Worked example E2: same pipeline, two workflow types

The same 6-state, 4-second, under-64-MB job, run 1,000 times:

More than 20x cheaper for Express at this shape, and the gap widens as duration grows, because Express bills duration while Standard bills only transitions. AWS's own 2022 cost-optimisation post illustrates the same asymmetry with a $0.42 Standard versus $0.01 Express comparison (AWS News Blog, 2022-08-30). The corollary exam question is the reverse: a workflow that waits 12 hours in a Wait state would burn transitions if you polled, whereas a callback token waits for free (see E9).

• 📚 Did you know? Express workflows are the only workflow type you can start with StartSyncExecution from the console, and that synchronous console result expires after 60 seconds, so anything longer belongs in an SDK/CLI call. Nesting is also officially endorsed: a Standard parent can start a nested Express child, and starting a nested workflow costs exactly one state transition (AWS Step Functions pricing and best-practices pages, accessed Oct 2026).

---

3. Amazon States Language: the eight state types

3.1 The vocabulary

State machines are defined in Amazon States Language (ASL): a JSON-based declarative language (states-language.net specification, accessed Oct 2026):

Every state declares Type and either Next xor End, plus the input/output filters InputPath, ResultPath, OutputPath, an optional TimeoutSeconds, and optional Retry and Catch arrays. The query language is JSONPath (paths written with a .$ suffix, for example "Bucket.$": "$.bucket") or JSONata ({% … %} blocks). Context paths reach the execution itself: $$.Execution.Id, $$.State.Name, $$.Task.Token and $$.Map.Item.*.

3.2 Worked example E3: a real ASL fragment

Read it as a story: the Glue job runs synchronously (.sync waits for the job to finish), timeouts are never retried while everything else retries up to 5 times with exponential backoff and full jitter, unrecoverable errors are caught into $.error and routed to an SNS alert, a Choice enforces a data contract, and the terminal state is an explicit Fail: the exam's definition of an error state.

3.3 Retry mechanics: arrays scan top-down

• Retriers (Retry[]) are scanned in array order; the first entry whose ErrorEquals matches governs. That is why ["States.Timeout"] with MaxAttempts: 0 must come before ["States.ALL"]; this is the canonical "retry everything except timeouts" configuration.
• Defaults (when you omit a field): IntervalSeconds 1, MaxAttempts 3 (so 4 total attempts), BackoffRate 2.0 (must be ≥ 1.0), JitterStrategy NONE unless you set FULL.
• Catchers (Catch[]) run only when there is no matching Retry entry or the retries are exhausted; each carries ErrorEquals, Next and usually a ResultPath such as $.error so the failure payload does not overwrite your data.

3.4 Built-in error names you must not mix up

3.5 Worked example E4: retry timing arithmetic

With the defaults (IntervalSeconds: 1, BackoffRate: 2.0, MaxAttempts: 3):

Set IntervalSeconds: 30, MaxAttempts: 5, BackoffRate: 2.0 and the waits become 30 + 60 + 120 + 240 + 480 = 930 seconds (15.5 minutes) before the task is declared failed: the configuration in E3. JitterStrategy: "FULL" randomises each delay across its window so a fleet of executions does not retry in lockstep (AWS Step Functions error-handling documentation, accessed Oct 2026).

3.6 Task tokens: waiting that costs nothing

For work no .sync pattern covers, a human approval, a third-party API, an external worker, the state machine injects the token and waits:

The external worker later calls SendTaskSuccess, SendTaskFailure or SendTaskHeartbeat. The wait is unbounded (up to the 1-year Standard limit), not billable, and Standard-only, so always set HeartbeatSeconds, otherwise a lost callback hangs until the state timeout instead of failing fast with States.Timeout.

• 📚 Did you know? A human-approval state that sits open for three days costs zero state transitions per hour: the transitions were spent entering and leaving the state, not waiting in it. That asymmetry is why the cheapest long-running designs put a .waitForTaskToken where a Wait-and-poll loop would otherwise live: 1,000 hourly polls cost $3.00 a month, a token callback costs a single transition (worked example E9; Step Functions pricing, as of Oct 2026).

3.7 Service integrations: which pattern each service supports

Step Functions ships optimized integrations (the service interprets the call) plus a generic AWS SDK integration covering over 220 AWS services (AWS What's New, 2026-03-26: +28 services / 1,100+ APIs):

Optimized ≠ SDK: the Lambda optimized integration auto-parses the response payload, Glue's optimized integration returns JobName + JobRunID, and EMR renames JobFlowId to ClusterId. Callback and .sync calls also need extra IAM permissions (glue:GetJobRun, ecs:RunTask plus iam:PassRole): an exam favourite when a pipeline "works in the console and fails in production".

---

4. The Map state: inline versus Distributed

4.1 When the switch happens

Map iterates an input array. Inline (the default) runs every iteration inside the parent's own execution history; Distributed mode runs each iteration as a child workflow execution with its own history (AWS Step Functions developer guide, Map state, accessed Oct 2026):

Configuration lives in ProcessorConfig: { "Mode": "DISTRIBUTED", "ExecutionType": "EXPRESS" | "STANDARD" } plus the fields below:

• ItemReader (+ ReaderConfig.InputType): where the items come from; MaxItems ≤ 100,000,000, and MaxItems XOR MaxItemsPath.
• ItemsPath / ItemSelector; which items, and how each is reshaped before the child runs.
• ItemBatcher: MaxItemsPerBatch, MaxItemsPerBatchBytes (≤ 256 KiB).
• MaxConcurrency / MaxConcurrencyPath: never set it above downstream capacity.
• ResultWriter: results land in S3, not in the parent history.

4.2 Worked example E5: ItemBatcher arithmetic

A Distributed Map fans out 1,130 objects with MaxItemsPerBatch: 100:

$$\text{batches} = \left\lceil \frac{1{,}130}{100} \right\rceil = \lceil 11.3 \rceil = \mathbf{12 \ batches}$$

Batching matters because each batch becomes one child execution, and each child execution costs at least one state transition (or, for Express children, one request). 1,130 children is 1,130 executions; 12 children is 12: same work, a fraction of the overhead. For a 1,000,000-object S3 backfill the arithmetic is different in kind: inline Map would exhaust the 25,000-event history long before the last object, so you need ItemReader on s3:listObjectsV2, ItemBatcher, MaxConcurrency sized to downstream capacity (for example 1,000), ExecutionType: "EXPRESS" children, ToleratedFailurePercentage: 1, and ResultWriter to S3.

---

5. Amazon EventBridge: rules, buses, retries and DLQs

5.1 Three bus types, one rule per bus

A rule watches exactly one bus. Schedules are never allowed on partner event buses; the API reference documents scheduled expressions on the default and custom buses (AWS EventBridge API PutRule, accessed Oct 2026), so treat "schedule on a partner bus" as always wrong.

5.2 Two rule shapes, and one schedule

A rule matches either an event pattern (content-based filtering over JSON fields, up to 2,048 characters) or a schedule expression, or both:

5.3 Worked example E6: writing the cron rule

Requirement: "kick off the warehouse load at 20:00 UTC daily, and also every time a file lands in the raw prefix." One rule cannot do both cleanly in a single expression the way an exam expects; you create two rules on the same bus:

Note the details the exam checks: schedule precision is 1 minute in UTC+0, so cron(0 20 * * ? *) is 20:00 UTC and not local time; Step Functions targets are invoked asynchronously; and a rule carries a maximum of 5 targets.

5.4 Retry policy and dead-letter queue

Per target you can override the delivery retry policy and attach a DLQ (AWS EventBridge Retry policy and DLQ, accessed Oct 2026):

5.5 Worked example E7: from 24-hour retries to a 10-minute DLQ

A team wants failures surfaced in 10 minutes, not a day:

The DLQ must be a standard SQS queue in the same Region: a FIFO queue cannot be used as an EventBridge DLQ. The messages that arrive carry attributes such as ERROR_CODE, EXHAUSTED_RETRY_CONDITION and RETRY_ATTEMPTS, and a KMS-encrypted queue needs kms:Decrypt for events.amazonaws.com.

• 📚 Did you know? Some failures are never retried at all: a missing permission, a deleted resource or bad DNS goes straight to the DLQ with zero retries, which is why "no retries happened" in a support ticket usually means an IAM or resource problem rather than a retry-policy bug. And EventBridge DLQs are plain standard SQS queues: you cannot use a FIFO queue, so a question offering "SQS FIFO as the EventBridge DLQ" is a built-in distractor (AWS EventBridge DLQ documentation, accessed Oct 2026).

5.6 Quotas worth memorising (as of Oct 2026)

• 📚 Did you know? EventBridge also orchestrates security pipelines, not just data ones: Amazon Macie emits its findings to the default bus, and a rule can route them into a Lambda or a Step Functions remediation flow with the same retry policy and standard-SQS DLQ you use for a warehouse load: the wiring behind Oportun's reported 95% sensitive-data discovery accuracy and 80% reduction in the time needed to discover sensitive data. The orchestration skill is unchanged (match on source / detail-type, bound the retries, make the remediation idempotent); only the producer changes (AWS Macie customer material and re:Invent SEC215 session deck, accessed Oct 2026).

---

6. EventBridge Pipes and EventBridge Scheduler

6.1 Pipes: source → filter → enrich → target, with no code

Pipes are point-to-point (not pub/sub): a source, an optional filter (only matching events are billed), an optional enrichment step (Lambda or Step Functions) and a target (AWS EventBridge Pipes documentation, accessed Oct 2026).

6.2 Worked example E8: Pipes billing with a filter

A pipe consumes 10,000,000 SQS messages, a filter lets 25% through, and batching is 5 messages per request (Step Functions pricing page example, as of Oct 2026):

$0.20 for ten million messages; because the filter runs before billing. The examinable principle: filter early, batch to the target's maximum, and remember the 5-minute pipe ceiling.

6.3 Scheduler: schedules without a rule

AWS recommends EventBridge Scheduler for invoking targets on a schedule, and the console labels scheduled rules "(legacy)" (AWS EventBridge Scheduler documentation, accessed Oct 2026).

Flexible time window is the feature exam questions hinge on: MaximumWindowInMinutes: 60 means a nightly job scheduled for 01:00 may fire anywhere between 01:00 and 02:00: perfect for batch windows that must absorb load spikes, wrong for a job that must start exactly at 01:00:00.

6.4 Four things that "run on a schedule" (do not mix them up)

---

7. Amazon MWAA: managed Apache Airflow

7.1 What an environment actually is

An Amazon MWAA environment is a bundle of managed components (AWS MWAA documentation, accessed Oct 2026):

You give up SSH, custom images, custom SQS queues and executor choice, Celery only, in exchange for AWS running the control plane, Aurora, Fargate, logging and your VPC wiring. Existing DAG updates are picked up in about 30 seconds.

7.2 Worker classes and auto scaling (as of Oct 2026)

Workers are added as (RunningTasks + QueuedTasks) / tasks-per-worker, up to max-workers. The per-class celery.worker_autoscale default (3 / 5 / 10 / 20 / 40 / 80) is authoritative, and celery.worker_concurrency is overridden: setting it does nothing.

7.3 Operators and dependencies

DAGs live in S3, dependencies in requirements.txt (pip3) and plugins.zip; a new requirements.txt requires an environment update. The operators that matter for this exam:

Versions available as of Oct 2026: Airflow 3.2.x (announced 2026-04-01), 3.0.x, and 2.11.0 (released 2026-01-07, Python 3.12), plus older 2.9.x/2.7.x/2.4.x lines. Airflow 3 adds asset- and partition-aware scheduling, human-in-the-loop audit history and async PythonOperator.

7.4 Worked example E9: pricing an MWAA environment

AWS's own worked example for an mw1.small environment for a 31-day month: 1 environment, 49 workers running 1 hour a day, 10 GB of metadata-database storage (MWAA pricing, us-east-1, as of Oct 2026):

For comparison, mw1.large is $0.99/h for the environment with extra workers and schedulers at $0.22/h and an extra web server at $0.11/h, and MWAA Serverless bills AWS Managed Tasks at $0.080 per hour of task time (1-minute minimum, 1-second billing). Other quotas: 10 environments per Region (5 of mw1.xlarge/mw1.2xlarge), 25 workers per environment (50 after a quota increase), 5 webservers, 20 GB task storage.

• 📚 Did you know? MWAA's Celery queue is an AWS-managed Amazon SQS queue that you cannot create, replace or configure; which is why questions offering "bring your own SQS queue to MWAA" or "tune celery.worker_concurrency" are both wrong. What you do tune is the worker class, and that single choice moves both the DAG capacity and the default task concurrency: mw1.small → mw1.medium changes the default from 5 to 10 tasks per worker (AWS MWAA documentation, accessed Oct 2026).

7.5 MWAA Serverless

MWAA Serverless removes the always-on environment: YAML DAGs plus a code package in S3, per-task pay-per-use scheduling backed by EventBridge Scheduler, a per-workflow execution role and isolated compute. It is the serverless answer inside the Airflow family, but the Airflow concepts, DAG file and operators are the same. (AWS MWAA Serverless documentation, accessed Oct 2026; Region availability is not repeated here because the launch list was not verifiable in Oct 2026.)

---

8. AWS Glue Workflows and triggers

8.1 One entity, two graphs

A Glue workflow tracks jobs + crawlers + triggers as a single entity with a graph view: a static graph (design) and a dynamic graph (runtime status per node) (AWS Glue Workflows and Triggers, accessed Oct 2026).

8.2 The three start triggers

Inside the graph, conditional triggers express dependencies: job J3 starts after J1 AND J2 SUCCEEDED; job J4 starts after J1 OR J2 FAILED; a crawler starts after the jobs complete. Workflow run properties are shared name/value pairs that jobs read and mutate mid-run: state passing without Airflow XCom.

8.3 The hard rule

All jobs or crawlers in a dependency chain must be descendants of a single scheduled or on-demand trigger. Dependents fire only if a trigger started them: a crawler you start by hand will not fire the next job in the graph. Each trigger supports ≤ 50 jobs and ≤ 2 crawlers, and a Glue workflow is a first-class EventBridge target, so an EventBridge rule or Scheduler schedule can start the whole graph.

8.4 Choosing between Glue Workflows and Step Functions for the same ETL

---

9. The orchestration selection matrix

9.1 The money diagram

9.2 The matrix in words

---

10. Retry, DLQ, idempotency and backfill patterns

10.1 The reliability ladder

10.2 Idempotency: the precondition for every replay

Delivery is at-least-once almost everywhere (Express executions, EventBridge targets, SQS, Pipes), so replay is only safe if processing is idempotent:

• DynamoDB conditional write: ConditionExpression: attribute_not_exists(pk) fails with DynamoDB.ConditionalCheckFailedException, which a Catch can absorb; parallel branches competing for a lock use exactly this trick.
• Lambda Powertools @idempotent with a TTL: the second invocation of the same event returns the cached result.
• Upstream Idempotency-Key header for API-driven pipelines.
• Glue job bookmarks: the transformation-side equivalent: state tracked between job.init and job.commit prevents reprocessing (Lesson 03).

10.3 Redrive, retry and replay are three different things

Step Functions also emits execution status-change events to the EventBridge default bus, so a rule can watch for ExecutionFailed and invoke a Lambda that calls RedriveExecution: orchestration healing itself.

10.4 Worked example E10: designing the recovery path

10.5 Backfill patterns

2026 Updates (as of October 2026)

• 📚 Did you know? The exactly-once / at-least-once wording has been stable since Express launched on 2019-12-05, while everything around it changed: integration coverage more than doubled to 220+ services, Distributed Map grew S3 and Athena sources, and MWAA moved through Airflow 2.4 → 2.11 → 3.0 → 3.2. Exam-wise this means the guarantee table in Section 2 is old, durable material, while integration names and version numbers are the part you must re-check every study cycle (AWS Step Functions announcement, 2019-12-05; Step Functions recent launches, accessed Oct 2026).

---

Real-World Case Studies

AWS publishes what these abstractions look like in production. Every figure below is customer- or AWS-claimed and unaudited, with the source named so you can check it: the examinable point is the pattern (which orchestration problem was solved, which service did the work, which number moved), not the marketing.

Case A, Edmunds.com: a serverless fan-out batch job (700 million derivatives in 8 days)

Read this as the orchestration shape behind fan-out: a driver enumerates work items (the S3 object list), each unit runs as an independent, idempotent, short-lived invocation, and the aggregate completes without a cluster. That is the same shape as a Distributed Map with ItemReader over an S3 prefix or an EventBridge Pipe from a stream to Lambda: the examinable idea is independent items + a bounded concurrency + a defined completion condition.

Case B, EOS Group: a staged migration pipeline with zero data loss

The examinable point is the stage discipline: full load, CDC, validation and load are separate, ordered, retryable steps with a validation gate between them: the same topology you built in Section 3 (Task → Choice → Task → Fail), just implemented with migration tooling. "Zero data loss" is what a Choice state on a row-count contract and an idempotent target buy you; "minimal downtime" is what running the stages as a retryable graph instead of one monolithic script buys you.

Case C, Nasdaq: a deadline-driven nightly batch across a lake house

Read this as a deadline problem, not a tool problem: because the write path (S3) and the read path (Redshift/Spectrum) are decoupled, loading and querying never contend for the same capacity, so the orchestrator can restart a failed stage without pausing the analysts, and a late run degrades one stage instead of the whole night. That is the same reasoning that makes a retryable graph with a validation gate (Case B, Section 3) the default answer when a question describes a batch that simply cannot be late.

Case D, AGCO: a streaming pipeline one person can operate

Here the trigger is the stream, not a cron: a continuously running pipeline is orchestrated by back-pressure and buffering rather than by a schedule, so the resilience checklist in Section 1.2 moves into the consumers: retries with backoff on the Lambda enrich step, a durable landing zone (S3) so a downstream failure replays from the buffer, and CloudWatch alarms instead of a human watching a dashboard. "One person instead of three to five" is the operational dividend the exam is really asking about when it tests skill 1.3.2.

• 📚 Did you know? Nasdaq's published jump from 30 billion to 70 billion records a day works out to roughly 810,000 records per second sustained (70,000,000,000 ÷ 86,400 s, derived arithmetic, not an AWS-published figure) with a hard stop at market open, while AGCO's equivalent streaming pipeline is run by one engineer instead of three to five. Put side by side they make the exam's real distinction: a batch orchestrator is judged by whether it finishes before its deadline, a streaming pipeline by whether one person can keep it healthy, the same two questions, asked of two different failure modes (AWS Nasdaq case study; AWS Architecture Monthly, December 2021, accessed Oct 2026).

• 📚 Did you know? Neither AWS case study names an orchestrator at all: AWS publishes the outcomes and leaves the wiring to you. That is deliberate exam design: DEA-C01 never asks you to recall a customer's tool choice, it asks you to recognise that 700 million independent items implies fan-out with a concurrency bound (Distributed Map or Lambda), and that zero data loss across a cutover implies ordered stages with validation and idempotent writes (AWS case studies, accessed Oct 2026).

---

Practice Questions