Orchestrating the Lakehouse: Step Functions, EventBridge, and MWAA

Ten lessons in, you can land files, convert them, curate them into Iceberg, query them with Athena and hand the result to Redshift. None of that answers the four questions that decide whether the lakehouse is a product or a liability: what starts the run, in what order do the steps run, what happens when a step fails, and how do you run it again without writing the same rows twice? That is orchestration, and it is the layer where most lakehouse incidents actually live: not in the SQL.

The teaching rule for this lesson is short: the orchestrator owns time, order, failure and replay; the compute owns bytes. Keep those two jobs separate and every later design decision gets easier.

In this lesson you will:

• separate Standard and Express workflows on delivery guarantee, duration ceiling, rate, supported patterns and cost per state transition;
• read and write Amazon States Language: the eight state types, the three integration patterns, and Task resources for Glue, EMR, Lambda and the AWS SDK;
• choose between inline Map and Distributed Map, and size ItemReader, ItemBatcher and ResultWriter;
• use Parallel, Retry, Catch, TimeoutSeconds and nested state machines to isolate phases and route failures;
• wire Amazon EventBridge buses, event patterns, schedules, Pipes with filtering and enrichment, API Destinations and Archive/Replay;
• size an Amazon MWAA environment and recognise the Airflow DAG patterns that justify it;
• chain AWS Glue Workflows and triggers, including EVENT triggers started from EventBridge;
• make re-runs safe with deterministic execution names, dedup, checkpoints and watermark handoff;
• run a backfill over date partitions with a Distributed Map;
• apply a scheduler decision matrix and least-privilege execution roles;
• work a bronze to silver to gold state machine with quarantine routing, ten worked examples, a sourced October 2026 update box, Transport for NSW case material, three interactive checks and fourteen exam-style questions.

---

1. The orchestration surface: five tools, five jobs

1.1 The five primitives

Every orchestration question reduces to the same five primitives. Each one owns a different thing, and every design smell in this space is a primitive used for a job it does not own.

The four prompts you must always answer before choosing: What starts the run? (schedule, event, API call, upstream success) What order do steps run in? (serial, parallel, fan-out, conditional) What happens on failure? (retry, catch, quarantine, alert) How do you run it again without corrupting data? (idempotent name, checkpoint, watermark, redrive).

• 📚 Did you know? Step Functions can reach "over 14,000 API actions from over 220 AWS services" as of the AWS Compute Blog of 2025-11-04: an earlier AWS What's New post of 2022-12-01 put the same figure at "over 200 AWS services". The number is a moving target, but the examinable fact is that the AWS SDK integration is the escape hatch for anything without an optimized integration (AWS Compute Blog, 2025-11-04; AWS What's New, 2022-12-01).

---

2. AWS Step Functions: Standard versus Express

2.1 Two workflow types, and no third one

There are exactly two state machine types: Standard (the default) and Express. The type is immutable: chosen at creation and never changed afterwards. The word "sync" names two different things and is the single most common confusion in this service:

1. the .sync integration pattern ("Run a Job"), where a Standard workflow calls a service and blocks until the job finishes; and
2. Synchronous Express (StartSyncExecution), where an Express execution is invoked synchronously from API Gateway, Lambda or the CLI and returns the result.

AWS states the rule without hedging: Standard Workflows "follow an exactly-once model… suited to orchestrating non-idempotent actions, such as starting an Amazon EMR cluster or processing payments", while Express Workflows "use an at-least-once model… better suited to orchestrating idempotent actions" (Step Functions Developer Guide, accessed 2026-10-11).

2.2 Worked example E1: a nine-transition daily pipeline

AWS's own pricing example arithmetic, reproduced with the published rates (us-east-1, accessed 2026-10-11):

The free tier is 4,000 state transitions per month, indefinite. Two lessons hide in that number: a daily pipeline almost never generates a Step Functions bill worth thinking about, and a high-frequency one does.

2.3 Worked example E2: the same pipeline, two meters

Fourteen states, 30,000 runs per month, no retries (derived from published rates, as of Oct 2026):

Express bills duration rounded up to the nearest 100 ms, in 64 MB chunks, and the memory footprint is computed as 50 MB + state machine definition size + execution data size x number of Parallel or Map steps. The three duration tiers AWS prints in its own Example 4 are $0.0600 per GB-hour for the first 1,000 GB-hours, $0.0300 for the next 4,000, and $0.01642 thereafter (Step Functions Pricing, accessed 2026-10-11).

---

3. Amazon States Language and the Task layer

3.1 The eight state types

Amazon States Language (ASL) is a JSON-based document. The state types AWS lists are Task, Choice, Wait, Pass, Succeed, Fail, Parallel and Map: eight of them, and the whole of a state machine is these states wired together with Next and End fields (Step Functions Developer Guide, accessed 2026-10-11).

• Task does work (calls Lambda, starts a Glue job, puts an item in DynamoDB).
• Choice branches on input; Wait sleeps; Pass reshapes data.
• Succeed and Fail terminate the execution deliberately.
• Parallel fans out over branches written into the definition.
• Map fans out over a dataset: inline or Distributed (Section 4).

Three quotas bound the definition itself: maximum state machine definition size is 1 MB (hard), maximum input or output size for a task, state or execution is 256 KiB of UTF-8 encoded data, and maximum execution history size is 25,000 events for a Standard workflow.

3.2 Three integration patterns, and which services get which

Every optimized integration can be invoked in one of three ways:

AWS's optimized-integration matrix splits the lakehouse services cleanly:

• .sync supported: athena, batch, emr, emr-serverless, glue, glue:databrew, sagemaker, states (nested Step Functions itself).
• Request Response plus callback (.waitForTaskToken), no .sync: lambda, sns, sqs, dynamodb, EventBridge.
• AWS SDK integration: Request Response across all supported services; .waitForTaskToken on Standard only; never .sync.

So the concrete Task resources you will actually write in a lakehouse state machine are:

• 📚 Did you know? startExecution.sync and startExecution.sync:2 are not interchangeable: .sync returns the child's output as a string, .sync:2 returns it as JSON. Parsing the wrong one silently corrupts every downstream Payload.$ reference, and the failure shows up much later as a missing field rather than a deployment error (Step Functions Developer Guide, accessed 2026-10-11).

3.3 Worked example E3: the retry budget you forgot to cost

Retries are not free: "Retries are treated as state transitions", and "If you include retry error handling in any steps of your workflow, each retry will be charged as an additional state transition" (Step Functions Pricing, accessed 2026-10-11).

MaxAttempts: 0 means never retry: a valid, deliberate choice for a permanently failing step that should route straight to a Catch.

3.4 Retry, Catch, Timeout and Heartbeat: the exact defaults

Built-in error names you must not mix up:

• States.ALL: must appear alone in a Catcher and cannot catch States.DataLimitExceeded or States.Runtime.
• States.TaskFailed: a wildcard for any known error except States.Timeout.
• States.Timeout, States.HeartbeatTimeout, States.Permissions, States.ItemReaderFailed, States.ResultWriterFailed, States.ExceedToleratedFailureThreshold.
• States.DataLimitExceeded is terminal: a task that emits a payload above 256 KiB dies and no catcher attached to that state can save it.

Three structural rules that catch experienced engineers:

1. Catchers exist on Task, Parallel and Map only: "not for top-level state machine execution failures". A whole-execution failure must be handled by the caller, a nested workflow, or an EventBridge rule on execution status (FAILED / TIMED_OUT).
2. If a nested state machine throws a States.Timeout, the parent receives States.TaskFailed instead.
3. In newer Lambda runtimes, timeouts surface as Sandbox.Timedout: match on Lambda.Unknown, Sandbox.Timedout and States.TaskFailed together.

3.5 Activities: the callback lane that waits up to a year

Activities are a Standard-only construct: an activity worker polls with GetActivityTask, keeps the task alive by sending a heartbeat with SendTaskHeartbeat inside the configured TimeoutSeconds, and replies with SendTaskSuccess or SendTaskFailure. With heartbeats, "an activity in Step Functions can wait up to a year for an execution to complete". After the task times out, Step Functions invalidates the task token and later calls raise TaskTimedOut.

3.6 Nested workflows: phases with their own history and retries

Four resources control nesting: states:startExecution (fire and forget), .sync (wait, string output), .sync:2 (wait, JSON output), .waitForTaskToken (callback). Nesting buys per-phase history isolation, per-phase retries and reuse, but the parent must be able to find out that the child finished, and that detection has two paths with two different permission sets:

• polling: needs states:DescribeExecution;
• EventBridge events: needs events:PutTargets, events:PutRule and events:DescribeRule.

Omit the second set and the parent hangs rather than failing loudly, which is the classic symptom.

3.7 Worked example F1: filling in a real ASL fragment

---

4. Map, Distributed Map, Parallel and ItemBatcher

4.1 When inline Map stops being enough

Parallel iterates over branches fixed in the definition; Map iterates over a dataset. Inline (default) Map runs its iterations inside the parent's execution history, and that creates three hard ceilings:

AWS's own stated conditions for switching: use Distributed mode when "the size of your dataset exceeds 256 KiB", "the workflow's execution event history would exceed 25,000 entries", or "you need a concurrency of more than 40 concurrent iterations" (Step Functions Developer Guide, accessed 2026-10-11).

Distributed Map specifics worth committing to memory:

• ItemReader sources: S3 objects (CSV, JSON, JSONL, Parquet), or an Athena data manifest: the 2025-11-04 Compute Blog confirms Distributed Map can "process AWS Athena data manifest and Parquet files directly, eliminating the need for custom pre-processing".
• ExecutionType on children: STANDARD or EXPRESS: use EXPRESS children for short, idempotent fan-out units.
• MaxConcurrency / MaxConcurrencyPath: if unset or zero, Step Functions "doesn't limit concurrency and runs 10,000 parallel child workflow executions", and AWS recommends not exceeding the capacity of a downstream service such as Lambda.
• ToleratedFailureCount / ToleratedFailurePercentage / their ...Path variants: the default percentage is zero, meaning "the workflow fails if any one of its child workflow executions fails or times out". Exceeding the threshold raises States.ExceedToleratedFailureThreshold, which needs its own Catch.
• Quotas: 1,000 open Map Runs per account/Region, 1,000 redrives per Map Run, 10,000 parallel child executions per Map Run (hard), and Step Functions dispatches express children at up to 1,000 TPS and standard children at up to 100 TPS.

4.2 ItemBatcher: the cheapest lever you have

ItemBatcher groups items so each child receives {"Items": [...]} plus an optional static BatchInput. Two knob pairs: MaxItemsPerBatch / MaxItemsPerBatchPath, and MaxInputBytesPerBatch / MaxInputBytesPerBatchPath: the byte ceiling is 256 KiB, the Step Functions payload limit itself.

AWS's own worked figure: "imagine you run an execution with an input JSON file that contains 1130 nodes. If you specify a maximum items value for each batch of 100, Step Functions creates 12 batches" (Step Functions Developer Guide, accessed 2026-10-11). Use batching "when processing large CSV files or JSON arrays, or large sets of Amazon S3 objects."

4.3 Worked example E4: batching arithmetic and its billing effect

• 📚 Did you know? The Distributed Map observability metrics announced on 2025-09-18, ApproximateOpenMapRunsCount, OpenMapRunLimit and ApproximateMapRunsBacklogSize, exist because fan-out depth is otherwise invisible until you hit the 1,000-open-Map-Run quota. Alarm on the backlog metric before the limit, not after (AWS What's New, 2025-09-18; AWS Compute Blog, 2025-11-04).

4.4 Sample projects worth reading before you build one

Two AWS sample projects make good teaching fixtures: a weather data pipeline where "weather data is recorded in over 12,000 CSV files" with MaxConcurrency set at 3,000, copying then analyzing then reducing into DynamoDB; and a CSV shipping-orders pipeline of "10,000 rows of a CSV file" processed in batches of 10 with MaxConcurrency 1,000, results written to S3 through ResultWriter (Step Functions sample projects, accessed 2026-10-11).

---

5. A worked lakehouse state machine: bronze to silver to gold

This is the shape you should be able to draw from memory. A Standard parent hosts a Distributed Map (Express could not), children run as EXPRESS executions with MaxConcurrency: 100 so Glue's account-level job concurrency is never starved, and partial failure routes to quarantine instead of failing silently.

Teaching notes, all of them load-bearing:

1. SilverMap must be Distributed: an inline Map's 40-iteration cap and shared 25,000-event history will kill a multi-thousand-partition run.
2. ItemBatcher.MaxItemsPerBatch: 20 turns thousands of files into tens of children. Children are the unit of both parallelism and billing.
3. Retry on the Map creates a new Map Run: cap MaxAttempts, and use redrive (14-day window, Standard only) for whole-Map replays.
4. Catch on States.ExceedToleratedFailureThreshold routes partial failure to quarantine. Without it, a threshold breach is an unhandled execution failure.
5. The watermark is written last. Record s3://lake/_checkpoints/silver/dt=2026-10-11.json only after PublishCatalog succeeds; that is what makes re-triggering the same day idempotent.
6. Quarantine is a destination, not a state. Land the failing partition keys somewhere a human or a ticket system can see them.
7. ValidateInput is a Choice state, and the FailFast branch ends in a Fail state: an early, cheap exit beats discovering a bad date range three hours into a fan-out.

5.1 Worked example E5: pricing this exact machine

That is the single most important cost lesson in this lesson: Step Functions is a rounding error next to the compute it schedules. Optimise for failure routing and replay safety, not for transition count: with the single exception of unbounded retry policies (Section 3.3).

---

6. Amazon EventBridge for the lakehouse

6.1 Buses, rules, quotas

AWS gives every account a default event bus that receives AWS service state-change events (management events ingest free; opt-in data events such as S3's must be enabled). The pricing page describes four types of event bus: the default bus; two custom buses, Custom event bus: Classic (July 2019) and an enhanced Custom event bus launched in September 2026, and a partner event bus for SaaS vendor events (Amazon EventBridge Pricing, accessed 2026-10-11).

6.2 Event patterns

Patterns match on source, detail-type, account, region and detail, with the operator set And, anything-but (including prefix, suffix, ignore-case and wildcard forms), prefix, suffix, empty, equals-ignore-case, exists, null, numeric ranges, $or, wildcard and cidr. Two sharp edges: "Currently the wildcard character is supported in event bus rules only", and any pattern where $or expands to more than 1,000 rule combinations makes PutRule, CreateArchive, UpdateArchive and TestEventPattern throw InvalidEventPatternException (EventBridge User Guide, accessed 2026-10-11).

6.3 Schedules: legacy rules versus EventBridge Scheduler

Scheduled rules are explicitly "a legacy feature of EventBridge" and AWS "recommend[s] that you use Scheduler to invoke targets on a schedule". Legacy cron has six required fields (minutes, hours, day-of-month, month, day-of-week, year) and "Cron expressions that lead to rates faster than 1 minute are not supported"; everything runs in UTC.

EventBridge Scheduler is the current answer: three schedule types, rate-based, cron-based and one-time, with "60 second precision", automatic daylight-saving-time handling (on a spring-forward gap the invocation is skipped), flexible time windows of 1 to 1,440 minutes, retry and event-age policies, and templated targets across hundreds of AWS services. Quotas: 10,000,000 schedules per Region, 5,000 TPS CreateSchedule, 1,000 TPS invocations.

Pricing: $1.00 per million invocations after 14,000,000 free invocations per month (EventBridge Pricing, accessed 2026-10-11).

6.4 Pipes: source to filter to enrich to target

A pipe is "choose the source, add optional filtering, define optional enrichment, and choose the target": with ordering preserved when the source enforces it, batching, partial-failure reporting, and its own IAM role under pipes.amazonaws.com.

• Sources: DynamoDB Streams, Kinesis streams, Amazon MQ, MSK, SQS, and Apache Kafka (self-managed, Confluent Cloud, CloudKarafka, Redpanda). Do not treat any published list as exhaustive: the AWS docs page is the authority.
• Targets: API destination, API Gateway, Batch job queue, CloudWatch log group, ECS task, same-account and same-Region event bus, Firehose delivery stream, Kinesis stream, Lambda (SYNC or ASYNC), Redshift Data API queries, SageMaker AI pipelines, SNS topic (FIFO not supported), SQS queue, Step Functions (Express SYNC or ASYNC; Standard must be ASYNC), Timestream.
• Payload ceilings: Pipes accept 6 MB, but the target constrains you: an event bus target is 1 MB and a Step Functions state machine target 256 KB. Enrichment output above the target's limit fails the invocation.
• Batch sizes are the minimum of source and target: Kinesis allows 10,000-record batches, SQS targets only 10, Firehose and Kinesis 500, CloudWatch Logs 10,000, event bus 10, SNS 10, so the pipe caps at the smaller number.
• Pipe execution ceiling: 5 minutes including enrichment and target processing: "This limit currently can't be increased."
• Concurrency defaults: SQS 1,250; Kafka up to the partition count (max 1,000); Amazon MQ 5; Kinesis and DynamoDB = ParallelizationFactor multiplied by the number of shards.
• Partial batch failure uses the payload shape {"batchItemFailures": [{"itemIdentifier": "id2"}]}.
• Billing: "You are only charged for those events that match the filter": $0.40 per million requests, measured in 64 KB chunks, post-filter.

6.5 API Destinations, Archive and Replay

API Destinations let a rule call an external HTTP endpoint over an AWS-managed connection: $0.20 per million events, with 3,000 destinations and 3,000 connections per account/Region and 300 invocations per second per destination.

Archive and Replay is the lake's event black box. "By default, EventBridge stores events in the archive indefinitely"; "Replaying events does not remove them from the archive… EventBridge only removes events when they exceed the archive retention period, or you delete the archive itself"; replays are deleted after 90 days; you may have at most 10 active concurrent replays per account per Region; and "you can replay events only to the same event bus that emitted the events."

6.6 The lakehouse event path: S3 straight into EventBridge

Once EventBridge is enabled, S3 delivers every object-level event without per-type configuration: a rule filters source: aws.s3, detail-type: Object Created, the bucket name, a prefix of bronze/orders/ and a suffix of _SUCCESS, and targets a state machine, a Pipe, a Lambda function or an SQS queue. This replaces per-type S3 event notifications plus a Lambda fan-out when you need many consumers, content filtering, archives and cross-account routing.

6.7 Worked example E6: the event-driven lake bill

• 📚 Did you know? S3 does not make you enumerate event types. Once EventBridge is enabled on the account, "you don't need to select which event types you want to deliver": all of them flow, including Object Created, Object Deleted, Restore Initiated / Completed / Expired, Storage Class Changed, Object Tags Added/Deleted, Retention Updated and Annotation Created/Removed. The filter belongs in the rule, not in a per-type notification configuration (Amazon S3 User Guide, accessed 2026-10-11).

---

7. Idempotent orchestration: surviving the second run

7.1 Four mechanisms, in the order you build them

Names are unique per account and Region, capped at 80 characters, and execution history is retained 90 days (reducible to 30 by quota request). A practical daily pattern:

7.2 Checkpoints and watermark handoff

Step Functions does not track data watermarks; this is an engineering pattern built on verified primitives, not a managed feature. Persist the watermark after the commit, in one of: a DynamoDB item, an S3 marker object such as _checkpoints/silver/dt=2026-10-11.json, an Iceberg snapshot id, or a Glue job bookmark. The reader must then take its start bound from that watermark, which makes re-reading a closed window a no-op rather than a duplicate write.

Three cheap determinism rules complete the picture: derive object keys and context identifiers from (layer, partition_date, source); keep every downstream Payload field referenced through .$ paths rather than assumed; and make the success marker the only thing a downstream rule filters on, so a partial write can never masquerade as completion.

7.3 Worked example E7: designing the safe re-run

7.4 Ordering the idempotent pipeline

---

8. Backfill with a Distributed Map over date partitions

Backfill is where the Distributed Map stops being a curiosity and becomes the cheapest tool you own: one parent execution, hundreds of date-partition children, concurrency bounded by what Glue can actually absorb.

8.1 Worked example E8: a 90-day backfill

Three rules for production backfills:

1. Bound MaxConcurrency to the downstream service. AWS's own guidance: "we recommend that you don't exceed the capacity of a downstream AWS service, such as AWS Lambda", and 10,000 children at once will hit Glue's account job quota instantly.
2. Set ToleratedFailurePercentage deliberately. The default is zero, so one corrupt partition fails the whole Map Run.
3. Prefer redrive over Retry on a Map. Retrying a Map state "creates a new Map Run"; redrive reuses the existing one within the 14-day window.

• 📚 Did you know? A backfill is also the cheapest way to test your idempotency design: re-run a window that is already committed and watch whether the writer converges or appends. If the second run changes row counts, your watermark handoff is broken before an incident finds it for you.

---

9. Amazon MWAA: managed Airflow for the lakehouse

9.1 What an environment actually is

Amazon MWAA runs Apache Airflow on AWS Fargate, workers, schedulers and a web server, backed by a single-tenant Aurora PostgreSQL metadata database in your own VPC, with DAGs and plugins stored in an S3 bucket. Sizes run mw1.micro, mw1.small, mw1.medium, mw1.large, mw1.xlarge and mw1.2xlarge. Auto-scaling: workers 10 default / 1 minimum / 25 maximum, scaling back to minimum when queued plus running tasks stay at zero for more than two minutes; schedulers 2 default / 2 minimum / 5 maximum.

AWS's sizing advice is blunt: "We recommend choosing the smallest size necessary to support your workload."

Metadata database storage: $0.10 per GB-month. All rates US East (N. Virginia) from the AWS Price List offer file published 2026-09-11, accessed 2026-10-11. mw1.xlarge gives 8 vCPU / 24 GB for scheduler and worker with a 40-task default and up to 2,000 DAGs; mw1.2xlarge gives 16 vCPU / 48 GB, 80 tasks, 4,000 DAGs; an environment size upgrade takes 20 to 30 minutes to complete.

You pay "for the time your Airflow Environment runs plus any additional auto-scaling", billed at one-second resolution; which is why the environment rate, not the worker rate, dominates the bill.

9.2 Version hygiene

As of October 2026 MWAA offers Airflow 3.3.1 (available since 2026-09-01, Python 3.12), 3.2.1 (2026-05-19), 2.11.2 (2026-07-23), 2.11.0 (2026-01-07) and 3.0.6 (2025-10-01). Effective December 30, 2025, Amazon MWAA ended support for Apache Airflow versions v2.4.3, v2.5.1 and v2.6.3. AWS commits to supporting at least three minor versions at any given time, at least 12 months per version, with 180 days of end-of-support announcement.

Two constraints that bite: MWAA supports minor-version upgrades in place, "but not to a new major version", so Airflow 2 to 3 is a migration, not an upgrade, and from Airflow v2.7.2 onward your requirements file must include a --constraint statement. AWS also documents that "Amazon MWAA does not currently support multi-team mode."

9.3 Which lakehouse DAG patterns MWAA earns

Airflow earns its always-on cost in exactly four situations:

1. Checkpoint resume over long windows. AWS's own MWAA-versus-Step-Functions blog of 2026-06-10 frames MWAA around "resume from the last successful checkpoint" for "long-running processes extending 2-12 hours" over "5-10 TB of financial transaction data daily", against Step Functions' "built-in retry logic with exponential backoff" and "parallel execution".
2. Human-readable cross-team DAGs with catchup semantics, sensors and a UI for backfills.
3. Seasonal or parameterised backfills where an analyst picks a date range and expects a progress view.
4. Heterogeneous task types, ECS, Bash, Python, external APIs, under one dependency graph.

9.4 Worked example E9: MWAA versus Step Functions per month

AWS's own large-environment example lands at $1,047.46 per month, of which the environment ($736.56) plus schedulers ($163.68) is roughly 86% fixed cost; which is exactly the gap MWAA Serverless addresses. A once-a-day pipeline on a medium environment costs 744 x $0.74 = $550.56 per month before a single worker runs.

• 📚 Did you know? MWAA Serverless (GA 2025-11-17, available in 15 AWS regions at launch) is a separate deployment option, not a size tier: workflows are defined in YAML (a Python DAG converted to YAML), started with start-workflow-run, scoped by per-workflow IAM, and billed "per task for its duration, for a minimum of 1 minute" at $0.08 per hour for the AWS-managed task: with "Automatic scaling based on workflow demand" and no environment to size at all (Amazon MWAA Serverless User Guide; AWS Big Data Blog, 2025-11-17).

---

10. AWS Glue Workflows and event triggers

Glue can orchestrate itself without Step Functions. "A trigger fires on demand, based on a schedule, or based on a combination of events", with Type values SCHEDULED, CONDITIONAL, ON_DEMAND and EVENT.

Four pitfalls worth memorising:

1. Conditional triggers only see runs a trigger started. If a job was started some other way, its completion never fires the conditional trigger.
2. An EventBridge-started workflow run that fails consumes its event: "that event or batch of events is no longer considered for starting a workflow run", and there is no redelivery.
3. Glue does not retry workflow runs that fail due to exceeded concurrency limits.
4. Crawlers and jobs are not interchangeable inside one trigger: the two-crawler cap is a hard rule, and exceeding 100 components can break resume and stop operations.

10.1 Worked example E10: Glue-only chain versus Step Functions

---

11. Security: least-privilege execution roles

Orchestration roles are a privilege-escalation surface, because a state machine can start jobs that write to your lake. AWS's own Distributed Map least-privilege sample grants exactly two things:

The shape of that second statement is the point: StartExecution takes a state machine ARN; DescribeExecution and StopExecution take an execution ARN. Granting DescribeExecution against stateMachine:... grants nothing.

A complete least-privilege checklist for a lakehouse orchestrator:

---

12. The scheduler decision matrix

12.1 The matrix

Two entries that belong on the board but are rarely taught:

• cron on an EC2 instance: self-operated, always-on, no execution history, no retry primitives, no audit trail beyond your own logs. It is the baseline everyone starts from, and the thing every managed option is arguing against. Choose it only when you already run that fleet for other reasons and the schedule is trivial.
• Third-party orchestrators (open-source Airflow you operate yourself, or a commercial orchestrator); you buy engine portability and a team's existing skill set, and you take on the scheduler, the metadata database, upgrades and the on-call. ZS published exactly this evaluation, comparing "Amazon MWAA, Azkaban on Amazon EMR, and AWS Step Functions" before choosing MWAA with per-client AWS accounts for tenant isolation (AWS Big Data Blog, 2022-09-16).

12.2 Matching check: pattern to tool

---

2026 Updates (as of October 2026)

• 📚 Did you know? The S3 to EventBridge to Glue/Firehose routing pattern did not change in 2026: what changed is where the transformed result can land: Firehose to S3 Tables went GA on 2025-03-14, so the same event-driven chain can now terminate in a managed Iceberg table instead of a raw prefix (AWS Big Data Blog, 2025-03-14). The orchestration shape is stable; the destinations widened.

---

Real-World Case Studies

AWS publishes what these abstractions look like in production. Every figure below is customer- or AWS-claimed and unaudited, with the source named so you can check it: the examinable point is the pattern, not the marketing.

Case A, Transport for NSW: 400+ workflows off a licensed ETL tool

"The team also used AWS Step Functions, to improve workflow orchestration and monitoring… For scalable storage, the team adopted Amazon Simple Storage Service (Amazon S3) as its data lake foundation.": AWS case study (accessed 2026-10-11)

The orchestration lesson is the one AWS itself puts in the benefits strip: removing the license bill was the financial win, but Step Functions was the mechanism that made orchestration and monitoring a shared competency instead of a specialist one. "More team members able to manage data workflows on cloud native tools, removing dependency on legacy platform expertise" is an orchestration outcome, not an ETL one.

Case B, TrueCar: removing the schedule, not tuning the engine

The orchestration point: TrueCar's 48x came from removing the batch schedule, not from a faster query engine. When the trigger changes from a nightly cron to an event, the orchestration layer changes with it; which is Section 6's argument in a single customer number. The TfNSW bars are index conversions of AWS's published "80%" cost reduction and "down from several months" to "Less than 3 weeks" lead time, shown against a legacy baseline of 100.

Case C, Cisco Webex: event-driven routing at 147 TB per day

The orchestration reading: Cisco's event volume (100+ billion messages per day) is precisely the regime where a polling cron cannot compete with a content-filtered event bus, and where Pipes-style source, filter, enrich, target beats a bespoke consumer fleet. Routing decisions at that rate belong in patterns, not in code.

• 📚 Did you know? Every number in this section is one customer's measurement on one workload, and the TfNSW and Cisco pages are undated: AWS publishes only "accessed October 2026" for them. Never restate them as current AWS specifications; they are dated snapshots of specific migrations (digest 18, sources S3, S8, S12).

---

Practice Questions