Lakeflow Jobs (formerly Databricks Workflows): Multi-Task Jobs as Code, Triggers, Retries and the Scheduler Decision Matrix

Every lakehouse eventually fails the same way: the data was fine, the cluster was fine, and nobody pressed the button, or three buttons, in the wrong order, twice. Orchestration is the difference between a lakehouse demo and a lakehouse, and on Databricks on AWS the native orchestrator is Lakeflow Jobs. If that name is new to you, that is the first lesson: "Databricks Workflows" was renamed Lakeflow Jobs inside the Lakeflow family: "Lakeflow Jobs (formerly known as Databricks Workflows)" (Databricks blog, 2025-07-24); while the REST paths, API objects and system tables keep saying jobs. Three names, one product, and a docs page updated October 9, 2026 that now opens with "Lakeflow Jobs is workflow automation for Databricks, providing orchestration for data processing workloads" (docs.databricks.com/aws/en/jobs/).

This lesson is the operations manual: how a job is built (tasks, DAG, compute), when it fires (six trigger types), what happens when capacity runs out (queue vs skip), how it recovers (retries, repairs, notifications), who it runs as (service principals), how it ships (Declarative Automation Bundles YAML, formerly Databricks Asset Bundles), and when you should not use it at all (Amazon MWAA/Airflow, AWS Step Functions). Every limit and price below is stamped with its source and dated to October 2026; anything Databricks does not publish is left out rather than guessed.

In this lesson you will:

• reconstruct the Workflows → Lakeflow Jobs rename and the job/tasks/triggers anatomy against the published scale envelope (tasks, jobs, triggers, concurrency, creations per hour);
• choose task types and compute from the serverless-first matrix, and price the same job at the $0.15 vs$0.40 per-DBU floors;
• wire the DAG: depends_on + the six run_if conditions, condition/for-each/Run Job control flow, and the Excluded-counts-as-success semantics that decide whether your alert task ever runs;
• configure all six trigger families: Quartz schedules with the 10-second floor and the DST → UTC fix, file-arrival triggers with 900 s cooldown / 60 s debounce, table-update triggers with their 60-second debounce floor;
• separate queueing (48-hour cap, default ON for UI jobs since 2024-04-15) from skipping, and forensically diagnose both in the jobs system tables;
• set retries, timeouts, repairs, partial runs and webhook notifications, knowing that classic tasks default to no retries and that job-level alerts stay silent while retries remain;
• run production jobs as service principals with OAuth M2M, not as the human who clicked Create;
• write the estate as Declarative Automation Bundles YAML with dev/prod targets and remote-Git task sources, and bridge enterprise calendars through Amazon MWAA with DatabricksRunNowOperator;
• pick Workflows vs Airflow/MWAA vs Step Functions from a priced decision matrix, study NAB and Asurion on AWS, read the 2026 update box, and practise with fourteen questions plus four interactive checks.

---

1. Anatomy: one job, one DAG, three concepts

The docs are terse on purpose: a job holds tasks, tasks are visually represented by a DAG, and triggers decide when the job fires. Job-level settings carry the rest: Trigger, Parameters, Notifications (emails or webhooks), Git (docs.../jobs/, Oct 9, 2026). Everything else in this lesson is a detail of one of those four surfaces.

1.1 The scale envelope (all accessed Oct 2026)

Two numbers do most of the operational damage. First, max_concurrent_runs defaults to 1: "By default, the maximum concurrent runs for all new jobs is 1" and a run that would exceed the cap is skipped unless queueing catches it (configure-job). Second, the workspace ceiling of 2,000 concurrent task runs is global: a fleet of fan-out jobs can starve each other, and the error surfaces as a 429 or as WORKSPACE_RUN_LIMIT_EXCEEDED in the system tables (jobs overview; system-tables/jobs, May 21, 2026).

• 📚 Did you know? The legacy Jobs API 2.0 reference (last updated Aug 3, 2026) still says "A job can contain up to 100 tasks" while the current overview and the 2024-08-22 "What's new in Workflows?" blog both say 1,000. The 1,000-task figure is the current product limit (the write endpoints enforce it); the 100 is a stale legacy page, and read endpoints simply return 100 per page with pagination. Always date-stamp the limit you quote.

---

2. Task types and compute: serverless-first, never all-purpose

2.1 The task-type palette

The current UI palette (configure-task page) reads:

• Notebook
• Visual data prep
• Clean Room notebook
• Python script
• Python wheel
• SQL (query/alert/file)
• Pipeline (Lakeflow)
• Database Table Sync pipeline
• Ingestion pipeline
• SQL Alert (Public Preview)
• Dashboards
• Power BI
• dbt
• dbt platform (Public Preview)
• JAR
• Spark Submit
• Run Job
• Genie Code (Beta)
• If/else
• For each

The REST mapping (Jobs API 2.2) is what your YAML actually writes: notebook_task, spark_jar_task, spark_python_task, python_wheel_task, sql_task, dbt_task, run_job_task, condition_task, foreach_task, pipeline_task: with spark_submit_task marked "Deprecated (Legacy) … Databricks recommends using the spark_jar_task instead".

Three entries deserve commentary before the compute matrix:

• dbt is first-class: a dbt task is "one or more dbt commands" plus a project_directory, and "The dbt task requires both Databricks SQL and the ability to use a serverless or a pro SQL warehouse": classic SQL warehouses are limited to scheduled single-task SQL jobs (Jobs API 2.2). Databricks injects DBT_ACCESS_TOKEN for the Run-as identity, which is why the run-as service principal needs warehouse access and UC grants.
• Condition (if/else) tasks "do not require a cluster to execute and do not support retries or notifications" (Jobs API 2.2; bundles docs, Aug 21, 2026). You cannot "retry a branch": retry the data task the branch guards.
• Run Job tasks trigger "other jobs configured in your Databricks workspace", same workspace only, with job parameters pushed down, and are hard-capped: "Databricks does not support jobs with circular dependencies or that nest more than three Run Job tasks" (docs.../jobs/tasks/run-job, Sep 11, 2026).

2.2 The compute decision tree

The compute-matrix page (last updated Sep 15, 2026) is explicit: "Serverless compute for workflows… is the default for all tasks that support it", and "All-purpose compute | You can optionally configure tasks using classic all-purpose compute. Databricks does not recommend this configuration for production jobs."

Billing follows the cluster choice, and this single rule anchors every cost lesson in the course: "When you run a job on a new jobs cluster, the job is treated as a Jobs Compute (automated) workload subject to Jobs Compute pricing. When you run a job on an existing all-purpose cluster, it is treated as an All-Purpose Compute (interactive) workload subject to All-Purpose Compute pricing" (Jobs API 2.0 reference, Aug 3, 2026). Job clusters also behave differently on the way out: compute "terminates and is unavailable for restarting when the job is complete", while a scheduled job pointed at a terminated all-purpose cluster autostarts it (docs.../compute/clusters-manage); which is how a "cheap" reused cluster quietly becomes a 24×7 line item.

2.3 Worked example E1: the same job, two SKUs

As of Oct 2026. A medallion job runs 2 hours a day and consumes 12 DBU/h (a modest multi-task fan-out on classic jobs compute). Monthly DBU cost at the official floors from the Databricks pricing page (snapshot 2026-10):

Serverless jobs are the third path: they "bill only when a command or query runs (plus a small session fee for the REPL VM); charges stop automatically after a few minutes of idle time" (Databricks Answers, 2026-08-27), and the serverless best-practices page adds that "Standard mode can reduce costs by up to 70% compared to performance-optimized mode": at the price of "4 to 6 minutes" of startup latency (run-serverless-jobs, Oct 9, 2026). Serverless jobs require a Unity Catalog-enabled workspace and standard access mode; auto-optimization (including retries, section 6) is on by default.

• 📚 Did you know? Serverless jobs carry automated rate-limit objects that cap execution at roughly 60 / 120 / 240 / 480 / 960 DBU per hour for Small through 2X-Large, and "Within a single job, the cap is shared across all the job's tasks" (docs.../compute/serverless/manage-serverless-compute, accessed Oct 2026). Rate limits apply only to automated workloads, interactive notebooks and Databricks Connect are exempt, and enforcing them requires Databricks Runtime 17.3.1 or above. A "why did my fan-out stall?" ticket often has this as the answer.

---

3. Dependencies and control flow: the DAG you actually get

Tasks after the first declare depends_on: [{task_key: …}], which draws the edges. What those edges mean is decided by Run if, and this is where production DAGs go wrong. The six conditions (configure run-if docs, Jun 25, 2026): All succeeded (default) · At least one succeeded · None failed · All done · At least one failed · All failed.

Three documented semantics are exam-grade and incident-grade at once:

1. "Upstream tasks with the Excluded result state are treated as successful when evaluating Run if conditions."
2. "Upstream tasks marked Upstream failed or Upstream canceled are treated as failed, not as skipped."
3. "If all of a task's dependencies are excluded, the task itself is also excluded… exclusion cascades along a linear dependency chain": regardless of that task's own Run if. Cancellation likewise "propagates through downstream tasks, and tasks with a Run if condition that handles failure are run".

Read the diagram as a policy, not a shape. dq_gate uses NONE_FAILED because the gold publish must not run on corrupt silver, but a canceled upstream counts as failed, so a human cancel still skips gold, which is what you want. The alert task uses AT_LEAST_ONE_FAILED so it survives silver's failure; because "Upstream failed" counts as failed, the alert fires on exactly the incidents you care about. Control-flow companions complete the toolbox: If/else condition tasks compare operands (EQUAL_TO, GREATER_THAN, …) against dynamic values such as {{job.repair_count}} or {{tasks.task_key.values.output}} (bundles job-task-types docs, Aug 21, 2026); For each loops over a JSON array, manual or produced upstream via dbutils.jobs.taskValues.set; Run Job composes whole jobs, same-workspace, no cycles, ≤3 levels of nesting.

• 📚 Did you know? Parameter propagation has a direction and a winner: job parameters "are pushed down automatically, and if a job parameter and a task parameter share a key, the job parameter takes precedence", but "Tasks that take JSON-array parameters don't receive job parameters automatically; reference them explicitly with {{job.parameters.<name>}}" (docs.../jobs/parameters, Jun 26, 2026). JSON-array tasks are Python script, positional wheel, JAR, Spark Submit and For each. Miss that and your Python-script task silently runs with empty argv.

---

4. Triggers: cron is a guess, data arrival is a fact

Six trigger behaviors ship today (triggers docs, Oct 6, 2026): Scheduled, Table update, File arrival, Model update (Beta), Continuous ("trigger another job run whenever a job run completes or fails"), and None (manual). Multiple triggers per job are Beta: up to five, each independently pausable, and "A continuous trigger cannot be combined with any other trigger". The API carries them in a triggers array; do not mix that array with the legacy schedule / trigger / continuous fields in one request.

4.1 Scheduled: Quartz cron, the 10-second floor, and DST

Schedules use Quartz cron syntax in a Java timezone id, with pause_status UNPAUSED/PAUSED (Run object schema, Jobs API 2.2: example "quartz_cron_expression": "20 30 * * * ?", "timezone_id": "Europe/London"). Three rules from the schedules page (last updated Jun 26, 2026):

• "Databricks enforces a minimum interval of 10 seconds between subsequent runs triggered by the schedule of a job regardless of the configuration in the cron expression."
• DST-observing timezones: "an hourly job is skipped or might appear to be delayed by an hour or two when daylight saving time begins or ends. To run at every hour (absolute time), choose UTC."
• "The job scheduler is not intended for low-latency jobs. Due to network or cloud issues, job runs might occasionally be delayed up to several minutes."

4.2 Worked example E2: the cron bug and the event-driven fix

File-arrival triggers "make a best effort to check for new files every minute", monitor "the root of a Unity Catalog external location or volume, or a subpath" recursively, and "do not incur additional costs other than cloud provider costs associated with listing files" (file-arrival docs, Oct 7, 2026). The sharp edges: only new files trigger: "Overwriting an existing file with a file of the same name does not trigger a run"; no wildcards; and "When the configured directory does not exist or is deleted from Amazon S3…, file arrival triggers continue to be evaluated with no error". Without file events enabled on the external location, the workspace caps you at "A maximum of 50 jobs" with file-arrival triggers on such locations and "10,000 files" in the watched path. The recommended pairing is Auto Loader with cloudFiles.useManagedFileEvents=true and trigger(availableNow=True): a foreachBatch variant "provides only at-least-once processing guarantees": with the checkpoint kept outside the watched path.

4.3 Table update triggers

A table-update trigger "checks for table updates, and when a table is updated, the job is run", monitoring up to 10 Unity Catalog Delta/Iceberg managed tables, Delta-backed external tables, materialized views, streaming tables and dependent views: detecting "updates, merges, and deletes" (table-update docs, Jun 26, 2026). wait_after_last_change_seconds has a 60-second floor: it is a debounce, not a sub-minute SLA. Like file arrival, it "do[es] not incur additional costs other than cloud provider costs", and "For best performance and scalability, enable file events on the external location."

• 📚 Did you know? A workspace can hold up to 12,000 active triggers across all jobs and trigger types, and paused triggers don't count toward that limit (triggers docs, Oct 6, 2026). That makes pause-status hygiene a capacity tool: a fleet of dormant backfill jobs should ship with pause_status: PAUSED; which is exactly what Declarative Automation Bundles' development mode does for you (section 8).

---

5. Queueing, concurrency and the skip trap

When a run cannot start, two different machines catch it, and only one is on by default. Queueing (default ON for jobs created through the UI after April 15, 2024) holds the run "for up to 48 hours if resources are unavailable", triggered when any of three limits is reached: the workspace's maximum concurrent active runs, the workspace's maximum concurrent Run Job task runs, or the job's own max_concurrent_runs (configure-job, Jul 7, 2026). Without queueing, a limit hit means the run is skipped, and a skipped scheduled run is a silent data gap.

Knobs and ceilings to keep straight: max_concurrent_runs default 1, hard max 1,000, 0 skips everything; lowering it never kills active runs; the workspace-wide 2,000 concurrent task runs returns 429 or WORKSPACE_RUN_LIMIT_EXCEEDED even when queueing is on (jobs overview; system tables). Queue depth beyond the documented 48-hour cap is not published: treat "how deep is the internal queue, and in what order does it dequeue?" as unknown, and design for the cap, not for fairness.

5.1 Worked example E3: queue forensics in the system tables

The same reference lists the failure states worth alerting on: WORKSPACE_RUN_LIMIT_EXCEEDED ("The workspace has reached the quota for the maximum number of concurrent active runs"), MAX_CONCURRENT_RUNS_EXCEEDED, MAX_JOB_QUEUE_SIZE_EXCEEDED, CLUSTER_REQUEST_LIMIT_EXCEEDED and REPOSITORY_CHECKOUT_FAILED: plus the honest footnote that creator_user_name and paused are "Not populated for rows emitted before early December 2025", so historical hygiene queries need a date floor.

---

6. Reliability: retries, timeouts, repairs, notifications

6.1 Retries differ by compute: the default is silence

"The default setting for task retries depends on the job configuration. For most configurations, the default setting does not retry any tasks on task failure" (configure-task). Against that: "Serverless jobs auto-optimize retries by default" (opt out via the "Enable serverless auto-optimization" checkbox, and do opt out for non-idempotent sinks), and "Continuous jobs use an exponential backoff retry policy" (same page). The knobs per task: max_retries, min_retry_interval_millis, retry_on_timeout; timeout_seconds is per task with "A value of 0 means no timeout" (Jobs API 2.2), and a timeout is not retried unless retry_on_timeout is set.

6.2 Worked example E4: retry policy as a design document

Recovery surfaces beyond retries: repair re-runs "the subset of unsuccessful tasks" in a multi-task run; partial runs (Lakeflow Jobs blog, 2025-07-24) "allow users to select which tasks to execute without affecting others": the operational answer to "re-run just the broken gold task"; and SQL query outputs can feed later tasks as parameters (same blog). Notebook tasks also carry hard output caps: combined cell output 30 MB, individual cell 8 MB: beyond which the run fails (docs.../jobs/tasks/notebook, Sep 11, 2026); chatty tasks need to write to Delta, not to stdout.

6.3 Notifications: three destinations, silent retries

Notification events (notifications docs, Oct 9, 2026):

• Start
• Successful completion
• Failure
• duration threshold exceeded
• streaming backlog threshold exceeded
• maintenance start/complete (continuous jobs)

Destinations are "email addresses or third-party destinations such as Slack, Microsoft Teams, PagerDuty, or any webhook-based service", where "An administrator must configure system destinations", and "For each job or task, you can configure a maximum of three system destinations for each notification event type". Webhook event_type codes are stable: jobs.on_start, jobs.on_success, jobs.on_failure, jobs.on_duration_warning_threshold_exceeded.

6.4 Worked example E5: the webhook payload and the run-now call

Use OAuth M2M tokens (client id + OAuth secret; access tokens valid 1 hour, secrets scopable to jobs) rather than personal access tokens; if a legacy integration forces a PAT, use a service-principal PAT, never a human's (section 7 and the Airflow seam in section 9).

---

7. Identity: owner, Run as, and the departure that breaks 3 a.m.

A job has two identities: the owner (job ACL; "Only workspace admins can edit the job owner") and the Run as identity whose permissions execute every task. Default Run as = the job creator, and the docs' recommendation is unambiguous: "Databricks recommends changing the Run as user to a service principal, so that privileges can be controlled separately from the owner's privileges, and so that jobs do not break when the owner leaves or has changed privileges" (docs.../jobs/privileges, accessed Oct 2026). Changing Run as needs CAN MANAGE or IS OWNER on the job, the service principal needs the Service Principal User entitlement, and "Run as a group" is Public Preview.

The audit nuance matters during incident review: identity_metadata.run_as records your configured identity (the group, in the group Run-as preview), while identity_metadata.run_by "always records" the Jobs service's application service principal (same page). The failure mode this prevents is the course's most common production breaker: the scheduled job keeps looking healthy at schedule time and dies at first UC read after the creator's access is revoked. Machine-to-machine auth for orchestration is OAuth M2M (1-hour tokens) or, for Airflow on AWS, the provider's AWS IAM OIDC token federation: "exchange for a Databricks OAuth token. Stores no secrets" (apache-airflow-providers-databricks 7.19.0 docs, accessed Oct 2026).

---

8. Jobs as code: Declarative Automation Bundles

Clicking jobs together does not survive a second engineer. The as-code surface is Declarative Automation Bundles, "formerly known as Databricks Asset Bundles" (docs.../dev-tools/bundles, accessed Oct 2026), which package "Source files… Definitions and settings for Databricks resources, such as Lakeflow Jobs, Lakeflow Spark Declarative Pipelines, Dashboards, Model Serving endpoints, MLflow Experiments". The loop: write databricks.yml + resources/*.job.yml, then databricks bundle validate → databricks bundle deploy --target dev → databricks bundle run --target dev <job>.

8.1 Worked example E6: a production multi-task job, full YAML

Everything in that file maps to a documented field: run_as.service_principal_name accepts "the email of an active workspace user or the application ID of a service principal" (Jobs API 2.2); queue.enabled mirrors the 48-hour queueing behavior (section 5); the retry block is E4; run_if is section 3; schedule is the verified API shape from the Jobs API job-settings example (accessed Oct 2026). Existing UI-built jobs can be reverse-engineered into this shape with databricks bundle generate job (bundles job-task-types docs, Aug 21, 2026): the fastest migration path from click-ops.

8.2 Dev/prod targets are the safety rail

Development mode presets (bundles deployment-modes docs): "Pauses all schedules and triggers on deployed resources such as jobs" (override with schedule.pause_status: UNPAUSED), prepends a [dev ${workspace.current_user.short_name}] name prefix, and "Enables concurrent runs on all deployed jobs". Production mode applies none of that; which is why the same YAML is safe to deploy from a laptop and terrifying to deploy from a laptop without targets. Promotion is databricks bundle deploy --target prod from CI, with the run-as service principal and secrets resolved per target.

8.3 Git sources: one repo, one commit, one snapshot

Notebook, Python-script, SQL-file and dbt tasks may source a remote Git repository (docs.../jobs/git, Sep 11, 2026): "All tasks in a job must reference the same commit… Databricks takes a snapshot of the specified branch or commit" when the run begins; "Only one remote Git repository can be used for all tasks in a job"; and "Tasks configured to use a remote Git repository cannot write to workspace files". Git folders (workspace-repo sync) work but must be synced manually and are recommended "only for rapid iteration and testing during development": staging and production tasks reference the remote repo with a pinned branch/tag/commit, and the run history shows the commit SHA.

---

9. The external seam: Airflow and Amazon MWAA

Some calendars are enterprise-owned: SAP windows, Redshift maintenance, third-party API SLAs. Databricks' own page documents the seam, "you can also schedule your jobs using tools such as Apache Airflow", with the caveat that the open-source packages "are not directly supported by Databricks" (docs.../jobs/how-to/use-airflow-with-jobs, Sep 11, 2026). The operator map:

Production hardening from the provider docs (7.19.0 / module 7.20.0, accessed Oct 2026): durable mode persists "the Databricks run id… to task state store before polling begins, so if the worker crashes… the operator reconnects to the run… instead of triggering a duplicate run"; deferrable variants exist; the connection should carry SP OAuth or AWS IAM OIDC federation, not a human PAT; and the operator "write[s] the job run page URL to the Airflow logs every polling_period_seconds (the default is 30 seconds)" (Databricks tutorial). Version hygiene: Databricks' tutorial still says "Airflow version 2.5.0 and later… tested with Airflow version 2.6.1", while provider 7.19.0 requires "Apache Airflow version supported by this provider distribution is 2.11.0": pin and test on MWAA rather than trusting either number alone.

9.1 Worked example E7: the MWAA-to-Databricks DAG, priced

The design rule that falls out of E7: one layer owns the DAG edges, the other owns the compute. Airflow's retries mirror, but must not multiply, Databricks' task retries (set Airflow retries 0 to 1 when the job already retries 2 to 3); the DatabricksRunNowOperator URL in the Airflow log is your cross-system breadcrumb; and multi-workspace estates hold one MWAA connection (or one set of OIDC credentials) per workspace, because the Run Job task and run-now are same-workspace primitives.

---

10. The scheduler decision matrix: Workflows vs Airflow vs Step Functions

Three orchestrators touch a Databricks-on-AWS estate, and each has a documented price shape.

The hybrid that most AWS shops land on: a single Databricks "ingest" job with a file-arrival trigger owns data readiness; MWAA owns the enterprise calendar and calls that job with DatabricksRunNowOperator; Step Functions coordinates the AWS-side steps around the Databricks call. Managed-Airflow-family alternatives (Prefect, Astronomer) integrate through the same REST/SDK seam: same architecture, different bill, so keep the comparison qualitative until you have their live quotes.

Comparative Verdict: Lakeflow Jobs vs Amazon MWAA/Airflow vs AWS Step Functions (AWS, as of Oct 2026)

• Lakeflow Jobs: the default for Databricks-native DAGs. No idle scheduler charge (vendor: "no orchestration surcharge; You pay only for compute used by your tasks", blog 2025-01-12), data-aware triggers on S3-backed external locations and UC tables, repairs/partial runs, run-as service principals, and job observability in system tables. DBU floor $0.15/DBU (Data Engineering, pricing page snapshot 2026-10) on classic job compute, or serverless that "bill[s] only when a command or query runs". Weakness: orchestration out of Databricks is still young (the June 2026 Lakeflow blog announces External Orchestration for Snowflake/REST/Slack/PagerDuty without stating its maturity), and Run Job chaining is same-workspace only, ≤3 levels, no cycles.
• Amazon MWAA/Airflow, the enterprise-calendar owner. Vendor-neutral DAGs, custom operators/sensors, mature backfill semantics, and the only comfortable home for fan-out that spends most of its steps outside Databricks. The honest cost: a $0.99/h large environment ≈ $736.56/month floor (AWS pricing page example) plus workers at $0.22/h and RDS metadata at $0.10/GB-month, orchestration with its own bill and its own on-call. Call Databricks through DatabricksRunNowOperator (SP OAuth or IAM OIDC, deferrable, durable) so runs stay visible in the jobs UI.
• AWS Step Functions: the AWS-glue orchestrator. Best when the workflow's centre of gravity is Lambda/ECS/Batch/Glue and Databricks is one API call among many; Standard pricing is per state transition ($0.025 per 1,000, AWS Compute Blog 2022-08-30: re-check the live table), Express meters GB-seconds ($0.00001667/GB-s, pricing page accessed Oct 2026). You give up task values, repairs and the jobs UI: Databricks-aware features live outside the console.
• Verdict in one line: Lakeflow Jobs for the Databricks-internal DAG → MWAA where the calendar or the fan-out is non-Databricks → Step Functions where AWS services are the workflow and Databricks is a callee, and in every hybrid, exactly one layer owns the DAG edges, run-now owns the seam, and the run-as identity is a service principal.

---

2026 Updates (as of October 2026)

---

Real-World Case Studies

📋 Case A: NAB (National Australia Bank, banking, Cloud: AWS) Challenge. A 26-year-old Teradata warehouse carrying ~1,000 use cases and ~140 reporting suites, plus hand-written Spark ETL with 5,000 to 8,000-line DML and up to 64 union-all per pipeline, maintained by 300 to 400 engineers with inconsistent practice. Stack on AWS. Databricks SQL, Auto Loader, Spark Declarative Pipelines on Lakeflow (the pipeline sibling of this lesson's Jobs), Unity Catalog, and asset bundles for CI/CD: Cloud: AWS on the story page. Reported outcome. "1,800 Spark Declarative Pipelines running on Databricks Lakeflow · 99.6% Job success rate · 80% Less transformation complexity"; end-to-end latency "under 15 minutes"; data quality "+38%" (https://www.databricks.com/customers/national-australian-bank-nab, undated page, accessed 2026-10). Quote. "The migration wouldn't have been possible without Databricks SQL": Gurry (https://www.databricks.com/customers/nab/dbsql, undated page, accessed 2026-10). Maps to this lesson. Sections 6 and 8: a 99.6% job success rate across 1,800 pipelines is a reliability metric: retries, repairs, run-as service principals and bundle-deployed definitions are what turn orchestration from heroics into a number an auditor can track. ⏳ Temporal note: undated page, accessed Oct 2026: a reported customer outcome, not a 2026 guarantee.

📋 Case B: Asurion (insurance / device protection, Cloud: AWS) Challenge. 8,000 tables, 10,000 views and ~600 data marts with per-table ETL mappings, plus batch/speed-layer duplication across thousands of jobs that would not fit a one-cluster-per-job scheduling model. Stack on AWS. Spark Structured Streaming, Auto Loader, foreachBatch into Delta; sources Kinesis, Kafka, SNS, SQS, CDC, APIs and flat files; Databricks SQL for the mart layer: the production data lake explicitly on AWS. Reported outcome. "The Asurion data lake executes 4,000+ streaming jobs and hosts over 4,000 tables in a production data lake on AWS" (https://www.databricks.com/blog/2021/05/10/aws-guide-to-data-ai-summit-featuring-disney-comcast-capital-one-and-mcdonalds.html, 2021-05-10); "unified approach to data movement from traditional ETL to streaming data pipelines between Delta tables" (https://www.databricks.com/blog/2021/09/16/large-scale-etl-and-lakehouse-implementation-at-asurion.html, 2021-09-16). Maps to this lesson. Sections 1 and 2: at 4,000+ streaming jobs, job design is the architecture: one parameterized framework with shared conventions (task keys, timeouts, run-as identity, compute choices) beats 4,000 bespoke ETLs, and the workspace-scale limits in section 1.1 are exactly the envelope such an estate must be designed against. ⏳ Temporal note: 2021 figures; treat as a reported scale milestone, and re-check any concurrency limit quoted in the blog against current docs before reusing it.

---

---

Practice Questions