Lakeflow Declarative Pipelines (formerly Delta Live Tables): Expectations, Modes and the DLT vs Jobs Decision

Every data estate eventually asks two questions of its ETL layer: is the data any good? and did the job actually run? On Databricks on AWS, the first question is answered inside Lakeflow pipelines: the product your 2023 tutorials called Delta Live Tables (DLT): by a feature called expectations: per-record SQL checks with three graded actions (warn, drop, fail) and machine-readable metrics. The second question is answered by orchestration, which in the Databricks world means Lakeflow Jobs (Workflows). The single most expensive mistake in this domain is confusing the two planes: expectations enforce data quality, jobs enforce orchestration, and the docs say so in as many words: "Expectations enforce data quality, not orchestration" (docs.../ldp/expectation-patterns, dated 2026-07-14).

This lesson teaches the product under its current name with its full rename history, because as of October 2026 your estate will contain all three names at once: notebooks with import dlt, invoices with DLT_ SKUs, event-log fields with dlt in them, and documentation that says "Lakeflow pipelines". "The product formerly known as Delta Live Tables (DLT) has been updated to Lakeflow pipelines. If you have previously used DLT, there is no migration required to use Lakeflow pipelines: your code will still work" (docs.../ldp/concepts/where-is-dlt, footer Sep 11, 2026). Three names, one product, zero forced rewrites, and a billing surface that still says DLT.

In this lesson you will:

• reconstruct the naming timeline (2024-06 Lakeflow announce → 2025-06 GA + DLT rename + Spark contribution → Oct 2026 docs name) and triage the DLT vestiges that survive it;
• write pipeline Python in the dp API and prove to yourself that the old @dlt.expect style still runs;
• apply all six expectation decorators and the three actions, and explain precisely why fail records no metrics;
• query expectation metrics from the event log and the Beta pipeline_events system table;
• build the quarantine pattern with NULL-safe rules so clean and bad rows partition the input exactly once;
• choose between triggered, continuous and real-time modes, and compute the $105 vs$2,520 per month cost shape of that choice;
• configure pipelines on AWS: Unity Catalog storage cascade, serverless vs classic, editions, run-as service principals, Declarative Automation Bundles;
• decide Lakeflow pipelines vs Workflows jobs with a decision matrix, study NAB and Shell on AWS, read the 2026 update box, and practise with twelve questions plus four interactive checks.

---

1. One product, three names: the timeline you will be tested on

1.1 The rename, with dates

Course wording to memorise: Delta Live Tables (DLT), renamed Lakeflow Declarative Pipelines at GA in June 2025 and now called Lakeflow pipelines in the docs: same product, three names, one timeline. Two related names must not be confused with the product name: Apache Spark™ Declarative Pipelines (SDP) is the open standard Databricks extended ("All Lakeflow pipelines Python APIs are implemented in the pyspark.pipelines module"; "Apache Spark™ includes declarative pipelines beginning in Spark 4.1", docs.../ldp/developer/python-dev, dated 2026-07-10), and Lakeflow is the family: Lakeflow Connect (managed ingestion), Lakeflow Declarative Pipelines (this lesson), Lakeflow Jobs (Workflows) (GA blog, 2025-06-12).

1.2 The vestiges: why your billing queries still grep for dlt

The FAQ page is explicit about what did not change: "There are still some references to the DLT name in Databricks. The classic SKUs for Lakeflow pipelines still begin with DLT, and event log schemas with dlt in the name have not changed. Python APIs that used dlt in the name can still be used, but Databricks recommends moving to the new names" (docs .../ldp/concepts/where-is-dlt, 2026-09-11).

• 📚 Did you know? A support ticket that reads "why is there a pipeline ID on our SQL bill?" has a documented answer: "When you create a materialized view in a Databricks SQL warehouse, a serverless pipeline is created to process refreshes", and the billing row appears as billing_origin_product = 'SQL' with usage_metadata.dlt_pipeline_id IS NOT NULL (docs.../admin/system-tables/serverless-billing, 2026-06-01). Pipelines are not only the thing you build; they are also machinery other products spin up behind your back.

1.3 Worked example E1, the migration: legacy dlt code vs current dp code

As of Oct 2026. Team A ships a 2023-era DLT notebook. Team B asks whether to rewrite it. Answer per the docs: no migration required, but the split of @dlt.table into two decorators matters.

The mapping table from the Python reference (dated 2026-08-06): learn the middle column, quote the left column only when reading old code:

Two traps visible in this example. First, the decorator split: @dlt.table used to cover both dataset kinds; @dp.table with a batch DataFrame (or @dp.materialized_view with a streaming one) mis-declares the dataset and breaks flow planning (python-ref mapping, 2026-08-06). Second, the NULL-safe predicate: amount >= 0 drops only rows where the predicate is false; a NULL amount evaluates to NULL and is kept by drop-mode expectations. Write amount IS NOT NULL AND amount >= 0 (section 5 returns to this with the quarantine partition proof).

⚠️ WARNING, do not copy-paste the rest of a 2023 DLT blog post. Old material also says "set continuous in the pipeline settings for production", "query LIVE.my_table", and "serverless is preview". As of Oct 2026: default publishing mode ignores LIVE (query catalog.schema.table), serverless is the default for new ETL pipelines, and Databricks discourages the built-in continuous setting for new pipelines, wrap the pipeline in a continuous job instead (docs.../ldp/concepts/pipeline-mode, accessed 2026-10;.../ldp/serverless, 2026-10-09;.../ldp/target-schema, 2026-09-11).

---

2. The declarative model: what you write vs what the platform does

2.1 Pipelines, flows, datasets, and an inferred DAG

"A pipeline is the main unit of development and execution of Apache Spark™ Declarative Pipelines (SDP) in Lakeflow. A pipeline is a collection of source code files and a configuration" (docs.../ldp/concepts/pipelines, dated 2026-10-09). The concepts page names the building blocks: "pipelines, flows, streaming tables, materialized views, and sinks" (docs.../ldp/concepts/, dated 2026-08-15). You declare datasets with decorators; the platform "automatically infer[s] dependencies between datasets and arrange[s] them in a directed acyclic graph (DAG)"; there is no @graph decorator; the graph is inferred from what your dataset functions read (concepts/pipelines, 2026-10-09).

A flow is the processing edge: "A flow reads data from a source, applies user-defined processing logic, and writes the result into a target … (Currently, only the Append and Update flows are exposed.)" (concepts, 2026-08-15). Append flows are the default, one per target; update flows are Python-only via @dp.update_flow.

2.2 Coding laws: the engine evaluates your function repeatedly

"Python pipeline code does not directly call the functions that perform data ingestion and transformation … Instead, Lakeflow pipelines interpret the decorator functions … and build a dataflow graph" (docs.../ldp/developer/python-dev, 2026-07-10). Dataset functions "must return a Spark DataFrame", and you must never call collect(), count(), toPandas(), save(), saveAsTable(), start(), toTable() inside them: the planner evaluates definitions multiple times, so actions and side effects break idempotency. Reading other datasets is done with ordinary Spark readers (spark.read.table, spark.readStream.table, SQL STREAM(...)), not a special source() helper.

2.3 Sources on AWS: where the bytes come from

Inside pipelines the platform owns the plumbing: "If you use Lakeflow pipelines, Databricks manages schema location and other checkpoint information automatically" (auto-loader schema docs, 2026-07-10), and "The event log location also serves as the schema location for any Auto Loader queries in the pipeline" (docs .../ldp/monitor-event-logs, 2026-10-09). The Auto Loader best-practices page states the production default plainly: "For most users and production pipelines, Auto Loader with Lakeflow pipelines is a good fit. However, if you need maximum control and customization, use Auto Loader with Structured Streaming" (docs, 2026-06-29).

---

3. Expectations: six decorators, three actions, one rulebook

3.1 The verified inventory: exactly six, no quarantine decorator

"Expectations are optional clauses in pipeline materialized view, streaming table, or view creation statements that apply data quality checks on each record passing through a query. Expectations use standard SQL Boolean statements" (docs.../ldp/expectations, footer Sep 11, 2026). The Python reference (dated 2026-09-11) is unambiguous: "The dp module includes six decorators to control expectations behavior", and the canonical order is after the dataset decorator and before the function:

SQL twin (for SQL-authored datasets): CONSTRAINT name EXPECT (expr) with optional ON VIOLATION DROP ROW or ON VIOLATION FAIL UPDATE. There is no expect_all_or_quarantine decorator: quarantine is a documented pattern you build yourself (section 5). Expectations require the Advanced product edition: "Core to run streaming ingest workloads … doesn't require advanced features such as change data capture (CDC) or expectations. Pro … plus … CDC. Advanced … and includes data quality constraints with expectations" (docs.../ldp/configure-pipeline, dated 2026-10-01).

3.2 The three actions, drawn

Three clauses from the expectations doc (2026-09-11) that decide production design:

1. "Because fail causes the update to fail when an invalid record is detected, metrics are not recorded." Fail is a stop sign, not a measurement instrument: never instrument a table with fail-mode expectations and then wonder why your quality dashboard is blank.
2. "An expectation name must be unique for a given dataset. You can reuse expectations across multiple datasets in a pipeline."
3. Constraints "must use valid SQL syntax and cannot contain … Custom Python functions, External service calls, Subqueries referencing other tables." No UDFs, no external APIs, no cross-table checks: if you need those, they belong in a validation notebook or a separate pipeline.

Run-mode semantics matter too: "In a triggered pipeline, the failure of a single flow does not cause other parallel flows to fail. In a continuous pipeline, an expectation failure stops the flow, all dependent flows, and the pipeline emits a message explaining why it stopped" (same page). The docs' own debugging artifact for fail-mode: [EXPECTATION_VIOLATION.VERBOSITY_ALL] Flow 'sensor-pipeline' failed to meet the expectation. Violated expectations: 'temperature_in_valid_range'. Input data: '{"id":"TEMP_001",…}': flow name, expectation name, and the offending record, in one error.

3.3 Worked example E2: one medallion pipeline, all three actions

As of Oct 2026, serverless, Advanced edition, Unity Catalog targets. The complete pipeline source (adapted from the docs' worked shape; @dlt.expect* aliases work identically):

Narrate the graph before you run it: bronze warns (metrics only, bronze is for forensics, not serving), silver drops (bad rows never reach consumers; only counts remain, section 5 shows what to do when counts are not enough), gold fails (a negative revenue total is impossible, so stop the update rather than publish it). The engine infers the DAG (bronze_orders → silver_orders → gold_daily_revenue), publishes all three to the pipeline's target schema in Unity Catalog (physically landing in that schema's managed S3 storage), and a triggered update processes the data available at start, then stops. Note the dataset-kind split in action: gold_daily_revenue is a batch aggregation, so it is @dp.materialized_view, not @dp.table.

---

4. Metrics: where expectation numbers actually live

4.1 The event-log payload is the metrics API

Metrics are not columns on your target table. Current docs define them purely as event-log payloads: "the metrics for the number of records that passed and failed an expectation are stored in the details:flow_progress.data_quality.expectations object. The metric for the number of dropped records is stored in the details:flow_progress.data_quality object. Events … have the event type flow_progress" (docs .../ldp/monitor-event-logs, 2026-10-09). The expectations array schema is published: array<struct<name: string, dataset: string, passed_records: int, failed_records: int>>, and dropped_records counts rows that "failed one or more expectations".

Three surfaces read the same numbers:

1. Pipeline monitoring UI → dataset → Data quality tab (warn/drop expectations; the human view).
2. The per-pipeline event log: a hidden Delta table queryable as SELECT * FROM event_log(<pipeline_id>) (formatted event_log_{pipeline_id}), which you can publish to a named Unity Catalog table for wider access (Advanced settings JSON {"event_log": {"catalog", "schema", "name"}}).
3. The pipeline_events system table (Beta): system.lakeflow_pipeline_events_preview.pipeline_events, which "contains events for all pipelines across the workspaces in a region" and is now the recommended query surface, complete with a downloadable Lakeview dashboard (docs.../ldp/monitor-and-query-events, 2026-10-09).

⚠️ Do not go hunting for _expectation_name columns on your silver table. Older community material shows expectation metadata columns; the current docs define the contract above and nothing else. If your monitoring SQL selects _expectation_* from a pipeline table, you are running stale code (docs .../ldp/monitor-event-logs, 2026-10-09; expectations doc, 2026-09-11). And remember: fail-mode expectations leave no metrics by design, which is exactly why fail belongs in orchestrated gates (section 7), not in monitored serving tables.

4.2 Worked example E3, DQ monitoring SQL: event log, system table, bill

Query (c) is the §1.2 vestige table in executable form: pipelines bill under usage_metadata.dlt_pipeline_id, and serverless rows carry product_features.performance_target ∈ {PERFORMANCE_OPTIMIZED, STANDARD} (docs .../admin/system-tables/serverless-billing, 2026-06-01). The Data quality tab in the UI is fine for a human checking one dataset; queries (a) and (b) are what you put in a scheduled alert or a Lakeview health dashboard.

• 📚 Did you know? Pipelines retain "60 days of past updates in the pipeline UI and the Pipelines API" (docs.../ldp/configure-pipeline, 2026-10-01), but "older ones remain only in the event log". Any audit or incident review that reaches back beyond 60 days is an event-log (or published event-log table) query, which is one more reason never to delete that table.

---

5. Quarantine: a pattern, not a decorator, and the NULL trap inside it

When drop-mode expectations are too destructive (regulated data, replay requirements) you need the bad rows kept. Databricks documents the recipe: a temporary streaming table partitioned by a computed is_quarantined flag, annotated with @dp.expect_all(rules) so metrics still flow, then two views that split the stream (docs.../ldp/expectation-patterns, dated 2026-07-14).

5.1 Worked example E4: the official quarantine shape, made NULL-safe

Then, in production, promote invalid_trips_data to a real @dp.table(name="trips_quarantine") (it must persist on S3 under Unity Catalog: for UC pipelines that means the schema's managed storage, the catalog location, or the metastore root, in that fallback order) and alert on its row count; nothing in the platform auto-replays or auto-expires quarantine tables.

Replay story (course practice, consistent with the documented pattern): triage quarantine rows → fix upstream or accept them → re-run the same filtered transform with a widened predicate or MERGE from the quarantine table after repair → the pipeline's next update continues (Delta ACID + pipeline state keep replays idempotent as long as you replay by key, not by blind append) → archive or drop quarantine rows under UC MODIFY rights with an audit trail.

---

6. Modes and guarantees: triggered, continuous, real-time

6.1 The mode table

"Triggered mode refreshes available data and stops, while continuous mode keeps tables fresh as new data arrives" (docs .../ldp/concepts/pipeline-mode, accessed 2026-10). Two facts students routinely get wrong: "Pipeline mode is independent of the type of table being computed" (both MVs and streaming tables update in either mode), and "Refresh operations for standalone materialized views and streaming tables always run using triggered pipeline mode".

6.2 Exactly-once: inside the envelope and at its edges

"Lakeflow pipelines give you exactly-once processing by default within the tables they manage, and at the edges you add idempotent writes to keep re-runs safe" (docs.../ldp/best-practices/processing-guarantees, dated 2026-08-15). Mechanism: "Streaming tables use Structured Streaming checkpoints combined with Delta Lake's transactional writes: each micro-batch commits its source offsets and its output together, so a retried batch after a failure either fully succeeds or is fully rolled back … never partially applied twice", for Auto Loader, Kafka, Kinesis and AUTO CDC upserts, "without any code from you". The rule of thumb: "if your entire pipeline is Delta-to-Delta … you already have exactly-once." Where it stops: foreach_batch_sink, Kafka/non-Delta sinks and custom Python sources are at-least-once, make the write idempotent or dedupe downstream.

6.3 The continuous-job rule (production hygiene)

Databricks' current guidance is a pattern, not a preference: "Databricks recommends running continuous pipelines with a continuous job rather than setting … Pipeline mode to continuous … the job manages the pipeline's execution lifecycle and unlocks serverless performance modes, such as Standard mode, that the pipeline's built-in continuous mode doesn't support." And when a job orchestrates a pipeline, "the job determines the execution mode and takes precedence … set the pipeline's Pipeline mode to triggered (the default) when you wrap it in a continuous job." The built-in setting "is not being removed, but Databricks discourages its use for new pipelines" (docs .../ldp/concepts/pipeline-mode, accessed 2026-10). Real-time mode rides the same stack: "Continuous mode is still required" (docs .../ldp/real-time, accessed 2026-10).

6.4 Worked example E5: the mode decision, in dollars

As of Oct 2026. Arithmetic on the documented rates: serverless Lakeflow pipelines are reported at $0.35/DBU (customer quotation of the pricing page confirmed in substance by a Databricks staff answer, "While the DBU price is the same for both modes, the Performance Optimized mode … can significantly increase total cost due to higher DBU consumption", Databricks Community, 2025-07-10); classic DLT SKUs corroborated by a third-party rate CSV (fetched 2026-04-10): DLT_CORE_COMPUTE $0.20, DLT_PRO_COMPUTE $0.25, DLT_ADVANCED_COMPUTE $0.36, treat that CSV as SKU-name corroboration and confirm the live card before you sign a budget.

The break-even framing is the discipline this course wants: start triggered, and buy continuous (or real-time) only with a latency SLA whose business value beats the always-on premium. The same maths extends to performance modes: standard serverless performance mode "typically start[s] within four to six minutes after being triggered" and "Both modes use the same SKU, but standard performance mode consumes fewer DBUs": the saving is in consumption, not rate, and standard mode for continuous pipelines is only reachable through a continuous job with "Performance optimized" cleared (docs.../ldp/serverless, 2026-10-09).

---

7. Topology: one pipeline or many? (and why expectations cannot gate)

7.1 One pipeline: the medallion graph in a single update

A single pipeline can host bronze→silver→gold: the engine infers the dependencies, publishes every non-private dataset to the target schema, and lets you hide intermediates with private=True (table ref, 2026-08-06). You get one update, one compute assignment, automatic ordering and shared incremental state; which is why the worked example E2 graph is one pipeline. But orchestration logic cannot live inside it: a validation dataset with expect_or_fail fails only its own flow (triggered mode) or its dependents (continuous mode), and "A validation table doesn't gate its downstream tables … To stop downstream processing when a validation fails, split the validation logic and the downstream work into separate pipelines and orchestrate them with a job" (expectation-patterns, 2026-07-14).

Practical rule for this course: one pipeline per bounded domain/SLA (e.g., orders ingest+clean; a second pipeline for the gated gold marts), jobs orchestrating them in order, run as a service principal. The job's default run_if = ALL_SUCCESS is what turns a fail-mode expectation into a real gate: "Because a pipeline task fails when its update fails, the downstream task doesn't run unless the validation pipeline succeeds" (expectation-patterns, 2026-07-14).

7.2 Development updates vs production updates

Interactive updates from the Lakeflow Pipelines Editor (the "IDE for data engineering" announced in the GA release, with code-DAG pairing, contextual previews and selective execution of a single file/table) use "fast-start, debugging-focused behavior"; updates from Jobs or the Pipelines API, and all continuous updates, use automatic retry and restart (cluster restart on recoverable errors such as stale credentials, retry on cluster-start failure, cluster torn down after the run) (monitor-event-logs, 2026-10-09; updates doc). Production hygiene checklist: schedule a pipeline task in Workflows (update triggers are "Manual … Scheduled (jobs) … Programmatic"), set run-as to a service principal ("Running pipelines as service principals is considered a best practice because they are not tied to individual users"), promote code with Declarative Automation Bundles, keep the pipeline triggered unless a proven sub-second SLA demands continuous/real-time, and remember the 60-day UI/API history window.

---

8. Configuring the pipeline on AWS: storage, compute, editions, identity

8.1 Where the data lands: the Unity Catalog storage cascade

"Unity Catalog is the default for newly created pipelines" and a requirement for serverless (docs .../ldp/unity-catalog, 2026-10-09). Storage follows one rule: "Data for materialized views and streaming tables are stored in the storage location for the containing schema. If a schema storage location is not specified, tables are stored in the catalog storage location. If schema and catalog storage locations are not specified, tables are stored in the root storage location of the metastore." Two overrides to know cold:

• The Python path= parameter on @dp.table(...)/create_table, "A storage location for table data. If not set, use the managed storage location for the schema containing the table" (table ref, 2026-08-06), the supported way to pin a pipeline table to an explicit location (e.g., a UC volume or an external-location-backed S3 prefix).
• The SQL LOCATION property is not supported when defining tables in UC pipelines (unity-catalog limitations, 2026-10-09). Legacy storage pipeline settings (pre-February 5, 2025 pipelines, Hive-metastore era) default to dbfs:/pipelines/ and "cannot be changed after a pipeline is created" (properties ref, 2026-07-14).

Deletion semantics also changed with UC: deleting a pipeline retains its tables by default (they go inactive but remain queryable); DELETE /api/2.0/pipelines/{id}?cascade=true removes them; UNDROP recovers dropped tables within 7 days; and backing data lives under a __databricks_internal catalog: "This is expected" (unity-catalog doc, 2026-10-09).

8.2 Serverless vs classic (AWS)

New ETL pipelines default to "Unity Catalog · Current channel · Serverless compute" (configure-pipeline, 2026-10-01). Serverless requirements: UC-enabled workspace, accepted serverless terms, a serverless-enabled region (check the region support matrix, do not assume), and "If you need to use an AWS PrivateLink connection with your serverless Lakeflow pipelines, contact your Databricks representative" (serverless doc, 2026-10-09). No cluster-creation permission is required. Serverless-only capabilities: "Incremental refresh for Materialized views", "Stream pipelining" (enabled by default) and "Vertical autoscaling". Classic remains for legacy Hive metastore pipelines, unsupported network setups and non-serverless regions; its knobs are enhanced autoscaling, worker/driver instance types, instance profile, cluster policy, cluster tags and the Photon toggle. You cannot mix: one pipeline writes to either UC or Hive metastore, never both, and "Existing pipelines that use the Hive metastore cannot be upgraded to use Unity Catalog … you must create a new pipeline and re-ingest data" (unity-catalog doc, 2026-10-09).

8.3 Worked example E6, production wiring: DAB bundle, pipeline config, continuous job, run-as

The run-as identity must hold, in Unity Catalog: "USE CATALOG on the target catalog; USE SCHEMA and CREATE TABLE on the target schema; MODIFY permission on existing tables that the pipeline updates" (configure-pipeline, 2026-10-01): plus cloud-storage access via storage credentials and external locations (the IAM-role-backed objects from lesson 03) for every S3 path the pipeline touches. Note every non-obvious line: pipeline_mode: TRIGGERED + continuous: true on the job is the §6.3 pattern (the job takes precedence anyway, but leaving the pipeline on triggered avoids surprises); edition: ADVANCED is the §3.1 gate for expectations; and the dev/prod targets promote the same code without forking it.

• 📚 Did you know? The Auto Loader file-notification mode has a hard scale limit you will hit eventually: "100 pipelines per S3 bucket" in classic file-notification mode (docs, 2026-10, via digest 07). Managed file events (DBR 14.3 LTS+) use one queue per external location instead; which is another reason external locations, not raw s3:// paths, are the unit of ingestion governance on AWS.

---

9. Lakeflow pipelines vs Workflows jobs: the decision matrix

The docs' framing: pipelines are "declarative datasets + flows" with managed state and incremental processing; classic jobs are for work the abstraction hides. Auto Loader best practices make the default explicit: "For most users and production pipelines, Auto Loader with Lakeflow pipelines is a good fit. However, if you need maximum control and customization, use Auto Loader with Structured Streaming" (2026-06-29).

In practice production estates combine them: a pipeline task inside a multi-task job for scheduling, dependencies, retries and notifications; separate pipelines coordinated by job dependencies for quality gates (§7.1); and classic jobs for the non-DataFrame work no dataset function may contain.

Comparative Verdict: Lakeflow pipelines (DLT) vs Workflows jobs (as of Oct 2026)

• Lakeflow pipeline, the default answer for incremental batch-or-stream ETL over Delta. You declare datasets and expectations; the platform infers the DAG, manages checkpoints, applies schema evolution, emits exactly-once on managed tables, records quality metrics and publishes to Unity Catalog, on serverless by default, with serverless-only extras (incremental MV refresh, stream pipelining, vertical autoscaling). Buy the Advanced edition when you need expectations; wrap continuous workloads in a continuous job; treat the built-in continuous setting as legacy for new pipelines.
• Workflows job: the default answer for orchestration and everything non-DataFrame: scheduling (pipeline tasks, notebook tasks, run_if gates), retries, notifications, run-as service principals, branching, JAR/Spark-Submit work, ML training, file moves. Jobs are also where cross-pipeline quality gates live, because expectations cannot gate a pipeline's own downstream datasets.
• Verdict in one line: pipelines for the data plane, jobs for the control plane: one pipeline per bounded domain/SLA, gated by a validation pipeline inside a multi-task job, running as a service principal, triggered by default, continuous only against a written latency SLA. Portability bonus: the pipeline half is open-standard SDP; the job half is Databricks-native.

---

10. The traps: twelve confusions this lesson will not let you keep

1. "DLT is gone." Half-right and operationally dangerous: the product is now Lakeflow pipelines, but DLT_ SKUs, dlt event-log fields, billing_origin_product='DLT' and import dlt all survive: new code uses dp, monitoring still greps dlt.
2. "@dlt.table covers both dataset kinds." Not in the current API: streaming DF → @dp.table, batch DF → @dp.materialized_view. The mismatch breaks flow planning.
3. "There is an expect_all_or_quarantine decorator." There is not. Six decorators exist; quarantine is the documented pattern in section 5.
4. "Fail-mode expectations populate my quality dashboard." "Because fail causes the update to fail … metrics are not recorded." Fail = gate, warn/drop = measurement.
5. "Expectations gate the whole pipeline." expect_or_fail fails only the offending flow (triggered) or its dependents (continuous); downstream datasets do not wait for a validation table's verdict. Cross-pipeline gating = separate pipelines + a job.
6. "NOT(rule) splits NULLs correctly." NULL is neither true nor false; both branches can miss the row. Make rules total or branch on one explicit flag.
7. "Continuous is the production default." It is the largest cost factor on the meter ($2,520 vs$105/month at 10 DBU/h in E5). Start triggered; prefer a continuous job over the built-in setting.
8. "Editing a live continuous pipeline is safe." "Continuous pipelines automatically restart with the updated configuration. This can interrupt an active update"; switching continuous→triggered cancels the running update.
9. "Raw s3:// paths work in UC pipelines." Auto Loader must go through a UC external location + storage credential + READ FILES; and the event-log location doubles as the schema location, so moving or deleting it breaks future updates.
10. "Deleting the pipeline deletes the data." UC pipelines retain tables by default (inactive, still queryable); cascade=true deletes; UNDROP has 7 days.
11. "Expectations work on any edition." Core = ingest, Pro = +CDC, Advanced = +expectations; CDC needs Pro+. Pick per pipeline, price per SKU.
12. "Standard serverless is a cheaper rate." Same SKU, fewer DBUs and a 4 to 6 minute start; and standard mode for continuous pipelines only exists via a continuous job.

2026 Updates (as of October 2026)

---

Real-World Case Studies

📋 Case A: National Australia Bank (NAB) (banking, Cloud: AWS) Challenge. A 26-year-old Teradata warehouse with ~1,000 use cases, plus hand-written Spark/legacy-SQL ETL carrying 5,000 to 8,000-line DML and up to 64 union-all per pipeline, maintained inconsistently by 300 to 400 engineers. Stack on AWS. Databricks SQL, Auto Loader, Spark Declarative Pipelines on Lakeflow, Unity Catalog (lineage, masking, access control) and asset bundles for CI/CD: Cloud: AWS on the story page (https://www.databricks.com/customers/national-australian-bank-nab, undated page, accessed 2026-10). Reported outcome. "1,800 Spark Declarative Pipelines running on Databricks Lakeflow · 99.6% job success rate · 80% less transformation complexity"; end-to-end latency "under 15 minutes"; data quality "+38%"; some pipelines went from 64 union-all to 3 (same URL, undated, accessed 2026-10). Companionship page: 26-year-old Teradata decommissioned, 456 use cases migrated to Databricks SQL (https://www.databricks.com/customers/nab/dbsql, undated, accessed 2026-10). Maps to this lesson. Sections 2, 7 and 9: 1,800 declarative pipelines is what "the DAG is inferred, the platform owns the state" looks like at bank scale; the 80%-less-complexity figure is the medallion + declarative model replacing tens of thousands of lines of imperative ETL. ⏳ Temporal note: undated page, accessed Oct 2026: treat every figure as what NAB reported, not a platform guarantee.

📋 Case B, Shell (energy, Cloud: AWS, Azure, dual-cloud tag) Challenge. Millions of industrial time-series tags, legacy infrastructure and exponential data growth: ETL at IoT scale with a shortage of data-engineering skills. Stack. Shell.ai's platform on Databricks: Delta Live Tables (historic name; today Lakeflow Declarative Pipelines), Delta Lake, Databricks SQL and the open-source RTDIP streaming framework. ⚠️ Cloud-scope note: the story page is tagged Cloud: AWS, Azure: the pattern (S3-backed Delta tables + streaming ingestion + declarative pipelines) is cloud-portable; the AWS-specific part is the substrate: S3, IAM storage credentials, VPC. Reported outcome. "1.3 trillion rows of sensor data processed with Delta Live Tables · millions of dollars saved with data-driven operations · 70+ AI-driven transformation use cases" (Shell DLT case-study PDF, August 2021: historic product name inside the period-correct source); "Real-time data consumption has increased 15-fold over the last 5 years at Shell" (https://www.databricks.com/blog/developing-time-series-lakehouse-shell, 2023-10-09); 600 data-science team members supporting 100+ AI applications (https://www.databricks.com/customers/shell, undated, accessed 2026-10). Maps to this lesson. Sections 2 and 6: declarative pipelines are what let a century-old industrial operator run trillions of sensor rows without a 100-person ETL team; the DLT name in the 2021 PDF is this lesson's rename timeline made concrete. ⏳ Temporal note: the 1.3-trillion figure is a 2021 report; the 15× figure is 2023: both are reported outcomes, not 2026 platform capabilities.

---

---

Practice Questions