Structured Streaming on the AWS Lakehouse: Triggers, Checkpoints, foreachBatch and Exactly-Once

A streaming query on Databricks is an unbounded table and a batch-style plan that the engine re-runs over the next slice of input. That model is why "converting existing batch jobs into streaming jobs requires minimal code changes" (Databricks blog, 2026-04-20), and it is also why every production incident in this space comes back to four objects: the trigger that decides when a micro-batch runs, the checkpoint that remembers what was read and what was committed, the sink contract that decides whether a replayed batch duplicates rows, and the watermark that decides when state may be dropped. Get those four right and the lakehouse fills itself in near-real-time; get one wrong and you reprocess a week, duplicate a silver table, or watch state grow until the cluster dies.

This lesson is the streaming half of the course. Lesson 05 taught you how data arrives (Auto Loader, MSK auth, file events); this lesson teaches how it keeps arriving correctly: triggers and their AWS cost physics, checkpoint hygiene on S3-backed volumes, foreachBatch MERGE with txnAppId/txnVersion, change-data-feed fan-out, watermark-bounded state, exactly-once end to end, and the production posture (Lakeflow Jobs, not all-purpose clusters) that Databricks documents today.

Every fact below comes from Databricks' AWS documentation: the triggers (footer 2026-10-08), checkpoints (2026-09-11), foreach, Delta streaming (both 2026-08-24), CDF, Kafka (2026-09-11), api-options (2026-10-05), stream-monitoring (2026-09-23), watermarks (2026-09-11), stateful-streaming (2026-10-09), production, schema-evolution (2026-08-05) and Databricks SQL streaming/dashboards pages: plus dated Databricks blogs: all accessed or dated October 2026. Platform scope is Databricks on AWS: Amazon MSK, S3, Kinesis and Unity Catalog volumes are the substrate; Azure Event Hubs does not exist here and is out of scope.

In this lesson you will:

• configure every supported trigger and defend the choice against latency, cost and serverless constraints: including why the OSS continuous trigger is dead on Databricks and which of the three different "continuous" features people confuse;
• dissect a checkpoint (offsets, commits, state, metadata), place it on a Unity Catalog volume path on S3, and recite the restart-change matrix: what you may alter on a live query versus what silently forces a fresh start;
• write the headline streaming job: Amazon MSK → parse → watermark + dropDuplicatesWithinWatermark → foreachBatch MERGE into a Delta silver table with txnAppId/txnVersion, plus a dead-letter path;
• stream change data feed into gold: readChangeFeed=true, _change_type semantics, startingVersion, and the table_changes() batch replay for backfill and audit;
• bound stateful aggregation with watermarks: append vs update vs complete, the mandatory watermark on outer joins, the global-min multiple-watermark policy, and the state-bloat failure mode with its RocksDB remedy;
• watch a stream like an operator: progress fields (numInputRows, batchDuration, stateOperators), the Spark UI Streaming tab, StreamingQueryListener, and df.observe;
• build the near-real-time dashboard stack honestly: DBSQL streaming tables on serverless pipelines, and why a Lakeview cache up to 24 hours old is not "real-time";
• study three named AWS customer streaming stories (Burberry, Shell, Asurion), read the 2026 update box, and practise with 14 exam-style questions plus four interactive checks.

---

1. The micro-batch model and the trigger menu

Structured Streaming plans your DataFrame transformations once and executes them incrementally. In the default micro-batch engine, the driver repeatedly asks the source "what is new since the last offset?", plans a batch, runs it on the cluster, commits the offset and the sink write, and checkpoints progress. Real-time mode is Databricks' separate, supported low-latency execution path: not the OSS continuous-processing experiment. Batch and streaming share one API surface; only the reader, the trigger and the checkpoint differ. That is the unification lesson 05 used to run the same bronze code as a nightly job and as a low-latency stream.

1.1 The verified trigger table (docs, footer 2026-10-08)

1.2 Worked example E1: choosing a trigger, with the AWS bill attached

Scenario: an S3 landing zone drops files in bursts; the product spec says "near-real-time, minutes OK". Three candidate configurations, priced on two axes: compute (jobs cluster) and S3 request cost (the default trigger's hidden multiplier).

Compute sketch (ours on documented semantics): the hourly availableNow run drains in 20 minutes on a 4-node jobs cluster: 12 DBU/h × 0.33 h = 4 DBU/run; at Jobs Compute $0.15/DBU (AWS Premium list shape as of Oct 2026, same documentary basis as lesson 05's E7: always re-check system.billing.list_prices in your account) that is $0.60/run → $14.40/month. A processingTime='10 seconds' stream on the same cluster runs continuously: 720 h/month × 12 DBU = 8,640 DBU ≈ $1,296/month, whether or not data arrives. The default trigger (checks "every few milliseconds") multiplies S3 LIST/GET API volume against the same $0.005-per-1,000-LIST rate lesson 05 computed: on a 2M-file prefix, directory-listing mode at second-level triggers cost $2,592 to $25,920/month in LIST alone versus $28.80 at a 15-minute availableNow. The trigger is a cost decision, not a latency preference.

• 📚 Did you know? Trigger.Once is deprecated since DBR 11.3 LTS, "Use Trigger.AvailableNow for all incremental batch processing workloads" (triggers, 2026-10-08), but Once still appears in the serverless support sentence because old notebooks still call it. AvailableNow is the strict upgrade: same drain-then-exit semantics, plus rate limiting across micro-batches (maxFilesPerTrigger, maxOffsetsPerTrigger), which Once lacked. Any tutorial that teaches Once as current is teaching a 2022 API.

---

2. Checkpoints: the query's memory, on S3

2.1 Anatomy

"A checkpoint directory contains Offsets, Commits, State, and Metadata" (checkpoints, 2026-09-11):

On AWS the checkpoint lives on durable storage: the documented example writes checkpointLocation under a Unity Catalog volume path (/Volumes/catalog/schema/volume/path), which lands on S3 through your storage credential. "Give each query its own checkpoint directory in durable cloud storage" (production page). Never local DBFS /tmp, never the landing prefix, never a lifecycle-policy path.

2.2 The restart-change matrix

Not every code change is restart-safe on an existing checkpoint. The checkpoints page's rules, compressed:

---

3. Sources on AWS: Kafka/MSK deep, Delta CDF in

Lesson 05 covered Auto Loader end to end; this section adds what a streaming operator needs: Kafka rate controls, the Delta table as a source, and CDF as the recommended change path.

3.1 Kafka / Amazon MSK: the four options that decide your week

Minimal read is spark.readStream.format("kafka") with kafka.bootstrap.servers and subscribe, and then the options that matter (api-options, 2026-10-05; Kafka doc, 2026-09-11):

The reader returns key and value as binary, always cast/from_json/from_avro explicitly. Auth on AWS, in Databricks' recommended order: Unity Catalog service credentials (databricks.serviceCredential, DBR 16.1+, preferred for shared clusters and serverless, do not also set sasl.* yourself), then instance-profile MSK IAM (AWS_MSK_IAM + shaded shadedmskiam.* classes, SASL_SSL, port 9098), then role-based JAAS (awsRoleArn=), then SASL/PLAIN/SCRAM, then mTLS with material in a UC volume and passwords in a secret scope. MSK security groups must allow the Databricks compute SG on 9092 (PLAINTEXT) / 9094 (SASL/SSL) / 9098 (IAM); VPC peering or PrivateLink bridges the two VPCs.

3.2 Worked example E2: MSK → Delta bronze, rate-limited and deduplicated

Two documented traps live in those twenty lines. startingOffsets=latest is the streaming default, on a brand-new checkpoint the query silently skips everything already in the topic; set earliest for the first run, then trust the checkpoint forever. And on DBR 17.1+ the latest Kafka offsets are fetched after each micro-batch completes, so backlog metrics (avgOffsetsBehindLatest etc.) "may show small, persistent non-zero values. This is expected behavior", do not page someone at 03:00 for a 12-offset "lag".

3.3 Delta as a source, and CDF as the recommended change path

"Structured Streaming only accepts append inputs and throws an exception if any modifications occur on the source Delta Lake table" (docs.../structured-streaming/delta-lake, 2026-08-24), which is why "Databricks recommends streaming from the CDC feed of a Delta Lake table rather than directly from the table whenever possible." Automatic change data feed "uses the same table_changes() and readChangeFeed APIs as legacy change data feed and works with batch queries, Structured Streaming…" (CDF page). Enable it at table creation (delta.enableChangeDataFeed=true), nothing is captured before that moment. Section 5 builds the full fan-out.

• 📚 Did you know? The DStreams-era "Spark Streaming" project "is now a legacy project and new streaming applications should use Structured Streaming instead" (Databricks blog, 2026-04-20). If your resume or runbook still says "Spark Streaming" or Dstream, you are advertising a technology its own creator retired. The Structured Streaming blog of the same date notes the API "became production-ready in Spark 2.2": the model you are learning here has been the default for the better part of a decade.

---

4. foreachBatch and the exactly-once story

4.1 The contract

"You must use foreachBatch for Delta Lake merge operations in Structured Streaming": MERGE is a batch operation, so an upsert from a stream can only happen inside the micro-batch callback (docs.../structured-streaming/foreach, 2026-08-24). The same page fixes the guarantee: "foreachBatch() provides only at-least-once write guarantees. However, you can use the batchId provided to the function as way to deduplicate the output and get an exactly-once guarantee." On DBR 14.0+ the Delta path requires the explicit idempotency API: "You must use the txnAppId and txnVersion write options, binding txnVersion to batchId", and "Do not catch and retry exceptions locally"; let errors propagate so the orchestrator retries the whole micro-batch cleanly. For custom/external destinations (OLTP DB, queue, HTTP API): "Implement your own idempotency": if the batchId is the same, your side effect must be the same.

Three corollaries the docs state and junior engineers violate constantly:

1. New checkpoint ⇒ new txnAppId. "If you delete the streaming checkpoint and restart the query with a new checkpoint, you must provide a different txnAppId." New checkpoints start at batch ID 0; with the old appId, Delta treats the replay as a duplicate and drops it.
2. One streaming write per sink. "Databricks recommends configuring a separate streaming write for each sink you want to update instead of using foreachBatch. Writes to multiple sinks in foreachBatch reduces parallelization and increases overall latency because writes to multiple tables are serialized."
3. Idempotent MERGE. "Verify that your merge statement inside foreachBatch is idempotent. Otherwise, restarts of the streaming query can apply the operation on the same batch of data multiple times." Narrow the ON clause to the batch's keys.

4.2 Worked example E3, the headline job: MSK → silver MERGE with a dead-letter path

4.3 The exactly-once decision tree

---

5. Change data feed out: the gold fan-out

5.1 The pattern

Create the silver table with CDF on; every downstream consumer that needs changes streams the CDC feed instead of re-reading the table. Each CDF record carries _change_type ∈ insert / update_preimage / update_postimage / delete. Batch replay uses table_changes('tbl', v1, v2) for backfill and audit: the same mechanics Databricks' own CDF blog (2021-06-09) used to build the medallion CDC pattern.

5.2 Worked example E4: CDF stream to gold plus a batch replay

"You must use Structured Streaming for Databricks to automatically track versions for your table's change data feed", version tracking is a streaming-consumer behaviour, which is why pairing CDF with availableNow on a schedule is the documented deterministic-replay pattern. Lakeflow pipelines add AUTO CDC (the Applier) for DB→Delta and Delta→Delta without hand-written MERGE, covered in lesson 07; this section is the raw-notebook version you now understand from the inside.

---

6. Watermarks, state and the bloat failure mode

6.1 What a watermark is for

"Watermarks automatically remove old state data to prevent memory errors and increased processing latency" (watermarks, 2026-09-11). You assert a maximum lateness, withWatermark("event_time", "10 minutes"), and the engine advances a global watermark past max(event_time) − delay. "Queries always process records that arrive within the threshold. Queries might still process records that arrive outside the threshold, but this isn't guaranteed." Output mode changes what you see:

Stream-stream joins "only support append mode"; "For outer joins, watermarking is mandatory"; for inner joins it is strongly recommended. With multiple watermarks "Structured Streaming uses the minimum as the global watermark" by default; spark.sql.streaming.multipleWatermarkPolicy=max lowers latency but "drops data from the slowest streams… apply with caution." Window durations run "from microseconds up to days. Month durations and longer are not supported."

6.2 Worked example E5: windowed aggregation with the state math visible

Sizing logic (ours on documented semantics): live keys ≈ ads × windows-within-watermark = 5,000 ads × 3 windows (10 min watermark ÷ 5 min window) ≈ 15,000 state rows, trivial. Remove the watermark and state retains every (window, ad_id) forever; add complete mode and every historical row is re-emitted each trigger. The dedup variant matters as much: plain dropDuplicates(["event_id"]) holds state for every key forever; dropDuplicatesWithinWatermark(["event_id"]) (DBR 13.3 LTS+, requires a watermark) only holds keys within the lateness window, "the correct default for idempotent event ingestion when duplicates arrive close in event time."

6.3 Worked example E6: production state defaults and custom state

For all stateful queries on current DBRs, Databricks' documented posture is: RocksDB state store + changelog checkpointing (DBR 13.3 LTS+), "Databricks recommends enabling changelog checkpointing for all Structured Streaming stateful queries"; async state checkpointing and async progress tracking on the production page; no autoscaling for raw Structured Streaming job clusters ("Compute auto-scaling has limitations scaling down"); and state rebalancing on resize (default-on in Lakeflow pipelines; config spark.sql.streaming.statefulOperator.stateRebalancing.enabled since DBR 11.3+). Custom keyed state beyond aggregation/dedup/joins: transformWithState (DBR 16.2+) supersedes the legacy mapGroupsWithState/flatMapGroupsWithState for new code. Shuffle partition count is frozen at checkpoint creation for stateful operators (on-demand repartitioning arrives in DBR 18 LTS+), a silent footgun when someone "just bumps parallelism" on an existing query.

• 📚 Did you know? VACUUM "skips any directories that begin with _", which is why the Delta streaming docs suggest storing checkpoints "using a directory structure such as /_checkpoints". A one-character naming convention is the difference between a routine maintenance command and a wiped streaming state: pair it with a UC volume path and an IAM policy that forbids s3:DeleteObject on the checkpoint prefix, and checkpoint deletion stops being a 3 a.m. possibility.

---

7. Schema evolution on streams: locked plans, additive sinks

"A streaming query's schema is locked in during the planning phase… If the source schema changes mid-execution, the query fails, and the user must restart the streaming query" (schema-evolution, 2026-08-05), which is why production streams must run under a job with auto-restart. On the sink side, Delta accepts additive change only: .option("mergeSchema", "true") on the streaming write "only supports adding new columns". Prefer the per-write option (or SQL WITH SCHEMA EVOLUTION) over the session-wide spark.databricks.delta.schema.autoMerge.enabled, which is legacy. On the source side: new columns flow with mergeSchema; renames and drops need spark.databricks.delta.streaming.allowSourceColumnRename / …allowSourceColumnDrop; type widening is DBR 16.4 LTS+; "Other type changes: Not supported." And the nuclear fact, "Updating a table schema terminates any streams reading from that table" (update-schema, 2026-08-19). Auto Loader's own evolution loop (fail-with-UnknownFieldException-then-restart, schemaEvolutionMode=addNewColumns / rescue) was lesson 05's E-series; the Delta-source story here is the same restart-driven discipline at a different layer.

---

8. Observability and the near-real-time dashboard

8.1 Three layers of stream monitoring (stream-monitoring, 2026-09-23)

1. Query progress: the notebook's returned StreamingQuery, the Spark UI Streaming tab (.queryName() to disambiguate), and the progress JSON. Key fields: numInputRows, inputRowsPerSecond, processedRowsPerSecond, batchId, batchDuration, the durationMs breakdown (getBatch, addBatch, commitOffsets, walCommit), eventTime.watermark, and stateOperators (numRowsTotal, memoryUsedBytes, numRowsDroppedByWatermark, RocksDB customMetrics). Source metrics include Delta/Auto Loader numBytesOutstanding / numFilesOutstanding; reservoirId maps to DESCRIBE DETAIL <table> id.
2. StreamingQueryListener (Python and Scala, DBR 11.3 LTS+; UC access-mode floors 15.1+ dedicated, 16.1+ standard Scala): push to Prometheus/Grafana/Kafka. Keep callbacks tiny: "Processing latency with listeners can significantly affect query processing speeds."
3. df.observe(name, agg…): arbitrary per-epoch aggregates emitted into observedMetrics (e.g. a malformed-row ratio that alerts before downstream freshness does).

8.2 Dashboards: pull-based, cached, honest

DBSQL/Lakeview dashboards are pull-based: datasets run on a SQL warehouse and results land in a cache that "can return results that are up to 24 hours old even when the underlying data has changed", only an explicit schedule (or a cache miss) refreshes it, and "Serving results from the dashboard cache does not start the SQL warehouse" (dashboards caching, 2026-07-16). The supported near-real-time shape on AWS is therefore: stream writes Delta in seconds; a standalone streaming table in Databricks SQL (CREATE OR REFRESH STREAMING TABLE … SCHEDULE EVERY … / TRIGGER ON UPDATE AT MOST EVERY INTERVAL 5 MINUTES, minimum TRIGGER ON UPDATE interval 1 minute, running on an automatically created serverless pipeline) keeps the table fresh; the dashboard queries that table on its own cron. Sub-minute dashboard push is not a documented DBSQL feature, do not promise it in a design review.

---

9. Production posture: Lakeflow Jobs, and the three "continuous"s

Databricks' production page is unusually direct: "Run production Structured Streaming workloads as scheduled Lakeflow Jobs on Databricks"; "Don't run Structured Streaming workloads using all-purpose compute"; "Schedule Lakeflow Jobs using Continuous mode. This refers to the Databricks Jobs scheduling feature, not the Structured Streaming trigger interval"; "Don't enable autoscaling for compute for Structured Streaming jobs" (compensated by RocksDB state, async state checkpointing and async progress tracking). And the platform recommendation: "Databricks recommends using Lakeflow pipelines for new Structured Streaming pipelines."

The naming trap that breaks architecture reviews: three different things called "continuous":

Two more incompatibilities from the same pages: async progress tracking "isn't compatible with the AvailableNow or Once triggers", you cannot combine the cost-optimized trigger with that latency optimization; and "If you try to recover from a query failure with an incremental batch, a trigger interval change doesn't solve the problem", a stuck failed micro-batch must still complete; scale up or start a new checkpoint instead.

---

10. The traps: fourteen confusions this lesson will not let you keep

1. Default trigger as a production default. 3 to 5 s latency with per-few-millisecond storage checks: the documented "unexpected charges" scenario. Set an explicit trigger; prefer availableNow for scheduled ingestion.
2. Deleting a checkpoint to "fix lag". Fresh start, full replay, watermark reset, and Delta silently drops the replay unless txnAppId rotates.
3. Sharing one checkpoint across queries. Explicitly forbidden: "Each query must have a different checkpoint location."
4. Same txnAppId after a new checkpoint. Batch IDs restart at 0; Delta ignores the writes. Document the appId per stream; rotate it deliberately.
5. Catching exceptions inside foreachBatch. Breaks at-least-once accounting; let the orchestrator retry the whole batch.
6. Fan-out inside foreachBatch. Multiple sinks serialize; prefer one streaming write per sink.
7. Missing persist() in foreachBatch. Read the micro-batch DataFrame twice without it and you re-read the source.
8. dropDuplicates without a bound. State for every key forever; use dropDuplicatesWithinWatermark (DBR 13.3 LTS+) with a watermark.
9. complete mode on a growing aggregation. State is never dropped by design; "can perform poorly as data scales."
10. Kafka first run on default startingOffsets. latest silently skips the backlog; set earliest once, then trust the checkpoint.
11. Treating small Kafka backlog metrics (DBR 17.1+) as an incident. Persistent non-zero *OffsetsBehindLatest is expected by design.
12. The three "continuous"s. Unsupported OSS trigger ≠ Lakeflow continuous mode ≠ Jobs continuous scheduling.
13. Serverless + non-availableNow trigger. Only AvailableNow/Once start on serverless: plan compute per trigger.
14. "The dashboard is real-time because the stream is." The cache is up to 24 hours stale; the schedule, or a DBSQL streaming table underneath, is what makes it fresh.

Comparative Verdict: raw Structured Streaming notebooks vs Lakeflow Declarative Pipelines (as of Oct 2026)

• Raw writeStream notebooks (this lesson's code), the transparent option. Full control over triggers, checkpoints, foreachBatch MERGE, txnAppId discipline and rate limits; the right teaching tool and the right choice for one-off jobs, bespoke sinks (OLTP upserts, queues, APIs) and maximum customization. Databricks still fully supports them, when they run as "scheduled Lakeflow Jobs on Databricks" with auto-restart, on jobs compute, without autoscaling. The cost of that control is yours: you own checkpoint placement, state-store configuration (RocksDB + changelog), restart-on-schema-change wiring, and every line of idempotency code.
• Lakeflow Declarative Pipelines (formerly Delta Live Tables; lesson 07): the recommended production path. "Databricks recommends using Lakeflow pipelines for new Structured Streaming pipelines" (production page). You declare sources, sinks and expectations; Databricks manages checkpoints and state, runs continuous mode for true continuous streaming (including serverless), applies AUTO CDC instead of hand-written MERGE, rebalances state by default on resize, and offers enhanced autoscaling that raw streaming jobs should not use. The trade-off: less fine-grained control of the micro-batch loop, a pipeline (not notebook) development model, and another concept to govern.
• Auto Loader ingestion (lesson 05) splits the same way: "Lakeflow Spark Declarative Pipelines is the recommended way to run Auto Loader for most production ingestion… If your workload doesn't have low-latency requirements and your priority is minimizing compute cost, you can instead schedule Auto Loader as a triggered batch job that uses Trigger.AvailableNow."
• Verdict in one line: new streaming pipelines → Lakeflow; bespoke sinks, teaching labs and tightly cost-tuned availableNow batch jobs → raw Structured Streaming on Lakeflow Jobs with the hygiene of this lesson, and never an all-purpose cluster either way.

---

2026 Updates (as of October 2026)

---

Real-World Case Studies

📋 Case A: Burberry (luxury retail, Cloud: AWS) Challenge. Daily clickstream batches arriving "anywhere between 2:00 and 7:00 PM GMT", and a cloud warehouse that pre-aggregated away the detail needed for Customer 360. Stack on AWS. Databricks Data Intelligence Platform with Snowplow Behavioral Data Platform for real-time Customer 360: the story page carries Databricks' own Cloud: AWS tag (https://www.databricks.com/customers/burberry, undated page, accessed 2026-10). Reported outcome. "99% Reduction in latency for customer clickstream data · 52x Increase in cookie duration"; "we've broken free from daily data batches and formed an accurate real-time view of our customers' digital activity" (same URL, accessed 2026-10; story origin 2023). Maps to this lesson. Sections 1 and 9: a clickstream latency SLA is a business requirement; Burberry's fix was an event-driven lakehouse path, not a bigger nightly window. The trigger choice (availableNow on file arrivals, or a seconds-interval stream from a bus) is exactly the E1 decision, and the dashboard freshness promise belongs to section 8's cache honesty. ⏳ Temporal note: a 2023-origin figure still displayed in 2026: quote it as what Burberry reported at the time, not a platform guarantee.

📋 Case B, Shell (energy, Cloud: AWS, Azure, dual-cloud story) Challenge. Millions of industrial time-series tags, legacy infrastructure, exponential data growth and a shortage of data-engineering skills: ETL at IoT scale. Stack. Shell.ai's platform on Databricks: "Delta Live Tables" (now Lakeflow / Spark Declarative Pipelines), Delta Lake, and structured streaming ingestion via the open-source RTDIP framework; the story page lists AWS and Azure, so we teach the AWS half as S3-backed Delta plus streaming ingestion (https://www.databricks.com/customers/shell, undated, accessed 2026-10). Reported outcome. "1.3 trillion rows of sensor data processed with Delta Live Tables · Millions of dollars saved with data-driven operations" (Shell DLT PDF, August 2021); "Real-time data consumption has increased 15-fold over the last 5 years at Shell"; RTDIP "downloaded 25,000+ times/month" (https://www.databricks.com/blog/developing-time-series-lakehouse-shell, 2023-10-09). Maps to this lesson. Sections 6 and 9: a trillion-row sensor estate survives only with watermark-bounded state and declarative continuous pipelines; the raw-notebook skills in this lesson are what let you debug the managed pipeline when it misbehaves. ⏳ Temporal note: DLT PDF figures are 2021-vintage under the historic product name; the RTDIP numbers are 2023. The story spans AWS and Azure: only the streaming pattern is cloud-portable.

📋 Case C: Asurion (insurance & device protection, Cloud: AWS) Challenge. 8,000 tables, 10,000 views and ~600 data marts with per-table ETL mappings and a duplicated batch/speed layer that could not absorb new sources cheaply. Stack on AWS. Spark Structured Streaming, Auto Loader and foreachBatch into Delta, ingesting "database CDC feeds, APIs and flat files to streams from Kinesis, Kafka, SNS and SQS": a production data lake explicitly "on AWS" (https://www.databricks.com/blog/2021/09/16/large-scale-etl-and-lakehouse-implementation-at-asurion.html, 2021-09-16). Reported outcome. "The Asurion data lake executes 4,000+ streaming jobs and hosts over 4,000 tables in a production data lake on AWS" (https://www.databricks.com/blog/2021/05/10/aws-guide-to-data-ai-summit-featuring-disney-comcast-capital-one-and-mcdonalds.html, 2021-05-10). Maps to this lesson. Sections 4 and 10: at 4,000+ streaming jobs, the architecture is a parameterised foreachBatch/checkpoint framework (one idempotent pattern, many feeds), and the AWS-native source list (Kinesis, SNS, SQS, MSK) is why this course teaches MSK IAM auth and S3 checkpoint hygiene as first-class skills. ⏳ Temporal note: 2021 figures reported at the time; re-check any workspace concurrency statement from that blog against current docs before reusing it.

---

---

Practice Questions

Hover a bar: availableNow runs ~20 minutes per hourly schedule (1 on the scale, it turns itself off); a processingTime='10 seconds' stream is up whenever data flows (36, roughly a full month of uptime against a 20-minute job); the unspecified default adds a per-few-millisecond storage poll on top of continuous compute (60, the documented API-call multiplier, which lesson 05 priced at $2,592 to$25,920/month in LIST alone on a 2M-file prefix versus $28.80 at a 15-minute availableNow); real-time mode requires dedicated compute and less frequent checkpoints (80, the price of sub-second latency, and the reason it is reserved for fraud and personalization); the OSS continuous trigger is free because you cannot run it (0, "Not supported"). The scale is a teaching model of documented semantics, not a platform metric, do the DBU arithmetic with your own cluster shape and system.billing.list_prices.