Ingestion on AWS: Auto Loader, File Events, Kafka/MSK and Idempotent Sinks

Everything downstream in this course, the Delta transactions of lesson 04, the declarative pipelines of lesson 07, the quality gates of lesson 10, assumes a bronze table that is already being filled correctly. Ingestion is where lakehouse projects quietly fail: a checkpoint dropped "to fix lag", a schema directory parked inside the landing prefix so the stream discovers its own state files, a Kafka query left on the default startingOffsets so the first run skipped three hours of backlog, or a directory-listing stream on a one-second trigger whose LIST bill grows with every hour of uptime.

The mechanics are not complicated. Databricks' own recommendation is one sentence: "Databricks also recommends Auto Loader whenever you use Apache Spark Structured Streaming to ingest data from cloud object storage" (docs.../auto-loader, last updated 2026-09-11). What is complicated is the hygiene: where state lives, what happens when a column appears, which discovery mode pays for itself, and how a retried micro-batch avoids writing rows twice. That is this lesson.

Every fact below comes from Databricks' AWS documentation (Auto Loader, directory-listing, file-notification, schema, production, clean-source, best-practices, FAQ, checkpoints, output-mode, triggers, foreach, Delta streaming, Kafka and JDBC pages), the Apache Spark and AWS documentation, and dated Databricks engineering blogs: all accessed or dated October 2026. Platform scope is Databricks on AWS: Azure Event Hubs does not exist on this substrate and is out of scope here.

In this lesson you will:

• write the Auto Loader bronze notebook itself, cloudFiles.format, schemaLocation, checkpointLocation, useManagedFileEvents, inferColumnTypes, schemaEvolutionMode, mergeSchema on the Delta sink, and narrate every default;
• place the schema location and checkpoint where they cannot be discovered, expired or shared, and defend the choice against the "put state next to the data" instinct;
• compute the three LIST bills for one landing prefix under three trigger policies, and choose between directory listing, classic SNS+SQS file notifications and managed file events with their permissions, caps and backfill behaviour;
• run a throughput sketch for a 1.8-million-file peak: arrival rate versus service rate, the hard file cap versus the soft byte cap, and what happens when someone "carefully" sets maxFilesPerTrigger too low;
• configure Amazon MSK with AWS_MSK_IAM, including the shaded class names and the port that must match the authentication mode, plus maxOffsetsPerTrigger sizing arithmetic;
• choose an output mode against the Delta support matrix, make foreachBatch idempotent with txnAppId/txnVersion, and defend the end-to-end exactly-once chain;
• unify batch and streaming on one code path with Trigger.AvailableNow, and price the nightly bronze job in DBUs as of October 2026;
• study two named AWS customer cases (Asurion and Burberry), read the 2026 update box, and practise with 13 exam-style questions plus four interactive checks.

---

1. Ingestion on the AWS lakehouse: three source families, one write path

Databricks-on-AWS ingestion has three families that matter in production, plus one toy:

Three things in that table are load-bearing. First, all four converge on the same write path: writeStream → checkpointLocation → output mode → Delta. The transformation code is identical for spark.read and spark.readStream because both return DataFrames, that is the batch/streaming unification story in section 9. Second, JDBC has no Databricks-documented streaming source: current docs describe batch reads, governed UC connections, and foreachBatch for writes; row-level DML into the source database is explicitly unsupported ("The Spark JDBC data source does not support arbitrary DML statements", docs.../connect/jdbc-connection, accessed 2026-10). Third, the checkpoint is the query, delete it and you get a fresh start: for Kafka that means startingOffsets applies again, for Auto Loader the discovered-file state is gone.

• 📚 Did you know? Auto Loader is not new. Databricks announced it on 2020-02-24 as "an optimized cloud file source for Apache Spark that loads data continuously and efficiently from cloud storage as new data arrives", naming Fivetran, Qlik, Infoworks, StreamSets and Syncsort as partner ingest tools at launch (databricks.com/blog/2020/02/24/introducing-databricks-ingest-easy-data-ingestion-into-delta-lake.html, 2020-02-24). The problem it solved has never changed: "Both cost and latency can add up quickly as more and more files get added to a directory due to repeated listing of files." The scale figures you will see quoted, "billions of files" for backfills and "millions of files per hour" for near-real-time ingest, are vendor capability claims from the docs page (2026-09-11), not customer benchmarks: treat them as design headroom, never as a SLA.

---

2. Auto Loader anatomy: the cloudFiles source

2.1 The options that matter (Oct 2026)

spark.readStream.format("cloudFiles") is an ordinary Structured Streaming source. What makes it Auto Loader is the discovery engine: as files are found, their metadata is persisted "in a scalable key-value store (RocksDB) in the checkpoint location", which "ensures that data is processed exactly once" (docs .../auto-loader, 2026-09-11).

Two defaults surprise everyone. includeExistingFiles=true means the first start of a fresh checkpoint walks the whole prefix (that is your backfill). And maxFilesPerTrigger=1000 is a hard limit while the byte limit is soft: "An individual file is never split across micro-batches" (docs.../spark/api-options, page date 2026-10-05).

2.2 Worked example E1: a production-shaped bronze Auto Loader notebook

What the numbers mean while it runs: the first start samples the first 50 GB or 1,000 files (whichever is crossed first, most recent by modification time) to infer the schema; every subsequent micro-batch is capped at 1,000 files unless the runtime auto-tunes it (DBR 18.0+); Trigger.AvailableNow processes only files that arrived before the query start time and turns itself off; and if a new column appears, the query stops with UnknownFieldException after _schemas has already been updated, so the job, not a human, restarts it (section 3).

---

3. Schema inference, evolution and rescue

3.1 What inference actually samples

On the first definition of the DataFrame, Auto Loader "lists your source directory and chooses the most recent (by file modification time) 50 GB of data or 1000 files, and uses those to infer your data schema" (Auto Loader FAQ, dated 2026-06-23). The result is written to a _schemas directory under cloudFiles.schemaLocation, and it is the schema history the stream reloads on every restart: "When the stream restarts, it picks up the evolved schema from the schema location and starts executing without any overhead from inference" (same FAQ).

Defaults that bite:

• JSON, CSV and XML infer all columns as strings unless you set cloudFiles.inferColumnTypes=true. Typed formats (Parquet, Avro) merge encoded types instead.
• If the source directory is empty, you must supply a schema; there is nothing to sample.
• cloudFiles.schemaHints lets you pin known types ("order_id=STRING, amount=DECIMAL(12,2)") without hand-writing the whole schema.
• Partition columns are inferred from Hive-style paths, but new partition columns are not evolved: declare them with cloudFiles.partitionColumns.
• Overrides exist: spark.databricks.cloudFiles.schemaInference.sampleSize.numBytes (e.g. 10gb) and ...sampleSize.numFiles.

3.2 Evolution modes and the restart loop

The sequence on a schema change is fixed, and worth memorising, because it is an alert-and-replay design rather than a silent fix:

Two supporting facts. The write side must agree: .option("mergeSchema", "true") on the Delta sink, or the table refuses the additive change. And a rename is treated as an add: the new name becomes a column while the old one is populated with NULL for new rows (docs.../data-engineering/schema-evolution, accessed 2026-10). A streaming query's schema is "locked in during the planning phase", so any mid-flight source change fails the query and requires a restart; that is why the docs recommend running Auto Loader streams as Lakeflow/Workflows jobs rather than as forever-running all-purpose notebooks.

3.3 Worked example E2: a never-fail events stream with pinned types

_rescued_data holds whatever the schema could not take, "column is missing from the schema / type mismatches / case mismatches", as a JSON blob alongside the source file path, which is how you keep a zero-loss lane and a measurable quality signal at the same time. Use rescue for lanes that must never stop (clickstream, IoT); use addNewColumns plus an auto-restarting job for lanes where a new column is an event you want to notice.

3.4 ⚠️ WARNING: never put schema or checkpoint state in the data path

⚠️ WARNING: cloudFiles.schemaLocation and checkpointLocation must never live inside the source landing directory, and neither may sit under a cloud object lifecycle policy.

1. Inside the data path means self-discovery. If _schemas/ or the checkpoint sits under s3://acme-landing/raw/orders/, Auto Loader will eventually list its own state files as inputs. Keep state on a different prefix or a Unity Catalog volume (/Volumes/<catalog>/<schema>/checkpoints/...), never under the table's data directory.
2. A lifecycle rule that expires state destroys the query. "Databricks recommends setting the checkpoint location to a location without a cloud object lifecycle policy. If files in the checkpoint location are cleaned according to the policy, the stream state is corrupted. If this happens, you must restart the stream from scratch" (docs .../auto-loader/production, footer 2026-09-18). The same sentence applies to schemaLocation.
3. Separate per workload. Each Auto Loader ingestion workload requires a separate streaming checkpoint. Two workloads sharing one checkpoint, or one _schemas directory, will fight over state.
4. State beside a Delta table must be underscore-prefixed. If you must co-locate a checkpoint with a table, use /_checkpoints: VACUUM "skips any directories that begin with _" (docs .../structured-streaming/delta-lake, dated 2026-08-24), so maintenance will not eat it.
5. Equality is fine, containment is not. The docs explicitly allow schemaLocation == checkpointLocation; what they never allow is state inside the input path or under expiry.

---

4. File detection: directory listing vs file notifications

Auto Loader "supports two file detection modes. By default, Auto Loader uses directory listing mode. However, Databricks recommends file notification mode using file events for most workloads" (docs .../auto-loader, 2026-09-11). Choosing between them is the cost-and-infrastructure decision of this lesson.

4.1 Directory listing (default)

Every trigger lists the input tree. Two properties make it viable at all: Auto Loader "uses native cloud APIs to get lists of files", and on S3 each LIST returns up to 1,000 keys, so the number of API calls scales as files ÷ 1,000 rather than directories × triggers. That is the difference against the OSS Spark file source, whose cost the docs model as "1 (base directory) + 365 (per day) × 24 (per hour) = 8761 calls" for a YYYY/MM/DD/HH layout (docs .../auto-loader/directory-listing-mode, 2026-09-11).

The documented warning is about frequency: with continuous triggers such as Trigger.ProcessingTime, directory listing mode "continuously lists the entire directory and can significantly increase LIST API costs". Three hygiene rules:

• keep lexical file naming (YYYY/MM/DD/HH, zero-padded: AWS DMS CDC files like LOAD00000001.csv and Kinesis Firehose object names are the docs' own examples), even though Auto Loader does not guarantee the order in which files are discovered or processed;
• prefer Trigger.AvailableNow over a continuous trigger in this mode;
• incremental listing is deprecated: in auto mode a full directory list runs after 7 consecutive incremental lists (tunable with cloudFiles.backfillInterval), and DBR 11.3 LTS+ lets you change the input path without a new checkpoint: a capability not supported for file notifications.

4.2 Classic file notifications: SNS + SQS

In this mode Auto Loader wires the S3 bucket notification (ObjectCreated) through an AWS SNS topic to an AWS SQS queue (resource prefix databricks-auto-ingest), then consumes the queue. The setup policy needs the documented action set, s3:GetBucketNotification, s3:PutBucketNotification, sns:CreateTopic, sns:Subscribe, sns:Publish, sqs:CreateQueue, sqs:ReceiveMessage, sqs:DeleteMessage, sqs:SendMessage, sqs:SetQueueAttributes, sqs:ChangeMessageVisibility, sqs:PurgeQueue on databricks-auto-ingest-*, and after setup the running stream needs only the reduced consumption policy (docs.../auto-loader/file-notification-mode, footer 2026-10-07). The same page caps the pattern at 100 per S3 bucket.

Two caveats travel with it. Event delivery is best-effort, "Cloud providers do not guarantee 100% delivery of all file events", so completeness SLAs in classic mode are bought with cloudFiles.backfillInterval asynchronous relists that ingest only missed files (no duplicates, because the checkpoint already knows what it saw). And you cannot move the input path without a new checkpoint in this mode.

4.3 Managed file events (recommended)

The modern form is enabled on the external location, not per stream: a single file notification queue for all streams that process files from a given external location, activated with cloudFiles.useManagedFileEvents=true. Requirements: a Unity Catalog-enabled workspace, permission to create a storage credential and external location, and Databricks Runtime 14.3 LTS or above (same page, 2026-10-07).

What you get: fewer IAM grants (one-time cloud permission, then no per-stream queue), automatic tuning and cleanup, automatic backfill, "Databricks automatically backfills those external locations with a full listing when file events are first enabled, and then continues to backfill about every 24 hours", and the explicit instruction not to set a backfill interval in this mode. What you must respect: "The file events service optimizes file discovery by caching the most recently created files. If Auto Loader runs infrequently, this cache can expire … invoke Auto Loader at least once every seven days." A nightly or hourly AvailableNow job clears that bar with room to spare; a quarterly manual run does not.

Mode switching is safe: "you can switch between file notifications and directory listing at any time and still maintain exactly-once data processing guarantees" (same page).

4.4 Worked example E3: the LIST bill, computed three ways

Inputs (all official): S3 returns 1,000 keys per LIST; "LIST requests for any storage class are charged at the same rate as S3 Standard PUT, COPY, and POST requests": $0.005 per 1,000 in us-east-1, while DELETE and CANCEL are free (aws.amazon.com/s3/pricing, accessed 2026-10). Landing prefix: 2,000,000 files. The arithmetic below is ours; the inputs are AWS's.

Read the shape, not just the dollars: the same 2M-file prefix costs 900× more at a one-second trigger than at a 15-minute AvailableNow, because listing frequency, not data volume, is the multiplier. Managed file events replace per-trigger listing with queue API calls, leaving only the automatic ~24-hour backfill listing. We deliberately do not price the SQS/SNS side here: this lesson's sources carry no verified AWS queue-price figure, so check aws.amazon.com/pricing yourself before you model it.

• 📚 Did you know? LIST is billed like a write, not like a read: "LIST requests for any storage class are charged at the same rate as S3 Standard PUT, COPY, and POST requests": $0.005 per 1,000; while DELETE and CANCEL are free (aws.amazon.com/s3/pricing, accessed 2026-10). And there is no general-purpose rename on S3: "S3 Express One Zone is the only storage class that supports the RenameObject API", so a MOVE-style archive is a copy + delete pair billed at PUT/COPY/LIST rates, and in classic notification terms that copy surfaces as yet another ObjectCreated event. That is a second reason managed file events, which do not re-fire on your own bookkeeping copies, are the recommended mode.

---

5. Throughput and production hygiene

5.1 Worked example E4: a throughput sketch for a 1.8-million-file peak

Scenario: a partner drops 1,800,000 JSON files of 120 KB into the landing prefix during a two-hour nightly window (216 GB), then keeps a trickle arriving afterwards. Arrival rate = 1,800,000 ÷ 7,200 s = 250 files/s.

Sizing the micro-batches:

• Default maxFilesPerTrigger=1000 → each micro-batch carries 1,000 files = 120 MB.
• Set maxBytesPerTrigger="500mb" instead and that soft byte cap would allow 500 MB ÷ 120 KB ≈ 4,166 files, but the file limit is a hard limit, so the effective batch stays at 1,000 files. Whichever limit is reached first wins.
• Assume the cluster sustains 1.0 s per micro-batch → service rate 1,000 files/s (120 MB/s). Utilisation ρ = 250 ÷ 1,000 = 0.625: the queue drains.
• Backlog of 1.8M files drains at 1,000 − 250 = 750 files/s net → 2,400 s = 40 minutes, having processed 2.4M files in total (≈288 GB).

Now the failure mode a "careful" engineer creates: setting maxFilesPerTrigger=100 "to be gentle on the cluster" caps service at 100 files/s < 250 files/s arrival, the queue grows by 150 files/s without bound, and no amount of cluster size fixes a bad admission-control knob. On DBR 18.0+ both cloudFiles.maxFilesPerTrigger and cloudFiles.maxBytesPerTrigger are "dynamically configured and do not need to be set manually" (docs.../spark/api-options, 2026-10-05); pinning them there fights the automatic tuning. And if the peak is genuinely huge and one-shot, run Trigger.AvailableNow as a Workflows task instead of a continuous trigger, it rate-limits across micro-batches and turns itself off when caught up.

5.2 Retention: maxFileAge, cleanSource, backfill

The economics of cleanSource are the same argument as section 4: "Archive processed files with cloudFiles.cleanSource … This reduces both storage costs and directory listing costs for long-lived streams"; you stop paying to re-list files you will never read again.

5.3 Monitoring the source

Three metrics tell you whether ingestion is healthy before downstream freshness alerts fire: numFilesOutstanding, numBytesOutstanding, and (file notifications) approximateQueueSize: plus the state table:

A growing numFilesOutstanding at a stable processing rate means you are under-sized (fix the rate limits, not the checkpoint); a stuck approximateQueueSize in classic notification mode means the consumer cannot drain the queue: permissions, or the 100-per-bucket cap.

---

6. Kafka and Amazon MSK as a source

6.1 The options that matter

Minimal read: spark.readStream.format("kafka").option("kafka.bootstrap.servers", "<server:9094>").option("subscribe", "orders").option("startingOffsets", "latest").load(), and then the four options that decide whether your first day goes well:

The row schema is a fixed seven columns: key binary, value binary, topic string, partition int, offset long, timestamp timestamp, timestampType int. Authentication on AWS is SSL or SASL_SSL with PLAIN/SCRAM/AWS_MSK_IAM, or, in the governed path, a Unity Catalog service credential / connection, in which case you must not set kafka.security.protocol or sasl.* yourself. If you do drive the client directly: always prefix client properties with kafka. and always use Databricks' shaded class names (kafkashaded., shadedmskiam.), or you will meet SaslException / RESTRICTED_STREAMING_OPTION_PERMISSION_ENFORCED.

6.2 Worked example E5: MSK IAM read with rate limiting

Sizing arithmetic (ours, on official semantics): peak arrival is 12,000 msg/s across 6 partitions at 400 bytes average. With processingTime="30 seconds" and maxOffsetsPerTrigger=300000, capacity is 300,000 ÷ 30 = 10,000 msg/s < 12,000 msg/s, the backlog grows by 2,000 msg/s. Raise it to 600,000 → 20,000 msg/s capacity, micro-batch payload 600,000 × 400 B = 240 MB, ρ = 0.6, stable. Two supporting facts for the ops runbook: the IAM role needs kafka-cluster:Connect and kafka-cluster:ReadData, and the port must match the mode (9092 PLAINTEXT, 9094 SASL/SSL, 9098 IAM), plus VPC peering or PrivateLink between the Databricks VPC and the MSK VPC (docs.../kafka/authentication, 2026-07-10).

6.3 JDBC and the rate source

JDBC is a batch pull on this platform: credentials via dbutils.secrets.get(scope="jdbc", key="username") (never in the URL), or the governed .option("databricks.connection", "prod_pg") path with the driver JAR on a Volume: driver memory is capped at 400 MiB, and the default externalOptionsAllowList permits only six read options. Because row-level writes into the source are unsupported, the upsert pattern is always pull → Delta with idempotent appends:

The rate source (spark.readStream.format("rate"), rowsPerSecond default 1, plus rampUpTime and numPartitions) emits synthetic timestamp + value rows and is "intended for testing and benchmarking": perfect for verifying checkpoint, output mode and Delta sink wiring in two minutes before any S3 or MSK permissions are involved.

---

7. Sinks: output modes, idempotency and MERGE

7.1 Output modes and the Delta matrix

"Delta Lake, which backs all Unity Catalog managed tables, supports append and complete modes but not update mode" (docs .../structured-streaming/output-mode, 2026-09-11). Also note "Joins only support the append output mode". Plain Delta appends are exactly-once through the transaction log: "The Delta Lake transaction log guarantees exactly-once processing, even when there are other streams or batch queries running concurrently against the table" (docs .../structured-streaming/delta-lake, 2026-08-24).

7.2 Worked example E6: idempotent upserts in foreachBatch

"You must use foreachBatch for Delta Lake merge operations in Structured Streaming", and foreachBatch() "provides only at-least-once write guarantees", so idempotency is your job (docs.../structured-streaming/foreach, dated 2026-08-24).

For plain appends written from foreachBatch, bind the batch identity explicitly: .option("txnAppId", app_id).option("txnVersion", batch_id), Delta "uses txnAppId and txnVersion to identify and ignore duplicate writes". Three rules from the same pages: if you delete the checkpoint and restart, you must provide a different txnAppId, because "New checkpoints start with a batch ID of 0"; do not catch and retry exceptions locally, let errors propagate so the orchestrator retries the whole batch and metrics stay honest; and prefer one streaming write per sink, since multiple sinks inside foreachBatch serialize your parallelism.

---

8. Exactly-once end to end

The guarantee is a chain, and every link is a component you have already met:

Checkpoint contents are fixed: offsets (what was read), commits (which micro-batches reached the sink: "enabling exactly-once semantics"), state, and metadata (the unique query ID plus configuration settings). Consequences you should be able to recite:

• one checkpoint per query, on durable storage, outside any lifecycle policy;

• deleting or moving it starts fresh: Kafka replays per startingOffsets, Auto Loader re-lists (and reprocesses the prefix if includeExistingFiles was true on the original first start), and Delta writes replay unless txnAppId changes;

• for non-Delta targets (an OLTP database, a queue, an HTTP API) you own idempotency keyed on batchId: "If the batchId stays the same, the result of your operation must remain the same";

• foreachBatch never runs in continuous processing mode, and on serverless only Trigger.AvailableNow() and the deprecated Trigger.Once() are supported.

• 📚 Did you know? A checkpoint stored beside a Delta table survives maintenance only if you name it with an underscore: VACUUM "skips any directories that begin with _", which is why the docs' own suggestion is "a directory structure such as /_checkpoints" (docs .../structured-streaming/delta-lake, 2026-08-24). It is a small naming convention standing between your stream state and a routine maintenance command.

---

9. Batch + streaming unification

One code path, two triggers. The same bronze reader runs as a nightly incremental job or as a low-latency stream; only trigger, checkpoint and cluster differ.

Trigger.AvailableNow "processes all files that arrived before the query start time", which is exactly the semantic a scheduled batch wants, and SQL users get the same thing through STREAM read_files(...); which itself uses Auto Loader for streaming tables (docs.../read_files, 2026-08-03).

9.1 Worked example E7: what the nightly bronze job costs

One AvailableNow run per night, 25 minutes per run, 30 runs/month → 12.5 cluster-hours/month on a 4-node job cluster (1 driver + 3 workers). For emission we use Databricks' own illustrative node: "an instance type which runs at 3 DBU/hour … a four node jobs cluster would be charged at $2.40 ($0.2 × 3 × 4)" (best-practices cost blog, 2022-10-18) → 12 DBU/h. Rates: Jobs Compute $0.15/DBU and All-Purpose $0.55/DBU (AWS, US, Premium list, as of Oct 2026: documentary source: Databricks G-Cloud 15 supplier record captured 2026-09-25, corroborated by independent reads in 2026-10; always re-check system.billing.list_prices in your own account). AWS side: m5.xlarge at $0.192/h (EC2 price list, read 2026-10) × 4 nodes.

The all-purpose version is 3.7× the DBU line, matching Databricks' own rule of thumb that pipelines belong on job compute, and it also bills for idle time you would have to manage yourself. Workers on spot cut the AWS column by up to 90% (driver stays on-demand, two-minute eviction notice); the DBU column is unchanged, because spot changes the instance price, never the emission rate.

• 📚 Did you know? The old "S3 list-after-write lag might make Auto Loader miss files" warning is history: since 2020-12-01 "all S3 GET, PUT, and LIST operations … are now strongly consistent" (AWS News Blog, 2020-12-01), confirmed in the S3 user guide (accessed 2026-10). The current miss risk is event delivery, cloud providers "do not guarantee 100% delivery of all file events", which is what cloudFiles.backfillInterval (classic mode) and the automatic ~24-hour backfill (managed file events) exist to cover. If a 2021 tutorial tells you to build an eventual-consistency workaround for S3, you can delete that paragraph.

---

10. The traps: twelve confusions this lesson will not let you keep

1. "startingOffsets=latest is fine; it is the default." On a fresh query it silently skips the backlog; resumed queries use the checkpoint anyway. Set earliest on the first run, then trust the checkpoint forever.
2. Deleting a checkpoint to fix lag. It begins fresh: Kafka replays or skips per startingOffsets, Auto Loader re-lists, and Delta replays unless txnAppId changes (new checkpoints restart at batch ID 0).
3. State in the data path or under a lifecycle rule. Schema/checkpoint inside the landing prefix → self-discovery; lifecycle expiry → "the stream state is corrupted".
4. "schemaLocation and checkpointLocation must always differ." Wrong: the docs allow them to be equal. What must differ is state vs data, and one checkpoint per workload.
5. Delta + outputMode("update"). Unsupported; use MERGE in foreachBatch. Delta gives you append and complete only.
6. Trusting foreachBatch for exactly-once. It is at-least-once: bind txnAppId/txnVersion, keep the MERGE idempotent, and never swallow exceptions locally.
7. Continuous trigger + directory listing mode. The classic cost blow-up: listing the whole tree every trigger. Switch to AvailableNow or file events.
8. Pinning maxFilesPerTrigger on DBR 18.0+. It is dynamically configured there, and manual pins fight the tuning; on older runtimes the 1,000-file default can be far too small for a multi-TB backfill.
9. maxFileAge means one thing. The file source uses it as a 7-day discovery filter; Auto Loader uses it as event-tracking state with a 14-day minimum and a 90-day recommendation, and on first run it bounds how far back you ingest.
10. cleanSource with two readers. The second reader finds nothing. Also: a MOVE destination inside the source path causes infinite re-ingestion, and cross-bucket moves error out.
11. Unshaded Kafka classes or a missing kafka. prefix. Use kafkashaded. / shadedmskiam., and when a UC service credential governs the connection, do not set kafka.security.protocol yourself.
12. "maxOffsetsPerTrigger resets on every restart." Admission controls apply to fresh queries; a resumed query reads its checkpoint. Size it once, deliberately, and watch lag rather than re-tuning after every incident.

Comparative Verdict: directory listing vs file notifications (as of Oct 2026)

• Choose managed file events (one queue per Unity Catalog external location, cloudFiles.useManagedFileEvents=true, DBR 14.3 LTS+) for most production workloads; that is Databricks' stated recommendation: "Databricks recommends file notification mode using file events for most workloads" (docs.../auto-loader/file-notification-mode, 2026-10-07). You get event-driven discovery with no per-trigger LIST (E3: $0.30/month vs $28.80 to $25,920/month on the same 2M-file prefix), one-time permissions instead of per-stream IAM, automatic full backfills at enablement and about every 24 hours, and the hard requirement to run the stream at least once every 7 days so the file-events cache does not expire.
• Stay on directory listing when the prefix is small and short-lived, when the job is a scheduled Trigger.AvailableNow run (96 listings/day ≈ $28.80/month at 2M files: a rounding error next to compute), when you cannot grant s3:GetBucketNotification / s3:PutBucketNotification plus SNS/SQS rights, or when you need to change the input path without a new checkpoint (a directory-listing-only capability). It is the default precisely because it needs nothing beyond read access.
• Classic SNS+SQS file notifications are the legacy middle: event-driven like file events, but with the published IAM action set, a resource prefix to govern (databricks-auto-ingest-*), a hard cap of 100 pipelines per S3 bucket, per-stream queues to clean up, best-effort delivery requiring cloudFiles.backfillInterval, and no input-path mobility. New designs belong on managed file events; classic mode survives in estates that predate Unity Catalog external locations.
• The trap answer: "file notifications cost more because you pay for SNS and SQS." The bill you can actually model, S3 LIST at $0.005 per 1,000, scales with listing frequency, and this lesson's sources carry no verified SQS/SNS price (so do not model one from memory). Decide on permissions, backfill behaviour, the 7-day cache rule and the 100-per-bucket cap; treat request pennies as secondary.

---

Real-World Case Studies

📋 Case A: Asurion (insurance & device protection, Cloud: AWS) Challenge. A legacy estate of 8,000 tables, 10,000 views and ~600 data marts, with per-table ETL mappings and a duplicated batch/speed layer that could not absorb new sources cheaply. Stack on AWS. Spark Structured Streaming, Auto Loader and foreachBatch into Delta tables, ingesting "database CDC feeds, APIs and flat files to streams from Kinesis, Kafka, SNS and SQS", with Databricks SQL feeding 600 data marts: a production data lake explicitly "on AWS" (https://www.databricks.com/blog/2021/09/16/large-scale-etl-and-lakehouse-implementation-at-asurion.html, 2021-09-16). Reported outcome. "The Asurion data lake executes 4,000+ streaming jobs and hosts over 4,000 tables in a production data lake on AWS" (https://www.databricks.com/blog/2021/05/10/aws-guide-to-data-ai-summit-featuring-disney-comcast-capital-one-and-mcdonalds.html, 2021-05-10). Architecturally: "an unified approach to data movement from traditional ETL to streaming data pipelines between Delta tables" (2021-09-16). Maps to this lesson. Sections 1, 6 and 7: at 4,000+ streaming jobs the architecture is a parameterised ingestion framework (one Auto Loader / foreachBatch pattern, many feeds), and the AWS-native source list (Kinesis, SNS, SQS) is exactly why the substrate choices in sections 4 and 6 matter. ⏳ Temporal note: 2021 figures reported at the time; re-check any workspace concurrency statement in that blog against current docs before reusing it.

📋 Case B: Burberry (luxury retail, Cloud: AWS) Challenge. Daily clickstream batches arriving "anywhere between 2:00 and 7:00 PM GMT", and a cloud warehouse that pre-aggregated away the detail needed for Customer 360. Stack on AWS. Databricks Data Intelligence Platform with Snowplow Behavioral Data Platform for real-time Customer 360: the story page carries Databricks' own Cloud: AWS tag (https://www.databricks.com/customers/burberry, undated page, accessed 2026-10). Reported outcome. "99% Reduction in latency for customer clickstream data · 52x Increase in cookie duration"; "we've broken free from daily data batches and formed an accurate real-time view of our customers' digital activity" (same URL, accessed 2026-10; story origin 2023). Maps to this lesson. Sections 4 and 9: a latency SLA on event files is a business requirement, met by event-driven discovery plus a low-latency trigger rather than by a bigger cluster. File-arrival cadence, not compute, was the bottleneck. ⏳ Temporal note: a 2023-origin figure still displayed in 2026: quote it as what Burberry reported at the time.

---

2026 Updates (as of October 2026)

---

---

Practice Questions