Databricks Runtime, Photon and Clusters: DBR/LTS, Policies, Pools and Compute Cost Shape

Every pipeline in this course runs on something. Delta tables do not optimise themselves, Auto Loader does not list a prefix for free, and a Lakeflow pipeline is only as cheap as the machines underneath it. This lesson is about that something: which Databricks Runtime version you pin, whether Photon is executing your plan, which kind of cluster (or warehouse, or serverless session) the work runs on, who is allowed to create it (cluster policies), how fast it can start (instance pools), which EC2 family it uses, and, the part that shows up on the invoice, which DBU rate applies to it.

That last point is the spine of the lesson. On AWS, the same EC2 instance bills at a different DBU rate depending on the compute type you attached it to. The Jobs API 2.0 states it without hedging: "When you run a job on a new jobs cluster, the job is treated as a Jobs Compute (automated) workload subject to Jobs Compute pricing" and "When you run a job on an existing all-purpose cluster, it is treated as an All-Purpose Compute (interactive) workload subject to All-Purpose Compute pricing" (docs.../jobs-2.0-api, page dated 2026-08-03). Same nodes, same Spark, same data, and at US list prices as of Oct 2026 a $0.15/DBU SKU versus a $0.55/DBU SKU. Everything else in this lesson is either a performance decision or a governance decision; this one is pure arithmetic, and it is the reason a 45-minute nightly ETL can cost either $1,027 or$2,947 a month (worked example E5).

In this lesson you will:

• read the supported DBR table as of Oct 2026 with every EoS date, and drive the Beta → GA → LTS → EoS → EoL lifecycle as a diagram;
• trace a query through Catalyst and Photon, name the four cases where Photon does not help, and date-stamp every benchmark you quote;
• separate all-purpose, job, SQL warehouse and serverless compute, including the pricing rule that decides the bill;
• work through six worked examples: an annotated cluster-policy JSON, an instance-pool configuration, instance-family selection for four workload shapes, the job-vs-all-purpose cost shape, a Photon-or-not triage, and a production notebook posture;
• apply cluster policies (fixed/allowlist/regex/range, dbus_per_hour, cluster_type, auto:latest-lts) and close the Unrestricted backdoor;
• set autoscaling windows, auto-termination, spot topology and first_on_demand without tripping the documented traps;
• study Comcast and Addepar on AWS, read the 2026 update box, and practise with twelve questions plus four interactive checks.

---

1. Databricks Runtime: the version ladder and the LTS clock

1.1 The supported set (release-notes page last updated Sep 11, 2026)

Databricks ships two lines at once: a current line that keeps receiving features, and an LTS line that trades novelty for a three-year support window. The supported-versions table on AWS (docs .../release-notes/runtime, last updated Sep 11, 2026, accessed 2026-10) is the only version list you should trust:

Each version also ships an ML variant (same number, MLflow plus ML libraries pinned to it): the MLflow matrix reads 19 → MLflow 3.12.0, 18 LTS → 3.8.1, 17.3 LTS → 3.0.1, 16.4 LTS → 2.21.3 (same page, accessed 2026-10).

Two 2026 wrinkles matter more than the numbers themselves:

1. Numbering changed at DBR 19. "Prior to Databricks Runtime 19, Databricks released runtimes as minor feature versions (for example, 18.0, 18.1, 18.2), each with its own Beta and GA release" (docs .../release-notes/runtime/databricks-runtime-ver, accessed 2026-10). From 19 the feature-development phase runs as a single major line.
2. A retired minor and a live LTS coexist. The DBR 18.0 release-note page is already labelled (EoS), "Support for this Databricks Runtime version has ended", even though that page also records "This version incorporates all features … Databricks released this version in January 2026", Spark 4.1.0 and default JDK 21 (docs.../release-notes/runtime/18.0, accessed 2026-10). Meanwhile 18 LTS is supported through Jun 10, 2029. Pin to the LTS line, never to a retired minor number.

1.2 The lifecycle, drawn

The four durations come straight from the support-lifecycle page (accessed 2026-10): a ~six-month GA feature phase, LTS backports for three years, "The end-of-support date is three years after the GA release date", and "Databricks makes a best effort to set the end-of-life date six months after the end-of-support date". At EoS the version "is no longer selectable using the UI when you create or update a compute resource"; at EoL it "becomes unusable … existing workloads running on these versions fail". That is the difference between a job that stops scheduling and a job that crashes, and it is why pinning is a production control, not a preference.

1.3 Which line for which workload

The compute-configuration page is explicit (docs .../compute/configure, last updated Oct 6, 2026): "For all-purpose compute, use the most current version … For job compute running operational workloads, consider using the Long Term Support (LTS) Databricks Runtime version." Translated:

• Interactive/exploratory (all-purpose): newest supported version; you want the features and you restart often anyway.

• Operational jobs: an LTS line, pinned by exact string (17.3.x-scala2.12), because a surprise Spark minor in the middle of a quarter is an outage.

• Data science / ML: the ML variant of whichever line you chose.

• In a cluster policy: spark_version with a special token, auto:latest-lts, auto:latest-ml, auto:prev-major, auto:prev-lts, which "does not make the compute auto-update when a new runtime version is released"; it resolves when a user edits the compute (docs.../admin/clusters/policy-definition, last updated Oct 1, 2026).

• 📚 Did you know? Databricks retains "cluster configuration information for terminated clusters for 30 days" (Clusters API intro, accessed 2026-10). That is long enough to answer "what runtime and node type did that failing job use last Tuesday?" and short enough that you should export the shapes you care about (via asset bundles or Terraform) rather than archaeology-in-the-UI.

---

2. Photon: what the vectorized engine does, and what it does not

2.1 One planner, two execution engines

"Photon is the Databricks-native vectorized query engine that accelerates your SQL workloads, DataFrame API calls, ETL pipelines, and stateless streaming workloads" (docs.../compute/photon, last updated Sep 11, 2026). The architecture sentence to memorise: "The Apache Spark query optimizer (Catalyst) still plans your query, but Photon takes over at the execution layer", and "When Photon encounters an unsupported operation … it transparently falls back to the Spark runtime for the remainder of that operation."

Photon processes "data in batches of thousands of rows at a time, enabling modern CPUs to use SIMD instructions", and features such as predictive I/O and dynamic file pruning in MERGE/UPDATE/DELETE require Photon (same page). It is "the default engine for all SQL warehouses", is enabled on serverless compute and serverless Lakeflow pipelines, and, since DBR 9.1 LTS, "Photon is enabled by default on compute running Databricks Runtime 9.1 LTS and above" for classic all-purpose and job compute created through the UI.

2.2 Where Photon does not help

The docs are unusually blunt, and every clause below is quotable:

There is also an economics clause: Photon is not free speed. "Photon instance types consume DBUs at a different rate than the same instance type running the non-Photon runtime" (Databricks blog, "Announcing Photon Public Preview", 2021-06-17). The operative rule: Photon pays only when it is more than proportional in speed, so benchmark your workload, and measure Photon share in the Spark UI rather than guessing.

2.3 The evidence, date-stamped

Quote these with their dates; they are not interchangeable:

• 📚 Did you know? The audited 100 TB TPC-DS record ran on this course's substrate: data in Amazon S3, 256 × i3.2xlarge (SIGMOD '22). Storage-optimized instances plus a vectorized engine is the reference "lakehouse beats warehouse" datapoint, and it is the same instance family that classic and pro SQL warehouses still standardise on today (all classic/pro warehouse workers are i3.2xlarge, docs.../warehouse-behavior, accessed 2026-10).

2.4 Worked example E1: Photon or not? triage two notebooks

As of Oct 2026. Two teams claim "Photon didn't help". Triage before you change anything:

---

3. Cluster anatomy: all-purpose vs job vs warehouse vs serverless

3.1 What runs where, on AWS

"Databricks components" (docs, page dated 2026-07-28) draws the line simply: "There are two types of clusters: all-purpose and job." Add SQL warehouses and serverless sessions and you have the four compute kinds this course uses.

Three AWS-specific facts from that picture: classic clusters and pro/classic warehouses run on EC2 in your VPC ("With a pro SQL warehouse (unlike a serverless SQL warehouse), the compute layer exists in your AWS account", docs .../warehouse-types, last updated Feb 12, 2026); serverless runs in the Databricks account with no public IPs; and with secure cluster connectivity on (the default deployment mode), nothing in the data plane needs an inbound public IP.

3.2 The lifecycle difference, and the price difference

Docs quote (components page, 2026-07-28): "The Databricks job scheduler creates a job cluster when you run a job on a new job cluster and terminates the cluster when the job is complete." And the pricing consequence, from the Jobs API 2.0 (page dated 2026-08-03): "When you run a job on a new jobs cluster, the job is treated as a Jobs Compute (automated) workload subject to Jobs Compute pricing"; while pointing a job at an existing all-purpose cluster is All-Purpose pricing, and per the components page "running a job on an existing all-purpose cluster is also treated as an interactive workload." The same API adds: "We suggest running jobs on new clusters for greater reliability."

Databricks' jobs-compute guidance goes further (docs.../jobs/compute, last updated Sep 15, 2026): "Serverless compute for jobs is the default for all tasks that support it", and "All-purpose compute; You can optionally configure tasks using classic all-purpose compute. Databricks does not recommend this configuration for production jobs."

One nuance worth carrying into lesson 09: a shared job cluster is "scoped to a single job run and cannot be used by other jobs or runs of the same job. Libraries cannot be declared in a shared job cluster configuration." Declare dependencies per task (or via job environments), not on the shared cluster object.

3.3 Worked example E2, the cost shape: one pipeline, three configurations

As of Oct 2026, US list, Premium tier, AWS. Scenario: a nightly 45-minute ETL that runs 400 cluster-hours per month, 4 nodes total (1 driver + 3 workers). Databricks' own worked example uses a node that "runs at 3 DBU/hour" (blog, Best practices for cost management in Databricks, 2022-10-18): treat that emission value as illustrative: per-node DBU emission is not published as an official table, so read yours from system.billing.list_prices or the pricing calculator.

Configuration (B) also carries idle hours: an all-purpose cluster bills while it is up, not while it works, which is exactly why auto-termination (section 8) and policies (section 6) exist. Notice what did not change between (A) and (B): nodes, runtime, Spark, data. Only the compute type, and with it the SKU. That is the single highest-leverage decision in this lesson, and it is why Databricks reports customers "saved tens of thousands of dollars by simply moving just ten percent of their workloads from all-purpose clusters to jobs clusters" (blog, 2022-10-18).

---

4. Serverless compute and SQL warehouses

4.1 The recommended-compute matrix

Databricks publishes a per-task recommendation table (docs .../jobs/compute, last updated Sep 15, 2026). Read it as "the default, unless your dependency list forbids it":

Serverless eligibility on AWS (docs.../sql/admin/serverless, last updated Jun 10, 2026): workspace on Premium plan or above, not a free trial, supported in all AWS regions, no public IPs, and "Cluster policies, including spot instance policies, are not supported". That last clause is the boundary of everything in section 6: policies govern classic compute; serverless is governed by account/workspace-level controls instead.

4.2 Serverless limitations: read this list before you migrate a job

From docs .../compute/serverless/limitations (accessed 2026-10), the blockers that force classic compute:

• "R is not supported." · "Scala and R are not supported in notebooks."
• "Only Spark Connect APIs are supported. Spark RDD APIs are not supported."
• "Maven coordinates are not supported." · "JAR libraries are not supported in notebooks … JAR tasks in jobs are supported." · "Task libraries are not supported for notebook tasks."
• "You must use Unity Catalog to connect to external data sources." · "Access to DBFS is limited."
• "Serverless compute has a maximum runtime of 7 days."
• Streaming triggers: Trigger.AvailableNow() (recommended) and deprecated Trigger.Once() only: Trigger.ProcessingTime and Trigger.Continuous are not supported.

The migration page (last updated 2026-07-10) lists the same blockers as things to replace first: custom images, ML Runtime variants, DBR < 13, instance profiles, and "dependencies such as eggs, JARs, and Maven libraries", with replacement patterns: RDD APIs → DataFrame APIs, DBFS paths → UC volumes, IAM instance profiles → UC external locations, custom JDBC JARs → Lakehouse Federation.

4.3 SQL warehouses: three types, two clocks

(docs .../sql-warehouse/warehouse-types, last updated Feb 12, 2026; auto-stop from docs .../sql-warehouse/create, accessed 2026-10)

Sizing for classic/pro is one cluster per 10 concurrent queries, from 2X-Small (1 driver + 1 worker) through X-Small, Small … X-Large (32 workers), 2X (64), 3X (128), 4X (256), with 5X-Large (512) in Public Preview (docs.../warehouse-behavior, accessed 2026-10). Databricks' standing guidance: "Databricks recommends using serverless SQL warehouses when available", and the billing sentence to memorise: "Idle SQL warehouses continue to accumulate DBU and cloud instance charges until they are stopped."

Comparative Verdict: job cluster vs all-purpose vs SQL warehouse vs serverless (as of Oct 2026)

• Job cluster (classic, Jobs Compute $0.15/DBU + your EC2): the default answer for production batch. Ephemeral per run, no restart, auto-terminates with the run, spot-capable, policy-governed, cheapest per-DBU classic SKU. Pay for hours it actually runs; you own EC2, EBS and warm-up time. Use when dependencies are classic-only (JAR/Spark Submit, RDD, instance profiles, DBFS) or when the job runs long and hot on a predictable shape.
• All-purpose (All-Purpose Compute $0.55/DBU + your EC2): interactive only: exploration, shared notebooks, teaching. 3.7× the per-DBU rate of jobs at US list, it bills while up, and pointing production jobs at it is both an interactive-billing and a reliability anti-pattern ("Databricks does not recommend this configuration for production jobs"). Mandatory controls: auto-termination, a policy, never a spot driver.
• SQL warehouse (SQL Classic $0.22 / Pro $0.55 / Serverless $0.70 per DBU), the BI serving path, billed while up, sized by T-shirt not by workers you choose. Serverless adds Predictive IO + IWM and a 10-minute auto-stop; pro/classic run in your VPC on i3.2xlarge with a 45-minute auto-stop and ~4-minute start. Choose by query concurrency and latency SLA, not by DBU rate alone, the classic/serverless gap ($0.48 per DBU-hour) is the machine+ops serverless absorbs for you.
• Serverless jobs/notebooks (Jobs Serverless $0.35/DBU, All-Purpose Serverless $0.75/DBU): the platform default wherever the limitations list permits: no EC2 bill, no pools, no warm-up, no policy surface, 7-day max runtime, per-second billing, zero idle cost. Costs roughly 1.3 to 2.3× more per DBU than classic but deletes the idle-time and admin columns of the bill.
• Verdict in one line: serverless by default → classic job clusters when a blocker or a long hot shape demands it → all-purpose for humans, never for pipelines → SQL warehouse for BI with auto-stop enforced. Every rate above is US list, Premium tier, AWS, as of Oct 2026; confirm against system.billing.list_prices in your account before you commit a budget to it.

---

5. Instance pools: warm start, not free compute

A pool is "A set of idle, ready-to-use instances that reduce cluster start and auto-scaling times" (docs .../getting-started/concepts, 2026-07-28). Attached clusters take their driver and workers from the pool and give them back on termination. Three sizing knobs (docs .../compute/pools, accessed 2026-10): min_idle_instances, max_capacity, idle_instance_autotermination_minutes.

The billing rule is the one students get wrong: "Databricks does not charge DBUs while instances are idle in the pool. Instance provider billing does apply" (docs .../pool-best-practices / pool-index, accessed 2026-10). Pools shift cost from the DBU column to the AWS column and buy you start-up minutes; they do not delete the bill. Constraints and rules from the same pages:

• A pool is "either all spot instances or all on-demand instances"; the Instance Pools API adds SPOT_WITH_FALLBACK: "the pool acquires spot instances first, and falls back to on-demand instances when spot capacity is unavailable".
• "By default, Databricks sets the max spot price at 100% of the on-demand price."
• Over-scale fails loudly: "If a cluster using the pool requests more instances than this number during autoscaling, the request fails with an INSTANCE_POOL_MAX_CAPACITY_FAILURE error." Keep max cluster size ≤ pool max_capacity.
• Clusters inherit the pool's availability zone; use separate pools when driver and worker requirements differ.
• Never spot the driver: "Do not use a pool with spot instances as your driver type. Select an on-demand driver type to prevent your driver from being reclaimed."
• Where serverless fits, Databricks' advice is blunt: "If your workload supports serverless compute, Databricks recommends using serverless compute instead of pools."
• Sizing guidance: spot pools for cost-tolerant workers; on-demand pools for "jobs with short execution times and strict execution time requirements".

5.1 Worked example E3: pool configuration + the idle bill

As of Oct 2026 (us-east-1 list, read 2026-10-12). A team wants warm starts for a 45-minute nightly job and for interactive dev clusters.

• 📚 Did you know? The pool's idle insurance costs about $136/month in this example; while one accidental day of that 4-node all-purpose cluster left up (12 DBU/h × 24 h × $0.55 = $158.40 of DBUs plus 4 × $0.192 × 24 ≈ $18.43 of EC2 ≈ $177) already costs more than a whole month of warm starts. Pools are for start latency and capacity, auto-termination is for cost; teams that install pools but skip auto-termination usually end up with both bills.

---

6. Cluster policies: governance and cost guardrails as JSON

"A policy is a tool workspace admins can use to limit a user or group's compute creation permissions based on a set of policy rules" (docs.../admin/clusters/policies, page dated 2026-06-10). The documented benefits: prescribed settings, limits on how many clusters a user may create, a simplified UI, "Control cost by limiting per-cluster maximum cost", and "Enforce cluster-scoped library installations". Databricks' own policy-onboarding blog (2025-04-17) names the two payoffs, "Simplification of platform usage and effective cost control are the two main benefits of cluster policies", and the four-family starter set: Personal Compute, Power User Compute, Shared Compute, Job Compute.

Mechanics you must know before you write JSON (docs .../admin/clusters/policy-definition, last updated Oct 1, 2026):

• Rules attach to "any of the attributes controlled with the Clusters API", and "There can only be one limitation per attribute."
• Limitation types: fixed, forbidden, allowlist, blocklist, regex, range, unlimited (each may carry defaultValue, isOptional, hidden).
• Virtual attributes: dbus_per_hour, "Calculated attribute representing the maximum DBUs a resource can use on an hourly basis including the driver node … Use with range", and cluster_type (all-purpose / job / dlt), which lets you block a compute class outright.
• runtime_engine accepts PHOTON or STANDARD; spark_version accepts the auto:* tokens (section 1); workload_type.clients.jobs / .notebooks gate which client type may attach.
• API trap: "Default values don't automatically get applied to compute created with the Clusters API. To apply default values using the API, add the parameter apply_policy_default_values … set it to true."
• Deleting a policy does not clean up: running compute continues but "cannot be edited unless the user has unrestricted cluster creation permissions."

6.1 Worked example E4: a job-only cost-guardrail policy

Built on the docs' own job-only example (policy-definition page, Oct 1, 2026) and extended with Photon, spot topology and tags. This is the JSON you would paste into a Terraform databricks_cluster_policy resource or the admin UI:

---

7. Libraries and notebook posture

7.1 Four scopes, one source matrix

(docs .../libraries, last updated Sep 11, 2026)

Sources: PyPI, Maven, CRAN, workspace files, Unity Catalog volumes, cloud object storage. The cut lines as of Oct 2026:

• Standard access mode needs DBR 13.3 LTS+ for PyPI, Maven, volumes and workspace files; "CRAN: Not supported" on standard access mode (dedicated/legacy keep it).
• Python eggs only on "Databricks Runtime 13.3 LTS and below"; requirements.txt needs DBR 15.0+.
• "storing library files in the DBFS root is deprecated and disabled by default in Databricks Runtime 15.1 and above": use workspace files, UC volumes or package repos.
• "Workspace libraries have been deprecated and should not be used": note that workspace files remain fully supported.
• Python import precedence (highest first): Git-folder cwd → Git-folder root → notebook-scoped (%pip) → compute-scoped → libraries included in DBR.

Notebook ergonomics worth memorising: "%pip, Install Python packages (notebook-scoped)" but "%pip commands do not automatically restart the Python process … you may need dbutils.library.restartPython()"; "%run, Execute another notebook, importing its functions and variables" and "%run must be in a cell by itself, because it runs the entire notebook inline." On serverless, %uv pip is the faster alternative (environment version 5+).

7.2 Worked example E5: the production notebook posture

• 📚 Did you know? Databricks' engineering guidance now reads: "Declarative automation is built on the idea that if something isn't in version control, it doesn't exist" and "Minimize business logic in notebooks … Use notebooks only as thin orchestration and visualization layers that call the underlying code" (docs .../notebooks/best-practices and .../developers/best-practices, both last updated 2026-09-11). The companion repo is github.com/databricks/notebook-best-practices.

---

8. Autoscaling, auto-termination and spot

8.1 Horizontal on classic, vertical on the serverless side

Classic all-purpose and job clusters autoscale workers only, between autoscale.min_workers and max_workers. Optimized autoscaling (Premium and above; Standard-plan workspaces get standard autoscaling) has four documented characteristics (docs .../compute/configure, last updated Oct 6, 2026):

• "Scales up from min to max in at most 2 scaling events."
• "On job compute, scales down if the compute resource is underutilized over the last 40 seconds. On all-purpose compute … 150 seconds." (spark.databricks.aggressiveWindowDownS, up to 600 s, slows that decay.)
• It can scale down non-idle nodes using shuffle-file state, so a warm-but-waiting node is not protected.
• Standard autoscaling instead starts at "+8 nodes, then exponential" and "scales down when 90% of the nodes are not busy for 10 minutes".

Vertical scaling does not exist on classic: "Scale vertically by adding or removing resources from a single machine … This typically means stopping the workload, moving it to a larger machine, and restarting it" (docs.../performance-efficiency, accessed 2026-10). Vertical instance selection is a serverless pipelines feature: "Vertical autoscaling is used for all serverless pipelines", picking cost-efficient instance types on OOM evidence (docs.../ldp/auto-scaling, accessed 2026-10).

Two documented traps: "Do not enable Apache Spark Dynamic Allocation (spark.dynamicAllocation.enabled) on compute resources that use Databricks autoscaling" (conflicting executors → NODES_LOST), and autoscaling "has limitations scaling down … for Structured Streaming workloads": long-lived streams want a floor, not a floor that moves.

8.2 Auto-termination and spot topology

Auto-termination kills an idle all-purpose cluster after autotermination_minutes of inactivity. The API accepts 10 to 10,000 minutes, with 0 disabling it (Databricks asset-bundle/job schema docs, accessed 2026-10), and the docs pages read in October 2026 print no numeric default, so set it explicitly rather than trusting a UI value you have not checked. SQL warehouse auto-stop defaults are documented (section 4.3): 45 min pro/classic, 10 min serverless.

Spot, from the same compute-configuration page: aws_attributes.availability ∈ SPOT / ON_DEMAND / SPOT_WITH_FALLBACK; aws_attributes.first_on_demand pins the driver (and N nodes) to on-demand; aws_attributes.spot_bid_price_percent defaults to 100% of on-demand. "The first instance will always be on-demand (the driver node is always on-demand) and subsequent instances will be spot instances." AWS prices Spot at "up to 90% off On-Demand prices" (aws.amazon.com/ec2/spot, accessed 2026-10) with a two-minute interruption notice, and the saving lands only on the AWS column: DBU rates and emission counts do not change with spot.

---

9. AWS node types: families, Graviton, fleet and disks

9.1 The families

From the EC2 instance-types documentation (accessed 2026-10): M general purpose, C compute-optimized, R memory-optimized, and D/I storage-optimized with local NVMe: "Storage optimized instances are designed for workloads that require high, sequential read and write access to very large data sets on local storage" (family classes span C5…C9g, R5…R9g, D3/D3en/I3en/I4i/I7i/I8g/Im4gn/Is4gen).

Databricks' own selection guidance (docs .../cluster-config-best-practices, page dated 2026-06-11): simple batch ETL that "don't require wide transformations" → "instances with lower requirements for memory and storage"; analytical workloads "will likely require reading the same data repeatedly, so recommended node types are storage optimized with disk cache enabled or instances with local storage"; and "For simple batch ETL jobs … use spot instances for workloads that have lax latency requirements."

9.2 Worked example E6: instance-family selection for four workload shapes

As of Oct 2026; EC2 Linux on-demand, us-east-1, read from the AWS price list 2026-10-12.

Two more hardware facts worth carrying: fleet instance types (m-fleet.xlarge, rd-fleet.4xlarge, …) are "a variable instance type that automatically resolves to the best available instance type of the same size" using AWS's Spot Placement Score, and "If you update the spot instance bid percentage using the API or JSON, it has no effect when the worker node type is a fleet instance type." Fleet types carry no GPUs. EC2 P2 is no longer supported (same page). For guaranteed GPU/instance windows, Capacity Blocks attach via the tag X-Databricks-AwsCapacityBlockId plus an AZ and on-demand-only.

---

10. The meter: same instance, different bill

10.1 The rate table you will be examined on

US list, Premium tier, AWS, as of Oct 2026: documentary sources (Databricks' own pricing pages render client-side; these figures come from the UK G-Cloud 15 supplier record captured 2026-09-25 plus six independent reads of the same rate card dated between 2026-06-03 and 2026-10-06, and are corroborated for SQL by databricks.com page snapshots). Confirm in your account with system.billing.list_prices before budgeting:

Three rules around that table:

1. Region: classic rates are flat across AWS regions (you pay local EC2 anyway); serverless rates are not: SQL Serverless is $0.70/DBU in us-east-1 vs $0.91 in eu-central-1 (page-data read, 2026-08-07). A US quote never transfers to Frankfurt.
2. The Standard tier is gone. "Databricks has announced end-of-life for the Standard tier. All current Standard tier workspaces will be upgraded to the Premium tier effective October 1, 2025 on AWS and GCP", so the old $0.10 jobs rate is history (databricks.com/product/pricing/lakeflow-jobs, accessed 2026-10).
3. Two bills on classic, one on serverless. Classic = DBUs + EC2/EBS in your account; serverless folds the machine into the DBU. system.billing.usage contains only the Databricks side: "The system.billing.usage only contains Databricks costs … cloud infrastructure spend is not included and would need to be reviewed via the relevant cloud console" (blog, 2026-09-22).

10.2 Attribution: give every DBU an address

system.billing.usage (page dated 2026-07-23) carries sku, usage_type, product_feature, workspace_name, region, tags, run_as, metered_quantity, list_price and unblended_cost, "When a classic compute resource's driver and worker nodes use different instance types, their usage appears as separate records". Group by sku + run_as and you can see exactly how much of your estate sits on All-Purpose versus Jobs. One boundary to remember when you build the dashboard: job cost-attribution queries "only calculate costs for jobs run on jobs compute and serverless compute. Jobs run on SQL warehouses and all-purpose compute are not billed as jobs" (docs.../system-tables/jobs-cost, accessed 2026-10). And budgets (up to 1,000 per account, 4 thresholds each, USD at list price) monitor spend, they do not cap it (budgets doc, 2026-10-07).

---

11. The traps: fourteen confusions this lesson will not let you keep

1. "LTS means the number with .x." As of Oct 2026 18 LTS is supported (to 2029-06-10) while 18.0 is EoS. Pin spark_version to the LTS line.
2. "auto:latest-lts keeps my clusters current." It resolves when compute is edited, not continuously: running clusters do not upgrade themselves.
3. "Our API-created clusters are on Photon." Not unless runtime_engine is "PHOTON": the UI default does not travel to the Clusters/Jobs API.
4. "Photon will fix my 100 ms dashboard." "Queries that normally complete in under two seconds don't see meaningful improvement."
5. "Photon handles everything Spark handles." No UDFs, no RDD/Dataset APIs, no stateful streaming; fallback is per-operation and silent.
6. "A job on an existing all-purpose cluster is the cheap option." It is billed at All-Purpose (interactive) pricing and flagged interactive: 3.7× the per-DBU rate at US list.
7. "Pools are free." 0 DBUs while idle, but AWS instance charges continue; set min_idle_instances deliberately.
8. "Spot workers, spot driver." Never: driver always on-demand (first_on_demand >= 1), and spot only discounts the AWS column.
9. "Dynamic Allocation + Databricks autoscaling = more safety." Explicitly warned against: executor churn and NODES_LOST.
10. "Bigger memory = bigger node, no restart." Classic clusters have no vertical autoscaling; changing node type means a restart.
11. "Policy defaults apply automatically." Not via the Clusters API; you must pass apply_policy_default_values: true.
12. "We deleted the policy, so the spend stopped." Existing compute keeps running and becomes un-editable (except for unrestricted users).
13. "%pip install in the notebook means the job has the package." Notebook-scoped libraries "do not persist": declare deps at task/cluster/environment level.
14. "The DBU rate is my total cost." On classic it is half the bill; the EC2/EBS/NAT/S3 half lives only in AWS Cost Explorer.

2026 Updates (as of October 2026)

---

Real-World Case Studies

📋 Case A: Comcast (communications & media, Cloud: AWS) Challenge. Petabytes of telemetry and billions of voice interactions from 20M+ voice remotes; fragile pipelines and small-file ingestion pain, with hundreds of models deployed to cloud, on-prem and edge. Stack on AWS. Databricks with Delta Lake for ingestion and small-file optimisation, managed MLflow, and autoscaling with spot instances: Cloud: AWS on the story page (https://www.databricks.com/customers/comcast, undated page, accessed 2026-10); also listed among customers "already leveraging Databricks on AWS" (https://www.databricks.com/company/newsroom/press-releases/databricks-strengthens-scope-of-work-with-aws, 2022-05-10). Reported outcome. "10x Reduction in overall compute costs to process data · 90% Reduction in required DevOps resources"; "Delta Lake has enabled Comcast to optimize data ingestion, replacing 640 machines with 64 while improving performance" (https://www.databricks.com/customers/comcast, undated, accessed 2026-10; story origin 2019). Maps to this lesson. Sections 8 to 10: spot workers plus horizontal autoscaling and structural right-sizing are exactly the levers in worked example E2: the saving came from which compute shape ran the work, not from a discount. ⏳ Temporal note: a 2019-origin story still displayed in 2026: teach it as what Comcast reported at the time, not as a guaranteed 10×.

📋 Case B: Addepar (wealth management, Cloud: AWS) Challenge. Engineering and data teams siloed, slow pipeline delivery and rising infrastructure cost on a platform modelling over $8 trillion of investments. Stack on AWS. Databricks running on AWS with Unity Catalog, shared notebooks and Databricks SQL: Cloud: AWS tag on the story page. Reported outcome. "Addepar models over $8 trillion in investments on Databricks running on AWS. Shared notebooks, Unity Catalog and collaborative data access bring engineering and data teams together in one place, accelerating pipeline delivery by 5x and cutting infrastructure costs by $2 million" (https://www.databricks.com/customers/addepar/aws, undated page, accessed 2026-10). Maps to this lesson. Sections 6 and 10: a $2M infrastructure reduction with 5× delivery speed is what "governance and compute standardisation" look like on an invoice: shared, policy-governed compute beats one-cluster-per-notebook sprawl, which is the personal-compute policy problem from section 6 solved at company scale. ⏳ Temporal note: undated page, accessed Oct 2026; the $2M is infrastructure cost, not DBU cost.

---

---

Practice Questions