Databricks SQL Warehouses and the Serving Layer: Serverless, Scaling, Dashboards, BI and Federation

Everything upstream in this course ends the same way: a Delta table, governed by Unity Catalog, that somebody has to read. Analysts read it in a SQL editor, executives read it in a dashboard, Tableau reads it over ODBC, and a reporting pipeline reads it through a federated catalog. On Databricks all of that reading happens on one piece of compute, the SQL warehouse, and the choice you make about which warehouse type, which size, which auto-stop window and which BI driver is simultaneously a latency decision, a governance decision and an invoice.

This lesson is that choice, worked end to end. The spine is a single distinction that the October 2026 docs state without hedging: with a pro or classic SQL warehouse the compute layer exists in your AWS account; with a serverless SQL warehouse it exists in the Databricks account (docs.../compute/sql-warehouse/warehouse-types, last updated Sep 11, 2026). Everything else, 2-to-6-second cold starts versus roughly four minutes, cluster policies that stop applying, a customer-managed VPC that stops mattering, the $0.70 "includes cloud instance cost" footnote, falls out of that one line.

In this lesson you will:

• read the Oct 2026 warehouse lineup, three GA types plus the Lakehouse//RT Beta, and the capability matrix that separates them (Photon, Predictive IO, Intelligent Workload Management);
• draw where the compute lives on AWS (Databricks account vs your VPC), including the serverless egress and outbound-PrivateLink story;
• work through seven worked examples: a warehouse-type decision table with the API incantation, concurrency sizing math, an auto-stop waste calculation, a monitoring-query pack, a BI connection card for Tableau and Power BI, a federated-query DDL sequence, and a two-column DBU + EC2 monthly cost sketch at Oct 2026 rates;
• size a warehouse from symptom to knob using the query profile and performance-insight codes;
• build AI/BI dashboards, saved queries and alerts and know exactly what happened to legacy dashboards;
• wire Tableau, Power BI, Excel and Partner Connect tools with the current driver names and minimums;
• sketch Lakehouse Federation with foreign catalogs, and name the requirement that silently excludes classic warehouses;
• study NAB, Punchh and nCino on AWS, read the 2026 update box, and practise with twelve questions plus four interactive checks.

---

1. What Databricks SQL is, and the two gates every query passes

1.1 One interface, many surfaces

Databricks SQL is "a cloud data warehouse built on lakehouse architecture. It runs directly on your data lake" with ANSI SQL and Delta Lake extensions (docs.../sql/, last updated Sep 11, 2026). The documented surfaces: SQL editor (AI-assisted authoring, code comments, version history), notebooks attached to a warehouse, jobs, dashboards, metric views, alerts, ETL (streaming tables and materialised views defined straight from SQL) and a REST API: plus the monitors: query history, query profile and query performance insights.

The compute rule worth memorising: "Databricks uses SQL warehouses for compute in the following interfaces: SQL editor · Databricks SQL queries · Dashboards · SQL alerts" and, optionally, notebooks, the file editor and Lakeflow Jobs (docs .../query, last updated Sep 11, 2026). The same page adds two constraints that shape everything downstream:

• "When you query data with SQL warehouses, you can use only SQL syntax."
• "For workspaces that are enabled for Unity Catalog, SQL warehouses always use Unity Catalog to manage access to data sources."

And from the warehouse-types page: "SQL warehouses do not support credential passthrough. Databricks recommends using Unity Catalog for data governance." There is no per-user-credential path on a warehouse: governance is Unity Catalog grants, full stop.

1.2 The two gates

Every query through a warehouse passes two independent checks, and students routinely conflate them:

1. The warehouse ACL, can this principal use this compute? (CAN USE to run, CAN MONITOR to see profiles, CAN MANAGE to reconfigure, section 9.)
2. The Unity Catalog grant: can this principal read this table? (SELECT on catalog/schema/table.)

Passing one says nothing about the other. A user with CAN USE on the warehouse and no SELECT on the table gets an authorization error; a user with SELECT and no CAN USE cannot even reach the engine. Teach both, every time.

That diagram is also the mental model for the rest of the lesson: sections 2 to 4 decide where the compute is and how much of it there is, sections 5 to 6 decide what the user sees, sections 7 to 8 decide which tools can talk to it, section 9 decides who may, and section 10 decides what it costs.

---

2. The warehouse lineup as of Oct 2026: serverless, pro, classic, and one Beta

2.1 Three GA types, one Beta

"Databricks SQL supports serverless, pro, and classic types" and "Additionally, Lakehouse Real-Time (Beta) is a specialized serverless warehouse type for sub-second, high-concurrency read queries" that "supports (SELECT) queries only" (docs .../warehouse-types, last updated Sep 11, 2026). The capability matrix from the same page:

Three consequences follow from the middle row alone:

1. Pro and classic inherit your AWS network. They run as EC2 in the workspace VPC, pick an availability zone through Auto-AZ, and are governed by cluster policies, spot settings and your VPC/PrivateLink design (Lesson 02's workspace network).
2. Serverless does not. "Customer-managed VPCs are not applicable to compute resources for serverless SQL warehouses", "Cluster policies, including spot instance policies, are not supported", and "serverless SQL warehouses do not have public IP addresses" (docs .../sql/admin/serverless, last updated Jun 10, 2026).
3. Classic is the API default while being the least capable type. Using the UI "the default SQL warehouse type is serverless"; using the SQL warehouses API with defaults "the default SQL warehouse type is classic. To use serverless, set the enable_serverless_compute parameter to true and warehouse_type to pro." Terraform and scripts that omit that flag silently create the one warehouse type that cannot federate, cannot create materialised views or streaming tables and cannot call AI Functions.

2.2 Serverless prerequisites on AWS (read before you promise it)

From docs .../sql/admin/serverless (last updated Jun 10, 2026): the account "must not be on a free trial"; the workspace "must be on the Premium plan or above"; "Serverless SQL warehouses are supported in all AWS regions"; the workspace "must not use S3 access policies" and must not use "an external Hive legacy metastore" (warehouse-types repeats the metastore clause: "If your workspace uses a legacy external Hive metastore, serverless SQL warehouses are not supported"). Identity is Unity Catalog, connectivity is private: serverless "use[s] private connectivity between the Databricks control plane and the serverless compute plane in nearly all cases."

• 📚 Did you know? Serverless egress to your resources is a separate problem from serverless's own lack of public IPs. For anything outside same-region Amazon S3 or DynamoDB, "serverless compute uses public IP addresses to reach your resources", and since mid-February 2026 Databricks "publishes outbound IPs in JSON format on a public endpoint", after May 25, 2026 the legacy IP lists were decommissioned (docs.../serverless-firewall, accessed 2026-10). For dedicated connectivity you use outbound PrivateLink through a network connectivity configuration: each account gets up to 10 NCCs per region, each region 30 private endpoints, and each NCC attaches up to 50 workspaces, and that path needs the Enterprise plan (docs.../pl-to-internal-network, accessed 2026-10).

2.3 Where the compute lives, drawn

The right-hand edge is the one students miss: federation from a serverless warehouse to a private database still has to traverse a network path, and serverless brings its own egress rules with it. Pro is the documented pick when "you have custom-defined networking and want to connect to databases in your network" (warehouse-types, Sep 11, 2026).

2.4 Worked example E1: warehouse-type decision table and the API incantation

Inputs (three forks, all AWS-specific):

1. Is the workspace eligible for serverless (Premium+, not a trial, no legacy external Hive metastore, no S3 access policies)?
2. Do you need to reach a private/on-prem database for federation, or enforce custom networking?
3. Do you need AI Functions, or materialised views / streaming tables created from SQL?

The API body that gets you serverless (docs .../api/warehouses/v1/warehouse, accessed 2026-10):

warehouse_type accepts only PRO or CLASSIC; serverless is PRO plus enable_serverless_compute: true. Omit the flag and you have built a pro warehouse; which is a perfectly good warehouse, just one with a four-minute cold start and an EC2 line on your AWS bill. Tags are capped at fewer than 45 per warehouse and exist, in the API's own words, "to monitor the cost of cloud resources used by users and groups."

---

3. Sizing: T-shirt sizes, the concurrency rule and the hard limits

3.1 The size ladder

Sizes run 2X-Small → 5X-Large, and for classic/pro the ladder is concrete: "All workers use i3.2xlarge instances" (docs .../compute/sql-warehouse/warehouse-behavior, last updated Oct 8, 2026).

Only the driver escalates, i3.2xlarge → i3.4xlarge → i3.8xlarge → i3.16xlarge, while every worker stays i3.2xlarge (same page). The docs note the table "can vary based on product or region availability and workload"; read system.compute.warehouses for the shape you actually got.

The default when you create a warehouse in the UI is X-Large. The disk cache lives on those workers' local SSDs (section 6), which is why the i3 storage-optimised family matters for query performance and why the AWS-side price of an i3 node matters for the cost sketch in section 10.

3.2 Concurrency is a cluster count knob, not a size knob

Documented rules (warehouse-behavior Oct 8, 2026; create Sep 11, 2026):

• "a cluster for every 10 concurrent queries": the sizing rule of thumb, stated on both pages.
• "The maximum number of queries in a queue for all warehouse types is 1,000."
• API bounds: max_num_clusters ≤ 40; min_num_clusters ≤ min(max, 30); default min 1. Raising min_num_clusters "will ensure that a larger number of clusters are always running and therefore may reduce the cold start time": at a continuous billing cost.
• Workspace cap: 1,000 SQL warehouses total, fixed (docs .../resources/limits, Oct 9, 2026).
• "Databricks recycles clusters that have been running for more than 24 hours" and may forcefully terminate them, so no single query or attached notebook session should assume immortal clusters.

The two-knob rule that fixes most dashboards: size (cluster size) buys power per query; max_num_clusters buys concurrency. Disk spill means the first knob; queueing means the second.

The flat series is the lesson: with max_num_clusters pinned at 4 you get 4 × 10 = 40 concurrent query slots at every size, from a 1-worker 2X-Small to a 512-worker 5X-Large. Going wider on the size axis multiplies per-query power (and cost) without admitting one extra concurrent query; going wider on max_num_clusters does the opposite. Warehouse sizes are identical in name across types, but serverless sizes are billed at a fixed DBU-per-hour rate that depends on the size you pick (docs answers, page dated 2026-08-27).

3.3 Worked example E2: sizing 30 concurrent dashboard users

Scenario. A BI team with 30 concurrent dashboard users, average query 4 s, dashboards running 09:00 to 18:00, 21 working days per month.

Answer: three clusters minimum under the documented ratio, sized Medium or Large from the spill evidence, with max_num_clusters set ahead of the peak burst, and on serverless, a single larger warehouse with IWM instead of hand-tuning three clusters.

3.4 Symptom → knob

Two hard limits sit behind that tree: the queue holds at most 1,000 queries, and the profile of a cached result does not exist (section 6), so benchmarking must defeat the cache first or the tree will look cleaner than reality.

---

4. Scaling, auto-stop and cold-start economics

4.1 Two scaling models, not one

Two documented scaling models: one per warehouse family:

• Classic and pro: min/max clusters plus load-based autoscaling, with the load bands quoted in E2 (add clusters after 2 to 6, 6 to 12, 12 to 22 minutes of load; scale up when a query waits 5 minutes; scale down after 15 consecutive minutes of low load). Availability zones resolve to Auto-AZ.
• Serverless: Intelligent Workload Management. "IWM uses machine learning models to dynamically manage compute resources"; when capacity is short "the query is placed in a queue", and "if wait times increase, the autoscaler quickly provisions more clusters" (warehouse-behavior, Oct 8, 2026). IWM is serverless-only: pro and classic get Photon and Predictive IO but not IWM.

4.2 Auto-stop: the cheapest optimization in this course

And the money sentence, verbatim: "Idle SQL warehouses continue to accumulate DBU and cloud instance charges until they are stopped." (same page). A warehouse bills while it is up: not while it is busy.

⚠️ WARNING: an auto-stop default is a billing decision you inherit. The pro/classic default of 45 minutes is chosen for comfort, not economy: every gap shorter than 45 minutes keeps a warehouse (and its EC2 instances, in your AWS account) billable. Conversely, a serverless warehouse at its 1-minute API minimum will stop between a user's two dashboard refreshes: cheap, and only a 2 to 6 second restart away. On a pro/classic warehouse that same restart is ~4 minutes, which users read as an outage. Match auto-stop to client behaviour: scheduled extracts and refreshes → longer auto-stop or min_num_clusters = 1; interactive humans on serverless → short auto-stop. And never set auto-stop to "never": the docs' idle-charge sentence is unconditional.

4.3 Worked example E3: what an auto-stop default costs per month

Scenario (declared assumptions: re-run with your own traffic): one pro warehouse serving dashboards with real traffic only in two bursts: 09:00 to 10:00 and 14:00 to 15:00: over 21 working days.

Rates: SQL Classic $0.22/DBU (page snapshot 2026-10) and AWS us-east-1 on-demand i3.2xlarge $0.624/h (AWS EC2 price list, read 2026-10-12). The 4 DBU/h emission is an illustrative per-size figure: the pricing page publishes $/DBU only, so read your own per-size DBU/hour from system.billing.list_prices (section 10). The point survives the substitution: twenty minutes of unnecessary auto-stop across twenty warehouses dwarfs a size-tier change.

• 📚 Did you know? Auto-stop is only half of the idle story. Raising min_num_clusters "reduce[s] the cold start time for new queries" (Warehouses API, accessed 2026-10) but keeps billable clusters warm around the clock; while on serverless the cold start is already 2 to 6 seconds, so min_num_clusters > 1 is rarely worth paying for. And because "Databricks recycles clusters that have been running for more than 24 hours", a warehouse that never idles still gets its clusters recycled, with a possible force-termination of long-running work (docs.../sql-warehouse/create, Sep 11, 2026).

---

5. Queries, dashboards and alerts: the DBSQL workbench

5.1 Saved queries have their own permissions

The SQL editor ships with AI assistance, code comments and version history (docs .../sql/, Sep 11, 2026). Saved queries are securables in their own right (docs .../sql/user/queries, last updated Apr 29, 2026):

• sharing a query requires CAN MANAGE on that query;
• two credential modes: "Run as viewer", "The viewer's credential is used to run the query. The viewer must also have at least CAN USE permissions on the warehouse", versus "Run as owner";
• "A Databricks workspace admin user has view access to all queries in the workspace."

The query-history UI keeps 30 days of history; you see your own queries as owner, or others' if you hold at least CAN VIEW on the warehouse that ran them. Anything longer-lived goes to system.query.history (section 6).

5.2 AI/BI dashboards: the Lakeview generation, now the only generation

"AI/BI dashboards feature AI-assisted authoring, an enhanced visualization library, and a streamlined configuration experience" (docs .../dashboards, last updated Sep 11, 2026). The documented capability set: datasets and visualisations; filters at global/page/widget level with cross-filtering and drill-through; custom calculations; draft/publish; "When published, your dashboards can be shared with anyone registered to your Databricks account, even if they don't have access to the workspace"; iframe embedding; schedule-and-subscribe by email or Slack; Git integration / source control; Declarative Automation Bundles; REST APIs; scheduling as a Lakeflow Job; Genie agents; usage monitoring via audit logs and system tables; and Unity Catalog lineage built in.

Legacy dashboards are gone. "End of support for legacy dashboards: April 10, 2025. You can no longer create or clone legacy dashboards using the UI or API" (docs.../ai-bi/release-notes/2025); the legacy documentation is retired, and the migration endpoint POST /api/2.0/lakeview/dashboards/migrate is marked Deprecated with the note "Use Lakeview (AI/BI) dashboards instead" (Lakeview API reference, accessed 2026-10).

• 📚 Did you know? The name you see in the API is a fossil. AI/BI dashboards "were previously known as Lakeview dashboards. The Lakeview API still retains that name", so every create/list/publish call is still /api/2.0/lakeview/dashboards/* even though the UI says AI/BI (docs.../dashboards, Sep 11, 2026; REST API reference, accessed 2026-10). Any script, Terraform resource or tutorial that talks about "migrating to Lakeview" is writing 2023 to 2024 history: the migration endpoint is deprecated and legacy dashboards cannot be created at all since 2025-04-10.

5.3 Alerts

"Use Databricks SQL alerts to periodically run queries, evaluate defined conditions, and send notifications if a condition is met" (docs.../sql/user/alerts, page dated Jul 28, 2026). Since May 2025 the editor is consolidated, query setup, conditions, schedules and notification destinations in one interface, and legacy alerts still work alongside it (release notes, May 2025). Alerts are the closing half of the monitoring loop: attach one to the fleet-hygiene query in E4 and a warehouse whose auto-stop creeps past 30 minutes pages someone instead of quietly billing.

---

6. Caching, the query profile and the system tables

6.1 Four caches, four lifetimes

(docs.../queries/query-caching and.../optimizations/disk-cache, accessed 2026-10). To benchmark honestly: SET use_cached_result = false;: documented as appropriate "only in testing or benchmarking". Deterministic queries hit caches; a NOW()-style predicate defeats them and then you pay full price every run.

6.2 The profile, the insights and their meanings

• Query profile; you must be the owner of the query or hold at least CAN MONITOR on the warehouse; "A query profile is not available for queries that run from the query cache. To circumvent the query cache, make a trivial change to the query, such as changing or removing the LIMIT" (docs.../query-profile, Jun 10, 2026).
• Query performance insights: coded recommendations (docs.../performance-insights, Sep 11, 2026): DATA_SPILL → "Increase the warehouse size to add memory"; EXCESSIVE_QUEUE_TIME → "Increase the maximum number of clusters"; IO_THROTTLING → request higher storage request limits; plus DATA_SKEW, MANUAL_DATA_LAYOUT, and applied accelerations AUTO_LIQUID_CLUSTERING, HISTORY_BASED_JOIN_STRATEGY, SHORT_QUERY_PRIORITIZATION.
• System tables: the same story as data: system.query.history (Public Preview, 365 days, regional, records "typically available within one hour"), system.compute.warehouses (config snapshots), system.compute.warehouse_events (start/stop/scale events), system.billing.usage and system.billing.list_prices (the price log, indefinite, global).

Access note on system.query.history: "By default, only admins have access to the system table… Databricks recommends creating a dynamic view for each user or group" (docs .../system-tables/query-history, page dated Sep 21, 2026). Its client_application column is what tells you whether a query came from the "Databricks SQL Editor, Tableau, and Power BI", and cache_origin_statement_id is how you prove a result came from cache.

6.3 Worked example E4: a monitoring pack for the warehouse fleet

Sources: monitoring-queries page (docs.../sql-warehouse/monitor/queries, Sep 11, 2026), query-history reference (Sep 21, 2026), billing system tables (updated 2026-10-08). Query (a) is the one to wire to an alert; query (d) is the one to publish on an AI/BI dashboard: Databricks ships ready-made cost queries and a prebuilt Usage Dashboard for exactly this (docs.../admin/usage/system-tables, Sep 11, 2026).

---

7. BI connectivity: drivers first, tools second

7.1 The driver situation as of Oct 2026

(docs .../integrations/odbc/download, page mirror dated Sep 11, 2026; driver download and archive pages, accessed 2026-10). The February 2026 rename matters operationally: "As of February 2026, new versions of the ODBC driver are being released by Databricks and the driver has been renamed to Databricks ODBC Driver." Install paths: Windows C:\Program Files\Databricks ODBC Driver, macOS /Library/databricks/databricksodbc, Linux /opt/databricks/databricksodbc. Some tools bundle the driver (recent Power BI Desktop releases); Tableau Desktop does not.

The common denominator across every tool: connection = Server Hostname + HTTP Path of a SQL warehouse, authentication = OAuth / Databricks login / personal access token / service principal. Username/password is dead: Tableau's own help records that "Basic authentication using a Databricks user name and password reached end of life as of July 2024."

7.2 The tool matrix

• 📚 Did you know? Driver versions are a support commitment, not just a download list: "Databricks supports each driver version for at least 2 years" (driver download page, accessed 2026-10). The archive also carries a warning worth reading once: Simba JDBC versions before 2.6.21 depend on a library with CVE-2021-44228 (Log4Shell): a concrete reason the docs "strongly recommend migrating" off the Simba lines.

7.3 Worked example E5: BI connection card (copy/paste spec)

Tableau Desktop → a serverless SQL warehouse

Power BI → the same warehouse

Both cards assume the two gates from section 1: CAN USE on the warehouse for the tool's identity, SELECT on the underlying tables for that same identity (a service principal needs both). Add the warehouse's auto_stop_minutes to the card: an extract scheduled every 15 minutes against a 10-minute auto-stop serverless warehouse will pay a 2 to 6 second restart each time, while the same schedule against a pro warehouse pays ~4 minutes.

---

8. Lakehouse Federation: querying other systems from Databricks SQL

8.1 What it is

"Lakehouse Federation is the Databricks query federation platform. It provides governed, read-only access to external data through Unity Catalog foreign catalogs, with automatic query pushdown and fine-grained access controls at the table level" (docs .../query-federation, last updated Sep 29, 2026). Two flavours:

• query federation: pushed down to the external database over JDBC; "The query runs on both Databricks and remote compute… Not supported (read-only)" for writes;
• catalog federation: "Runs directly against object storage on Databricks compute only. More cost-effective and performance-optimized."

Verified source lists (Oct 2026). Query federation: MySQL, PostgreSQL, Teradata, Oracle, Amazon Redshift, Salesforce Data 360, Snowflake, Microsoft SQL Server, Azure Synapse, Google BigQuery, Databricks. Catalog federation: legacy Databricks Hive metastore, external Hive metastore, AWS Glue metastore, Salesforce Data 360, Snowflake, Palantir Foundry, Workday Data Connect, Google Cloud Lakehouse.

8.2 The requirement that excludes classic warehouses

From the MySQL and SQL Server federation pages (Aug 19 / Sep 11, 2026): "Databricks compute must use Databricks Runtime 13.3 LTS or above and Standard or Dedicated access mode"; "SQL warehouses must be pro or serverless and must use 2023.40 or above"; the workspace must be Unity-Catalog enabled (workspaces created after November 8, 2023 are). MySQL additionally requires SSL. Pushdown is uneven by design: aggregates, arithmetic, boolean/bitwise and sort-with-limit push down on 13.3 LTS+; joins need DBR 17.2+ (MySQL: a Public Preview toggle); window functions are not supported on MySQL. And when both paths exist, "When your source supports both Lakehouse Federation and Lakeflow Connect, Databricks recommends Lakeflow Connect if performance on higher data volumes and lower latency are priorities."

8.3 Worked example E6: federated query, MySQL on RDS → Databricks SQL

The language reference (docs.../sql-ref-federated-queries, dated Jun 29, 2026) frames it as exactly three prerequisites: create a foreign connection, register foreign catalogs, grant access, and notes it applies to "Databricks SQL · Databricks Runtime 13.3 LTS and above · Unity Catalog only". The join in step 4 is the interesting half: local Delta joined to remote MySQL, with the remote side pushed down over JDBC and governed by the same grants as your native tables. It is read-only, it is not a substitute for ETL at volume, and the network path from your warehouse to RDS is yours to build (VPC peering, PrivateLink or VPN: section 2.3).

---

9. Who may touch the warehouse: ACLs, entitlements and the group pattern

Warehouse permissions, from the ACL matrix (docs .../access-control, last updated Jul 10, 2026) and the create-warehouse page (Sep 11, 2026):

Two entitlements sit above all of that: the databricks-sql-access entitlement gates who may use Databricks SQL at all, and creating warehouses needs workspace admin or the Allow unrestricted cluster creation entitlement: "Most users cannot create SQL warehouses, but can restart any SQL warehouse they can connect to." Deleted warehouses can be restored by support within 14 days.

The delegation pattern to teach: data-analysts → CAN USE on the BI warehouse plus SELECT on the gold schemas; bi-viewers → CAN VIEW; data-engineers → CAN MANAGE on the platform warehouse and its auto-stop; service principals for Power BI/Tableau → CAN USE plus SELECT. Groups, not individuals: otherwise workspace admins' blanket CAN MANAGE becomes the only working model and admin sprawl follows.

---

10. What it costs: rates, meters and the two-column bill

10.1 The rates (US list, AWS, page snapshot 2026-10)

Caveats you must print next to any number: "Databricks prices and Cloud Infrastructure prices may vary based on geographic region and cloud service provider" (pricing FAQ, 2026-10); serverless rates track regional infrastructure while classic rates are flat across AWS regions because you pay the local EC2 price (a secondary read of the pricing page's data file reported $0.91/DBU for SQL Serverless in eu-central-1 vs $0.70 in us-east-1, 2026-08-07: verify in-account); materialised-view and streaming-table refreshes bill at DLT serverless rates, not your warehouse's rate (pricing page footnote 3, 2026-10); AI Function inference bills separately under MODEL_SERVING / AI_FUNCTIONS (docs.../ai-functions, Oct 9, 2026). The authoritative number is always your own: system.billing.list_prices (indefinite retention, global) joined to system.billing.usage.

10.2 Worked example E7: a two-column monthly cost sketch

Declared assumptions: one 2X-Small warehouse (driver + 1 worker = 2 × i3.2xlarge), up 193 h/month (≈9.2 h/day × 21 working days of business-hours BI), illustrative emission 4 DBU/h, AWS us-east-1 on-demand i3.2xlarge $0.624/h (AWS price list, read 2026-10-12), rates as of 2026-10.

Read it as three lessons, not as a ranking. (A) is cheapest only if you accept a ~4-minute cold start, no Predictive IO, no federation, no AI Functions and no materialised-view creation. (C) beats (B) here because at this light, steady shape the pro warehouse's EC2 column ($240.86) exceeds the entire serverless premium ($540.40 − $424.60 = $115.80). And (A)'s sensitivity line is the real headline: idle hours, not list rates, dominate; which is section 4's auto-stop arithmetic showing up again on the invoice. The 4 DBU/h figure is illustrative per-size emission; the pricing page publishes $/DBU only, so pull your own per-size emission from system.billing.list_prices for your contract before you quote anyone a number.

---

11. The traps: fourteen confusions this lesson will not let you keep

1. "Serverless runs in my VPC." It runs in the Databricks account: "Customer-managed VPCs are not applicable", no cluster policies, no public IPs.
2. "The API creates the same warehouse as the UI." UI default = serverless; API default = classic unless enable_serverless_compute is true and warehouse_type is PRO.
3. "Classic is the budget option." It is also the option that cannot federate, cannot create MVs/STs, and cannot run AI Functions.
4. "A bigger size means more concurrent queries." Concurrency is max_num_clusters × 10; size is power per query. Spill → size; queue → clusters.
5. "The autoscaler will handle the 9 a.m. burst." Classic/pro scale up only after a query waits 5 minutes; the queue cap is 1,000.
6. "Auto-stop 45 minutes is harmless." "Idle SQL warehouses continue to accumulate DBU and cloud instance charges until they are stopped."
7. "Serverless auto-stop can be zero." Minimum 5 min in the UI, 1 min via API; pro/classic minimum is 10 min.
8. "I got a query profile, so I understand the run." No profile exists for a cached result: change the LIMIT.
9. "Cache is cache." UI cache (≤7 days, not AI/BI) ≠ local result cache (dies with the cluster) ≠ remote result cache (serverless-only, survives stop) ≠ disk cache (files, ≤50% of SSD).
10. "Legacy dashboards still work." End of support 2025-04-10: no create, no clone, migration endpoint deprecated.
11. "Lakeview is a different product from AI/BI." Same product, 2023 preview name retained in the REST path /api/2.0/lakeview/dashboards.
12. "Password auth works for Tableau." Basic auth "reached end of life as of July 2024"; and the Simba ODBC line is deprecated since the February 2026 rename.
13. "Federation lets me write back." Read-only, JDBC pushdown with uneven support; use Lakeflow Connect when volume and latency matter.
14. "The $0.70 rate is the whole serverless bill." MV refreshes bill at DLT serverless rates, AI inference under MODEL_SERVING/AI_FUNCTIONS, and networking to your VPC can carry its own charges.

---

Comparative Verdict: classic vs pro vs serverless SQL warehouses (AWS, as of Oct 2026)

• Classic, $0.22/DBU plus cloud instance charges, ~4-minute start. The API default and the floor of the lineup: Photon only, no Predictive IO, no IWM. Cannot federate, cannot create materialised views or streaming tables from SQL, cannot run AI Functions. Runs in your VPC with cluster policies, spot settings and the disk cache on i3 local SSD. Choose it when the workload is simple SQL at a steady shape, cost per DBU is the binding constraint, and none of the three missing features apply, and set auto-stop to 10 minutes, not 45.
• Pro, $0.55/DBU plus cloud instance charges, ~4-minute start. Everything classic has plus Predictive IO, and the documented pick when "you have custom-defined networking and want to connect to databases in your network", i.e. federation from inside your VPC. Still your instances, your Auto-AZ, your policies, your EC2 column. Choose it when private-network reach matters more than cold-start latency.
• Serverless: $0.70/DBU, machine included, 2 to 6-second start. The documented default for most workloads: Photon + Predictive IO + IWM, no cluster policies (so no policy-based guardrails either), no customer-managed VPC, no public IPs, auto-stop default 10 minutes, per-second billing with no idle-charge beyond the auto-stop window. Choose it unless a fork in E1 says otherwise.
• Verdict in one line: serverless by default → pro when you need the compute inside your AWS account for federation or custom networking → classic only for narrow, steady, feature-light SQL where $0.22 wins the arithmetic. Every rate is US list, AWS, page snapshot 2026-10; the per-size DBU/hour emission must come from system.billing.list_prices, not from a tutorial.

---

2026 Updates (as of October 2026)

---

Real-World Case Studies

📋 Case A: NAB (National Australia Bank, banking, Cloud: AWS) Challenge. A 26-year-old Teradata warehouse carrying ~1,000 use cases and ~140 reporting suites, with hand-written ETL and inconsistent practice across 300 to 400 engineers. Stack on AWS. Databricks SQL as the warehouse replacement, plus Auto Loader, Spark Declarative Pipelines (Lakeflow), Unity Catalog and asset bundles: Cloud: AWS on the story page. Reported outcome. "26-year-old Teradata warehouse decommissioned · 456 Use cases migrated to Databricks SQL · 1,200+ Users enabled"; "95% of critical workloads migrated"; ~140 reporting suites retired (https://www.databricks.com/customers/nab/dbsql, undated page, accessed 2026-10). Quote. "The migration wouldn't have been possible without Databricks SQL": Gurry, same URL. Maps to this lesson. Sections 3, 7 and 9: replacing a warehouse is a report rationalisation plus a sizing/ACL exercise: 1,200 users need CAN USE by group, and 456 use cases need a concurrency budget across the fleet, not one giant warehouse. ⏳ Temporal note: undated page, accessed Oct 2026: a reported customer outcome, not a 2026 guarantee.

📋 Case B: Punchh (retail loyalty / restaurant tech, Cloud: AWS) Challenge. Outgrown AWS MySQL, a warehouse too costly to keep, and Tableau reporting slowed by 12-table joins executed fresh for every report. Stack on AWS. Databricks SQL over bronze/silver/gold Delta tables, notebooks, and Tableau on top: Cloud: AWS on the story page. Reported outcome. "10x Faster time-to-insight · 30% Lower operational costs · 12% Increase in customer satisfaction" (https://www.databricks.com/customers/punchh, undated page, accessed 2026-10; story origin 2021). Quote. "When you're joining 12 tables on the fly every time a report is run by a customer, the cost to generate the report can be high. Databricks not only increased ETL performance but lowered overall costs": Phylis Savari, same URL. Maps to this lesson. Sections 7 and 10: the external-BI pattern in its purest form: Tableau over ODBC to a SQL warehouse, pre-joined gold tables doing what a bigger warehouse would otherwise have to brute-force. ⏳ Temporal note: a 2021-origin story still displayed in 2026: teach the mechanism, date the metrics.

📋 Case C: nCino (banking software, Cloud: AWS) Challenge. Consolidating 13 years of banking data and "trillions in loan history" while keeping sensitive financial information under full control. Stack on AWS. Databricks on AWS with Unity Catalog and AI/BI Genie: Cloud: AWS; use cases list Analytics and BI, AI, Data Warehousing, Governance and Security. Reported outcome. "Keeping data within a single AWS ecosystem reduces complexity, strengthens compliance and gives teams full control over sensitive financial information"; "With Unity Catalog governing access and AI/BI Genie democratizing insights, executives, analysts and relationship managers all benefit from domain-driven AI solutions" (https://www.databricks.com/customers/ncino, undated page, accessed 2026-10). Maps to this lesson. Sections 5 and 9: AI/BI dashboards plus Genie as the governed serving layer, with the two gates (warehouse ACL + UC grant) doing the compliance work. ⏳ Temporal note: undated page, accessed Oct 2026.

---

---

Practice Questions