AWS Foundation for Databricks: VPC Design, Root Bucket, PrivateLink, KMS and IAM

Every Databricks lakehouse in this course runs on one AWS-shaped assumption: the data plane lives in your VPC, the data lives in your S3, and the control plane talks to both over paths you can name on a whiteboard. Get the substrate wrong and nothing downstream saves you: cluster launches fail with cryptic assume-role errors, a 100-node cluster cannot start because a /26 subnet ran out of addresses, an S3 bill balloons because 1 TB a month of lake traffic hairpinned through a NAT gateway at $0.045/GB, or a TLS-inspecting firewall silently kills every cluster the moment it touches the pinned control-plane certificates.

Every fact below comes from Databricks' AWS documentation (customer-managed VPC page last updated Oct 8, 2026; well-architected network phase Sep 11, 2026; PrivateLink pages Oct 9, 2026), the Databricks Security Reference Architecture, and AWS pricing pages: all accessed October 2026. Do the arithmetic yourself; it is the only way to defend a network design in a review.

In this lesson you will:

• dissect workspace anatomy: the four account-console objects a Databricks-on-AWS workspace is assembled from, and the cross-account trust that binds them;
• size a customer-managed VPC with the documented (addresses − 5) ÷ 2 rule, from /26 (29 nodes) to /17 (16,381 nodes), and pick subnets against a real node target;
• run secure cluster connectivity (SCC) correctly: what it removes, what it does not, and why certificate pinning makes TLS inspection a cluster-killer;
• design egress the Databricks way: S3 gateway endpoints (free), STS + Kinesis interface endpoints, NAT economics, and the air-gapped Option 2 build;
• wire all three PrivateLink flavors (inbound, classic back-end, outbound/serverless) with the correct prerequisites and NCC limits;
• encrypt the root bucket and control-plane artifacts with the right KMS use cases, and read the key policy line by line;
• write the annotated root-bucket policy: the aws:PrincipalTag/DatabricksAccountId condition and the unity-catalog/* deny;
• choose between instance profiles, storage credentials and service credentials, and see why the modern path attaches IAM to principals, not to boxes;
• hand-compute seven worked examples: subnet capacity, the NAT-vs-endpoints bill, endpoint-inventory costing, KMS monthly cost, the trust-policy JSON, the security-group matrix and the root-bucket policy;
• study two named AWS customer cases (Block and Amgen), read the 2026 update box, and practise with 12 exam-style questions plus four interactive checks.

---

1. Workspace anatomy: the four objects you actually create

A Databricks workspace on AWS is not "an EC2 fleet". It is an account-console object assembled from four pieces: three supplied or approved by you, one created by Databricks (customer-managed VPC page, Oct 8, 2026; create-a-workspace guide, Sep 11, 2026):

Automated vs manual creation. Automated setup (recommended) uses AWS IAM temporary delegation so you never hand-copy trust policies. Manual setup makes you build every role and policy yourself; which is exactly why the manual guide publishes the trust-account IDs and the bucket-policy shape you will see in section 7 (create-a-workspace guide, Sep 11, 2026).

• 📚 Did you know? Databricks' AWS account IDs are public and partition-specific: 414351767826 for commercial regions, 044793339203 for AWS GovCloud, 170661010020 for GovCloud DoD (create-a-workspace guide, Sep 11, 2026). The AMI source account for workspace images is 601306020600. Any trust policy you write for Databricks names one of these, and always pairs it with aws:ExternalId equal to your Databricks account ID, so a second Databricks customer cannot assume your role.

1.1 Worked example E1: the cross-account trust policy, line by line

Manual setup asks for a role in your account trusted by Databricks. The essential shape (pattern from the Databricks create-a-workspace guide and the Databricks Security Reference Architecture, accessed 2026-10):

Three checks a reviewer must run on this JSON: (1) the account in the ARN is the commercial Databricks account 414351767826 (or the GovCloud ID if you are GovCloud); (2) the external ID equals your Databricks account ID, not a made-up string; (3) the role's permission policy scopes to EC2/EBS/Spot actions tagged for Databricks where your org standards require it: the SRA reference build uses Vendor=Databricks tag conditions so only Databricks-launched instances carry the profile.

---

2. The customer-managed VPC: private subnets, sizing, and the capacity formula

2.1 The rules that are hard failures, not advice

The 0.0.0.0/0 route is the part that confuses people: the workspace subnets are private, but in Option 1 (the recommended standard configuration) their route table still points default traffic at a NAT gateway sitting in its own public subnet that routes quad-zero to an internet gateway. Private means "no public IPs on my nodes", not "no egress".

2.2 Worked example E2: sizing subnets for 450 concurrent nodes per AZ

Target: a platform team wants clusters of up to 450 nodes per availability zone, two AZs, plus headroom for instance pools and SQL warehouses. Apply the documented formula nodes = (addresses − 5) ÷ 2:

Two traps live in that arithmetic. First, the /26 floor is legal but useless: 29 nodes per AZ dies on the first real workload, yet it satisfies the letter of the netmask rule, so workspace-creation validation passes and the failure arrives later as "clusters stuck pending". Second, capacity is consumed by idle things too: instance pools keep warm capacity, SQL warehouses hold nodes between queries, and an all-purpose cluster someone forgot to terminate holds IPs for days. Size for peak plus the idle fleet.

• 📚 Did you know? The halving is not arbitrary: "Databricks assigns two IP addresses per node, one for management traffic and one for Spark applications. The total number of instances for each subnet is equal to half of the number of IP addresses that are available" (customer-managed VPC page, Oct 8, 2026). The five subtracted addresses are AWS-reserved (network, router, DNS, future-use). So a /23 is not "512 nodes"; it is 253, and every capacity review that quotes raw CIDR math is wrong by 2×.

2.3 Security groups: the two-SG matrix

One to five security groups per workspace; the documented egress and ingress shape (customer-managed VPC page, Oct 8, 2026):

Do not port-copy from Azure NSG templates. 3306 exists only for the legacy Databricks Hive metastore (disabled by default on new Unity Catalog workspaces), 6666 is PrivateLink-only SCC, 8444 is the Unity Catalog lineage/logging stream, and 2443 is FIPS. A "secure" rule set that drops 8443/8444 produces launch failures that look like product bugs.

---

3. Secure cluster connectivity: the default, and what it does not buy you

SCC is the default deployment mode and the default recommendation (well-architected network phase, Sep 11, 2026). The mechanism reverses the call direction: each cluster initiates a connection out to the SCC relay in the control plane, and cluster-management traffic comes back through that tunnel. Result: no open inbound ports and no public IP addresses on classic compute-plane nodes.

Three consequences engineers get wrong:

1. SCC ≠ no internet. SCC removes inbound needs. Databricks still requires outbound reachability to the SCC relay FQDNs, s3.amazonaws.com:443, sts.amazonaws.com:443, Kinesis, and library repositories. Private subnets with neither NAT nor endpoints give you clusters that never start and pip/Maven installs that hang.
2. Allowlist FQDNs, not IP addresses. "Always allowlist the provided domain names (FQDNs) for secure cluster connectivity (SCC) relay endpoints, rather than individual IP addresses" (IP addresses and domains page, accessed 2026-10). Control-plane NAT IPs are published per region (e.g. us-east-1 3.237.73.224/28) and matter for bucket policies, not for SCC egress.
3. SCC is a prerequisite for classic (back-end) PrivateLink. If PrivateLink is on your roadmap, SCC must be on from day one, and an existing workspace without SCC is a rebuild conversation, not a toggle.

3.1 Interactive check: match each egress port to its job

---

4. Egress architecture: gateway endpoints, interface endpoints, NAT

4.1 What a customer-managed VPC must reach

"There are four AWS services that a Databricks workspace with a customer-managed VPC must reach: STS, S3, Kinesis, and RDS" (customer-managed VPC page, Oct 8, 2026). RDS is only for the legacy Databricks Hive metastore: a Unity Catalog workspace does not need it. So the modern required set is STS + S3 + Kinesis.

The endpoint-type mechanics that decide your bill:

Gateway endpoints exist only for S3 and DynamoDB (AWS VPC user guide). Everything else that you want off the public path is an interface endpoint billed per AZ, per hour, whether or not traffic flows.

4.2 Worked example E3: the NAT-vs-endpoints bill

Scenario from the Databricks network guidance: a workspace moving 1 TB/month of S3 traffic plus 500 GB/month of other AWS API traffic, two AZs, rates as of Oct 2026 (NAT gateway $0.045/hour + $0.045/GB; interface endpoints ~$0.01/AZ-hour; gateway endpoints free):

The teaching point is not the $6 swing; it is the shape. NAT charges hourly and per GB forever; the S3 gateway endpoint is free and removes the largest data path from NAT entirely; interface endpoints are the only way to keep STS/Kinesis private but you pay for them per AZ even idle. Databricks' own words: "If most traffic through your NAT gateway is to AWS services that support interface endpoints or gateway endpoints, consider creating an interface endpoint or gateway endpoint for these services" (AWS NAT gateway guide). And per GB through PrivateLink: first 1 PB $0.01/GB, next 4 PB $0.006, over 5 PB $0.004 (AWS PrivateLink pricing, accessed 2026-10).

4.3 Option 1 vs Option 2: the egress decision

The classic PrivateLink page defines the two official configurations: Option 1 (standard, recommended) keeps a NAT gateway + internet gateway and adds optional S3/STS/Kinesis endpoints; Option 2 (fully private / air-gapped) eliminates NAT and IGW and requires the S3, STS and Kinesis endpoints plus Databricks back-end PrivateLink (classic PrivateLink page, Oct 9, 2026).

Air-gapped builds still owe outbound reachability to s3.amazonaws.com:443, sts.amazonaws.com:443 and the SCC relay FQDNs: via endpoints, not the internet. The trade-off is explicit in the docs: Option 2 removes NAT cost and public paths, and hands you every endpoint as an operational dependency.

4.4 Interactive check: order an air-gapped build

---

5. PrivateLink at Databricks: three flavors, one prerequisite matrix

Databricks supports three types of Private Link connectivity (Private Link concepts page, Oct 9, 2026):

A fourth, narrower type, Service Direct, exists for performance-intensive services (Zerobus Ingest, Lakebase Autoscaling). Databricks "recommends at most one endpoint of each endpoint type" within the same VPC.

NCC limits worth memorising (Private Link concepts page, Oct 9, 2026): up to 10 NCCs per region per account; 50 workspaces per NCC; 30 S3 private endpoints per region; 100 private endpoints per region for VPC resources. VPC endpoint registrations are immutable once created. Port 3306 (legacy HMS) does not traverse PrivateLink. And per the private access settings page (May 4, 2026), private access level ANY is deprecated: use Account (only VPC endpoints registered in your Databricks account) or Endpoint (explicit allow list).

5.1 Worked example E4: endpoint inventory and idle cost

Take the two-AZ workspace from E2 adding classic back-end PrivateLink. Interface endpoints at the commonly published US rate of ~$0.01/AZ-hour (≈ $7.30 per AZ-month; verify your region on aws.amazon.com/privatelink/pricing, accessed 2026-10):

The anti-pattern at the bottom is the one design reviews keep catching: interface S3 endpoints charge hourly per AZ and per GB, while the gateway endpoint is free and covers in-VPC S3. Four interface endpoints across three AZs idles at ~$87.60/month; the same S3 access through a gateway endpoint idles at $0. Databricks says it directly: prefer gateway "unless strictly required for compliance reasons" (Sep 11, 2026).

---

6. Encryption: the root bucket, managed services, and the key policy

Databricks encrypts at rest by default (SSE on the root bucket; control-plane artifacts encrypted in the Databricks-managed layer). Customer-managed keys (CMK) on the Enterprise tier extend that control to two distinct use cases (customer-managed keys pages, accessed 2026-10):

Unity Catalog separately "supports the ability to read from and write to S3 buckets with KMS encryption enabled": each storage credential's IAM role gets the kms: actions it needs on the bucket's key.

6.1 Worked example E5: KMS cost for a three-key estate

The Databricks Security Reference Architecture reference build uses three KMS keys: workspace storage, managed services, and the Unity Catalog workspace catalog. AWS KMS pricing (accessed 2026-10): $1.00 per key per month (prorated hourly) plus $0.03 per 10,000 requests, with a free tier of 20,000 requests/month.

6.2 The key policy line you must get right

The customer-managed-keys configuration guide requires the key policy to allow the Databricks principal: with a tag condition, not open access (accessed 2026-10):

Plus a separate statement scoped to your own cross-account role for EBS-related use. The tag condition is the guardrail: it ensures only Databricks principals tagged with your account ID can use the key: not every principal in the Databricks AWS account. Pair it with MANAGED_SERVICES vs STORAGE use-case configuration so notebooks-secrets-SQL-history and bucket-EBS are independently rotatable.

• 📚 Did you know? In transit, Databricks uses TLS 1.2+ between users/APIs/BI tools and between control and compute planes; worker-to-worker shuffle encryption is opt-in (Enterprise tier: via an init script on AES-256/TLS 1.3, or by picking instance families that encrypt inter-node traffic natively). So "encryption at rest via CMK" and "encryption in transit everywhere" are three separate checkboxes, not one.

---

7. The root bucket: what lives in it, and the policy that guards it

The root bucket (workspace storage) is an S3 bucket in your AWS account that Databricks provisions (automated setup) or you register (manual setup: bucket name + IAM role ARN). It stores workspace assets "such as data, libraries, and logs", DBFS root (FileStore, MLflow models, Lakeflow pipeline data), workspace system data (job results, Databricks SQL results, notebook revisions), and, per the create-a-workspace guide (Sep 11, 2026), "the workspace's default catalog".

Two facts about it that stop two classic disasters:

1. It is customer-owned but Databricks-operated. The Databricks control plane reaches it as principal arn:aws:iam::414351767826:root, which your bucket policy permits only under the account-tag condition.
2. It is not your analytics lake. DBFS root and DBFS mounts are deprecated: "New accounts are provisioned without access to these features. Databricks recommends using Unity Catalog volumes, external locations, or workspace files instead" (DBFS + UC best-practices page, accessed 2026-10). And the same page: "You should never load a storage account used as a DBFS root as an external location in Unity Catalog."

7.1 Worked example E6: annotated root-bucket policy

Manual-setup shape from the create-a-workspace guide (Sep 11, 2026), annotated:

Hardening moves seen in production designs (patterns from the Databricks SRA and the documented condition-key list): add aws:SourceIp conditions for your region's control-plane NAT range (a brand-new workspace provisioning fails if the us-west control-plane NAT IP is not allowed), and/or aws:sourceVpce bound to your S3 gateway endpoint so the bucket answers only from inside the VPC. Databricks notes that control-plane access uses VPC IDs for same-region S3 and NAT IPs for cross-region S3: design the condition for the region pair you actually run.

---

8. IAM: instance profiles vs storage credentials vs service credentials

This is the single most important conceptual shift from 2021-era Databricks-on-AWS material to the Unity Catalog era:

Databricks states the shift plainly: service credentials are "the Unity Catalog alternative to instance profiles, with the advantage that access is not tied to a specific compute resource but instead to users, groups, or service principals" (service-credentials docs, accessed 2026-10). A storage credential is defined as "a Unity Catalog credential used to access external locations and tables" (storage-credential SQL reference, Sep 11, 2026).

8.1 Worked example E7: modern data-access path, end to end

The examinable difference: step 4 grants follow the group, so the same notebook works on a jobs cluster, a SQL warehouse and a serverless compute without any instance-profile attachment, and revocation is one REVOKE, not a cluster-patch plus a bucket-policy edit.

8.2 Interactive check: fill the terms

---

9. Cross-account patterns, hub-and-spoke, and the network-configuration traps

Three root-bucket/account patterns appear in official material (digest-grounded, create-a-workspace guide Sep 11 2026):

1. Single account: bucket, roles and workspace all in one AWS account (default automated flow).
2. Tag-gated bucket policy: bucket in your account, allow for 414351767826:root conditioned on aws:PrincipalTag/DatabricksAccountId, plus the unity-catalog/* deny (E6).
3. Network-restricted bucket: the same policy plus aws:SourceIp (control-plane NAT IPs / corporate VPN) and/or aws:sourceVpce (your S3 gateway endpoint) conditions.

For many business units, Databricks' documented topology is hub-and-spoke: "You can use a VPC hub-and-spoke architecture, where all VPC endpoints, NAT gateways, firewalls, and so on, are installed in the hub VPC. Each workspace is associated with subnets … in a different VPC, within the same or a different AWS account. The spoke VPCs are connected to the hub VPC using a Transit Gateway" (well-architected network phase, Sep 11, 2026). Remember the constraint from section 1: each workspace still registers its own network configuration, even when subnets are shared.

Operational traps collected from the documented failure modes (pitfalls section of the customer-managed VPC and PrivateLink pages, 2026):

• Blocked STS = dead clusters. STS is mandatory. Enforce regional endpoints or a new firewall rule without an STS interface endpoint (or an allow for sts.amazonaws.com:443) and assume-role failures stop cluster launches and S3 access. Manual workspace creation additionally requires the STS endpoint activated for us-west-2.

• NACLs that "help". Databricks validates that NACLs allow 0.0.0.0/0 inbound as the top rule; a deny on 8443/8444/6666 causes cryptic launch failures. Re-associate route tables if you recreate subnets.

• Kinesis in us-west-1 is cross-region (target streams live in us-west-2), so a regional Kinesis endpoint there behaves differently (customer-managed VPC note).

• Databricks-managed VPC is on the way out: "In a future release, Databricks plans to deprecate classic workspace creation with a Databricks-managed VPC. New classic workspaces will require a customer-managed VPC" (Oct 8, 2026), and an existing Databricks-managed-VPC workspace cannot be converted in place; plan a rebuild.

• 📚 Did you know? The Databricks-managed-VPC → customer-managed-VPC migration story has a hard Unity-Catalog edge: DBFS root/mounts must never become a UC external location, instance-profile access is replaced by storage/service credentials, and PrivateLink requires Enterprise + SCC. The realistic path is provision-new-workspace-in-customer-VPC → attach metastore → recreate access via storage credentials + external locations → migrate tables: the full sequence is lesson 3's topic.

---

10. Classic vs serverless networking, and the 2026 deltas

2026 Updates (as of October 2026)

---

Real-World Case Studies

Customer pages publish architecture outcomes, not VPC diagrams: "Cloud: AWS" tells you the substrate is this course's substrate, never which CIDR or KMS layout a company chose. Every figure below is customer-claimed, with the source and its date named so you can check it.

Case A, Block (Square, Cash App, TIDAL): 12 PB on S3, IAM plumbing collapsed into Unity Catalog

⏳ Temporal note: undated customer page accessed Oct 2026; treat the 12×/20%/12 PB figures as reported outcomes, not guarantees.

Case B, Amgen: from 120+ IAM roles to one or two, via Unity Catalog

⏳ Temporal note: May 2023 figures; the mechanism (principals governed in UC, roles collapsed) remains the documented pattern in 2026.

Case C, Nubank: S3 as the storage layer of a regulated fintech lakehouse

⏳ Temporal note: session-abstract evidence, not an audited case study; teach the migration pattern, not a promised discount.

---

Practice Questions