Unity Catalog Fundamentals: Metastore, Hierarchy, Grants, Storage Credentials and External Locations

Lesson 02 built the AWS substrate: VPC, root bucket, PrivateLink, KMS, IAM. This lesson puts the governance plane on top of it. Unity Catalog is where that substrate stops being plumbing and starts being policy: one GRANT that follows a group across every workspace in a region, one IAM role wrapped in a securable so that revoking access is a REVOKE instead of a bucket-policy edit, and one three-level namespace (catalog.schema.table) that finally gives S3 paths a name you can put in a review.

The teaching rule for this lesson: Unity Catalog decides who, your AWS account still holds what. Data files never leave your S3 account, IAM still has to let Databricks assume a role, and every grant you write is only as good as the chain of USE CATALOG → USE SCHEMA → SELECT that delivers it. Confuse those two sentences and every symptom below becomes a mystery.

In this lesson you will:

• place the metastore correctly: one per region, created at account level, attached to many workspaces, with account / workspace / metastore admin roles separated;
• draw the securable hierarchy and name the objects that deliberately hang outside the catalog tree;
• master privileges and inheritance: why USE CATALOG is required but grants no data, why metastore-level grants do not cascade, and why ownership does not inherit downward;
• operate deny-by-default with its two documented default-open carve-outs (main, the workspace catalog) and the workspace binding rule that overrides even a perfect SELECT grant;
• build the worked AWS setup end to end: IAM trust policy with external ID → CREATE STORAGE CREDENTIAL → CREATE EXTERNAL LOCATION → four GRANTs → an external Delta table;
• separate managed from external tables and volumes, predict where a managed table physically lands, and know exactly what DROP deletes in each case;
• choose a hive_metastore migration tool per table type (SYNC, upgrade wizard, CREATE TABLE CLONE, DEEP CLONE via UCX, federate-then-SET MANAGED);
• study three named customer cases (Amgen, Rivian, nCino), read the October 2026 update box, and practise with 12 exam-style questions plus four interactive checks.

1. The metastore: a regional, account-managed object

Unity Catalog's own reference opens with the definition everything else hangs off:

"The metastore is the top-level securable object in Unity Catalog. A metastore contains all securable objects registered in Unity Catalog in a single cloud region.": Databricks docs, Unity Catalog securable objects (last updated Oct 8, 2026)

Three consequences follow, and each one is examinable:

1. One metastore per cloud region. "A metastore is scoped to a single cloud region. Your organization requires one metastore per region in which it operates." A single metastore "can be attached to multiple workspaces in the same region" (same page, Oct 8, 2026).
2. One workspace, one metastore. "Each workspace must be assigned to exactly one metastore" (Databricks well-architected deployment guide, Design Unity Catalog architecture, accessed 2026-10). A workspace cannot straddle two metastores, so cross-region reads are never "just a grant".
3. Grants travel with the metastore, not the workspace. "Permission grants in a metastore apply to all workspaces attached to that metastore. In other words, a privilege granted in one workspace is effective in all other workspaces sharing that metastore" (Oct 8, 2026). That is the whole reason to consolidate: one policy surface, many workspaces.

The metastore itself is "hosted as a multi-tenant service in the Databricks control plane" while "the data itself still lives in your cloud account": the control plane holds metadata, your S3 holds bytes (deployment guide; Managed versus external assets, Sep 11, 2026).

1.1 Who administers what

Auto-provisioned metastores (workspace auto-assignment, or the automatic enablement for AWS accounts created after Nov 8, 2023) may have been "created without a metastore admin", so do not assume the role is populated just because Unity Catalog works.

• 📚 Did you know? Enabling an existing workspace for Unity Catalog is a one-way door: "enabling a workspace for Unity Catalog cannot be reversed. Once you enable the workspace, you will manage users, groups, and service principals for this workspace using account-level interfaces" (Databricks docs, Enable a workspace for Unity Catalog, last updated May 20, 2026). Identity federation, account-level users and groups, is switched on at the same time, which is why grants in this lesson always name account-level identities, never workspace-local ones.

1.2 Worked example E1: the two-region topology, decided on paper

Scenario: a retailer runs production and development in us-west-2 and a European entity in eu-west-1. Four workspaces, one platform team, one data-ownership model.

Reading: environments are separated by catalogs and bindings inside one metastore, not by extra metastores. Databricks' own new-workspace guidance says the same: "adopt a single metastore per region with multiple catalogs… Isolate environments with workspace bindings" (Databricks Answers, 2026-08-27). One caveat worth storing: that official Q&A also advises planning "the storage root carefully because it cannot be changed after creation": a Databricks Answers post, not a docs page, so treat the immutability claim as official-Q&A-grade evidence.

2. The hierarchy: what hangs where, and why it matters

Unity Catalog exposes "a three-level namespace (that is, catalog.schema.table)" (docs, Enable a workspace, May 20, 2026). Data and AI assets, "tables, views, volumes, functions, models, and services", follow it, while a second family of objects sits directly under the metastore, outside the tree entirely (What is Unity Catalog?, Jul 7, 2026; securable-objects, Oct 8, 2026).

Two structural facts students miss repeatedly:

• Containers are the catalog and the schema. "Every object in this hierarchy is a securable object", and privileges granted on a parent "automatically apply to all current and future child objects", but that inheritance engine starts at the catalog, not at the metastore (permissions-concepts, Aug 23, 2026; privileges blog: "The highest level object that privileges are inherited from is the catalog", 2023-05-05).
• Storage credentials and external locations are not under a catalog. They live at metastore level, which is why creating one is a metastore-scoped operation and why you grant on them with ON STORAGE CREDENTIAL … / ON EXTERNAL LOCATION … syntax rather than a three-level name.

The legacy Hive metastore is additive: "The Unity Catalog metastore is additive, meaning it can be used with the per-workspace Hive metastore in Databricks. The Hive metastore appears as a top-level catalog called hive_metastore in the three-level namespace" (docs, Work with the legacy Hive metastore, accessed 2026-10). Section 8 turns that fact into a migration plan.

• 📚 Did you know? Unity Catalog was open-sourced on 2024-06-12/13 under Apache 2.0, built on an OpenAPI spec and "compatible with Apache Hive's metastore API and Apache Iceberg's REST catalog API", with support announced from AWS, Google Cloud, Microsoft, NVIDIA and Salesforce (Databricks blog, 2024-06-13; press release, 2024-06-12). So the securable vocabulary you are learning in this lesson is not Databricks-UI-deep; it is the same model served through the Iceberg REST and Hive Metastore interfaces.

3. Privileges: the chain, the inheritance rules, the owner trap

"Securable objects in Unity Catalog are hierarchical, from the metastore at the top, through catalogs and schemas, down to the data objects they contain (tables, views, volumes, functions, and models).": Databricks docs, Unity Catalog privileges (last updated Oct 8, 2026)

That reference documents Privilege Model version 1.0; a metastore "created during the public preview (before August 25, 2022)" may sit on an earlier model without today's inheritance behaviour. Assume 1.0 for everything below.

3.1 The privileges you will actually type

Two navigation privileges carry the whole read path:

"USE CATALOG: Required to interact with any object inside a catalog. Does not itself grant access to data." … "USE SCHEMA: Required to interact with any object inside a schema. Does not itself grant access to data.": privileges reference, Oct 8, 2026

3.2 Inheritance versus ownership: two mechanisms, one confusion

Read that diagram as three separate laws:

1. Catalog and schema grants cascade to "all current and future schemas, tables, views, volumes, and functions"; that is why granting SELECT on a catalog is the efficient bulk move (securable-objects, Oct 8, 2026).
2. Metastore grants do not cascade, with the single documented exception of READ METADATA: "privileges granted at the metastore level do not inherit to child objects… This differs from the privilege inheritance behavior for catalog and schema grants" (Oct 8, 2026). Holding CREATE CATALOG on the metastore gives you zero readable tables.
3. Ownership is not inheritance. "The principal that creates an object becomes its initial owner"; "Owners of an object are automatically granted all privileges on that object"; but "Ownership doesn't inherit downward in Unity Catalog": a schema owner does not automatically get privileges on the schema's tables, though they "can grant themselves privileges" (permissions-concepts, Aug 23, 2026; GCP ownership wording, Mar 5, 2026).

Who may grant at all: "The owner of the object. The owner of the catalog or schema that contains the object. A user with the MANAGE privilege on the object. A metastore admin. Account admins can also grant privileges directly on a metastore" (manage-privileges, accessed 2026-10).

3.3 Worked example E2: the grant chain, and a SHOW GRANTS autopsy

Target: the group analysts must read prod.sales.orders.

Forensics. Suppose SHOW GRANTS ON SCHEMA sales_staging returns exactly one row: OWNERSHIP. Why can the owner still SELECT from sales_staging.t? Because ownership is all privileges, rendered as a single row, the reference is explicit that for an ALL PRIVILEGES holder "only ALL PRIVILEGES is returned, not the individual implied privileges like SELECT or MODIFY" (Oct 8, 2026). Why can a second user not? Deny by default (section 4). What must the schema owner run for a colleague? An explicit GRANT SELECT ON SCHEMA …, ownership does not cascade, so the colleague gets nothing from the owner's status.

3.4 Interactive check: match the privilege to its level

• 📚 Did you know? ALL PRIVILEGES sounds absolute and is not: "ALL PRIVILEGES does not include the EXTERNAL USE SCHEMA, EXTERNAL USE LOCATION, MANAGE, or READ METADATA privileges" (privileges reference, Oct 8, 2026). And to avoid accidental data exfiltration, "schema owners do not have the EXTERNAL USE SCHEMA privilege by default and external location owners do not have the EXTERNAL USE LOCATION privilege by default" (permissions-concepts, Aug 23, 2026). EXTERNAL USE LOCATION can only be granted by a principal that holds MANAGE on the location.

4. Deny by default, the two default-open carve-outs, and bindings that beat grants

"Initially, users have no access to data in a metastore. Databricks account admins, workspace admins, and metastore admins have default privileges for managing Unity Catalog.": Databricks docs, Manage privileges in Unity Catalog (accessed 2026-10)

Databricks' own Q&A restates it as a "deny-by-default model" (answers.databricks.com, 2026-08-27). Everything a user can reach, they were granted. Two documented exceptions keep new workspaces usable, and both are worth naming precisely, because they are what makes a "locked down" estate leak in its first week:

1. The main catalog: "All users have USE CATALOG on the main catalog by default" (privileges reference, Oct 8, 2026).
2. The workspace catalog: "All workspace users receive the USE CATALOG privilege on the workspace catalog. Workspace users also receive the USE SCHEMA, CREATE TABLE, CREATE VOLUME, CREATE MODEL, CREATE FUNCTION, and CREATE MATERIALIZED VIEW privileges on the default schema in the catalog" (manage-privileges, accessed 2026-10).

4.1 Bindings are a second, independent gate

"By default, a catalog is accessible from all workspaces attached to the same metastore. You can restrict this by binding the catalog to specific workspaces, optionally as read-only. Workspace binding supersedes individual privilege grants. Even a user with an explicit SELECT grant cannot access an object in a catalog that is not bound to their workspace.": securable-objects, Oct 8, 2026

Bindings exist for catalogs, storage credentials and external locations (workspace-catalog binding page, Apr 29, 2026), which is the mechanism that keeps a production credential from being usable in a sandbox workspace. The same page notes access "from an unbound workspace is denied, even for users with explicit privilege grants on the catalog".

A second default worth knowing: "A metastore admin must enable external data access for each metastore you need to access externally. This option is disabled by default to prevent unauthorized external access" (docs, Enable external data access, May 14, 2026).

4.2 Worked example E3: three broken grants, diagnosed

Pattern (b) is the one that generates tickets: a SELECT grant is a key to a door, and USE CATALOG/USE SCHEMA are the corridor that leads to it: without the corridor, the key is unused.

5. Storage credentials: your IAM role, wrapped in a securable

On AWS, "a credential is a securable object representing an AWS IAM role" (Databricks SQL credentials reference, accessed 2026-10), and the storage-credential flavour is specifically the one that reaches S3:

"The first is a storage credential, which specifies an AWS IAM role that allows access to the S3 location. You need this storage credential for the second required object: an external location, which defines the path to your S3 storage location and the credentials required to access that location.": Databricks docs, Connect to cloud object storage using Unity Catalog / S3 external location (Sep 11, 2026)

What the object actually stores (Storage Credential API, accessed 2026-10):

Three hard rules on the AWS side:

• The external ID has been mandatory since 2023-11-30. "IAM policies for storage credentials now require an external ID: November 30, 2023. To create a storage credential, you now must add the external ID for your storage credential in the IAM role's policy in AWS" (release notes, Nov 2023). The docs' trust example carries "sts:ExternalId": "0000" as a placeholder for your returned value.
• The role must live in the same AWS account as the bucket, and "You must have iam:CreateRole permissions to create the IAM role"; automated setup "uses AWS IAM temporary delegation" and is the recommended path for "most deployments" (S3 external-location pages, Sep 11 / Aug 7, 2026).
• Self-assuming roles are enforced. "Databricks blocks new and existing storage credentials based on IAM roles that are not self-assuming" (S3 connect page, Sep 11, 2026). Trust is a cross-account sts:AssumeRole into Databricks' fixed Unity Catalog master role.

5.1 Worked example E4: the trust policy, line by line

Reviewer's checklist: (1) the principal is the Databricks Unity Catalog master role for the commercial partition 414351767826 (GovCloud uses different account IDs: lesson 02); (2) sts:ExternalId equals the value Databricks returned for this credential, not a made-up string; (3) the role's permission policy scopes s3:* to your prefix only; (4) if the bucket uses SSE-KMS, add kms:Decrypt, kms:Encrypt, kms:GenerateDataKey* on that key to the role policy and update the key policy to allow the role with kms:ViaService: s3.<region>.amazonaws.com (docs, Configure encryption for S3 with KMS, accessed 2026-10).

Legacy contrast, in one line: an instance profile attaches an IAM role to a cluster, so everyone using that box inherits it; a storage credential attaches the role to a securable governed by grants. Databricks positions service credentials, the sibling object for AWS Glue and Secrets Manager, as "the Unity Catalog alternative to instance profiles, with the advantage that access is not tied to a specific compute resource" (docs, accessed 2026-10), and labels instance-profile KMS configuration "legacy configuration".

• 📚 Did you know? OAuth 2.0 machine-to-machine authentication, service-principal client credentials, one-hour access tokens, up to five OAuth secrets per service principal each valid up to two years, is how a service principal authenticates to Databricks (API, CLI, JDBC). It is not how Unity Catalog reaches S3: on AWS, storage access is always IAM-role vending with an external ID (OAuth M2M + CLI authentication docs, accessed 2026-10). Teach M2M as the identity layer for jobs calling Databricks, and IAM roles as the identity layer for Databricks calling S3.

6. External locations: credential plus path, governed by grants

"An external location is a securable object that combines a storage path with a storage credential that authorizes access to that path.": Databricks SQL external-locations reference (accessed 2026-10)

Privileges on it decide who may touch files: READ FILES to read directly, WRITE FILES to write (securable-objects, Oct 8, 2026), plus CREATE EXTERNAL TABLE, CREATE EXTERNAL VOLUME, EXTERNAL USE LOCATION and BROWSE. Two structural rules cause most CREATE EXTERNAL LOCATION failures:

• No nesting. "The storage path of any external location cannot be contained within another external location's storage path"; "External locations can't overlap other external locations" (external-locations reference; Path rules and access, accessed 2026-10).
• Both gates for creation. "CREATE EXTERNAL LOCATION: Create an external location that associates a cloud storage path with a storage credential. Required on both the metastore and the storage credential" (privileges reference, Oct 8, 2026).

And the operational fact that surprises everyone: flipping a location to read-only "can take up to 60 minutes … to take effect across all compute resources", while the flag itself "Prevents users from writing to files in those external locations, regardless of any write permissions granted by the IAM role that underlies the storage credential" (manage-external-locations, accessed 2026-10).

6.1 Worked example E5, the worked setup: credential → location → grants → external table

The centrepiece of this lesson. Students fill in their own role ARN and bucket; everything else is documented syntax (credentials reference; external-locations reference; S3 manual page, Aug 7, 2026).

Variant A: read-only landing zone. Mark the location read-only in Catalog Explorer (or the API's read_only flag), then re-test writes after the 60-minute propagation window; do not declare victory at minute five.

Variant B, SSE-KMS bucket. Add the three KMS actions to the storage-credential role policy, allow the role in the key policy with kms:ViaService, and set the encryption algorithm on the external location, a documented option since 2024-10-28 (release notes, Oct 2024).

Variant C: the group that should never write. Grant READ FILES only, and rely on the location's read-only mode plus binding to a production workspace as belt-and-braces: cloud administrators "often bind storage credentials to specific workspaces to ensure production credentials are only used to create external locations within production workspaces" (manage-storage-credentials, accessed 2026-10).

6.2 Interactive check: fill the terms

• 📚 Did you know? File-event plumbing for streaming over external locations is provisioned for you in AWS: the documented IAM policy resources for file events include arn:aws:sqs:*:*:csms-* and arn:aws:sns:*:*:csms-*: the managed file-event queues and topics that drive notification-based ingestion (manage-external-locations, accessed 2026-10). And governance of non-tabular data increasingly goes through volumes instead of raw path grants: "When you define an external volume, cloud URI access to data under the volume path is governed by the privileges granted on the volume, not the privileges granted on the external location where the volume is stored" (docs, What are Unity Catalog volumes?, accessed 2026-10).

7. Managed versus external: same governance, different lifecycle

"Managed assets: Unity Catalog controls both governance (access control, auditing, lineage) and the underlying file storage lifecycle (file optimization, how they are organized, and when they are deleted). External assets: Unity Catalog controls governance only.": Databricks docs, Managed versus external assets in Unity Catalog (Sep 11, 2026)

Resolution order for managed storage: "Managed storage locations at lower levels in the hierarchy override storage locations defined at higher levels… Databricks recommends that you assign managed storage at the catalog level for logical data isolation" (docs, Specify a managed storage location, accessed 2026-10). Hashed subdirectories keep tenants apart: schemas land under <storage-root>/__unitystorage/schemas/<uuid> and catalogs under <storage-root>/__unitystorage/catalogs/<uuid>.

7.1 Worked example E6, placement math: where does the table actually land?

Setup: metastore root s3://acme-uc/metastore-root/; catalog prod created with MANAGED LOCATION 's3://acme-uc/prod/'; schema prod.sales created without a managed location; a second schema prod_sandbox created with MANAGED LOCATION 's3://acme-uc/sandbox/'.

Managed versus external DDL, side by side (Q5):

8. Migrating off hive_metastore: the documented ladder

Unity Catalog is additive, so migration starts from coexistence: the legacy metastore "appears as a top-level catalog called hive_metastore", and queries can join a legacy table with a Unity Catalog table in one statement while you cut over (docs, Work with the legacy Hive metastore, accessed 2026-10: a behaviour this course calls dual-read, for the record, not a docs product name).

Two documented paths exist (same page): "Upgrade all tables registered in the Hive metastore to Unity Catalog" or "Federate your Hive metastore to Unity Catalog using Hive Metastore federation for a more gradual approach."

The recommended sequence is explicit:

"The recommended approach is to first federate your Hive metastore or AWS Glue catalog as a foreign catalog, then upgrade the foreign tables in place. This two-step process allows you to migrate tables without data movement while preserving table history, configuration, permissions, and views.": Databricks docs, Upgrade a Databricks workspace to Unity Catalog (Jul 30, 2026)

The in-place conversion command is ALTER TABLE <foreign_catalog>.<schema>.<table_name> SET MANAGED, which "Databricks recommends… to unlock Unity Catalog predictive optimization, which includes automatic maintenance (compaction, clustering, vacuuming)" (same page).

8.1 The four tools, by what they produce

Constraints to memorise: "If you want to perform a bulk upgrade of Hive tables to Unity Catalog managed tables, this is your only option": UCX (migrate page); UCX "is a public GitHub repo and not supported directly by Databricks" and carries no SLA; and "Hive managed tables must be in Delta or Parquet format to be upgraded" (UCX page, accessed 2026-10).

8.2 Worked example E7: pick the tool per table type

Reading: rows 1 and 4 are the cheap ones (metadata only, SET MANAGED where supported); row 2 is the expensive one: DBFS-root storage forces a physical DEEP CLONE, which is exactly why "DBFS root and DBFS mounts are deprecated and not recommended by Databricks. New accounts are provisioned without access to these features" (docs, Best practices for DBFS and Unity Catalog, accessed 2026-10). Row 6 is the safety net: because hive_metastore keeps working until you disable it, old and new can be joined in a single query while jobs migrate one by one.

8.3 Interactive check: order the cutover

• 📚 Did you know? Unity Catalog gives you a recovery window the Hive metastore never had: in Unity Catalog "you can UNDROP a managed table for 7 days, and data is permanently deleted within 30 days", whereas "When you drop a managed table in Hive metastore, all underlying data files are deleted immediately" (docs, Database objects in the legacy Hive metastore, accessed 2026-10). That difference is the single best argument for finishing the migration, and the single best reason to stop treating DROP as an undoable button in legacy scripts.

2026 Updates (as of October 2026)

Real-World Case Studies

Databricks publishes governance outcomes, not grant matrices. Every figure below is customer- or Databricks-claimed, unaudited, with its URL and date so you can check it: the examinable point is the pattern, not the marketing.

Case A, Amgen: from 120+ IAM roles to one or two, via Unity Catalog

⏳ Temporal note: May 2023 figures: a snapshot of that deployment at the time. The mechanism (principals governed in UC, roles collapsed) is still the documented pattern in Oct 2026.

Case B, Rivian: one governed catalog across teams on AWS

⏳ Temporal note: mixed dates: an undated customer page (accessed Oct 2026) plus dated 2024 and 2025 posts. Teach the "centralized catalog across teams" claim as reported, not guaranteed.

Case C, nCino: banking data inside one AWS boundary

⏳ Temporal note: undated customer page accessed Oct 2026; architecture claims are current, the loan-history figure is as published.

Practice Questions