CI/CD for the Databricks Lakehouse on AWS: Declarative Automation Bundles, Terraform and GitHub Actions

Every lakehouse in this course eventually meets the same cliff: the notebooks work, the pipelines are green, and then someone hand-clicks a job into the production workspace on a Friday afternoon, or worse, someone's departure from the company takes the 3 a.m. schedule with them. The cliff is the deployment seam, and the fix is boring on purpose: everything that lives in the workspace is declared in Git, reviewed in a pull request, validated by CI, and deployed by a service principal that is not any human being.

This is now the documented position, not folklore. "Declarative Automation Bundles are the recommended approach to CI/CD on Databricks" (docs.databricks.com/aws/en/dev-tools/ci-cd, 2026-07-10), and the same page's cloud-specific flow for AWS reads "GitHub Actions and Declarative Automation Bundles or Terraform". Note the name you must learn: the product formerly called Databricks Asset Bundles is documented since 2026 as Declarative Automation Bundles (bundles overview page, last updated 2026-09-11): same CLI verbs, same databricks.yml, new name. And the filename matters more than you think: "A bundle must contain one (and only one) configuration file named databricks.yml at the root of the bundle project folder" (bundles settings, 2026-03-16). There is no bundle.yaml. Ever.

Every fact in this lesson is stamped with its Databricks or AWS source and dated to October 2026; anything the docs do not say is left out rather than guessed.

In this lesson you will:

• separate the four layers of the Databricks-on-AWS delivery stack and assign exactly one owner to every resource class (no job defined twice);
• read a full databricks.yml, bundle name, include, artifacts, variables, workspace, run_as, permissions, resources, targets, and deploy it through the validate → deploy → run loop;
• apply development mode vs production mode semantics, the presets that override them, the substitution vocabulary, and the exact five-step variable-resolution precedence;
• split deploy identity from run identity with run_as service principals, including the "only jobs and pipelines are supported" rule when the two identities differ;
• provision the AWS substrate with Terraform: the databricks.mws alias module (credentials, networks, storage configuration, PrivateLink objects, workspace) kept in a separate module from the workspace-level resources (storage credentials, external locations, grants), with S3 + DynamoDB remote state;
• build the PR pipeline in GitHub Actions with databricks/setup-cli, bundle validate (validation runs as part of every deploy), and workload identity federation (OIDC) so no personal access token ever enters CI;
• test at five layers: local pytest, Databricks Connect, databricks-labs-pytester fixtures, the DBR 18.1+ in-pipeline test framework, and dbt SlimCI;
• design the environment topology, one metastore per region, 3 to 10 catalogs, ISOLATED production binding, 2 to 3 workspaces, and hand-compute a monthly CI DBU sketch at the published Oct 2026 floors;
• study three named AWS cases (NAB, Addepar, Mastercard), read the 2026 update box, and answer twelve exam-style questions plus four interactive checks.

---

1. The four-layer stack: one owner per resource class

The single most expensive CI/CD mistake on Databricks is not a missing test; it is defining the same job in two systems and letting them fight. Before any YAML, fix the ownership model. Databricks' documented stack has four layers that must stay distinct:

Two rules make the diagram hold. First, one owner per resource class: jobs, pipelines and dashboards belong to bundles; the workspace, storage credentials, external locations and metastore grants belong to Terraform. Defining a databricks_job in Terraform and the same job in a bundle creates import/ownership fights that surface as mysterious diffs, not as clean errors. Second, bundles are the recommendation for everything inside the workspace, while Terraform is the documented instrument for provisioning: "You use the Databricks Terraform provider to provision Databricks workspaces as well as the AWS Provider to provision required AWS resources for these workspaces" (Terraform provider docs, 2026-09-11).

📚 Did you know? The bundle's own config can declare bundle.engine: terraform | direct, and "The default value is direct" (configuration reference, 2026-08-18, added in CLI 0.295.0), with release notes confirming "Direct deployment engine is GA" and "Deploy without Terraform". That internal engine is plumbing for bundle deploys only; it is emphatically not your infrastructure Terraform state, and confusing the two is a documented team antipattern (pitfall 5 in the CI/CD digest). Older CLIs used the Terraform engine by default; current CLIs do not.

---

2. Bundle anatomy: the one file, its keys, and the resources map

2.1 The filename is a validation gate

Copy this sentence onto your team wiki: a bundle root contains exactly one databricks.yml (bundles settings, 2026-03-16). Additional configuration is pulled in via the include array of globs, and "include globs are relative to the databricks.yml that declares them". bundle validate fails immediately on the classic copy-paste from other tools that assumes bundle.yaml; there is no documented alias, now or historically.

2.2 Top-level keys (the whole surface, as of Oct 2026)

The supported lookup object types (fetch an existing workspace object by name instead of declaring it) are: alert, cluster_policy, cluster, dashboard, instance_pool, job, metastore, notification_destination, pipeline, query, service_principal, warehouse (variables docs, 2026-03-16).

A healthy project layout for one repository hosting one project (Databricks developer best practices, 2026-06-11: "use a single repository for all of your code… If you have multiple bundles for separate deployment lifecycles, keep them in a single repository"):

2.3 Worked example E1: the minimal production-shaped bundle

This is the skeleton from the Databricks bundles docs pattern (settings + deployment-modes + run-as pages, all accessed Oct 2026): one job, one schema, two targets, an SP run identity in prod.

The local loop is three commands, and it is the same loop CI runs:

Three details in E1 that exam questions love: notebook_path is relative to the bundle (never hard-code /Workspace/Users/... paths from your laptop, they break under a different root_path); the wheel is declared where the job consumes it (libraries: - whl:) while the build command lives under artifacts:; and the prod schema variable is deliberately not set per user, prod schemas come from the target's variables, not from whoever deployed.

📚 Did you know? databricks bundle init scaffolds a project from a template, and the release notes record that "The default-python template now uses uv" and "The default-python template defaults to use serverless" (bundles release notes, 2026-04-29). Two companion verbs save days of migration: bundle generate reverse-engineers YAML from resources that already exist in the UI ("bundle generate job --as-code" against a click-built job), and bundle deployment bind links your config to a pre-existing workspace object when names collide. bundle schema (CLI ≥ 0.218.0) emits the JSON schema your IDE uses for completion and inline validation.

---

3. Targets, modes, presets, variables: the safety rails

3.1 Development mode vs production mode

Every practical bundle declares targets.dev and targets.prod (optionally staging), exactly one marked default: true. The mode: on each target is not decoration: the two modes have opposite safety semantics (deployment-modes docs, last updated 2026-09-11):

Production mode also "recommends that you deploy your bundle to an immutable, read-only folder" (the immutable_folder setting). Promotion, in one sentence: the same immutable Git commit is redeployed to the next target: nothing is rebuilt, and data promotion happens by running the job against that target's catalog.

3.2 Presets tune the mode; resources override the presets

Presets customize the mode defaults: name_prefix, pipelines_development, trigger_pause_status, jobs_max_concurrent_runs, tags, and (release notes, 2026-04-29) artifacts_dynamic_version. The precedence inside the platform is one sentence: "if both mode and presets are set, presets override the default mode behavior, and settings of individual resources override the presets" (deployment-modes, 2026-09-11).

3.3 Substitutions, variables, and the resolution order

Built-in substitutions resolve at deploy time: ${bundle.name}, ${bundle.target} (use this, not the older ${bundle.environment}), ${workspace.host}, ${workspace.current_user.userName}, ${workspace.current_user.short_name}, ${workspace.file_path}, ${workspace.root_path}, and even ${resources.jobs.<key>.id} for cross-references. Custom variables are ${var.<name>} and resolve by walking this list, stopping at the first hit (variables docs, 2026-03-16):

1. --var="name=value" on the CLI
2. Environment variables starting with BUNDLE_VAR_ (e.g. BUNDLE_VAR_catalog=sales_prod)
3. variable-overrides.json under .databricks/bundle/<target>/
4. targets.<name>.variables
5. Top-level variables.<name>.default

3.4 Worked example E2: overrides in practice

Same bundle as E1, three different promotion moments:

⚠️ WARNING: dev-mode side effects cut both ways. Forgetting mode: development on the dev target means a PR-triggered deploy can unpause real schedules against shared resources; expecting a dev-deployed job's exact name to still exist in prod fails, because dev renamed everything [dev <user>]. And remember what dev mode buys you: paused schedules, development: true pipelines and concurrent runs exist so that twenty pull requests can deploy to one shared dev workspace without firing each other's calendars (deployment-modes, 2026-09-11). If parallel PRs collide on the same resource keys, give each PR its own root_path/variables or serialize with a GitHub concurrency: group.

---

4. Identity: deploy as one thing, run as another

4.1 The run_as rules that decide what your pipeline may do

run_as can be set to a user_name or a service_principal_name, and "Non-admins can only set this field to their own email" (run-as docs, last updated 2026-10-07). Four rules matter in production:

1. "Setting a run_as identity to a service principal for production targets is the most secure way of running a production workflow"; it "decouples the permission to run the production workflow from the identity that created or deployed the bundle".
2. If the deploy identity differs from run_as, "only jobs and pipelines are supported": any other resource type in the bundle then fails to deploy cleanly.
3. Job-level run_as may accept a group_name; bundle/target-level run_as may not.
4. Model serving endpoints do not support run_as at all.

4.2 Worked example E3: the identity split

What happens at 3 a.m. when the on-call engineer is asleep: the prod job still runs, because its identity is the service principal, and the run is auditable as that SP in system.lakeflow.job_run_timeline (CI/CD digest, lineage section; cross-ref lesson 09). What happens when Alice leaves the company: nothing in prod breaks, because Alice was never prod's identity: only dev's.

📚 Did you know? OAuth machine-to-machine tokens are short by design: "Each access token is valid for one hour, after which a new token is automatically requested", a service principal may hold up to five OAuth secrets, and each secret is valid "up to two years (maximum 730 days)", and secrets can be scoped to APIs such as sql or jobs rather than all-apis (OAuth M2M docs, accessed Oct 2026). Scope your CI secret to the APIs the pipeline touches; a scoped secret that leaks is a smaller incident than an all-apis one.

---

5. Terraform for the substrate: two providers, two modules, separate state

5.1 Worked example E4: the workspace-provisioning module (provider = databricks.mws)

Workspace creation on AWS uses account-level resources under a dedicated provider alias. The official guide is unambiguous about the shape: "Initialize provider with alias = \"mws\" and use provider = databricks.mws for all databricks_mws_* resources. We require all databricks_mws_* resources to be created within a dedicated terraform module of your environment. Usually, this module creates VPC and IAM roles as well" (AWS workspace guide, accessed Oct 2026).

The VPC itself, private/public subnets, NAT, the S3 gateway endpoint, and STS/Kinesis interface endpoints, comes from the AWS provider (the guide uses terraform-aws-modules/vpc/aws plus its endpoints module). For PrivateLink, the verified example set (aws-databricks-modular-privatelink in databricks/terraform-databricks-examples, accessed Oct 2026) adds aws_vpc_endpoint for backend REST + relay, then databricks_mws_vpc_endpoint, databricks_mws_private_access_settings (with public_access_enabled), and wires private_access_settings_id into databricks_mws_workspaces with pricing_tier = "ENTERPRISE".

5.2 Worked example E5: the workspace-management module (workspace-level provider)

The hard rule from the same guide: "Code that creates workspaces and code that manages workspaces must be in separate terraform modules": separate state, separate provider aliases (databricks.mws vs a workspace-level provider that, in CI, authenticates with OAuth M2M env vars DATABRICKS_HOST + DATABRICKS_CLIENT_ID + DATABRICKS_CLIENT_SECRET). Mixing them means an apply can authenticate against the wrong endpoint.

Facts you must carry out of that block: databricks_external_location "can only be used with a workspace-level provider" and "requires databricks_storage_credential"; databricks_storage_credential on AWS is the aws_iam_role { role_arn = ... } form, importable by name at workspace level or <metastore_id>|<name> at account level; and credentials/locations carry an isolation_mode (ISOLATION_MODE_ISOLATED / OPEN) that interacts with workspace binding (lessons 02 and 03).

⚠️ WARNING, databricks_grants is authoritative and will silently strip other people's grants. "This resource is authoritative for grants on securables. Configuring this resource for a securable will OVERWRITE any existing grants and changes made outside of Terraform will be reset" (Terraform Registry, resources/grants, published 2026-10-06). If anyone has granted access through the UI or through Unity Catalog SQL since the last apply, the next apply removes it. Use the singular databricks_grant for incremental management when other actors share ownership, and remember the account-level companion resources databricks_metastore + databricks_metastore_assignment (one metastore per region) belong to the account module, not the workspace module.

5.3 Versions and state: two hygiene rules with receipts

The provider moves fast: "several releases per month" (Registry releases page, accessed Oct 2026): v1.137.0 (2026-10-06), v1.136.0 (2026-10-05), v1.135.0 (2026-09-28), v1.134.0 (2026-09-23), v1.128.0 (2026-08-19), with v1.65.0 (2025-02-04) a useful yardstick of how far the surface has moved in 20 months. Pin with a pessimistic constraint (version = "~> 1.137"); never float to latest. State hygiene, from Databricks' own examples repo: "We recommend using remote storage, like S3, for state storage, instead of using default local backend" with backend "s3" { bucket, key, region, dynamodb_table for locks, encrypt = true }, and "You should create the infra for remote backend in another Terraform Project" (aws-remote-backend-infra). A laptop-local terraform.tfstate is unrecoverable drift and a secret leak, because state stores resource attributes.

---

6. GitHub Actions: the PR pipeline and the federation that secures it

6.1 The documented pattern

The official GitHub Actions page (dated 2026-03-16) lists exactly one official action: databricks/setup-cli, "A composite action that sets up the Databricks CLI in a GitHub Actions workflow", followed by raw CLI verbs. The flow it documents: PR against main → "Deploy bundle" job runs databricks bundle deploy with DATABRICKS_BUNDLE_TARGET: dev → "Run pipeline update" job runs the job/pipeline. "Bundle validation is automatically performed as part of this deployment. If validation fails, this workflow fails." Production is the same shape on push to main with target prod, serialized with concurrency: 1. Trunk-based branching plus commit-hash versioning is the recommended Git posture (CI/CD flows page, 2026-07-10).

6.2 Worked example E6: the full PR + prod workflow with OIDC

Java/Scala estates swap one step for a versioned artifact upload to a Unity Catalog Volume before deploy: the documented shape: mvn clean verify → databricks fs cp target/my-app-1.0.jar dbfs:/Volumes/artifacts/my-app-${{ github.sha }}.jar --overwrite → bundle validate → bundle deploy --target prod, with the deploy gated by if: github.event_name == 'push' && github.ref == 'refs/heads/main' (GitHub Actions docs, 2026-03-16). Volumes, not legacy dbfs:/mnt mounts, are the governed artifact store (lesson 12).

6.3 Workload identity federation: no secret in the vault at all

"Databricks OAuth token federation, also known as OpenID Connect (OIDC), allows your automated workloads running outside of Databricks to securely access Databricks without Databricks secrets" (provider-github docs, 2026-10-02), and "Databricks strongly recommends using workload identity federation to authenticate to Databricks from automated workloads whenever possible" (oauth-federation-provider, 2026-06-16). Supported IdPs beyond GitHub Actions: Azure DevOps, GitLab CI/CD, CircleCI, AWS IAM workloads, Jenkins, Terraform Cloud, Bitbucket Pipelines.

The verified wiring: an SP federation policy with issuer https://token.actions.githubusercontent.com and a subject Databricks "recommends using Environment", e.g. repo:my-github-org-or-user/my-repo:environment:Prod (reusable workflows use subject_claim: job_workflow_ref); the workflow then needs only three env vars: DATABRICKS_AUTH_TYPE: github-oidc, DATABRICKS_HOST, DATABRICKS_CLIENT_ID (the SP application ID): plus permissions: id-token: write. And the docs call out the trap explicitly: "Set permissions: id-token: write on the calling workflow, not the reusable workflow." The SDKs and CLI exchange the OIDC token for a Databricks OAuth token automatically: nothing to rotate, nothing in the repo's secret store, and a federation subject that must match the token's sub claim exactly (a policy written for environment:Prod rejects tokens from workflows lacking environment: Prod).

⚠️ WARNING: never put a personal access token in CI, and never paste any token into workflow YAML. A PAT couples production deployments to one employee's account lifecycle and over-privileges the pipeline; when that person leaves, the pipeline dies with their account, and the audit trail points at a human instead of a service. Databricks' own guidance: "As a security best practice, Databricks recommends that you do not enter a Databricks access token directly into the body of a GitHub Actions file. You should register the Databricks access token with GitHub Actions by using GitHub encrypted secrets" (service-principals docs, 2026-01-24), and that is the floor of the ladder (SP token as DATABRICKS_TOKEN), not the ceiling. Prefer OIDC federation; fall back to OAuth M2M (DATABRICKS_CLIENT_ID/DATABRICKS_CLIENT_SECRET in GitHub encrypted secrets); treat an SP access token as a last resort. And debug auth failures before rewriting YAML: "No conflicting variables set such as DATABRICKS_TOKEN, DATABRICKS_USERNAME. Run env | grep DATABRICKS and unset conflicts" (OAuth M2M docs): a stale token env var on a self-hosted runner shadows OIDC/M2M every time.

The legacy documented variant of E6 swaps the env: federation block for DATABRICKS_TOKEN: ${{ secrets.SP_TOKEN }} (an SP-generated access token in a GitHub encrypted secret): still in the docs, still a long-lived secret, and the wrong default for a 2026 design review. For the human approval gate, GitHub Environments carry the vars and "lets you require manual approval before deployments run": configure the prod environment with required reviewers (Deployment protection rules), and pin the workflow's environment: key per target, exactly as E6 does (Databricks Apps CI page, accessed Oct 2026, same pattern).

📚 Did you know? databricks bundle deploy performs validation for you, "Bundle validation is automatically performed as part of this deployment. If validation fails, this workflow fails" (GitHub Actions docs, 2026-03-16), which is why E6's explicit bundle validate step is belt-and-braces rather than load-bearing. Keep it anyway: it fails faster, before any sync begins, and it gives reviewers a named gate in the checks UI.

---

7. Testing: five layers, one pyramid, all in the pipeline

CI/CD on a lakehouse is only as good as what it can prove. The verified stack (CI/CD digest section 9, sources dated 2023 to 2026):

Two constraints stop teams from mis-using layer 4 as a CI hook. The in-pipeline test framework "must run on Databricks Runtime 18.1 or above", only in triggered mode, "Editor-only execution", "Python tests only": CI cannot call it headlessly today, so CI uses the external pytester pattern instead (unit-testing docs, accessed Oct 2026; community blog 2026-07-28 notes the framework "is currently in preview and heading into beta": check current status before betting prod CI on it). And its isolation has holes worth knowing before you trust a green run: test_spark "redirects table operations to a temporary test schema" by name: "reads or writes by path or connector… act on production data", the event_log() table-valued function is off-limits from tests, and "Mock data does not inherit row filters or column masks", so green tests do not prove governed behavior.

The older but still-canonical integration pattern for Delta Live Tables/Lakeflow is different: add extra expectations with fail so the pipeline itself is the assertion: "recommended approach… no additional compute resources" (DevOps for DLT blog, 2023-04-28). dbt teams get the SQL-side equivalent in SlimCI: "dbt tracks the state of what's running in your production environment… only the modified data assets in your pull request and their downstream dependencies are built and tested in a staging schema", with a dedicated test catalog and a dbt_test_sp service principal (dbt docs, 2026-08-19 and 2026-10-06).

7.1 Worked example E7: what a month of CI compute costs

Databricks publishes no surcharge line item for bundle deploys or GitHub Actions usage; what the pipeline triggers is regular DBU-metered compute. At the published floors (databricks.com/pricing, snapshot 2026-10: jobs classic from $0.15/DBU, DBSQL from $0.22/DBU):

Two cost levers belong to the pipeline design itself: dev-mode's paused schedules mean a PR deploy never doubles a prod calendar, and dbt SlimCI's "running and testing only modified models… reduces compute spend" (dbt docs, 2026-08-19) keeps PR runs proportional to the diff, not the estate.

Read the bar heights as a design argument, not a quote: the prod nightly dwarfs PR runs by an order of magnitude, which is why auto-stop, scaling policy and right-sizing (lesson 15) matter far more to the bill than shaving minutes off PR validation; while the PR runs are what keep the nightly from shipping a regression.

---

8. Environments: one metastore, ISOLATED catalogs, 2 to 3 workspaces

Bundles give you targets; Unity Catalog gives those targets something to point at. The documented guidance (UC best practices 2026-06-03, developer best practices 2026-06-11, lakehouse deployment guide 2026-06-26):

• "You can have only one metastore per region. All workspaces in that region share that metastore", and "Metastores provide regional isolation but are not intended as default units of data isolation. Data isolation typically begins at the catalog level."
• Use separate catalogs per SDLC scope (dev / staging / prod, or hybrid sales_dev/sales_prod), keeping "typically 3 to 10 per metastore"; "Use your single-region metastore for all software development lifecycle scopes."
• "Bind the production catalog in ISOLATED mode to the production workspace only… ensures that production data is unreachable from development or staging environments, even if an identity is misconfigured."
• "Always assign ownership of production catalogs and schemas to groups, not individual users."
• "Each bundle should cover all environments for a given project… rather than using separate bundles per environment."
• Workspace topology scales with team size: ≤5 data engineers → 2 workspaces (dev, prod); 5+ → 3 (add staging); regulated industries → "Physically isolate workspaces and cloud accounts". "One metastore per environment" exists but is "less common".

The bundle-to-catalog seam is a pair of variables: catalog: sales_dev on the dev target, catalog: sales_prod on prod, referenced everywhere as ${var.catalog}: the same YAML, the same code, two governed destinations. Promotion is then literally a redeploy: same commit, next target, different catalog.

---

9. Lineage of code → jobs: the audit chain you can query

The chain auditors should be able to walk, end to end (CI/CD digest section 12):

1. Git commit (trunk-based, commit-hash versioned artifacts).
2. PR checks: unit tests, bundle validate, dev deploy, bundle run on sample data; status posted back to the PR (dbt posts its own status the same way).
3. Merge → optional GitHub Environment approval (required reviewers on prod).
4. bundle deploy -t prod as the deploy identity, with run_as set to the prod service principal.
5. Job/pipeline objects in the prod workspace, owned by that SP, tagged with bundle name/target via presets, their runs recorded with identity_metadata.run_as in system.lakeflow.job_run_timeline, and their compute metered in system.billing.usage.
6. Terraform plan/apply history in the S3+DynamoDB backend covering the slower-moving substrate: workspace, storage credentials, external locations, grants.

If any link is missing, a job with no bundle tag, a run whose identity is a departed human, a grant change with no Terraform plan, the fix is process, not tooling: add the missing layer, do not add a second owner.

---

10. The traps: thirteen confusions this lesson will not let you keep

1. bundle.yaml does not exist. Exactly one databricks.yml at the bundle root; include globs are relative to it.
2. A personal access token in CI. Never; at best an SP token in an encrypted secret; prefer OIDC federation.
3. Stale auth env vars. A leftover DATABRICKS_TOKEN on a self-hosted runner shadows OIDC/M2M: env | grep DATABRICKS before you debug anything else.
4. Dev-mode amnesia. No mode: development on dev → PR deploys can unpause prod schedules; expecting prod to keep dev's [dev user] names → broken references.
5. Two different "Terraforms." The bundle's internal engine (default direct) is not your infra state; and one job defined in both Terraform and a bundle is an ownership fight, not redundancy.
6. Laptop Terraform state. Remote S3 backend + DynamoDB lock + encrypt = true, backend infra in its own project.
7. databricks_grants clobber. Authoritative overwrite of every grant on the securable; use databricks_grant for incremental ownership.
8. Host/subject mismatches. Workspace URL (no /api) for workspace APIs, https://accounts.cloud.databricks.com for account APIs; federation sub must match the token claim exactly.
9. External location scope. Workspace-level only; create storage credentials and locations only after metastore assignment, and mind isolation_mode.
10. dbfs: paths for artifacts. Docs CI examples upload to UC Volumes (/Volumes/...); legacy mounts are not governed the same way, and absolute dev notebook_paths break under prod root_path.
11. Pipeline test-mode illusions. Name-based redirection only; path/connector reads and event_log() still hit production; editor-only, so CI uses pytester.
12. PR-to-shared-dev concurrency. Dev mode drops the deployment lock and enables concurrent runs for iteration speed: parallel PRs on the same bundle/target need per-PR paths or a concurrency: group.
13. run_as gaps. Deploy-as-Alice/run-as-SP works for jobs and pipelines only; other resource types fail; non-admins can set run_as only to themselves.

---

Comparative Verdict: Declarative Automation Bundles vs Terraform-only vs Git-folder-only (AWS, as of Oct 2026)

• Declarative Automation Bundles, the documented recommendation for everything inside the workspace. "Declarative Automation Bundles are the recommended approach to CI/CD on Databricks" (docs, 2026-07-10). One databricks.yml plus include globs declares jobs, Lakeflow pipelines, SQL warehouses, schemas, catalogs, secret scopes, dashboards, alerts and permissions, things a Git folder structurally cannot declare, with dev/prod targets, mode guardrails, run_as identities, preset-based tagging, variable substitution, artifact builds and validation baked into every deploy. The direct deployment engine (default since CLI 0.295.0, GA per release notes 2026-04-29) removed the old Terraform-engine friction. Weakness: it does not provision workspaces, VPCs, root buckets or metastore assignments, Layer 1 is simply out of its scope.
• Terraform-only: the substrate king, the workload laggard. The databricks.mws module plus the AWS provider is the verified way to create credentials, networks, storage configurations, PrivateLink objects and the workspace itself, with plan/apply history in encrypted S3 state; the workspace-level resources (databricks_storage_credential, databricks_external_location, databricks_grant(s), cluster policies) are equally first-class. Weakness for workloads: every notebook edit becomes a Terraform cycle, notebook paths and artifacts are clumsier, dev-mode conveniences (paused schedules, [dev] prefixes, per-user schemas) do not exist, and an authoritative databricks_grants block will strip grants other teams made in the UI. Use it where audit and idempotent infra matter most; do not use it as your inner loop.
• Git-folder-only, the documented fallback, and a fallback only. "Use this approach when you don't have access to external CI/CD pipelines" (docs, 2026-07-10): a workspace-mirrored repo plus databricks repos update /Workspace/<path> --branch <branch> on push (WIF-authenticated). Excellent for interactive development with the IDE/Databricks extension. It versions files, not resources, no warehouse, no cluster policy, no secret scope, no permissions, no run_as, no multi-environment promotion. Databricks' own hybrid advice: Git folders for rapid iteration, bundles for anything that reaches staging or prod, in "a single repository" hosting all your bundles.
• Verdict in one line: Terraform owns the substrate (workspace, credentials, external locations, grants) → Declarative Automation Bundles own everything declared inside the workspace (jobs, pipelines, schemas, warehouses) → Git folders are the dev-loop fallback, with exactly one owner per resource class, service-principal identities throughout, and OIDC federation so no long-lived secret exists in CI at all.

---

2026 Updates (as of October 2026)

---

Real-World Case Studies

📋 Case A: National Australia Bank (banking, Cloud: AWS) Challenge. A 26-year-old Teradata warehouse with ~1,000 use cases and 140 reporting suites, plus hand-written Spark/legacy SQL ETL "5,000 to 8,000-line DML and up to 64 union-all per pipeline", maintained inconsistently by 300 to 400 engineers. Stack on AWS. Databricks SQL, Auto Loader, Spark Declarative Pipelines on Lakeflow, Unity Catalog (lineage, masking, access control), and, explicitly listed on the story page, asset bundles for CI/CD, Cloud: AWS on the customer-story page. Reported outcome. "26-year-old Teradata warehouse decommissioned · 456 Use cases migrated to Databricks SQL · 1,200+ Users enabled" (https://www.databricks.com/customers/nab/dbsql, undated, accessed 2026-10); "1,800 Spark Declarative Pipelines running on Databricks Lakeflow · 99.6% Job success rate · 80% Less transformation complexity", end-to-end latency "under 15 minutes", data quality "+38%" (https://www.databricks.com/customers/national-australian-bank-nab, undated, accessed 2026-10). Maps to this lesson. Sections 2 to 4 and 6: NAB is the named AWS example of the whole stack working together: thousands of pipelines can only be operated at a 99.6% success rate when every one is bundle-declared, PR-reviewed and service-principal-run, not click-ops'd. The 80%-less-complexity number is what standardisation looks like when the deployment unit is a repo, not a notebook. ⏳ Temporal note: undated customer-story pages, accessed Oct 2026: reported outcomes at the time of the story, not a 2026 guarantee.

📋 Case B: Addepar (wealth management, Cloud: AWS) Challenge. Engineering and data teams siloed on a platform modelling over $8 trillion in investments, with slow pipeline delivery and rising infrastructure cost, needing a governed AI foundation. Stack on AWS. Databricks running on AWS, Unity Catalog, shared notebooks, Databricks SQL: Cloud: AWS on the story page. Reported outcome. "Addepar models over $8 trillion in investments on Databricks running on AWS… accelerating pipeline delivery by 5x and cutting infrastructure costs by $2 million" (https://www.databricks.com/customers/addepar/aws, undated, accessed 2026-10). Maps to this lesson. Sections 6 and 9: a 5x delivery acceleration is the classic signature of the PR-pipeline loop replacing hand deployment: the pipeline is the delivery mechanism, and the $2M reduction is what unification plus attribution (bundles tagged with cost centers, compute metered in system.billing.usage) looks like on a bill. ⏳ Temporal note: undated page, accessed Oct 2026: the delivery and cost figures are the customer's reported outcome, not platform benchmarks.

📋 Case C: Mastercard (financial services, Cloud: AWS) Challenge. A governance framework that must prove trust, accessibility and compliance across data systems while generative-AI applications move to production. Stack on AWS. Databricks Data + AI Platform on AWS with Unity Catalog governance; two AWS re:Invent 2024 breakouts: "Unified and Open Governance for Data and AI" and "Build and Deploy Production-Ready Generative AI Applications". Reported outcome. "Databricks was highlighted in 4 of the 5 keynotes" at re:Invent 2024 (https://www.databricks.com/blog/databricks-and-aws-partnership-took-reinvent-2024-storm, 2024-12-11). No numeric customer metric is published on that page: cite the sessions, not invented numbers. Maps to this lesson. Sections 4 and 9: governed AI on AWS needs the same identity chain this lesson builds: an SP run_as, OIDC-authenticated CI, and an audit trail through system tables. It is also the evidence-literacy case: session titles are qualitative evidence, and a lesson may say what the sessions covered, never what they measured. ⏳ Temporal note: conference recap dated 2024-12-11: session evidence, not a metrics page.

---

---

Practice Questions