Building a Databricks Data Lakehouse on AWS: Unity Catalog to Production

End-to-end course for building and operating a production Databricks Lakehouse deployed on AWS, Unity Catalog governance, Delta Lake (ACID, CDF, liquid clustering), Auto Loader and Structured Streaming ingestion, Databricks Runtime/Photon transformation, Lakeflow/DLT pipelines, Databricks SQL serverless warehouses and Lakeview, workflows-as-code with asset bundles, data quality expectations, Delta Sharing and Marketplace, performance and DBU cost optimization, CI/CD with Terraform, and a full capstone reference architecture, with architecture diagrams, real customer cases, worked pricing examples, and practice questions.

Intermediate12 Weeks17 LessonsNuniversity
Start

Course Content

  1. 01Lakehouse Architecture on Databricks: Planes, Pillars, and the AWS Substrate
  2. 02AWS Foundation for Databricks: VPC Design, Root Bucket, PrivateLink, KMS and IAM
  3. 03Unity Catalog Fundamentals: Metastore, Hierarchy, Grants, Storage Credentials and External Locations
  4. 04Delta Lake Transactions on AWS: Log Anatomy, Time Travel, CDF, Clustering and VACUUM
  5. 05Ingestion on AWS: Auto Loader, File Events, Kafka/MSK and Idempotent Sinks
  6. 06Databricks Runtime, Photon and Clusters: DBR/LTS, Policies, Pools and Compute Cost Shape
  7. 07Lakeflow Declarative Pipelines (formerly Delta Live Tables): Expectations, Modes and the DLT vs Jobs Decision
  8. 08Databricks SQL Warehouses and the Serving Layer: Serverless, Scaling, Dashboards, BI and Federation
  9. 09Lakeflow Jobs (formerly Databricks Workflows): Multi-Task Jobs as Code, Triggers, Retries and the Scheduler Decision Matrix
  10. 10Data Quality in the Databricks Lakehouse: Constraints, Expectations, Quarantine and Monitoring
  11. 11Unity Catalog Security: Row and Column Controls, Tags, Auditing, Lineage and Secrets
  12. 12Data Sharing on Databricks on AWS: OpenSharing, Marketplace, and Lakehouse Federation
  13. 13Structured Streaming on the AWS Lakehouse: Triggers, Checkpoints, foreachBatch and Exactly-Once
  14. 14Performance Optimization on Databricks/AWS: Liquid Clustering, File Layout, Skew and Warehouse Tuning
  15. 15Cost Optimization and DBU Economics on AWS
  16. 16CI/CD for the Databricks Lakehouse on AWS: Declarative Automation Bundles, Terraform and GitHub Actions
  17. 17Capstone Reference Architecture: Observability, ML Activation and the 90-Day Plan for Databricks on AWS

Lessons: 17 · 12 Weeks