Production dbt — CI/CD, Environments, Scheduling

This lesson on Production dbt — CI/CD, Environments, Scheduling is hands-on and example-driven. You will be able to build a robust, production-grade CI/CD pipeline in dbt Cloud connecting to separate development and production Databricks workspaces. You will learn to isolate environments, enforce feature-branch git workflows, and automate production model runs using pull request merge triggers.

What You'll Be Able To Do

• Configure isolated Dev and Prod warehouse connections using distinct Databricks workspace credentials.
• Link a dbt Cloud project to a GitHub repository to enforce version control workflows.
• Implement a feature branch workflow to develop and test model changes in Dev without altering Prod.
• Create a deployment environment restricted to the main branch with dedicated production tokens.
• Automate production deployments using a run-on-merge CI/CD job triggered by GitHub Pull Requests.

Detailed Concept Walkthrough

1. Environment and Connection Segregation

A production-ready data platform requires complete architectural isolation between development and production environments. dbt Cloud separates physical database credentials (Connections) from execution context and git branch bindings (Environments).

• Mechanism: Connections store physical warehouse metadata including Server Hostname, HTTP Path, and Catalog definitions for each Databricks workspace. Environments then bind these connections to target branches, credentials, and dbt versions.
• Under the Hood: When a developer runs a query in the IDE, dbt Cloud uses the Dev connection token and schema prefix to sandbox changes. Deployment jobs target the Prod connection using production service tokens.
• Best Practice: Never reuse the same personal access token or target schema across Dev and Prod environments to avoid accidental overwrites of production tables.

# Conceptual dbt Cloud connection payload (Databricks)
connection:
  name: "Databricks prod"
  server_hostname: "dbc-xxxx-prod.cloud.databricks.com"
  http_path: "/sql/1.0/warehouses/abcdef1234567890"
  catalog: "analytics_prod"
  schema: "public"

Key Takeaway: Connections define where data lives, while Environments define how, when, and from which branch code executes against that data.

2. Feature Branching and Dev Execution

All development work occurs on isolated git feature branches targeting development data assets, preventing direct commits to production-bound branches.

• Execution Flow: Developers create a dedicated branch (e.g., test or feature/xyz) from main in the dbt Cloud IDE. The IDE environment automatically defaults to running against the Databricks Dev connection.
• Under the Hood: Executing dbt run compiles SQL models into DDL/DML targeting the developer's personal or dev schema in the Dev Databricks workspace, leaving the production warehouse untouched.
• Syntax Rule: Model-level materializations can be updated dynamically in SQL config blocks or YAML files to alter target objects from tables to views during iteration.

-- models/marts/fct_orders.sql
{{ config(
    materialized = 'view'  -- modified on feature branch from 'table'
) }}

select
    order_id,
    customer_id,
    order_date,
    status
from {{ ref('stg_orders') }}

Key Takeaway: Never develop directly on main; use feature branches in the Dev environment to validate transformations before requesting deployment.

3. Automating Deployments with Merge Triggers

Continuous Deployment (CD) automates running production dbt transformations whenever validated code is merged into the default deployment branch.

• Mechanism: A dbt Cloud Deployment Job is created inside the Prod environment, locked to the main branch, and configured with execution steps such as dbt run.
• Under the Hood: The job listens to webhook events from the linked GitHub repository. When a Pull Request merges into main, dbt Cloud automatically clones the latest commit and triggers the production job run.
• Best Practice: Ensure production jobs use a dedicated production deployment token rather than individual developer credentials to ensure unbroken execution and secure access control.

# dbt Cloud Job Run Configuration
job_name: "deploy to prod"
environment: "Prod environment"
target_branch: "main"
trigger:
  type: "run_on_merge"
commands:
  - "dbt run"

Key Takeaway: Merging a Pull Request into main acts as the single source of truth that triggers automated, unattended production builds.

Topics Covered in Production dbt — CI/CD, Environments, Scheduling

1. Project and Environment Architecture (0:46 - 2:02) — Establishes the dbt Cloud project and outlines the architectural need for separate Dev and Prod Databricks workspaces.
2. Configuring Database Connections (2:02 - 3:36) — Sets up persistent connection definitions for both Dev and Prod Databricks environments using workspace endpoints and catalogs.
3. Dev Linking and GitHub Integration (3:36 - 5:19) — Associates the development environment with the Dev connection, adds access tokens, and connects the project to GitHub.
4. Model Execution in Dev (5:19 - 6:14) — Runs models inside the dbt Cloud IDE to verify transformations execute properly against the Databricks Dev target.
5. Branching and Code Changes (6:14 - 7:25) — Creates a test branch, modifies a model materialization from table to view, and commits changes under version control.
6. Creating the Production Environment (7:25 - 8:43) — Configures a deployment environment restricted to the main branch and connects it to the Databricks Prod warehouse.
7. Configuring the Deployment Job (8:43 - 10:09) — Sets up an automated dbt Cloud job configured with run-on-merge triggers to execute models in production.
8. Triggering Production Deployment (10:09 - 11:26) — Merges the GitHub pull request into main and monitors the automated execution of the production deployment job.

dbt Cheat Sheet

• dbt run — Executes compiled SQL models against the active environment connection

  dbt run
  

• config(materialized = 'view') — Configures model materialization as a database view instead of table

  {{ config(materialized = 'view') }}
  

• config(materialized = 'table') — Configures model materialization as a persistent physical table

  {{ config(materialized = 'table') }}
  

• git checkout -b <branch_name> — Creates and checks out a new feature branch

  git checkout -b test
  

• Run on merge — dbt Cloud trigger executing jobs when PRs merge into main

  Run on merge: true
  

Comparison Table

Common Pitfalls

• Mistake: Committing changes directly to the main branch from the IDE. Avoid: Create dedicated feature branches for development and open PRs to merge.
• Mistake: Sharing database credentials and tokens between Dev and Prod environments. Avoid: Generate distinct Databricks access tokens for each environment.
• Mistake: Pointing deployment jobs to a feature branch instead of main. Avoid: Lock the production environment to the main branch exclusively.

FAQs

• What is the primary difference between a Connection and an Environment in dbt Cloud? A Connection specifies the physical warehouse host, path, and catalog details, whereas an Environment binds a Connection to a deployment type, target branch, and credentials.
• Why is the production deployment environment locked to the main branch? Restricting production to main guarantees that only reviewed, approved, and merged code is executed against live production data tables.
• How does dbt Cloud know when to trigger a production run on merge? dbt Cloud receives a webhook event from the linked GitHub repository whenever a pull request is merged into the designated target branch.