CI/CD for dbt with GitHub Actions: Slim CI and Safe Deployments

Once a dbt project has more than a handful of models and more than one developer, "run it locally and push to main" stops being safe. A change to a staging model can break a mart three levels down, and nobody finds out until the morning run fails or, worse, succeeds with wrong numbers.

This post shows the CI/CD setup I use for dbt with GitHub Actions: build and test only what changed on every pull request, in an isolated schema, and deploy to production on merge. The examples use Databricks, but the workflow is the same for any adapter.

The goals

• Every pull request is built and tested before it can be merged.
• CI only builds what changed and what depends on it, not the whole project.
• CI runs in its own schema, so it never touches production tables.
• Production deploys happen from main only, never from a laptop.

Slim CI: build only what changed

dbt can compare the current project with the manifest from production (manifest.json, written by every run) and select only modified models:

1: dbt build --select state:modified+ --defer --state ./prod-artifacts

• state:modified+: models whose SQL or config changed, plus everything downstream of them (the +).
• --defer: when a selected model refers to a model that wasn't rebuilt, read the production version instead. So CI doesn't need to build the whole upstream graph.
• --state: where to find the production manifest.

A change to fct_claims builds that model and its downstream marts, reading unchanged dimensions from production. A pull request that touches two models finishes in minutes instead of rebuilding years of transactions.

Isolated schemas for each pull request

Give CI its own target in profiles.yml, with the schema taken from an environment variable:

 1: analytics:
 2:   outputs:
 3:     ci:
 4:       type: databricks
 5:       host: "{{ env_var('DATABRICKS_HOST') }}"
 6:       http_path: "{{ env_var('DATABRICKS_HTTP_PATH') }}"
 7:       token: "{{ env_var('DATABRICKS_TOKEN') }}"
 8:       catalog: ci
 9:       schema: "{{ env_var('DBT_CI_SCHEMA') }}"
10:       threads: 8
11:     prod:
12:       type: databricks
13:       host: "{{ env_var('DATABRICKS_HOST') }}"
14:       http_path: "{{ env_var('DATABRICKS_HTTP_PATH') }}"
15:       token: "{{ env_var('DATABRICKS_TOKEN') }}"
16:       catalog: prod
17:       schema: reporting
18:       threads: 8

Each pull request gets a schema like pr_142 in a separate CI catalog, so two developers' pull requests can't overwrite each other, and none of them can touch production.

Keeping the production manifest

Slim CI needs the latest production manifest.json. The simplest approach is to have the production deploy upload it as a build artifact, or copy it to cloud storage, and have CI download it. I prefer storage, because artifacts expire:

1: aws s3 cp target/manifest.json s3://my-dbt-artifacts/prod/manifest.json

The pull request workflow

 1: # .github/workflows/dbt-ci.yml
 2: name: dbt-ci
 3: on:
 4:   pull_request:
 5:     branches: [main]
 6: 
 7: jobs:
 8:   build-and-test:
 9:     runs-on: ubuntu-latest
10:     env:
11:       DATABRICKS_HOST:      ${{ secrets.DATABRICKS_HOST }}
12:       DATABRICKS_HTTP_PATH: ${{ secrets.DATABRICKS_HTTP_PATH }}
13:       DATABRICKS_TOKEN:     ${{ secrets.DATABRICKS_CI_TOKEN }}
14:       DBT_CI_SCHEMA:        pr_${{ github.event.pull_request.number }}
15:       DBT_PROFILES_DIR:     ./ci
16:     steps:
17:       - uses: actions/checkout@v4
18:       - uses: actions/setup-python@v5
19:         with: { python-version: '3.11' }
20:       - run: pip install dbt-databricks==1.10.*
21:       - run: dbt deps
22: 
23:       - name: Get production manifest
24:         run: |
25:           mkdir -p prod-artifacts
26:           aws s3 cp s3://my-dbt-artifacts/prod/manifest.json prod-artifacts/ || echo "no manifest, full build"
27: 
28:       - name: Lint SQL
29:         run: pip install sqlfluff sqlfluff-templater-dbt && sqlfluff lint models
30: 
31:       - name: Build and test changed models
32:         run: |
33:           if [ -f prod-artifacts/manifest.json ]; then
34:             dbt build --target ci --select state:modified+ --defer --state prod-artifacts
35:           else
36:             dbt build --target ci
37:           fi

(The AWS step needs credentials too. With GitHub's OIDC integration you can assume an IAM role without storing access keys.)

Make the workflow a required status check on the main branch, so a pull request can't be merged until its models build and its tests pass.

The deploy workflow

 1: # .github/workflows/dbt-deploy.yml
 2: name: dbt-deploy
 3: on:
 4:   push:
 5:     branches: [main]
 6: 
 7: jobs:
 8:   deploy:
 9:     runs-on: ubuntu-latest
10:     environment: production
11:     env:
12:       DATABRICKS_HOST:      ${{ secrets.DATABRICKS_HOST }}
13:       DATABRICKS_HTTP_PATH: ${{ secrets.DATABRICKS_HTTP_PATH }}
14:       DATABRICKS_TOKEN:     ${{ secrets.DATABRICKS_PROD_TOKEN }}
15:       DBT_PROFILES_DIR:     ./ci
16:     steps:
17:       - uses: actions/checkout@v4
18:       - uses: actions/setup-python@v5
19:         with: { python-version: '3.11' }
20:       - run: pip install dbt-databricks==1.10.*
21:       - run: dbt deps
22:       - run: dbt build --target prod --exclude test_type:unit
23:       - run: aws s3 cp target/manifest.json s3://my-dbt-artifacts/prod/manifest.json

Unit tests already ran in CI, so production skips them. The GitHub environment adds an approval step, which also gives you an audit trail of who released what and when.

Many teams don't run the whole project on every merge. Instead, the scheduled production job (in Databricks Workflows, Airflow or similar) runs dbt build on its normal timetable, and the merge simply makes the new code available to it. Either works. What matters is that production only ever runs code from main.

Cleaning up

CI schemas pile up. A small workflow on pull_request: closed can drop the schema for that pull request:

1:       - run: dbt run-operation drop_schema

Here drop_schema is a short macro in the project that runs DROP SCHEMA ... CASCADE.

Small habits that help

• Pin dbt and package versions so CI and production behave the same.
• Use conventional commits (feat:, fix:) and you can generate release notes and version tags automatically.
• Use separate tokens or service principals for CI and production, with CI unable to write to production catalogs.
• Keep CI fast. If a pull request takes more than about ten minutes, people start bypassing it. Slim CI and deferral are what keep it quick.

None of this is specific to insurance, but it matters more when the output feeds Lloyd's returns and finance reconciliations. A tested, reviewed, repeatable release process is much easier to explain to an auditor than "it worked on my machine".