1 of 20

CI for dbt: Beyond the basics!

23rd August 2023

​

CI for dbt: Beyond the basics!

2 of 20

Introduction

Pádraic Slattery

​

Analytics Engineer @ Xebia Data

Lecturer @ University College Dublin

​

github.com/pgoslatara/dbt-beyond-the-basics

2

CI for dbt: Beyond the basics!

3 of 20

Expectations

3

CI for dbt: Beyond the basics!

4 of 20

What is Continuous Integration?

4

?

CI for dbt: Beyond the basics!

5 of 20

CI starts before pushing to your repo!

​

​

5

CI for dbt: Beyond the basics!

6 of 20

dbt-checkpoint

github.com/dbt-checkpoint/dbt-checkpoint

A dbt-native collection of pre-commit hooks

6

CI for dbt: Beyond the basics!

7 of 20

dbt Artifacts

7

CI for dbt: Beyond the basics!

8 of 20

The power of pytest

8

CI for dbt: Beyond the basics!

9 of 20

The power of pytest

  • Columns:
    • DATE data types must end with “_date”
  • Lineage:
    • Sources can only have 1 downstream stg model
  • Models:
    • Must have defined primary key (“_pk”)
  • Sources:
    • `loaded_at_field` must exist in source
  • etc.

9

CI for dbt: Beyond the basics!

10 of 20

Bonus points!

- Use the “state:modified” method to compare manifests

- Easily assess downstream impact on marts in large projects

- Emojis make everything better

10

CI for dbt: Beyond the basics!

11 of 20

11

CI for dbt: Beyond the basics!

12 of 20

Dedicated dataset per CI run

12

CI for dbt: Beyond the basics!

13 of 20

But building all the models is expensive!

13

CI for dbt: Beyond the basics!

14 of 20

Dedicated dataset per CI run: The sequel

​

14

CI for dbt: Beyond the basics!

15 of 20

Mart Monitor!

15

CI for dbt: Beyond the basics!

16 of 20

To be approve, or not to be approve

16

CI for dbt: Beyond the basics!

17 of 20

Summary

  • Use pre-commit to automate formatting and basic checks
  • Run pytest on dbt artifacts to check naming conventions and internal standards
  • Building your dbt project in a dedicated CI dataset permits close monitoring of impacted marts

​

github.com/pgoslatara/dbt-beyond-the-basics

17

CI for dbt: Beyond the basics!

18 of 20

18

CI for dbt: Beyond the basics!

19 of 20

Colors

Primary

  • #000000

Secondary

  • #F56551
  • #EEB62A
  • #BADC3F
  • #B861A4
  • #3FCDDE

19

CI for dbt: Beyond the basics!

20 of 20

Assets

20

CI for dbt: Beyond the basics!