1 of 25

Maintaining your conventions amid the chaos of dbt

Pádraic Slattery

5th December 2024

1

Intro #2

PowerPoint template

2 of 25

Speaker

Pádraic Slattery

​

Analytics Engineer @ Xebia Data

​

2



Journey #1

PowerPoint template

3 of 25

⚠️ Warnings:

​

​

Hot takes 🌶️

​

Product placement 💸

​

Minions -

3

Title Only #1

PowerPoint template

4 of 25

Working with a growing dbt project

4

Chapter #4

PowerPoint template

5 of 25

“dbt is easy, it’s just SQL”

  • jaffle_shop: 3 seeds, 5 models, 20 tests

5

Text 100% #2

PowerPoint template

6 of 25

6

Text 2/3 columns

CHOOSING TEXT LEVELS

1

Plain text

2

Under the tab ‘Home’, use the list level-buttons to choose a text level. �Choose from:

  • Bullet

3

  • Sub-bullets

4

Subtitle

5

  1. Numeric bullet

6

  • # Bullets

7

Cursive

8

Alt. Subtitle

9

  1. # Abc

WANT TO KNOW MORE?�Go to the instruction slide ‘VIDEO INSTRUCTIONS’.�You’ll find it at the beginning of the presentation�or insert it via ‘Home’ > ‘New Slide’.

Home

Text level up

Text level down

PowerPoint template

7 of 25

🫣

7

Text 100% #2

PowerPoint template

8 of 25

Finer details

  • Are columns like is_active and is_deleted BOOLEAN or STRING?
  • Do TIMESTAMP columns always end with _date?
  • Do I have to document all the columns in my intermediate model?
  • Can my marts model reference another marts model?
  • Am I required to document the arguments to my macro?
  • Can I build views on top of views on top of views … ?

8

Text 100% #2

PowerPoint template

9 of 25

Causes

  • People:
    • New developers have different approaches
    • Lack of consistency amongst developers
  • Architecture:
    • Integration with other tools (orchestrators, etc.)
    • Moving to multiple dbt projects
    • Data mesh is complex

9

Text 100% #2

PowerPoint template

10 of 25

Solutions?

  • Documentation
  • Pull request checklists
  • Required approvers
  • Approval processes
  • Never-ending re-factors

​

Are these really solutions? Or are they just responses?

10

handbook.gitlab.com/handbook/enterprise-data/platform/dbt-guide/

Text 100% #2

PowerPoint template

11 of 25

Solutions

  • dbt brings software engineering best practices to data transformations
  • How does software engineering deal with growing projects:?
    • Type checking (mypy)
    • Style (flake8)

​

Do similar tools exist for dbt?

11

Text 100% #2

PowerPoint template

12 of 25

Tackling complexity with automation - now with extra dbt

12

Chapter #4

PowerPoint template

13 of 25

Existing tooling

  • ​

13

github.com/dbt-labs/dbt-project-evaluator

github.com/PicnicSupermarket/dbt-score

github.com/dbt-checkpoint/dbt-checkpoint

​

Text 100% #2

PowerPoint template

14 of 25

dbt-bouncer

  • Python CLI
  • Configurable via YML
  • Parses dbt artifacts
  • Large number of checks against models, sources, macros, exposures, tests, unit tests, semantic models, etc.
  • Open-sourced by Xebia Data

14

github.com/godatadriven/dbt-bouncer

Text 100% #2

PowerPoint template

15 of 25

dbt-bouncer

15

Text 100% #2

PowerPoint template

16 of 25

dbt-bouncer in action

16

Text 100% #2

PowerPoint template

17 of 25

dbt-bouncer in action

17

Text 100% #2

PowerPoint template

18 of 25

dbt-bouncer in action

18

Text 100% #2

PowerPoint template

19 of 25

dbt-bouncer: what can it do?

A single tool to configure:

  • Documentation standards
  • Lineage rules
  • Testing conventions
  • Performance standards
  • And much more…

19

Text 100% #2

PowerPoint template

20 of 25

So many tests…

20

Text 100% #2

PowerPoint template

21 of 25

Summary

21

​

dbt-project-evaluator

dbt-score

dbt-checkpoint

dbt-bouncer

Maintainer

dbt Labs

Picnic

Datacoves

Xebia Data

# Tests

30

5

39

54

dbt package

✅

❌

❌

❌

Python package

❌

✅

❌

✅

Runs without database access

❌

✅

✅

✅

Configurable in YML

✅ and ❌

❌

✅

✅

Extendable

❌

✅

❌

✅

Text 100% #2

PowerPoint template

22 of 25

Conclusion

  • Maintaining order amidst the chaos of dbt is tough
  • Automation is the solution

​

“Running a single dbt command … is satisfyingly simple … the hard part is in the maintenance of the system”

  • Julia Schottenstein (formerly Product Manager @ dbt Labs)

22

getdbt.com/blog/seemingly-easy-but-in-reality-hard

Text 100% #2

PowerPoint template

23 of 25

Thank you!

Xebia Data

Oostenburgermiddenstraat 184

1018 LL Amsterdam

www.xebia.com

Pádraic Slattery

Analytics Engineer

Xebia Data

linkedin.com/in/padraicslattery

github.com/pgoslatara

23

Logo & text

PowerPoint template

24 of 25

Let’s start with dbt Cloud

24

Chapter #4

PowerPoint template

25 of 25

Icon set

25

Title Only #5

PowerPoint template