1 of 55

Building an open source data stack 🥞

A guide on open source tools from event capture to visualization

2 of 55

I’m Katie, and I’m here to tell you that the modern data stack is great.

2

3 of 55

I used to work at a bank

3

4 of 55

I used to work at a bank

4

https://media.giphy.com/media/3otPoBRKROJl9UbqMM/giphy.gif?quality=90&strip=all&zoom=1&resize=480%2C270

5 of 55

I used to work at a bank...and some people had never seen a modern data stack.

5

https://images.squarespace-cdn.com/content/v1/51eacd5be4b0161a067dfde4/1565878430123-QBWQBSDL49Y52JJQE67W/Chart-sure-its-complicated.PNG?format=1000w

6 of 55

The quest to spread the data stack love �(and understanding)

It sucks when I don’t understand things - I feel left out and frustrated.
I’m way more likely to use something if I understand why it’s useful.
I can’t help improve a system if I don’t understand how the system works.

6

https://memegenerator.net/img/instances/57304991.jpg

7 of 55

I believe that:

The modern data stack is way better at getting us from raw data → insights than what we used to have.

7

8 of 55

I believe that:

The modern data stack is way better at getting us from raw data → insights than what we used to have.
Open source software lets us build a stack that’s modifiable, extensible and reusable.

8

9 of 55

Let’s set the scene...

10 of 55

You’re at a company

without the modern data stack

10

11 of 55

Events get ingested...sometimes

11

?

12 of 55

Your customer growth is tracked in an Excel sheet (featuring lots of complicated VLOOKUPs)

12

13 of 55

Only one of the engineers knows how to share the customer growth data.

13

14 of 55

They’ve ended up in a situation where data-informed decision making is not an option.

14

15 of 55

Enter...the modern data stack

16 of 55

I’ve broken this down into three “steps”:

Going from App-land to Data-land:

Extract + Load

Finding America’s Next Top (Data) Model:

Transform

Keeping our data dashing:

Visualize/Analyze

16

17 of 55

From App-Land to Data-Land

How Rudderstack streams data from an app to a data warehouse

18 of 55

Ideal world: When a user does something in the app, we know about it.

Event data

19 of 55

Rudderstack lets us easily track actions that a user takes in the app.

Event data

20 of 55

Engineers build features that log information about what they’re doing (event data!)

Source: rudderstack.com

21 of 55

Rudderstack lets us send our data wherever we want it to go.

Source: rudderstack.com

22 of 55

Extract + Load:

Going from App Land to Data Land

Event data

App Land

Data Land

Raw event data

23 of 55

Extract + Load:

Going from App Land to Data Land

Event data

App Land

Data Land

Raw event data

24 of 55

Britain’s Next Top �Data Model

How we organize raw data using data models in dbt.

25 of 55

We have raw data, but it’s not nearly as nice as transformed data.

Event data

App Land

Data Land

Raw event data

Transformed data

?

26 of 55

26

We use data models to standardize relationships between raw events

identifies.user_created�identifies.user_activated

identifies.user_updated

identifies.user_deleted

deployments-prod.query_executed

deployments-prod.chart_saved

….

}

users.sql

users table

1. Raw Events

2. Data Model

3. Output

4. Inserted into Database

27 of 55

We write data models using dbt and their framework.

Raw event data

Transformed data

Data

Models

Data Land

28 of 55

Why not just use SQL + some version control?

Doesn’t Looker already do this?

28

29 of 55

What would happen if we just used SQL?

29

If we added a new column to the customer growth data model.

There’s nowhere to document what this new column does/means (context is lost)
We’ve accidentally duplicated every row (oh shit)
We have to build some sample dataset to test our change (or run it on our entire dataset)

30 of 55

30

fixes all of these problems at once.

dbt is the tool for data transformation
dbt makes it easy to:

manage relationships between models (using “ref”)
visualize these relationships (documentation)
assure the quality of our transformations through testing (model validation)

31 of 55

31

We write data models using dbt & their framework

Event data

App Land

Data Land

Raw event data

Transformed data

32 of 55

Keeping our Data Dashing

How we use Lightdash to share data insights

33 of 55

There are 3(ish) ways to let people answer their questions with data

33

34 of 55

Give everyone access to all the data.

34

Raw data

data

Raw data

data

Raw data

data

35 of 55

Give everyone access to all the data.

35

Raw data

data

Raw data

data

Raw data

data

36 of 55

2. Give complete answers to everyone.

36

37 of 55

2. Give complete answers to everyone.

37

38 of 55

3. Give people useful building blocks.

38

39 of 55

gives users meaningful building blocks to answer their own data questions.

39

Explore meaningful business entities

Users, Orders, Clicks, Emails

Choose from relevant metrics

Revenue, sales, conversion,

Filter data to relevant subsets

Geography, time, business area

Segment data into groups

Geography, time, status, product

https://www.mentalfloss.com/article/92127/how-many-combinations-are-possible-using-6-lego-bricks

40 of 55

40

What is the total order count split by status?

metric

dimension

41 of 55

41

What is the total order count split by status?

metric

dimension

42 of 55

now supports dbt metrics!

42

43 of 55

gives users meaningful building blocks to answer their own data questions.

You have a curated set of metrics & dimensions so you can cut your data in different ways.
Keep all of your business logic in one place - metrics + dimensions defined directly in your dbt project
Build trust in your data - show freshness, testing and other metadata

43

44 of 55

lets users answer their data questions

44

Event data

App Land

Data Land

Raw event data

Transformed data

Business insights

45 of 55

Back to our original scenario...

46 of 55

: Events get tracked and loaded into your data warehouse

46

Event data

App Land

Data Land

Raw event data

47 of 55

Your customer growth model is

built in

47

Event data

App Land

Data Land

Raw event data

Transformed data

48 of 55

Everyone in the team has access to the

customer growth chart in

48

Event data

App Land

Data Land

Raw event data

Transformed data

Business insights

49 of 55

The modern data stack has all of the main bits to go from raw data → insights

...and it’s way better than what we used to have.

49

50 of 55

I skipped a lot of potential steps and other tool options...

50

Reverse ETL

ELT

Ad hoc analysis

Orchestration

51 of 55

How do we integrate all of these modular parts together to create an end-to-end data experience?

51

52 of 55

As data builders, we feel like we’ve got a Ferrari

52

53 of 55

But data consumers see something more like this...

53

54 of 55

How do we integrate all of these modular parts together to create an end-to-end data experience?

54

55 of 55

55

#tools-lightdash (in dbt Slack)

lightdash/lightdash

@lightdash_devs

Skip the queue for

lightdash.com/coalesce2021