The Struggles We Skipped:

Data Engineering for the TikTok Generation

The Struggles We Skipped:

Data Engineering for the TikTok Generation

The TikTok generation of coders

The TikTok generation of coders

The TikTok generation of coders

The world with …

A tool for your

data engineering challenges:

A tool for your

data engineering challenges:

What to expect:

• Hiba’s experience with as a new-generation data professional

What to expect:

• Hiba’s experience with as a new-generation data professional
• Why it matters for us all
• How it works

Some of my roles in the past

Data Analyst

Business Analyst

Data Scientist

Computer Science Researcher

ETL

ETL issues someone like me has to face

Ad hoc sources

Learn Spark on the fly

Unstructured data

Ad hoc analysis

Ad hoc analysis of unstructured data

Streaming (read: non python based)

Time to data…

I’m sorting my

unstructured data into dataframes

Time to data

1. Structuring data (nested jsons *wink wink*)

Time to data

1. Structuring data (nested jsons *wink wink*)

Time to data

2. Data models in one line of code?

Time to data

3. Directly plug in my source for ad hoc sources.

Ad hoc

data

sources

Enter dlt

Raw data -> Analysis. I envision a plug-and-play solution.

One that is preferably

not like this:

Enter dlt

My pip-install-solution. The data-load-tool; dlt.

Code Demo

Smart device + dlt + duckdb

Code Demo

Code Demo

Step 1. Declare a pipeline

Code Demo

Step 2: Declare resources

Code Demo

Step 3: Run the pipeline

Code Demo

Step 4: View your schema and query your data

Okay, wait!!!!

But what if you’re a total noob?

We've all been there, and let's face it,

many of us are right in that boat at this very moment 👀.

EVERYONE ELSE AROUND YOU

YOU

I’m no different either …

• Gen Z
• Less than a year of real experience
• Fear of annoying seniors
• SELECT DOLLARS * 1000 AS CENTS

So let’s start from the beginning

What’s data engineering at all?

So let’s start from the beginning

What’s data engineering at all?

So let’s start from the beginning

What’s data engineering at all?

“I'm a plumber. I pipe data from one place to the other and make sure it's clean for the business to use.”

• A Data Engineer on Reddit

In other words:

In other words:

But there’s a plot twist:

Data scientists deal with data engineering tasks too

The reality

Every data job

is a data engineering job.

The problem

Every data job

is a data engineering job,

but not everybody is a data engineer.

The solution

As mentioned before…

​

dlt - Data Load Tool:

• Open-source
• Integrates and scales well
• Pythonic

TLDR: An in-house data engineer in a box

​

​

Why dlt?

From a junior’s perspective:

​

dlt - Data Load Tool:

• Open-source
• Integrates and scales well
• Pythonic

Why dlt?

From a junior’s perspective:

​

dlt - Data Load Tool:

• Community ⭐
• Integrates and scales well
• Pythonic

Why dlt?

From a junior’s perspective:

​

dlt - Data Load Tool:

• Community ⭐
• Saves time for other things ⭐
• Pythonic

Why dlt?

From a junior’s perspective:

​

dlt - Data Load Tool:

• Community ⭐
• Saves time for other things ⭐
• Smooth learning curve ⭐

Why dlt?

From a senior’s perspective:

​

dlt - Data Load Tool:

• Open-source
• Integrates and scales well
• Pythonic

Why dlt?

From a senior’s perspective:

​

dlt - Data Load Tool:

• Cost effective 👌
• Integrates and scales well
• Pythonic

Why dlt?

From a senior’s perspective:

​

dlt - Data Load Tool:

• Cost effective 👌
• No excuses from juniors 👌
• Pythonic

Why dlt?

From a senior’s perspective:

​

dlt - Data Load Tool:

• Cost effective 👌
• No need to reinvent the wheel 👌
• Pythonic

Why dlt?

From a senior’s perspective:

​

dlt - Data Load Tool:

• Cost effective 👌
• No need to reinvent the wheel 👌
• Works where Python runs 👌

So how exactly is it making your life easier?

Code Demo

Step 1. Declare a pipeline

Code Demo

Step 1. Declare a pipeline

Declaratively create

a connection that moves data to a specified destination.

Code Demo

Step 2: Declare resources

Code Demo

Step 2: Declare resources

resource is a logical grouping of data of similar structure and origin - basically, a table in a dataset.

Code Demo

Step 2: Declare resources

resource is a logical grouping of data of similar structure and origin - basically, a table in a dataset.

resource is a decorator function that yields/generates data on-the-fly, instead of loading all data into memory at once.

Code Demo

Step 2: Declare resources

resource is a logical grouping of data of similar structure and origin - basically, a table in a dataset.

​

resource is a decorator function that yields/generates data on-the-fly, instead of loading all data into memory at once.

resource is passed to the pipeline for the generated data to be loaded at the destination.

Code Demo

Step 2: Declare resources

The default write_disposition is “append”.

Code Demo

Step 2: Declare resources

The default write_disposition is “append”.

You can also use “replace” to overwrite existing data, or “merge” to deduplicate or upsert data.

Code Demo

Step 3: Run the pipeline

Code Demo

Step 3: Run the pipeline

Alternatively, run the pipeline once with a single source.

Code Demo

Step 3: Run the pipeline

Alternatively, run the pipeline once with a single source.

source is a logical grouping of multiple resources.

Basically, a dataset with multiple tables/resources.

Code Demo

Step 2: Declare resources

Code Demo

Step 2: Declare a source with multiple resources

Code Demo

Step 3: Run the pipeline with a single source

Code Demo

Step 3: Run the pipeline with a single source

What’s the point of grouping resources?

Code Demo

Step 3: Run the pipeline with a single source

What’s the point of grouping resources?

Imagine you have 100 different API

endpoints, each representing a separate table.

​

Code Demo

Step 3: Run the pipeline with a single source

What’s the point of grouping resources?

Imagine you have 100 different API endpoints, each representing a separate table.

​

Grouping them into single source brings:

• Efficiency: Run the pipeline once = load everything all at once.
• Easier control: Run the pipeline with selected resources from the source, apply any transformations.

Code Demo

Step 3: Run the pipeline with a single source

What’s the point of grouping resources?

• Reusability:

Code Demo

Step 4: View and query your data

​

Code Demo

Step 4: View and query your data: A pleasant surprise

​

​

Code Demo

Step 4: View and query your data: A pleasant surprise

​

​

Parent table

Code Demo

Step 4: View and query your data: A pleasant surprise

​

​

Parent table

Child table

Code Demo

Step 4: View and query your data: A pleasant surprise

​

​

Parent table

Child table

recursively

unpacked & normalised

data into

relational tables

​

Code Demo

Step 4: View and query your data: A pleasant surprise

​

​

Parent table

Child table

No need to worry about

Messy nested data!

​

The Bottom Line: pip install dlt

​

​

The Bottom Line: pip install dlt

​

​

BEFORE

AFTER

Try out our demo:

https://github.com/dlt-hub/dlt_demos/sengled_plug_demo

​

Get in touch:

hiba@dlthub.com & anuun@dlthub.com

Learn more at:

​