dlt
data load tool
Introduction
So let’s start from the beginning
What’s data engineering at all?
So let’s start from the beginning
What’s data engineering at all?
So let’s start from the beginning
What’s data engineering at all?
“I'm a plumber. I pipe data from one place to the other and make sure it's clean for the business to use.”
In other words
But there’s a plot twist:
Data scientists deal with data engineering tasks too
The reality
Every data job
is a data engineering job.
The problem
Every data job
is a data engineering job,
but not everybody is a data engineer.
The problems
And every data engineer has to be ready to solve problems
… whether they arise from new business requirements
… or pipelines breaking
The problems … Unstructured/Nested Data
Sorting unstructured data into data models.
The problems … Schema Evolution
The process of modifying the structure of a database schema over time.
The problems … Data Contracts
Data contracts define and enforce the functionality, manageability, and reliability of data.
… like enforcing rules on schema changes.
The problems … Too many/ad hoc data sources
Unifying different data sources in a single data pipeline.
The problems … Incremental Loading
Appending, replacing, or merging data periodically.
The problems … Performance Issues
Streaming and managing large amounts of data.
The solution
dlt - Data Load Tool:
What is dlt?
dlt History
Adrian
Adrian
Adrian
What is dlt?
The dlt is an open-source Python library that simplifies data loading by automating complex tasks like schema creation, data normalization, incremental loading, etc. Much like dbt democratized the T(Transform) layer of ELT to SQL users, dlt democratizes the EL (Extract/Load) aspect of data handling with Python.
What is dlt?
import dlt
pipeline = dlt.pipeline(
pipeline_name='my_pipeline',
destination='bigquery',
dataset_name='my_data',
)
pipeline.run(data, table_name='users')
>> pip install dlt
What is dlt?
dlt contains all DE best practices. Anyone who ever worked with Python can create the pipeline on a Senior level:
What is dlt?
dlt contains all DE best practices. Anyone who ever worked with Python can create the pipeline on a Senior level:
With dlt schema evolution is handled automatically. When modifications occur in the source data’s schema, dlt detects these changes and updates the schema accordingly.
What is dlt?
dlt contains all DE best practices. Anyone who ever worked with Python can create the pipeline on a Senior level:
You can use data contracts modes to tell dlt how to apply contract for a particular entity:
What is dlt?
dlt contains all DE best practices. Anyone who ever worked with Python can create the pipeline on a Senior level:
Incremental loading is a crucial concept in data pipelines that involves loading only new or changed data instead of reloading the entire dataset.
What is dlt?
dlt contains all DE best practices. Anyone who ever worked with Python can create the pipeline on a Senior level:
dlt provides several mechanisms and configuration options to manage performance and scale up pipelines:
What is dlt?
Database agnostic
Create custom sources & destinations
Source
Create custom sources & destinations
Source
Create custom sources & destinations
Source
dlt ecosystem
30+ existing well-tested sources, such as Postgres CDC, SQL databases, REST API connector, Google Sheets, Zendesk, Stripe, Notion, Hubspot, GitHub and others.
16 destinations, such as DuckDB, Postgres, Delta tables, BigQuery, Snowflake, and others.
Reverse ETL – build your own destination
Integrations:
dbt-runner, deploy helpers, Streamlit build-in app, etc.
Be it a Google Colab notebook, AWS Lambda function, an Airflow DAG, or your local laptop — dlt can be dropped in anywhere.
Why dlt?
From a junior’s perspective:
dlt - Data Load Tool:
Why dlt?
From a junior’s perspective:
dlt - Data Load Tool:
Why dlt?
Why dlt?
Why dlt?
From a senior’s perspective:
dlt - Data Load Tool:
Why dlt?
From a senior’s perspective:
dlt - Data Load Tool:
Why dlt?
From a senior’s perspective:
dlt - Data Load Tool:
Why dlt?
From a senior’s perspective:
dlt - Data Load Tool:
Why dlt?
From a senior’s perspective:
dlt - Data Load Tool:
Why would you need dlt?
(In case you haven’t been convinced yet)
How does dlt work?
A basic dlt pipeline has three main steps:
extract → fetch data from source system and write to local disk
normalize → read extracted data from local disk infer schema and transform data in memory write transformed data to local disk
load → read normalized data from local disk and ingest into destination system