1 of 38

Dagster Components

Low Code and AI-Native DX Without The Mess

Nick Schrock (@schrockn) - CTO & Founder - 2025-04-16

2 of 38

Data leaders today �face a critical tradeoff between�accessibility �and power

2

3 of 38

Accessibility in a data platform

Without accessibility there is a massive coordination burden that grinds everything to a halt.

What does it mean?

Accessibility means practitioners on the platform can self-serve to build and operate data pipelines.

Why does it matter?

First is accessibility. Accessibility means that practitioners who use the platform can self-serve to build and operate data pipelines. As you know, data platforms are heterogeneous places. There are many technologies, many tools, many different types of people using the data platform. You want those uses to be as close to the business and the people that rely on their end products, the data assets, as possible. And you want them to own and self-serve the data pipelines business logic directly.

This is absolutely critical for scale. Without this property there is a massive coordination burden between the business and the engineers. The users that know the business, the subject-matter experts, need to communicate requirements to the engineers. They need to ensure those requirements are implemented correctly. They need to get the engineer’s attention and time, competing with all the other SMEs who have the same incentives. Everything grinds to a half.

4 of 38

A powerful data platform

A powerful platform can adapt to business needs and technological change through shared tech, process, and workflows—enabled at scale by great engineering and a centralized, leveraged team.

A leveraged data platform team can drive organization-spanning outcomes. E.g:

Productivity improvements
Enforcing standards
Cost reductions
Deploying new technologies
Cross-cutting policy initiatives.

What does it mean?

Why does it matter?

In order to fix this dynamic, you need a powerful data platform. What does this mean?

It means that you can scale data engineering across the organization, and can do so in the context of a world that is constantly changing, both in terms of business requirements and an evolving technology landscape.

When you can scale data engineering across the whole organization, the data platform team and scope changes. It becomes leveraged. It is an organization that can drive organization-spanning outcomes. This means productivity improvements, enforcing standards and qualify, implementing cost reductions, deploying new technologies, cross-cutting policy initiatives, and more. Without the data platform team, these outcomes can only be driven by heavyweight process and manual coordination, and those efforts typically fail.

A leveraged, focused team can do so much with the proper mentality and tools. It is the way to uplevel your team and your career.

5 of 38

The Practitioner and the Data (Platform) Engineer

Data Practitioner

Subject-matter expert
Titles: E.g., Data Scientist, Analytics Engineer, Researcher
Typically uses a single, domain-specific tool: E.g. notebook, dbt, ML tool

Data (Platform) Engineer

Knows technology and engineering process
Typically have limited context on the business domain

5

6 of 38

The Practitioner and the Data (Platform) Engineer

There are very few people who can fulfill both of those roles.

They are unicorns.

The collaboration truly works only with the right systems and abstractions in place.

🦄

Data Practitioner

Data (Platform) Engineer

6

7 of 38

Approaches That

We See To Achieve

Self-Serve Pipelines

Verticalized low-code tool or platform

Engineer-only production data stack

🍰

Build a bespoke DSL and framework over the orchestrator

👨🏽‍💻

🪦

Fragile tooling, maintenance overhead, and subpar experiences for stakeholders

Not powerful and flexible enough

Inaccessible to many SMEs and practitioners

7

People have been working on this problem for a long, long time. It is not a new insight that it is important for subject-matter experts to self-service their own data pipelines.

These are common approaches in the industry that we have also experienced firsthand, we real at-scale customers. We have the scar tissue to prove it!

An approach whose popularity rises and falls with each success generation of technology is the vertically integrated, all-in-one, low-code tool or platform. The idea here is to remove the data platform engineer from the loop entirely, and rely exclusively on the vendor tool for the self-serve data pipelining experience. However this approach is not powerful and flexible enough once you go beyond the most basic needs or scale.

Next is an engineer-only production stack. While this can work better and is more adaptable, it often fails to serve the practitioners, SMEs, and stakeholder teams. These systems, such as Airflow, try to please both constituencies but end up not serving either. The result are poorly structured systems that end up with accessibility problems anyways.

Last is a very common pattern that we see to “fix” the accessibility problem, which is building bespoke tools, frameworks, DSLs stacked on top of that orchestrator. This ends in a road with fragile, brittle tooling, maintenance burden and subpar experiences that again leaves us with a platform that in reality is not that accessible and not that powerful.

8 of 38

Phase One: Low-code

• Demos well, and people happy, at first�

• However, not powerful or flexible enough�

• Engineers had to build parallel infrastructure for the business-critical pipelines�

• Three years in and the situation is untenable. Engineers are bottleneck, practitioners can’t do their work, everyone is unhappy.

A Platform Journey at BigCo

�Siloed, all-in-one tool

SME / Practitioner

8

Let’s go through a data platform journey at a company, that we’ll refer to as BigCo, starting nearly a decade ago.

They were getting their data journey starting and didn’t want to hire a bunch of data engineers. They felt their path forward was a low-code tool like Informatica or Azure Data Factory. The demos were awesome and people were happy at first.

However, the business changed, the requirements got more complex, and the ecosystem advanced and left them behind. The platform was not powerful or flexible enough. Productivity slows, integrations were out-of-date, the platform destabilized.

They had to sprint to hire data engineers, who built parallel infrastructure and took over a small set of mission-critical pipelines.

Fast forward a few years and the situation is untenable. The engineers can’t handle the load and are the bottleneck. The practitioners cannot do anything beyond simple use cases or they are begging for engineer time. The low-code tool is not scaling, breaks all the time, and does not integrate with the company infrastructure and technical processes. The situation is untenable.

9 of 38

Phase Two: Airflow

• Now BigCo migrated to Airflow �

• Engineers can get involved and write Python, stack on infrastructure

�• Practitioners tried to participate, project degraded, and most eventually failed, leading to Shadow IT and silo’ing�

• Central data team had to take over and become bottleneck, unhappy, and drowning in toil�

• Two years in, something must be done…

A Platform Journey at BigCo

DAGs / Business logic

Loosely structured project

Infrastructure

Engineer

DevOps/Infra Engineer

9

The data team is in crisis mode and they need to make a big change. They decide to adopt Airflow. Sounds great.

The engineers can get much more involved as it is Python-native. It will integrate more easily with existing company infrastructure because it is more flexible than the low-code tool. It promises to serve the practitioner as well as the engineer.

However as it is rolled out the reality doesn’t live up to promise. By trying to serve two masters with a single API and poorly structured projects, it satisfies neither. The data engineers cannot truly build an effective platform, and in reality many of the practitioners try and fail to onboard onto the platform effectively.

Increasingly the central team has to, again, assume more and more responsibility for the data platform. Some practitioner teams go rogue and self-serve their own tools via Shadow IT.

We are, ironically, left in state with similar tradeoffs and compromises as in the previous configuration. The same problems, again.

Two years in, and we need a new initiative.

10 of 38

Phase Three: DSL �over Airflow

• Ambitious engineer has idea to build custom YAML DSL on top of Airflow

• Practitioners happy with demos and initial, simple use cases

• Custom DSL had limited tooling and delivered bad, difficult-to-debug, and eventually unowned experience

• Two years later, central data team had to again take over everything and become bottleneck, drowned in toil

A Platform Journey at BigCo

DAGs / Business logic

Loosely structured project

Infrastructure

Engineer

DevOps/Infra Engineer

SME / Practitioner

YAML

10

An ambitious engineer has an idea: We are going to build a custom framework and DSL, in house, stack on top of the Airflow API.

The demos again are super impressive. It seems like the way out.

However, the Custom DSL has many of the same maladies. It handles the simple use cases but does not scale to the more demanding ones. It doesn’t have great tooling or a great testing story.

Worse the ambitious engineer leaves, and the system ends up with no true owner, and is fragile, brittle, and largely unmaintained.

Practitioners ship bugs and encounter maddening usability problems. Onboarding new practitioners is very difficult, as it is totally custom and not an industry standard.

Two years later. We are again in a similar state, but with different tradeoffs. But the one constant remains: data teams are overwhelmed, practitioners are unhappy, people don’t trust the platform, and everyone is drowning in toil.

This has been a decade-long journey! And where are we?

11 of 38

There are decades where nothing happens; and there are weeks where decades happen.

, Data Engineer at BigCo

Vladimir Lenin

12 of 38

Excruciating tradeoffs

12

13 of 38

Data leaders are navigating this grid

Powerful

Not Powerful

Not Accessible

Accessible

+

DAG-only

Modern

Low Code

DSL

Bespoke DSL

13

I like to think about it as a grid. On the X axis there a spectrum from inaccessible to accessible. On the Y-axis is a spectrum of not powerful to powerful. The journey of BigCo is moving around this grid.

The legacy low-code solutions were somewhat accessible but definitely not powerful.

They then adopted Airflow which led to a tradeoff where power and adaptability were exchanged for much less accessibility.

The adoption of the bespoke DSL made the system marginally more accessible. But it degraded the architecture, increased the surface area, and was not designed with proper “Escape Hatches”. There was more to deal with, and again the ground truth reality is that they again exchanged accessibility for adaptability and power.

The company is now considering a new generation of low code tools that have modernized UIs and some architectural improvements, but still making the same fundamental tradeoffs.

14 of 38

Power/Accessibility Frontier

Powerful

Not Powerful

Not Accessible

Accessible

DAG-only

Modern

Low Code

Bespoke DSL

Breaking through this barrier requires innovation

14

15 of 38

Goal of Dagster Components ↗️

Powerful

Not Powerful

Not Accessible

Accessible

DAG-only

Modern

Low Code

Bespoke DSL

Dagster Components

15

16 of 38

Demo: The Practitioner Journey Onboarding Onto a Components Project

16

17 of 38

Demo Recap

Practitioner can onboard easily by cargoculting and scaffolding

Robust support with tooling and integrated documentation

Tribal knowledge of the platform and workflow spread through scaffolding and built-in documentation

Straightforward integration into broader platform

17

18 of 38

Innovation doesn’t happen �in a vacuum

19 of 38

Where do Components fit?

We’re entering a new era.

AI is coming.

20 of 38

The data ecosystem

Big Data Era

Modern Data Era

AI Era

Map-Reduce

Hadoop

Data Lakes

Cloud Data Warehouse

Modern Data Stack

AI-Assisted Codegen

???

Map-Reduce

Hadoop

Data Lakes

Cloud Data Warehouse

Modern Data Stack

AI-Assisted Engineering

Lakehouses

???

2006 - 2016

2016 - 2024

2024 - ???

20

Let’s frame this with eras of data technologies. While these dates are not precise and we could argue the boundary conditions, I think these are largely correct.

We’ll start with the Big Data, starting with 2006, the release of Hadoop. The era as defined by the challenge of processing the new scale of data unleashed by Internet-scale data and digital native business processes. As the cloud data warehouses matured and handled scaled effortlessly, the big data era came to a close.

Next was the modern data era, defined by the the cloud data warehouses, the mass migration to the Cloud, and a whole set of new tools, collectively known as the modern data stack.

And now we are just entering the era of AI. We are just at the beginning and don’t know the set of technologies that will define it. But we know a couple. We know that AI-assisted engineering is not just coming, but is already here. And Lakehouses that seek to provide the same usability as Cloud Datawarehouse, but over open table formats, heterogeneous compute, and unstructured data. It’s an exciting time.

21 of 38

AI Mainstreaming Now in Software Engineering

🤯 AI-assisted engineering exploding

🚀 Model Context Protocol

MCP hit inflection in March. Gamechanger for tools.

Generalized AI tools in software engineering are exploding (Cursor, Co-Pilot 1M+ users)

21

22 of 38

MCP Server 🚀🌙

Dec 1 2024

Mar 30 2025

Source: Google Trends

Mar 1 2025

23 of 38

Shadow AI: Vibe Coders are Coming…

You will have vibe coding stakeholders,

whether you like it or not.

24 of 38

AI Risk in Software Systems

Unconstrained AI can be a

hallucinating demon

🐛 Easy to introduce bugs difficult or impossible to understand

🦠 Technical Debt Superspreader

😏 The wrong person working with AI without guardrails is dangerous

24

25 of 38

Shadow AI Vibecoding Can Be a Bad Time

25

26 of 38

Natural language is an insufficiently precise input to produce reliable software in a completely unsupervised manner.

I said this

CTO, Dagster

27 of 38

27

28 of 38

Components is the first AI-optimized framework for data engineering

Data engineering can have

nice things too

29 of 38

Guardrails

Modular code containers and constrained target language

29

30 of 38

Guardrails

Modular code containers and constrained target language

30

31 of 38

Guardrails are essential

Modular Code Containers:

Technical blast radius for AI slop
Limits necessary context window
More replaceable

Constrained codegen target:

E.g. SQL, Yaml, Structured Framework
Humans can evaluate correctness
Engineers and tools can impose constraints

31

32 of 38

Dagster Components

Stack

Infrastructure

Modular Project Structure and Framework

Engineer

DevOps/Infra Engineer

SME / Practitioner

Tooling & Ecosystem

Components DSL

Components

32

It is a stack.

Going from the bottom:

It integrates with your Infrastructure
It is built on a new project structure and framework for Dagster, that better organizes and modularizes your code
On top of that are both out-of-the-box components and as we saw the ability to build custom components, which is crucial.
And lastly there is the DSL which is accessible to all of your stakeholders, and an ideal target for AI codegen.
And we have tooling support and will have an ecosystem that supports this entire stack.

All this means that this integrates with your companies stack. It plugs into existing processes and tools. And it means that your engineers can support your practitioners via a well-supported API, assisted by an abstraction layer with tooling. And that API is an ideal code generation target.

33 of 38

What gives us the right to call ourselves an

AI-optimized framework for data engineering?

Unique hub of context

Guardrails

Visibility Across Dev Lifecycle

Integrations, Source Code, Lineage, Metadata, Operational Data

Modular Components defined using high-level, introspectable DSL

Dev → Test → CI/CD → �Branch → Prod

01

02

03

33

There are probably a lot of tools and frameworks that will position themselves as the right place to do AI in data, but we think that an engineering-forward data orchestrator is uniquely positioned to be the right tool for data pipelines.

It is a unique hub of context and can integration information about all the tools in the stack, the source code in your stack, the structure of your platform, and the operational metadata.
We have built this with guardrails in mind, not just a cherry on top. Compared to relatively unstructured DAGs, we believe Components are much more suited to incremental changes (which is nearly all of them in practice) in an existing pipeline or platform
And the data orchestrator has visibility and drives the workflow across the entire lifecycle, which is necessary to do all of this safely.

34 of 38

Demo: AI Components

34

35 of 38

Data Platform Uniquely Good AI �Use Case

Diverse Practitioners that need precise outcomes: e.g. Analysts, Data Scientists, Engineers

Wide surface area: Tons of tools and configuration knowable by LLM

Tribal Knowledge: Company-specific integrations, legacy tech, and uneven documentation

35

36 of 38

Conclusion

Data platform teams have historically had to make an excruciating trade offs between leverage and accessibility.

Components increase accessibility over a adaptable, powerful platform and abstraction layer

Components is built for the future, as an AI-optimized framework

36

37 of 38

What’s launching today

Integrations for Embedded ELT, DBT, and Pipes

Early access to Components for select partners

Coming soon: �The Components AI experience

37

38 of 38

Thank you!

Want to learn more?

Request a demo��dagster.io/request-a-demo

Join us on Slack��#dg-components

38