1 of 31

1

2 of 31

A leading Data & AI Consultancy in the Netherlands

Define data &

AI strategies

Build data

platforms and

data infrastructure

Develop data &

AI solutions

…to execution

From strategy…

Analytics Translators

Analytics Engineers�Data Scientists�ML Engineers

Data Engineers

Data Architects

2

3 of 31

Open positions
Account managers
Support Engineer
Analytics Engineers
Data Engineers
ML Engineers
Data Scientists
Data Solutions Architect

3

4 of 31

4

5 of 31

Intro to DataFusion: Technology, Community, and Not Quite Enough Time

Andrew Lamb

Staff Engineer, InfluxData

January 23, 2025, Data & Drinks: Building Next-Gen Data Systems with Apache DataFusion

5

11am-noon @ UC Berkeley

Title: Implementing InfluxDB 3.0 and other systems without starting from "scratch" using Apache DataFusion

Abstract: In this talk, Andrew will describe the high level architecture of Apache DataFusion, and how it is used in InfluxDB 3.0's new high performance timeseries database engine. He will also describe how other systems use DataFusion, including LanceDB, Apache DataFusion Comet (a native Spark Runtime), and the Rust implementations of Delta Lake and Apache Iceberg to provide fast and highly interoperable analytic systems without a tightly integrated codebase built from scratch.

Bio:

Andrew Lamb is a Staff Engineer at InfluxData, working in Rust on InfluxDB 3.0, focused on query processing, the Apache DataFusion query engine and the Apache Arrow ecosystem. He serves on the Apache DataFusion PMC (2024 Chair), and on the Apache Arrow PMC (2023 Chair), and actively contributes to DataFusion and the Arrow Rust implementations. He earned a BS and MEng in Course VI from MIT. More details are available at http://andrew.nerdnetworks.org/

6 of 31

DataFusion: Technology

6

7 of 31

Top Level Project, Apache Software Foundation

Apache 2.0 Licensed

https://datafusion.apache.org/

7

8 of 31

Analogy: DataFusion is LLVM for Databases

Clang

Rust

Swift

C/C++ frontend

LLVM

Rustlang frontend

LLVM

Swift frontend

LLVM

…

LLVM enabled innovation in programming languages:

High quality reusable optimizer, code generator, debugger, lsp integration, etc.
Focus on language design, ecosystem, libraries, etc

8

9 of 31

Analogy: DataFusion is LLVM for Databases

Analytic Application

Domain Specific Language

Specialized Database

Application Logic

Catalog

Analysis Engine

Multiple SQL Dialects

Data Flow Analysis

Custom Operators

File System Interface

…

DataFusion enables innovation in data intensive systems

High quality reusable SQL planner, optimizer, function library, vectorized operators, etc
Focus on language design, data management, use case specific features

9

10 of 31

Architecture

SQL

Front Ends

DataFrame

LogicalPlan

ExecutionPlan

Plan Representations and Rewrites

Expression Eval

Optimizations / Transformations

HashAggregate

Sort

…

Execution Engine

Join

Catalog and

Data Sources

Parquet

CSV

…

Extension

Catalog / Table

Extension

Frontend

Extension

LogicalPlan Rewrite

Extension ExecutionPlan Rewrite

Extension

Stream

Extension Node

Streams

10

11 of 31

Recognized Tech

Apache Arrow DataFusion: A Fast, Embeddable, Modular Analytic Query Engine

In SIGMOD 2024

We wrote this paper entirely in the open

11

12 of 31

Top of the Line Performance

Apache DataFusion is now the fastest single node engine for querying Apache Parquet files

Speed (and underlying techniques) similar to other top engines such as ClickHouse + DuckDB

12

13 of 31

Top of the Line Performance (is fleeting!)

ClickBench Results as of Jan 23, 2025

Looking for help rerunning ClickBench on 44 (and 45!)

Update ClickBench benchmarks with DataFusion 44.0.0 #13983

13

14 of 31

Trending

https://db.cs.cmu.edu/seminar2024/

Who’s who of DataFusion users

14

15 of 31

Architecture

Design Goals:

Work “out of the box” (fast time to awesome)
Customize everything via APIs
Architecturally Boring 🥱 (“Industrial best practice”)

Results for Users

Quickly start with a basic, high-performance engine
Specialize to suit their needs and available engineering capacity
Easy to try out new ideas (operators, rewrites, etc)

15

16 of 31

Use Case: File Formats (Lance)

Courtesy of Weston Pace

Decoder uses DataFusion Expr simplification to calculate zone pruning (source)

Encoder uses DataFusion aggregators to calculate min/max (source)

Lance Format

Arrow

Lance file format uses DataFusion to implement pushdown filtering

Encoder

Decoder

16

17 of 31

Use Case: Table Formats

delta-rs uses DataFusion for various features (and provides a TableProvider for reading)

delta-rs / deltalake python package

write

Predicate Evaluation (in overwrite mode)

Optimize (compact)

Source (link)

Load

Delete

…

Z-Order evaluation

sort

TableProvider:

Projection pushdown,

Limit pushdown

Predicate pushdown

Predicate Evaluation

Load_CDF

Custom plan nodes

for change data

deltafeeds

17

18 of 31

Use Case: SQL Analysis (frontend)

SDF

Assembly

Intelisense

Tests

Reports

Business Value

Catalog

BigQuery

Redshift

Metadata

Unified Logical Plan

SDF

Static Analysis

SDF Development Framework

Ingest

Deploy

Analyze

Guarantee

Snowflake

Trino/Presto

SDF uses complete ANTLR Grammars to define many SQL dialects - notably, proprietary ones like Snowflake.

All SQLs compile to a unified Intermediate Representation: the Datafusion Logical Plan.

This gives SDF Executable Semantics.

SDF’s transformation layer statically analyzes many logical plans at once for correctness and generates rich metadata.

Courtesy of SDF / Lukas Shute

18

19 of 31

Use Case: Execution Engine

Integration Layer with Spark

DataFusion’s ExecutionPlan Streams

Use Spark Planner / Executor machinery

19

20 of 31

Use Case: Specialized Database Systems

Examples

GreptimeDB (observability)
InfluxDB 3.0 (timeseries)
LanceDB (AI)
HoraeDB (observability)
Coralogix (observability)
Exon (bioinformatics)
Arroyo (streaming)
Synnada (streaming/AI)
…

Domain Specific Language

Catalog

Custom Operators

See more: Apache DataFusion documentation

20

21 of 31

DataFusion: Community + Not Quite Enough Time

21

22 of 31

“To achieve great things, two things are needed; a plan, and not quite enough time.”

- Leonard Bernstein (according to the internet)

22

23 of 31

Who Controls Project / Roadmap

DataFusion development is NOT directly funded

Users contribute together (including engineers paid by companies who build using DataFusion)
No one paid full time to work on DataFusion

⇒ Always a bit short on time, especially:

Reviews, Documentation, Ticket Triage

Pro: forced prioritization

Plug: Join us to have fun and have a say 🎣

🤑💰

23

24 of 31

Community

Not started / donated by a company: founded by Andy Grove

Community:

694 distinct contributors*
14 PMC members + 42 Committers
Distinct contributors 96 in 43.0.0, 94 in 44.0.0

Velocity:

Monthly releases for the last 3 years
Multiple commits a day (😅 still!)

* Caveat: some distortion due to tortured git history

24

25 of 31

“Community over Code” - The Apache Way

Non profit governance of open source communities

25

26 of 31

Apache: Benefits for DataFusion

⇒ Predictable Foundation
Stable License: (ASL 20 years old) low risk of changes, (ahem OpenTofu)
Communication: Predictable and open (if slow)
Multi-Vendor Participation: Shared investment reduces individual risk
Long Term Maintenance: Hedged against life changes, corporate strategy shifts, VC funding cycles
⭐⭐⭐⭐⭐: Works far better than could be reasonably expected

26

27 of 31

My Personal / Professional Goal

1,000+ projects!

( Used to be a crazy number I just made up. Not so crazy anymore…)

27

28 of 31

Thank You: On with the Talks

https://datafusion.apache.org/

https://github.com/apache/datafusion

Communication (Slack/Discord/etc)

28

29 of 31

T H A N K Y O U

29

30 of 31

Backup

30

31 of 31

DataFusion / Query Engine: Input / Output

31

Data Batches

SQL Query

SELECT status, COUNT(1)

FROM http_api_requests_total

WHERE path = '/api/v2/write'

GROUP BY status;

Data Batches

DataFrame

ctx.read_table("http")?

.filter(...)?

.aggregate(..)?;

Catalog information:

tables, schemas, etc

31