1 of 30

Intro to DataFusion: Technology, Community, and Not Quite Enough Time

Andrew Lamb | Staff Engineer, InfluxData

January 15, 2025, Boston DataFusion Meetup

1

11am-noon @ UC Berkeley

Title: Implementing InfluxDB 3.0 and other systems without starting from "scratch" using Apache DataFusion

Abstract: In this talk, Andrew will describe the high level architecture of Apache DataFusion, and how it is used in InfluxDB 3.0's new high performance timeseries database engine. He will also describe how other systems use DataFusion, including LanceDB, Apache DataFusion Comet (a native Spark Runtime), and the Rust implementations of Delta Lake and Apache Iceberg to provide fast and highly interoperable analytic systems without a tightly integrated codebase built from scratch.

Bio:

Andrew Lamb is a Staff Engineer at InfluxData, working in Rust on InfluxDB 3.0, focused on query processing, the Apache DataFusion query engine and the Apache Arrow ecosystem. He serves on the Apache DataFusion PMC (2024 Chair), and on the Apache Arrow PMC (2023 Chair), and actively contributes to DataFusion and the Arrow Rust implementations. He earned a BS and MEng in Course VI from MIT. More details are available at http://andrew.nerdnetworks.org/

2 of 30

Outline

Acknowledgements
Agenda
Intro to DataFusion

2

3 of 30

Thank you to our Sponsors

Room and facilities

(thanks Justin & Gladys!)

3

4 of 30

Thank you to our Sponsors

Food

4

5 of 30

Talks

6:00 - 6:20: Intro to DataFusion: Technology, Community, and Not Quite Enough Time

Andrew Lamb is a Staff Engineer at InfluxData, working on InfluxDB 3.0, focused on query processing, the Apache DataFusion query engine and the Apache Arrow ecosystem. He is a member of the Apache Software Foundation and the Apache DataFusion PMC (2024 Chair), and Apache Arrow PMC (2023 Chair) (LinkedIn).

6:20 - 6:40: Embucket: a Snowflake-compatible lakehouse based on DataFusion

Camuel Gilyadov is Founder and CEO of Embucket (LinkedIn)

6:40 - 7:00: Stateless engines, table formats, and the future

Jake Thomas is Manager, Data Foundations at Okta (LinkedIn)

6:40 - 7:00: Building InfluxDB 3.0 with DataFusion

Andrew Lamb

Update: Jake is unable to make it tonight 🤧

5

6 of 30

DataFusion: Technology

6

7 of 30

Top Level Project, Apache Software Foundation

Apache 2.0 Licensed

https://datafusion.apache.org/

7

8 of 30

Analogy: DataFusion is LLVM for Databases

Clang

Rust

Swift

C/C++ frontend

LLVM

Rustlang frontend

LLVM

Swift frontend

LLVM

…

LLVM enabled innovation in programming languages:

High quality reusable optimizer, code generator, debugger, lsp integration, etc.
Focus on language design, ecosystem, libraries, etc

8

9 of 30

Analogy: DataFusion is LLVM for Databases

Analytic Application

Domain Specific Language

Specialized Database

Application Logic

Catalog

Analysis Engine

Multiple SQL Dialects

Data Flow Analysis

Custom Operators

File System Interface

…

DataFusion enables innovation in data intensive systems

High quality reusable SQL planner, optimizer, function library, vectorized operators, etc
Focus on language design, data management, use case specific features

9

10 of 30

Recognized Tech

Apache Arrow DataFusion: A Fast, Embeddable, Modular Analytic Query Engine

In SIGMOD 2024

We wrote this paper entirely in the open

10

11 of 30

Top of the Line Performance

Apache DataFusion is now the fastest single node engine for querying Apache Parquet files

Speed (and underlying techniques) similar to other top engines such as ClickHouse + DuckDB

11

12 of 30

Trending

https://db.cs.cmu.edu/seminar2024/

Who’s who of DataFusion users

12

13 of 30

Architecture

Design Goals:

Work “out of the box” (fast time to awesome)
Customize everything via APIs
Architecturally Boring 🥱 (“Industrial best practice”)

Results for Users

Quickly start with a basic, high-performance engine
Specialize to suit their needs and available engineering capacity
Easy to try out new ideas (operators, rewrites, etc)

13

14 of 30

Use Case: File Formats (Lance)

Courtesy of Weston Pace

Decoder uses DataFusion Expr simplification to calculate zone pruning (source)

Encoder uses DataFusion aggregators to calculate min/max (source)

Lance Format

Arrow

Lance file format uses DataFusion to implement pushdown filtering

Encoder

Decoder

14

15 of 30

Use Case: Table Formats

delta-rs uses DataFusion for various features (and provides a TableProvider for reading)

delta-rs / deltalake python package

write

Predicate Evaluation (in overwrite mode)

Optimize (compact)

Source (link)

Load

Delete

…

Z-Order evaluation

sort

TableProvider:

Projection pushdown,

Limit pushdown

Predicate pushdown

Predicate Evaluation

Load_CDF

Custom plan nodes

for change data

deltafeeds

15

16 of 30

Use Case: SQL Analysis (frontend)

SDF

Assembly

Intelisense

Tests

Reports

Business Value

Catalog

BigQuery

Redshift

Metadata

Unified Logical Plan

SDF

Static Analysis

SDF Development Framework

Ingest

Deploy

Analyze

Guarantee

Snowflake

Trino/Presto

SDF uses complete ANTLR Grammars to define many SQL dialects - notably, proprietary ones like Snowflake.

All SQLs compile to a unified Intermediate Representation: the Datafusion Logical Plan.

This gives SDF Executable Semantics.

SDF’s transformation layer statically analyzes many logical plans at once for correctness and generates rich metadata.

Courtesy of SDF / Lukas Shute

16

17 of 30

Use Case: Execution Engine

Integration Layer with Spark

DataFusion’s ExecutionPlan Streams

Use Spark Planner / Executor machinery

17

18 of 30

Use Case: Specialized Database Systems

Examples

GreptimeDB (observability)
InfluxDB 3.0 (timeseries)
LanceDB (AI)
HoraeDB (observability)
Coralogix (observability)
Exon (bioinformatics)
Arroyo (streaming)
Synnada (streaming/AI)
…

Domain Specific Language

Catalog

Custom Operators

See more: Apache DataFusion documentation

18

19 of 30

DataFusion: Community + Not Quite Enough Time

19

20 of 30

“To achieve great things, two things are needed; a plan, and not quite enough time.”

- Leonard Bernstein (according to the internet)

20

21 of 30

Who Controls Project / Roadmap

DataFusion development is NOT directly funded

Users contribute together (including engineers paid by companies who build using DataFusion)

⇒ Always a bit short on time, especially:

Reviews, Documentation, Ticket Triage

Pro: forced prioritization

Plug: Join us to have fun and have a say 🎣

🤑💰

21

22 of 30

Community

Not started / donated by a company: founded by Andy Grove

Community:

694 distinct contributors*
14 PMC members + 42 Committers
Distinct contributors: 73 in 42.0.0, 96 in 43.0.0

Velocity:

Monthly releases for the last 3 years
Many commits a day

* Caveat: some distortion due to tortured git history

22

23 of 30

“Community over Code” - The Apache Way

Non profit governance of open source communities

23

24 of 30

Apache: Benefits for DataFusion

⇒ Predictable Foundation
Stable License: (ASL 20 years old) low risk of changes, (ahem OpenTofu)
Communication: Predictable and open (if slow)
Multi-Vendor Participation: Shared investment reduces individual risk
Long Term Maintenance: Hedged against life changes, corporate strategy shifts, VC funding cycles
⭐⭐⭐⭐⭐: Works far better than could be reasonably expected

24

25 of 30

My Personal / Professional Goal

1,000+ projects!

( Used to be a crazy number I just made up. Not so crazy anymore…)

25

26 of 30

Thank You: On with the Talks

https://datafusion.apache.org/

https://github.com/apache/datafusion

Communication (Slack/Discord/etc)

26

27 of 30

T H A N K Y O U

27

28 of 30

Backup

28

29 of 30

DataFusion / Query Engine: Input / Output

29

Data Batches

SQL Query

SELECT status, COUNT(1)

FROM http_api_requests_total

WHERE path = '/api/v2/write'

GROUP BY status;

Data Batches

DataFrame

ctx.read_table("http")?

.filter(...)?

.aggregate(..)?;

Catalog information:

tables, schemas, etc

29

30 of 30

Architecture

SQL

Front Ends

DataFrame

LogicalPlan

ExecutionPlan

Plan Representations and Rewrites

Expression Eval

Optimizations / Transformations

HashAggregate

Sort

…

Execution Engine

Join

Catalog and

Data Sources

Parquet

CSV

…

Extension

Catalog / Table

Extension

Frontend

Extension

LogicalPlan Rewrite

Extension ExecutionPlan Rewrite

Extension

Stream

Extension Node

Streams

30