1 of 31

MotherDuck:

DuckDB in the cloud & in the client

NWDS 2024

Jordan Tigani

co-founder & chief duck-herder @MotherDuck

2024-02-09

2 of 31

Remembrance of NWDS Meetings Past

NWDS 2022

3 of 31

What is DuckDB?

�A lightweight, in-process SQL Analytics Engine that is taking the data world by storm.

4 of 31

Embedded Database Execution w/ DuckDB

Client / Server Database Execution

DuckDB Execution

driver

results!

Database

Server

Application

SQL

data

DuckDB

results!

Application

SQL

data

API

5 of 31

What is MotherDuck?

�A serverless DuckDB platform for low-cost, low-friction analytics that will scale to support complex apps and data

6 of 31

MotherDuck Architecture

7 of 31

MotherDuck Architecture

Client Extension Server Extension

8 of 31

What do the Extensions do?

remote

(duckling)

​

MotherDuck

Server

Extension

local

(client)

​

MotherDuck

Client

Extension

DuckDB

Par

se

Bind

Optimize

Execute

Bind

Optimize

Execute

Execute

results

Parse

Bind

DuckDB

Parse

Bind

Optimize

Execute

9 of 31

What do the Extensions do?

local (client)

​

remote (duckling)

​

​

Extension

DuckDB

Parse

Bind

Optimize

Execute

Bind

Optimize

Execute

Execute

results

Parse

Bind

10 of 31

What do the Extensions do?

local (client)

​

remote (duckling)

​

​

Extension

DuckDB

Parse

Bind

Optimize

Execute

Bind

Optimize

Execute

Execute

results

Parse

FROM parquet_scan(“/home/tab.parquet”) t SELECT sum(t.c)

“recognize tables &

where they

come from”

Bind

11 of 31

What do the Extensions do?

local (client)

​

remote (duckling)

​

​

Extension

DuckDB

Parse

Bind

Optimize

Execute

Bind

Optimize

Execute

Execute

results

Parse

FROM parquet_scan(“s3://tab.parquet”) t SELECT sum(t.c)

​

Bind

“recognize tables &

where they

come from”

12 of 31

What do the Extensions do?

local (client)

​

remote (duckling)

​

​

Extension

DuckDB

Parse

Bind

Optimize

Execute

Bind

Optimize

Execute

Execute

results

Parse

FROM local_db.tab t SELECT sum(t.c)

Bind

“recognize tables &

where they

come from”

13 of 31

What do the Extensions do?

local (client)

​

remote (duckling)

​

​

Extension

DuckDB

Parse

Bind

Optimize

Execute

Bind

Optimize

Execute

Execute

results

Parse

FROM remote_db.tab t SELECT sum(t.c)

Bind

“recognize tables &

where they

come from”

14 of 31

What do the Extensions do?

local (client)

​

remote (duckling)

​

​

Extension

DuckDB

Parse

Bind

Optimize

Execute

Bind

Optimize

Execute

Execute

results

Parse

FROM remote_db.tab t SELECT sum(t.c)

Virtual catalog (skip network hop)

Bind

“recognize tables &

where they

come from”

15 of 31

What do the Extensions do?

local (client)

​

remote (duckling)

​

​

Extension

DuckDB

Parse

Bind

Optimize

Execute

Bind

Optimize

Execute

Execute

results

Parse

FROM range(1,100000000) LIMIT 100

“location agnostic” data source

Bind

“recognize tables &

where they

come from”

16 of 31

Query Pipelines

SELECT count(*) FROM t1 JOIN t2 ON t1.a = t2.a JOIN t3 ON t3.b = t2.b

17 of 31

Query Pipelines

SELECT count(*) FROM t1 JOIN t2 ON t1.a = t2.a JOIN t3 ON t3.b = t2.b

aggregate

join

join

scan(t1)

scan(t2)

scan(t3)

18 of 31

Query Pipelines

SELECT count(*) FROM t1 JOIN t2 ON t1.a = t2.a JOIN t3 ON t3.b = t2.b

aggregate

join

join

scan(t1)

scan(t2)

scan(t3)

19 of 31

What do the Extensions do?

local (client)

remote (duckling)

​

Extension

DuckDB

Parse

Bind

Optimize

Execute

Bind

Optimize

Execute

Execute

results

Parse

“decide which pipeline to run where”

Bind

20 of 31

Local-Remote Planning

SELECT count(*) FROM t1 JOIN t2 ON t1.a = t2.a JOIN t3 ON t3.b = t2.b

aggregate

join

join

scan(t1)

scan(t2)

scan(t3)

21 of 31

Local-Remote Planning

​

SELECT count(*) FROM t1 JOIN t2 ON t1.a = t2.a JOIN t3 ON t3.b = t2.b

SELECT count(*) FROM t1 JOIN t2 ON t1.a = t2.a JOIN t3 ON t3.b = t2.b

aggregate

join

join

scan(t1)

scan(t2)

scan(t3)

local

remote

agnostic

22 of 31

Local-Remote Planning

SELECT count(*) FROM t1 JOIN t2 ON t1.a = t2.a JOIN t3 ON t3.b = t2.b

SELECT count(*) FROM t1 JOIN t2 ON t1.a = t2.a JOIN t3 ON t3.b = t2.b

aggregate

join

join

scan(t1)

scan(t2)

scan(t3)

23 of 31

Bridge Operators

​

SELECT count(*) FROM t1 JOIN t2 ON t1.a = t2.a JOIN t3 ON t3.b = t2.b

SELECT count(*) FROM t1 JOIN t2 ON t1.a = t2.a JOIN t3 ON t3.b = t2.b

aggregate

join

join

scan(t1)

scan(t2)

scan(t3)

bridge operator

24 of 31

What do the Extensions do?

local (client)

remote (duckling)

​

Extension

DuckDB

Parse

Bind

Optimize

Execute

Bind

Optimize

Execute

Execute

results

Parse

“bridge operators transport data in flow”

Bind

25 of 31

DEMO TIME!

26 of 31

Exploring Hybrid Topologies

Future Work

27 of 31

Parallel Scans

Note similarity to Aggregators & leaves in CDW

Server

Client

Query Engine

Object Store

Query Engine

Server

Query Engine

Server

Query Engine

28 of 31

Edge Cache

Cache can be recent data, sampled data, or region-fenced data

Server

Server

Query Engine

Query Engine

Client

Query Engine

Local Region / PoP

Central Region

Object Store

Cache

29 of 31

Cross-Region Joins

Server

Server

Query Engine

Object Store

Query Engine

Server

Query Engine

Client

Query Engine

Region 2

Region 1

Object Store

30 of 31

Reverse the polarity

“Find Security Threats”�Minimize Egress

Client

Query Engine

Local DB

Client

Query Engine

Local DB

Client

Query Engine

Local DB

Server

Query Engine

Admin

Query Engine

31 of 31

Thank you!