1 of 30

Faster conclusions using in-memory columnar SQL and machine learning

Hortonworks – May 3, 2016

Apache Arrow

DREMIO

2 of 30

Who

Wes McKinney

  • Engineer at Cloudera, formerly DataPad CEO/founder
  • Wrote bestseller Python for Data Analysis 2012
  • Open source projects
    • Python {pandas, Ibis, statsmodels}
    • Apache {Arrow, Parquet, Kudu (incubating)}
  • Mostly work in Python and Cython/C/C++

Jacques Nadeau

  • CTO & Co-Founder at Dremio, formerly Architect at MapR
  • Open Source projects
    • Apache {Arrow, Parquet, Calcite, Drill, HBase, Phoenix}
  • Mostly work in Java

DREMIO

3 of 30

Arrow in a Slide

  • New Top-level Apache Software Foundation project
    • Announced Feb 17, 2016

​

  • Focused on Columnar In-Memory Analytics
    1. 10-100x speedup on many workloads
    2. Common data layer enables companies to choose best of breed systems
    3. Designed to work with any programming language
    4. Support for both relational and complex data as-is

​

  • Developers from 13+ major open source projects involved
    • A significant % of the world’s data will be processed through Arrow!

​

Calcite

Cassandra

Deeplearning4j

Drill

Hadoop

HBase

Ibis

Impala

Kudu

Pandas

Parquet

Phoenix

Spark

Storm

R

DREMIO

4 of 30

Agenda

  • Purpose
  • Memory Representation
  • Language Bindings
  • IPC & RPC
  • Example Integrations

DREMIO

5 of 30

Purpose

DREMIO

6 of 30

Overview

  • A high speed in-memory representation
  • Well-documented and cross language compatible
  • Designed to take advantage of modern CPU characteristics
  • Embeddable in execution engines, storage layers, etc.

DREMIO

7 of 30

Focus on CPU Efficiency

  • Cache Locality
  • Super-scalar & vectorized operation
  • Minimal Structure Overhead
  • Constant value access
    • With minimal structure overhead
  • Operate directly on columnar compressed data

Traditional

Memory Buffer

Arrow

Memory Buffer

DREMIO

8 of 30

High Performance Sharing & Interchange

Today

With Arrow

  • Each system has its own internal memory format
  • 70-80% CPU wasted on serialization and deserialization
  • Similar functionality implemented in multiple projects
  • All systems utilize the same memory format
  • No overhead for cross-system communication
  • Projects can share functionality (eg, Parquet-to-Arrow reader)

DREMIO

9 of 30

Shared Need > Open Source Opportunity

  • Columnar is Complex
  • Shredded Columnar is even more complex
  • We all need to go to same place
  • Take Advantage of Open Source approach
  • Once we pick a shared solution, we get interchange for “free”

“We are also considering switching to a columnar canonical in-memory format for data that needs to be materialized during query processing, in order to take advantage of SIMD instructions” -Impala Team

“A large fraction of the CPU time is spent waiting for data to be fetched from main memory…we are designing cache-friendly algorithms and data structures so Spark applications will spend less time waiting to fetch data from memory and more time doing useful work – Spark Team

DREMIO

10 of 30

In Memory Representation

DREMIO

11 of 30

Columnar data

persons = [{

name: 'wes',

iq: 180,

addresses: [

{number: 2, street 'a'},

{number: 3, street 'bb'}

]

}, {

name: 'joe',

iq: 100,

addresses: [

{number: 4, street 'ccc'},

{number: 5, street 'dddd'},

{number: 2, street 'f'}

]

}]

DREMIO

12 of 30

Simple Example: persons.iq

DREMIO

13 of 30

Simple Example: persons.addresses.number

DREMIO

14 of 30

Columnar data

DREMIO

15 of 30

Language Bindings

DREMIO

16 of 30

Language Bindings

  • Target Languages
    • Java (beta)
    • CPP (underway)
    • Python & Pandas (underway)
    • R
    • Julia
  • Initial Focus
    • Read a structure
    • Write a structure
    • Manage Memory

DREMIO

17 of 30

Java: Creating Dynamic Off-heap Structures

FieldWriter w= getWriter();

w.varChar("name").write("Wes");

w.integer("iq").write(180);

ListWriter list = writer.list("addresses");

list.startList();

MapWriter map = list.map();

map.start();

map.integer("number").writeInt(2);

map.varChar("street").write("a");

map.end();

map.start();

map.integer("number").writeInt(3);

map.varChar("street").write("bb");

map.end();

list.endList();

{

name: 'wes',

iq: 180,

addresses: [

{number: 2, street 'a'},

{number: 3, street 'bb'}

]

}

Json Representation

Programmatic Construction

DREMIO

18 of 30

Java: Memory Management (& NVMe)

  • Chunk-based managed allocator
    • Built on top of Netty’s JEMalloc implementation
  • Create a tree of allocators
    • Limit and transfer semantics across allocators
    • Leak detection and location accounting
  • Wrap native memory from other applications
  • New support for integration with Intel’s Persistent Memory library via Apache Mnemonic

DREMIO

19 of 30

RPC & IPC

DREMIO

20 of 30

Common Message Pattern

  • Schema Negotiation
    • Logical Description of structure
    • Identification of dictionary encoded Nodes
  • Dictionary Batch
    • Dictionary ID, Values
  • Record Batch
    • Batches of records up to 64K
    • Leaf nodes up to 2B values

​

Schema Negotiation

Dictionary Batch

Record Batch

Record Batch

Record Batch

1..N Batches

0..N Batches

DREMIO

21 of 30

Record Batch Construction

Schema Negotiation

Dictionary Batch

Record Batch

Record Batch

Record Batch

name (offset)

name (data)

iq (data)

addresses (list offset)

addresses.number

addresses.street (offset)

addresses.street (data)

data header (describes offsets into data)

name (bitmap)

iq (bitmap)

addresses (bitmap)

addresses.number (bitmap)

addresses.street (bitmap)

{

name: 'wes',

iq: 180,

addresses: [

{number: 2,

street 'a'},

{number: 3,

street 'bb'}

]

}

Each box is contiguous memory, entirely contiguous on wire

DREMIO

22 of 30

RPC & IPC: Moving Data Between Systems

RPC

  • Avoid Serialization & Deserialization
  • Layer TBD: Focused on supporting vectored io
    • Scatter/gather reads/writes against socket

​

IPC

  • Alpha implementation using memory mapped files
    • Moving data between Python and Drill
  • Working on shared allocation approach
    • Shared reference counting and well-defined ownership semantics

DREMIO

23 of 30

Real World Examples

DREMIO

24 of 30

Real World Example: Python With Spark or Drill

DREMIO

25 of 30

Real World Example: Feather File Format for Python and R

  • Problem: fast, language-agnostic binary data frame file format
  • Written by Wes McKinney (Python) Hadley Wickham (R)
  • Read speeds close to disk IO performance

DREMIO

26 of 30

Real World Example: Feather File Format for Python and R

library(feather)

path <- "my_data.feather"

write_feather(df, path)

df <- read_feather(path)

import feather

path = 'my_data.feather'

feather.write_dataframe(df, path)

df = feather.read_dataframe(path)

R

Python

DREMIO

27 of 30

More on Feather

array 0

array 1

array 2

...

array n - 1

METADATA

Feather File

libfeather

C++ library

Rcpp

Cython

R data.frame

pandas DataFrame

DREMIO

28 of 30

Feather: the good and not-so-good

  • Good
    • Language-agnostic memory representation
    • Extremely fast
    • New storage features can be added without much difficulty

​

  • Not-so-good
    • Data must be convert to/from storage representation (Arrow) and in-memory “proprietary” data structures (R / Python data frames)

DREMIO

29 of 30

What’s Next

  • Parquet for Python & C++
    • Using Arrow Representation
  • Available IPC Implementation
  • Spark, Drill Integration
    • Faster UDFs, Storage interfaces

DREMIO

30 of 30

Get Involved

  • Join the community

DREMIO