1 of 29

DEVELOPING ELASTIC DATA PIPELINES

FAST DATA 101

1

© 2016 Mesosphere, Inc. All Rights Reserved.

2 of 29

2

In the beginning, there was batch

© 2016 Mesosphere, Inc. All Rights Reserved.

3 of 29

3

MapReduce was a step forward

© 2016 Mesosphere, Inc. All Rights Reserved.

4 of 29

4

But we’re not bone-crunching any more

© 2016 Mesosphere, Inc. All Rights Reserved.

5 of 29

FAST DATA

​

EVERY MINUTE ...

YouTube: 300 hours of video uploaded [1]

Google: 3.4M searches [2]

Twitter: 443k tweets [2]

Email: 152M messages [2]

Facebook: 3.3M pieces of content shared [3]

​

​

​

​

​

[1] http://fortunelords.com/youtube-statistics/

[2] http://www.internetlivestats.com/one-second/

[3] https://zephoria.com/top-15-valuable-facebook-statistics/

5

© 2016 Mesosphere, Inc. All Rights Reserved.

6 of 29

WEB SCALE? A380 SCALE!

6

A380-1000: 10,000 sensors in each wing;�produces more than 7Tb of IoT data per day

​

[1] https://goo.gl/2S4q5N

​

© 2016 Mesosphere, Inc. All Rights Reserved.

7 of 29

Streaming

Spark

DC/OS

Real-time

Hadoop

IoT

Batch

Container

Mesos

Message Queue

Let’s Play Fast Data Buzzword Bingo

7

© 2016 Mesosphere, Inc. All Rights Reserved.

8 of 29

THE FOUNDATIONS OF FAST DATA

8

Sensors�& Sources

Data Processing

Modern�Apps

Message�Queue

© 2016 Mesosphere, Inc. All Rights Reserved.

9 of 29

MESSAGE QUEUES

Message Brokers

  • Apache Kafka
  • ØMQ, RabbitMQ, Disque

Log-based Queues

  • fluentd, Logstash, Flume

see also queues.io

9

© 2016 Mesosphere, Inc. All Rights Reserved.

10 of 29

APACHE KAFKA

Typical Use: A reliable buffer for stream processing

�Why Kafka?

  • High-throughput, distributed, persistent publish-subscribe messaging system
  • Created by LinkedIn; used in production by 100+ web-scale companies [1]

​

[1] https://cwiki.apache.org/confluence/display/KAFKA/Powered+By

10

© 2016 Mesosphere, Inc. All Rights Reserved.

11 of 29

DELIVERY GUARANTEES

  • At most once—Messages may be lost but are never redelivered
  • At least once—Messages are never lost but may be redelivered (Kafka)
  • Exactly once—Messages are delivered once and only once (this is what everyone actually wants, but no one can deliver!)

11

Murphy’s Law of Distributed Systems:

�Anything that can go wrong, will go wrong … partially!

© 2016 Mesosphere, Inc. All Rights Reserved.

12 of 29

STREAM PROCESSING

Microbatching

  • Apache Spark (Streaming)

Native Streaming

  • Apache Flink
  • Apache Storm/Heron
  • Apache Apex
  • Apache Samza

12

© 2016 Mesosphere, Inc. All Rights Reserved.

13 of 29

APACHE SPARK (STREAMING)

Typical Use: distributed, large-scale data processing; micro-batching

�Why Spark Streaming?

  • Micro-batching creates very low latency, which can be faster
  • Well defined role means it fits in well with other pieces of the pipeline

13

© 2016 Mesosphere, Inc. All Rights Reserved.

14 of 29

STORAGE

NoSQL

  • ArangoDB
  • mongoDB
  • Apache Cassandra
  • Apache HBase

SQL

  • MemSQL

Filesystems

  • Quobyte
  • HDFS

Time-Series Datastores

  • InfluxDB
  • OpenTSDB
  • KairosDB
  • Prometheus

see also iot-a.info

14

© 2016 Mesosphere, Inc. All Rights Reserved.

15 of 29

APACHE CASSANDRA

Typical Use: No-dependency, time series database

�Why Cassandra?

  • A top level Apache project born at Facebook and built on Amazon’s Dynamo and Google’s BigTable
  • Offers continuous availability, linear scale performance, operational simplicity and easy data distribution

15

© 2016 Mesosphere, Inc. All Rights Reserved.

16 of 29

AN EXAMPLE STACK: �THE “SMACK”

16

Apache Spark: distributed, large-scale data processing

Apache Mesos: cluster resource manager

Akka: toolkit for message driven applications

Apache Cassandra: distributed, highly-available database

Apache Kafka: distributed, highly-available messaging system

© 2016 Mesosphere, Inc. All Rights Reserved.

17 of 29

Streaming

Spark

DC/OS

Real-time

Hadoop

IoT

Batch

Container

Mesos

Message Queue

A Few More To Cross Off ...

17

© 2016 Mesosphere, Inc. All Rights Reserved.

18 of 29

LET’S GET IT WORKING

Challenges

  • Distributed Systems Are Hard
  • Setup of Components
  • Elasticity
  • Efficient Use of Cluster Resources
  • Monitoring
  • Debugging

18

© 2016 Mesosphere, Inc. All Rights Reserved.

19 of 29

DISTRIBUTED ARCHITECTURES

19

​

​

​

​

CaaS

PaaS

Traditional Approach

​

​

​

​

Big Data Analytics

Stateful Service

Modern Approach

Distributed Operating System

​

Container App

​

Container App

Big Data Analytics #2

Stateful Service #1

Big Data Analytics #1

Stateful Service #2

​

Container App

​

Container App

BIG DATA SERVICES

MICROSERVICES

CaaS

PaaS

© 2016 Mesosphere, Inc. All Rights Reserved.

20 of 29

LINUX CONTAINERS

The Why and What:

  • Containers vs. VMs
  • App-level dependency management
  • Lightweight (startup time, footprint, runtime)
  • Isolation and security

namespaces

  • Isolate PIDs between processes
  • Isolate process to network resources
  • Isolate the hostname to fake it out (UTS)
  • Isolate the filesystem mount points (chroot)
  • Isolate inter-process communication (IPC)
  • Isolate specific users to specific processes

cgroups

20

© 2016 Mesosphere, Inc. All Rights Reserved.

21 of 29

APACHE MESOS

Typical Use: The premier resource manager and negotiator

�Why Mesos?

  • 2-level scheduling
  • Fault-tolerant, battle-tested
  • Scalable to 10,000+ nodes
  • Created by Mesosphere founder @ UC Berkeley; used in production by 100+ web-scale companies [1]

​

[1] http://mesos.apache.org/documentation/latest/powered-by-mesos/

21

© 2016 Mesosphere, Inc. All Rights Reserved.

22 of 29

MESOS: FUNDAMENTAL ARCHITECTURE

22

Mesos Master

Mesos Master

Mesos Master

Mesos Agent

Mesos Agent Service

Cassandra Executor

Cassandra Task

Cassandra Scheduler

Container Scheduler

Spark Scheduler

Spark Executor

Spark� Task

Mesos Agent

Mesos Agent Service

Docker Executor

Docker� Task

Spark Executor

Spark� Task

Two-level Scheduling

  1. Agents advertise resources to Master
  2. Master offers resources to Framework
  3. Framework rejects / uses resources
  4. Agent reports task status to Master

© 2016 Mesosphere, Inc. All Rights Reserved.

23 of 29

Streaming

Spark

DC/OS

Real-time

Hadoop

IoT

Batch

Container

Mesos

Message Queue

So… What’s Left?

23

© 2016 Mesosphere, Inc. All Rights Reserved.

24 of 29

DC/OS ENABLES MODERN DISTRIBUTED APPS

24

Datacenter Operating System (DC/OS)

Distributed Systems Kernel (Mesos)

Big Data + Analytics Engines

Microservices (in containers)

Streaming

Batch

Machine Learning

Analytics

Functions & Logic

Search

Time Series

SQL / NoSQL

Databases

Modern App Components

Distributed systems kernel to abstract resources

Ecosystem of frameworks & apps

​

Consistent architecture to run on top of kernel

User Interface (GUI & CLI)

​

Core system services �(e.g., distributed init, cron, service discovery, package mgt & installer, storage)

Any Infrastructure (Physical, Virtual, Cloud)

© 2016 Mesosphere, Inc. All Rights Reserved.

25 of 29

DC/OS BENEFITS

All-In-One Cluster

  • Stateless services (web servers, app servers)
  • Stateful services (MemSQL, Kafka, Cassandra)
  • Elastic data processing (Spark, Akka)
  • CI/CD (Jenkins)

Dynamic partitioning of the cluster

Increased utilization

100% open source

  • An umbrella for ~30 OSS projects
  • A big, diverse community
  • Not limited in any way

25

© 2016 Mesosphere, Inc. All Rights Reserved.

26 of 29

26

A COMPLETE�CLI & GUI

© 2016 Mesosphere, Inc. All Rights Reserved.

27 of 29

27

THE UNIVERSE

© 2016 Mesosphere, Inc. All Rights Reserved.

28 of 29

28

© 2016 Mesosphere, Inc. All Rights Reserved.

29 of 29

ANY QUESTIONS?

THANK YOU!

29

@dcos

users@dcos.io

/groups/8295652

/dcos

/dcos/examples

/dcos/demos

chat.dcos.io

© 2016 Mesosphere, Inc. All Rights Reserved.