1 of 52

Advanced Big Data: Real-Time Data Processing

Dan Zaratsian

AI/ML Solutions Architect, Gaming @ Google

d.zaratsian@gmail.com

https://github.com/zaratsian

2 of 52

Session 1: Intro Big Data Architectures and Services
Session 2: SQL and NoSQL
Session 3: Spark ML
Session 4: Cloud Machine Learning
Session 5: Real-time Data Processing
Session 6: Cloud Services

TOPICS

3 of 52

Next up...

Session 2

SQL & NoSQL

Session 3

Distributed ML

Session 4

Cloud ML

Session 5

Real-time Systems

Session 6

Cloud

SQL

ML

GCP PubSub

Serverless

Vertex AI

4 of 52

Events

(Data In)

Events

(Data Out)

In-Stream

Data Processing

Re-route to

Dashboard / 3^rd Party Systems

Email

Alerts

4

Page 4

5 of 52

Events

(Data In)

Events

(Data Out)

Filtering

Aggregation

Pattern Detection

Computation

Merging / Joins

Retention Window

Text Analytics

Clustering

Machine Learning

…more…

Detect Events & Patterns of Interest

5

Page 5

6 of 52

Streaming Analytics Ecosystem

Edge

Analytics

In-Motion

Analytics

At-Rest

Analytics

Connected Systems, Devices

Monitor equipment on

for failures and safety

issues, and take action.

Identify fraudulent

transactions and be

alerted in real-time.

Intelligently integrate customer

information with real-time

streaming data

Strategic Data Integration

Transactions, Logs, Clickstreams

6

Page 6

7 of 52

Streaming Analytics: Types

Real-time Data Movement (Kafka, PubSub, etc)

Streaming Analytics (incremental, online, ESP)

Real-time Decisions (request & response) - Not quite streaming but can fit into real-time use case discussions.

7

Page 7

8 of 52

Where are the Opportunities?

Competitive Pressure (Technology, Sensors, Analytics)

Risk & Compliance

Safety

Security

Personalization

* In many applications, the value of data has a shelf life

8

Page 8

9 of 52

Batch Analytics

Logs

Clickstream

Social Media

Sensor Data

Call Center / Chat

Voice / Image / Video

Transactional

Documents

Get Data

Database

Historical Analysis

Adapters

FTP

APIs

Apache NiFi

Reports & Dashboards

Hive

BigQuery

Hbase

Cassandra

MongoDB

9

Page 9

10 of 52

Streaming Analytics – Process Flow

Logs

Clickstream

Social Media

Sensor Data

Call Center / Chat

Voice / Image / Video

Transactional

Documents

Get Data

Database

Historical Analysis

Adapters

FTP

APIs

Apache NiFi

Reports & Dashboards

Real-time Analytics

Alerts

Storm

Spark

Beam

Flink

SAS ESP

Hive

BigQuery

Hbase

Cassandra

MongoDB

10

Page 10

11 of 52

Streaming Analytics – Process Flow

Logs

Clickstream

Social Media

Sensor Data

Call Center / Chat

Voice / Image / Video

Transactional

Documents

Get Data

Database

Historical Analysis

Adapters

FTP

APIs

Apache NiFi

Reports & Dashboards

Real-time Analytics

Alerts

Bus

Kafka

RabbitMQ

PubSub

Kinesis

Storm

Spark

Beam

Flink

SAS ESP

Hive

BigQuery

Hbase

Cassandra

MongoDB

11

Page 11

12 of 52

Streaming Analytics – Process Flow

Logs

Clickstream

Social Media

Sensor Data

Call Center / Chat

Voice / Image / Video

Transactional

Documents

Get Data

Database

Historical Analysis

Adapters

FTP

APIs

Apache NiFi

Reports & Dashboards

Real-time Analytics

Alerts

Bus

Kafka

RabbitMQ

PubSub

Kinesis

Storm

Spark

Beam

Flink

SAS ESP

Hive

BigQuery

Hbase

Cassandra

MongoDB

12

Page 12

13 of 52

Streaming Analytics – Process Flow

Logs

Clickstream

Social Media

Sensor Data

Call Center / Chat

Voice / Image / Video

Transactional

Documents

Get Data

Bus

Real-time Analytics

Bus

Database

Historical Analysis

Adapters

FTP

APIs

Apache NiFi

Reports & Dashboards

3rd Party Service

Alerts

Internal Service (trigger offer)

Storm

Spark

Beam

Flink

SAS ESP

Kafka

RabbitMQ

PubSub

Kinesis

Hive

BigQuery

Hbase

Cassandra

MongoDB

13

Page 13

14 of 52

Developed in the NSA (code name NiagaraFiles)

Open-sourced in 2014 (NSA team left and started Onyara, which was acquired by Hortonworks).

Control, automate, and audit the bi-directional flow of data between software systems.

Confidential & Proprietary

15 of 52

Drag and drop processors to build a flow
Start, stop, and configure components in real time
View errors and corresponding error messages
View statistics and health of data flow
Create templates of common processor & connections

Confidential & Proprietary

16 of 52

Kafka

17 of 52

Developed by Linkedin

“Apache Kafka is used for building real-time data pipelines and streaming apps. It is horizontally scalable, fault-tolerant, wicked fast...” -- https://kafka.apache.org/

Up to 2 million writes/sec on 3 cheap machines (in 2014)

https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines

Confidential & Proprietary

18 of 52

Confidential & Proprietary

19 of 52

Kafka decouples Data Pipelines

Apache Kafka - Distributed Streaming Platform

Source System

Hadoop

Security Systems

Real-time monitoring

Data Warehouse

Kafka

Producers

Brokers

Consumers

Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem

Proprietary + Confidential

20 of 52

Kafka Topic - Category or feed name to which records are published.

Topics in Kafka are always multi-subscriber; that is, a topic can have zero, one, or many consumers that subscribe to the data written to it.

Producer (Publisher) - Publishes data to a topic

Consumer (Subscriber) - Receives data from a Topic either through pull or push methods.

Confidential & Proprietary

21 of 52

Options for Stream Analytics on GCP

Edge

Activate

Analyze

Process

Collect

Web

IoT

Mobile

Data Store

Trigger

Pub/Sub

Dataflow

BigQuery

OR

Managed Services

Options for Integration or Hosting

Data Studio

(Managed)

3rd

Party BI

Other Cloud Pipelines

Other Cloud Warehouse

(Managed)

Bigtable

22 of 52

Kafka helps companies achieve their real-time goals

67%

of the Fortune 500 uses Kafka

Kafka is in Production at 200,000+ Companies

23 of 52

PubSub

24 of 52

Without Pub/Sub, computer systems must be online at the same time to get anything done. If one is unavailable, there is no business. It’s the world of telephones.

Pub/Sub is like email for machines

With Pub/Sub, similar to email, no one has to be online at the same time. Pub/Sub delivers messages instantly if everyone is online and safely holds them until they can be delivered otherwise.

Without Pub/Sub

With Pub/Sub

Confidential & Proprietary

25 of 52

How does Pub/Sub simplify systems?

Pub/Sub

Confidential & Proprietary

26 of 52

High-Level Architecture - PubSub as the Central Service

Confidential & Proprietary

27 of 52

PubSub Architecture - Flow of Events

Confidential & Proprietary

28 of 52

Spark Streaming

29 of 52

Spark Streaming

Spark Streaming is an extension of the core Spark API that enables scalable, high-throughput, fault-tolerant stream processing of live data streams.

DStream API (powered by RDDs)
Micro batch
Fault tolerance Checkpointing

Structured Streaming

Structured Streaming is a scalable and fault-tolerant stream processing engine built on the Spark SQL engine.

Uses Spark Dataframes
No batch concept
Fault tolerance Checkpointing
Built on Spark SQL library

Confidential & Proprietary

30 of 52

Spark Streaming Architecture

Confidential & Proprietary

31 of 52

Structured Streaming Architecture

Confidential & Proprietary

32 of 52

Confidential & Proprietary

33 of 52

Structured Streaming Concepts

Sources (Input)
Sinks (Output)

Modes

Complete - entire results table will be output
Append - append events added since last trigger
Update - update events added since last trigger

Operations

Basic Operations (aggregation, filtering, etc)
Window Operations
Join Operations

Triggers - Specifies fixed batch interval or as a continuous processing
Streaming Dedup
Fault tolerance - Structured Streaming can ensure end-to-end exactly-once semantics under any failure. (checkpointing and write-ahead-logs)

Confidential & Proprietary

34 of 52

Apache Beam

35 of 52

Same code deploys to Batch and Real-time

Apache Beam

(Google Dataflow)

36 of 52

Example architecture for data analytics

Tableau

QlikView

Notes:

Popular mobile games can attract millions of players and generate terabytes of game-related data in a short burst of time. This places extraordinary pressure on the infrastructure powering these games and requires scalable data analytics services to provide timely, actionable insights in a cost-effective way.

To address these needs, a growing number of successful gaming companies use Google’s web-scale analytics services to create personalized experiences for their players. They use telemetry and smart instrumentation to gain insight into how players engage with the game and to answer questions like: At what game level are players stuck? What virtual goods did they buy? And what's the best way to tailor the game to appeal to both casual and hardcore players?

A new reference architecture describes how you can collect, archive and analyze vast amounts of gaming telemetry data using Google Cloud Platform’s data analytics products. The architecture demonstrates two patterns for analyzing mobile game events:

Batch processing: This pattern helps you process game logs and other large files in a fast, parallelized manner. For example, leading mobile gaming company DeNA moved to BigQuery from Hadoop to get faster query responses for their log file analytics pipeline.
Real-time processing: Use this pattern when you want to understand what's happening in the game right now. Cloud Pub/Sub and Cloud Dataflow provide a fully managed way to perform a number of data-processing tasks like data cleansing and fraud detection in real-time. For example, you can highlight a player with maximum hit-points outside the valid range. Real-time processing is also a great way to continuously update dashboards of key game metrics, like how many active users are currently logged in or which in-game items are most popular.

Some Cloud Dataflow features are especially useful in a mobile context since messages may be delayed from the source due to mobile Internet connection issues or batteries running out. Cloud Dataflow's built-in session windowing functionality and triggers aggregate events based on the actual time they occurred (event time) as opposed to the time they're processed so that you can still group events together by user session even if there's a delay from the source.

But why choose between one or the other pattern? A key benefit of this architecture is that you can write your data pipeline processing once and execute it in either batch or streaming mode without modifying your codebase. So if you start processing your logs in batch mode, you can easily move to real-time processing in the future. This is an advantage of the high-level Cloud Dataflow model that was released as open source by Google.

Cloud Dataflow loads the processed data into one or more BigQuery tables. BigQuery is built for very large scale, and allows you to run aggregation queries against petabyte-scale datasets with fast response times. This is great for interactive analysis and data exploration, like the example screenshot above, where a simple BigQuery SQL query dynamically creates a Daily Active Users (DAU) graph using Google Cloud Datalab.

37 of 52

Apache Beam: Data Processing SDK

Multiple Modes

Multiple SDKs

Multiple Runners

Batch

Streaming

Java

Python

Direct: local for testing

Apache Apex: local, on-premise, cloud

Apache Flink: local, on-premise, cloud

Apache Spark: local, on-premise, cloud

Cloud Dataflow: fully managed service on Google Cloud

One Model

38 of 52

Apache Beam Terminology

Pipeline Contains all of the data and steps in your data processing task

PCollection Data, it is a distributed, multi-element data set

Transforms The operations in your pipeline to process data

Windowing Divides a PCollection based on the timestamps of event(s)

Triggers Determines when to emit the aggregated results of each window

Runner The software that accepts a pipeline and executes it

Watermarks The system's notion for when all data has arrived in a window

39 of 52

Apache Beam Terminology

Pipeline Contains all of the data and steps in your data processing task

PCollection Data, it is a distributed, multi-element data set

Transforms The operations in your pipeline to process data

Windowing Divides a PCollection based on the timestamps of event(s)

Triggers Determines when to emit the aggregated results of each window

Runner The software that accepts a pipeline and executes it

Watermarks The system's notion for when all data has arrived in a window

Types of Transforms:

ParDo (custom transform, similar to a “map”)
GroupByKey
CoGroupByKey
Combine
Flatten
Partition

40 of 52

Apache Beam Pipelines

41 of 52

{a->[apple, art, argentina], ar->[art, argentina, armenia],...}

Count

ExpandPrefixes

Top(3)

Write

Read

ExtractTags

{a->(argentina, 5M), a->(armenia, 2M), …, ar->(argentina, 5M), ar->(armenia, 2M), ...}

{#argentina scores!, watching #armenia vs #argentina, my #art project, …}

{argentina, armenia, argentina, art, ...}

{argentina->5M, armenia->2M, art->90M, ...}

Tweets

Predictions

Auto-Complete Example

42 of 52

Count

ExpandPrefixes

Top(3)

Write

Read

ExtractTags

Tweets

Predictions

Pipeline p = Pipeline.create();

p.begin()

p.run();

.apply(ParDo.of(new ExtractTags()))

.apply(Top.largestPerKey(3))

.apply(Count.create())

.apply(ParDo.of(new ExpandPrefixes())

.apply(TextIO.Write.to(“gs://…”));

.apply(TextIO.Read.from(“gs://…”))

43 of 52

Apache Beam Windowing

Fixed

Sliding

1

2

3

5

4

Sessions

2

4

3

1

Key 2

Key 1

Key 3

1

2

3

4

Windowing divides data into time-based finite chunks

Often required when doing aggregations over unbounded data

44 of 52

Apache Beam Windowing

Fixed-time windows

from apache_beam import window

fixed_windowed_items = (

items | 'window' >> beam.WindowInto(window.FixedWindows(60)))

Python

Sliding time windows

from apache_beam import window

sliding_windowed_items = (

items | 'window' >> beam.WindowInto(window.SlidingWindows(30, 5)))

Python

Session windows

from apache_beam import window

session_windowed_items = (

items | 'window' >> beam.WindowInto(window.Sessions(10 * 60)))

Python

45 of 52

Google Cloud Dataflow 101

46 of 52

Google Cloud Dataflow

Dataflow is a fully managed streaming analytics service that minimizes latency, processing time, and cost through autoscaling and batch processing.

Serverless

Unified Batch & Stream Processing

Open Source - Apache Beam

Windowing, Exactly-once processing, SDKs

47 of 52

Google Cloud Dataflow - Service Architecture

Bounded

Compute and Storage

Unbounded

Resource Management

Resource Auto-scaler

Dynamic Work Rebalancer

Work Scheduler

Monitoring

Log Collection

Graph Optimization

Auto-Healing

Intelligent Watermarking

SOURCE

SINK

48 of 52

Google Cloud Dataflow - Reference Architecture

Cloud Pub/Sub

Cloud Storage

Raw logs, files, assets, Google Analytics data, and so on

Events, metrics, and so on

Stream

Batch

Cloud Dataflow

Cloud Dataproc

Batch

BigQuery

Bigtable

B

C

A

Cloud Machine Learning

Applications

and Reports

Cloud Datalab

Visualization

and BI

Co-workers

Spanner

49 of 52

Count

ExpandPrefixes

Top(3)

Write

Read

ExtractTags

Tweets

Predictions

Pipeline p = Pipeline.create();

p.begin()

p.run();

.apply(ParDo.of(new ExtractTags()))

.apply(Top.largestPerKey(3))

.apply(Count.create())

.apply(ParDo.of(new ExpandPrefixes())

.apply(TextIO.Write.to(“gs://…”));

.apply(TextIO.Read.from(“gs://…”))

50 of 52

Pipeline p = Pipeline.create();

p.begin()

.apply(TextIO.Read.from(“gs://…”))

.apply(ParDo.of(new ExtractTags())

.apply(Count.create())

.apply(ParDo.of(new ExpandPrefixes())

.apply(Top.largestPerKey(3))

.apply(TextIO.Write.to(“gs://…”));

p.run();

Google Cloud Dataflow

51 of 52

Creating a Dataflow Pipeline

Use pre-built Dataflow Templates

Write custom Apache Beam code (Java, Python, Scala, Go)

Use Google Dataflow SQL (in Alpha)

Hybrid of any of the above methods

52 of 52

Advanced Big Data: Real-time Data Processing

Dan Zaratsian

AI/ML Solutions Architect, Gaming @ Google

Thanks!