1 of 25

Apache Beam

Unified Data Processing with Apache Beam

2 of 25

​

I am Solution Architect !

​

I tweet politics at @rajrakesh2690

​

Tech Stuffs on Medium at @raj.rakesh

Hello !

3 of 25

What is Apache Beam ?

Unified programming model for expressing efficient

and extensible, portable data processing pipelines.

Apache Beam

Batch

Streaming

4 of 25

Apache Beam - Unified

Single programming model for both batch and

stream data processing.

5 of 25

Apache Beam - Portable

Execute data processing pipeline on multiple

runtime environments.

​

“Write Once, Run anywhere”

6 of 25

Apache Beam - Extensible

Write and share new SDK’s, IO connectors and

transformation libraries.

7 of 25

Apache Beam : Open source Data Processing APIs

  • Expresses data-parallel batch and streaming algorithms using one unified API.

​

  • Cleanly separates data processing logic from runtime requirements.

​

  • Supports execution on multiple distributed processing runtime environments.

8 of 25

Apache Beam: Evolution

MapReduce

Colossus

Megastore

Millwheel

BigTable

Pub/Sub

Flume

Dremel

Spanner

Cloud Dataflow

Apache

Beam

9 of 25

Batch Processing

​

Stream Processing

​

Google Cloud

Dataflow

​

Apache Flink

Apache Spark

Direct Pipeline

Beam Model & SDK

Multiple

Modes

One

Pipeline

Many

Runtimes

Local or

Cloud

Local

Cloud

Beam Model - Flexibility

  • Batch or Steaming just one pipeline.

​

  • Choose where you want to run.

​

  • Python , Java and Go supported

10 of 25

​

​

Beam Model -Generations beyond MapReduce

Batch

Stream

  • Improved abstractions lets you focus on application logic and swap backends.

​

  • Batch and Stream both are first-class citizens no need to choose.

​

  • Clearly separates event and processing time.

11 of 25

Beam Model

Pipeline

PTransform

PCollection (Bounded or Unbounded)

Source

Sink

12 of 25

A Simple batch Pipeline

with beam.pipeline as p:

​

lines = p | 'read' >> beam.io.ReadFromText(‘/path/to/files’)

​

words = line | ‘LinetoWord’ >> beam.FlatMap(lambda line : re.findall(‘\w+’, line) )

totals = words | ‘WordCount’ >> count()

​

totals | ‘WriteOutput’ >> beam.io.WriteToText(‘path/to/output’)

​

totals | ‘Top100’ >> CombinePerKey(Largest(100))

​

| ‘WriteTop100’ >> beam.io.WriteToText(‘path/to/another/output’)

​

​

Transformations

Source

Sink

Sink

Transformations

13 of 25

A Simple streaming Pipeline

with beam.pipeline as p:

​

lines = p | 'read' >> beam.io.ReadPubSub(..).WindowInto(...)

​

words = line | ‘LinetoWord’ >> beam.FlatMap(lambda line : re.findall(‘\w+’, line) )

totals = words | ‘WordCount’ >> count()

​

totals | ‘WriteOutput’ >> beam.io.WriteToText(‘path/to/output’)

​

totals | ‘Top100’ >> CombinePerKey(Largest(100))

​

| ‘WriteTop100’ >> beam.io.WriteToText(‘path/to/another/output’)

​

​

Transformations

Source

Sink

Sink

Transformations

Only code Change

14 of 25

Beam Model : Asking the Right Questions

What results are calculated ?

​

Where in event time the results are calculated ?

​

When in processing time the results are materialized ?

​

How do refinements of results relate ?

15 of 25

Event Time vs Processing Time

16 of 25

Classic Batch : What results are calculated ?

17 of 25

Windowed Batch : Where in event time results are calculated ?

18 of 25

Streaming : When in processing time results are materialized ?

19 of 25

Customizing: What Where When How

1. Classic Batch

2. Windowed Batch

3. Streaming & Accumulation

20 of 25

Beam - Built-in I/O Connectors

  • Beam Python SDK’s supports Apache HDFS, Google Cloud Storage, and local filesystems.

Avroio, parquetio, textio , tfrecordio, vcfio

  • It also supports various Google Database offerings.

Google BigQuery, Google Datastore, Google BigTable

  • Streaming Messaging service Google Cloud Pubsub is supported.

​

21 of 25

Usage

  • Just install and get started.

​

  • pip install apache-beam

​

Beam python SDKs is growing fast to provide support to all kind of data storage systems.

​

22 of 25

Sample Beam Pipeline - On Cloud Dataflow

23 of 25

Sample Beam Pipeline - On Cloud Dataflow

​

  • Beam data pipeline is sequence of steps stitched for data processing.

​

  • Can run on multiple runtime environments.

​

  • Integrate any custom UDF’s inside beam pipeline .

24 of 25

What does Apache Beam Provide ?

  • Beam Model : What/Where/When/How
  • API(SDKs) for writing Beam Pipelines
  • Runners for existing distributed Processing backends
    • Apache Flink
    • Apache Spark
    • Apache Gearpump
    • Samza
    • Apache Apex
    • Apache Nemo
    • Google Cloud Dataflow

​

​

​

25 of 25

Questions ?

Thank You !