1 of 37

Statistician

B.Ed (Maths)

  • PhD research-Big Data
  • Freelance Data Scientist
  • Part-time ML Engineer-Startup

GIS-lead

Data Scientist, WB

  • Mobile phone data and development
  • Machine learning and other ad hoc DS work
  • Teaching DS with AIMS in Rwanda
  • NLP and AI for local languages in Malawi

NSO, Malawi

1

2 of 37

Big Data Basics

​

Big Data Analytics with Python, AIMS-Rwanda 2022

Dunstan Matekenya, PhD

3 of 37

Outline

  1. What is Big Data?
  2. What can we do with Big Data?
  3. What are the sources of Big Data?
  4. Big Data tools and ecosystem
  5. Distributed data processing
  6. How to work with large datasets in Python
  7. Further readings

4 of 37

Whats Big Data?

  1. How big is big?
  2. Whats makes Big Data different from small data?
  3. Is there such a thing as small data?
  4. What are the characteristics of Big Data?

5 of 37

Exercise: Tracking the Big Data Hype

Using the Google trends site check how the popularity of the search term “Big Data” has declined compared with similar search terms: machine learning and artificial intelligence

6 of 37

Popularity of ‘Big Data’ Search Term on Google

7 of 37

What exactly is Big Data?

  • Big data is larger, more complex data sets, especially from new data sources. This data is characterized by the fact that it contains greater variety (texts, images, audios, videos etc), arriving in increasing volumes and with more velocity.
  • These data sets are so voluminous that traditional data processing software just can’t manage them

8 of 37

Characteristics of Big Data-The 4 V’s

9 of 37

Characteristics of Big Data-The 4 V’s

  • Volume: The amount of data matters. For Big Data, we are often talking about large volumes of data. Although the question often comes, how big is big?
  • Velocity: Velocity is the fast rate at which data is received and (perhaps) acted on.
  • Variety: Variety refers to the many types of data that are available. Traditional data types were structured and fit neatly in a relational database. With the rise of big data, data comes in new unstructured data types. Unstructured and semistructured data types, such as text, audio, and video, require additional preprocessing to derive meaning and support metadata.
  • Veracity: The data is often noisy

10 of 37

History of Big Data

  • Previously, large datasets were available through satellite images and others sources.
  • Explosion in size of datasets came with the coming of internet companies such as Facebook, Youtube and others around 2005
  • Hadoop and Mapreduce framework were created in 2006 specifically to deal with large datasets at Google
  • Since then, the advent of IoT, more mobile devices and increased access to internet has contributed to ever increasing datasets

11 of 37

Examples of Big Data

  • This old article has some interesting stats BUT no way to know how accurate it is
  • Yet another article

Articles talking about Big Data volumes

Very Big Data at Netflix

12 of 37

Applications of Big Data?

13 of 37

Industries Using Big Data

What Do they Do with the Data?

Online Retail

Search

Finance

Manufacturing

Automobile

Telecom

Entertainment

Medicine

Insurance

  1. Amazon uses its massive data to build recommendation systems

​

  1. Netflix uses data to predict customer viewer behavior

​

  1. Google uses search data to improve search results and optmize ranking of results
  2. Google uses its data for advertising
  3. ​
  4. And more and more

14 of 37

Big Data Tools and Ecosytem

15 of 37

What Do We Want to Do with Big Data?

Store, manage and retrieve

Analyze and visualize

Build models (ML)

The Big Data tools, methodologies, frameworks and ecosystems address these tasks. Most technologies combine several aspects. For instance, with Hadoop, you have both storage and analysis capabilities

16 of 37

Tools and Platforms

Cloud providers

  • Data storage, management
  • Compute for data processing, analysis and visualization
  • Examples: AWS, Google Cloud Platform (GCP), Microsoft Azure

​

Hadoop, HDFS, Apache Spark, Mapreduce

  • Provides distributed storage
  • Data analytics, machine learning
  • Parallel data processing
  • Example vendors: Cloudera, Hortonworks, Databricks

NoSQL Databases

  • Store large datasets at scale
  • Store data in JSON instead of relational tables
  • NoSQL database types include pure document databases, key-value stores, wide-column databases and graph databases
  • Examples: MongoDB, CouchDB, Cassandra and Redis
  • Data lake is a relatively new architecture in data storage

17 of 37

How Some of these Components Look in a Full System

18 of 37

Summary on Big Data

  • Big Data, when it comes to size is a relative term BUT you will know you have a large dataset if you can’t process it on a single computer or you can’t use traditional software such as Excel to handle the data
  • The Big Data technology landscape is now dominated by cloud providers such as AWS, Google, Azure and others. Other notable vendors involved include Databricks, Cloudera/Hortonworks, MapR
  • Big Data has penetrated many industries including government

19 of 37

Further Reading on Big Data Basics

  1. This PDF book on Big Data is a good resource to look through
  2. If you want to understand data lakes and data warehouses more, take a look at this eBook

20 of 37

Big Data and Parallel Processing

21 of 37

What is Parallel Processing

  • Parallel computing is a computation style of carrying out multiple operations simultaneously using one (by means of multi-threading) or many machines. Parallel computing works on the principal that large problems can often be divided into small problems and these small problems can be executed concurrently.
  • Task parallelism-This form of parallelism covers the execution of computer programs across multiple processors on same or multiple machines. It focuses on executing different operations in parallel to fully utilize the available computing resources in form of processors and memory.
  • Data parallelism- This form of parallelism focuses on distribution of data sets across the multiple computation programs. In this form, same operations are performed on different parallel computing processors on the distributed data subset.

Most Big Data Frameworks utilize both data and task parallelism

22 of 37

Similar Concepts to Parallelism

  • Distributed computing
  • Concurrent processing

23 of 37

Why Parallel Processing?

  • In this single node/single setup, if you have too much data and/or too many computations to perfom, its difficult to scale

​

  • Parallel or distributed processing is the cornerstone of working with Big Data

Compute (CPU)

Storage

Single node setup

24 of 37

Parallel vs. Linear Processing

Linear/sequential processing

Parallel processing

Problem

Instruction-1

Instruction-2

Instruction-N

Output

Problem

Instruction-1

Instruction-2

Instruction-N

Output

Error

Error

25 of 37

Advantages of Parallel Processing

  • Faster: Parallel processing can process large datasets in a fraction of time
  • Less compute: Less memory and compute requirements needed as compute instructions are distributed to smaller execution nodes
  • Scalability: More execution nodes can be added or removed from the execution network depending on the problem

26 of 37

Vertical Scaling Vs. Horizontal Scaling

Compute

Eventually vertical scaling fails

Compute

Compute

27 of 37

Horizontal Scaling is Better

Storage

Compute

Storage

Compute

Storage

Compute

Storage

Compute

Storage

Compute

A Cluster of computers

28 of 37

Concurrency and Parallelism in Python

Before we turn to the big guns (Big Data frameworks) to handle our Big Data, we will look at how to achieve simple parallelism with vanilla Python

29 of 37

Concurrency and Parallelism in Python

  • In Python, strictly speaking, concurrency and parallelism are different
  • Concurrency-processing happens on a single processor, although its still possible to run multiple things through some tricks
  • Parallelism-Python creates several processes which run simultaneously on different processors

30 of 37

Summary of Different Concurrency Types in Python

31 of 37

CPU-Bound and I/O-Bound Programs in in Python

  • I/O-bound problems cause your program to slow down because it frequently must wait for input/output (I/O) from some external resource. For example, scraping data from many websites is an I/O bound problem. What limits the program from running faster is time spent interacting with the file system and network.
  • ​
  • CPU-bound problems: classes of programs that do significant computation without talking to the network or accessing a file. These are the CPU-bound programs, because the resource limiting the speed of your program is the CPU, not the network or the file system.

32 of 37

I/O Bound Vs. CPU Bound Processes

33 of 37

How to Achieve Concurrency in Python

  • There are several libraries which tackle this problem in Python depending on whether the problem is IO or CPU bound and also expressiveness of the library
  • asyncio-For IO bound programs such as downloading data from the web, this package is useful to run concurrent tasks
  • Multiprocessing: This is a builtin Python library for speeding up CPU bound programs by creating many processes to take advantage of multiple cores on your computer
  • Other libraries: Joblib, concurrent.futures which provide similar functionality but at a higher level

34 of 37

How to Solve Big Data Problems in Python

35 of 37

Working with data in a distributed fashion is inherently difficult, therefore make sure that you exhaust all options before jumping into using Spark, Hadoop or other Big Data frameworks

36 of 37

Advice on Tackling a Big Data Problem

  1. Can I optimize pandas to solve the problem?: If you are using Pandas for data munging, you can optimize pandas to load large datasets depending on the nature of your problem
  2. How about drawing a sample from the large dataset? Depending on your use case, drawing a sample out of a large dataset may or may not work. Just be careful that you sample correctly.
  3. Can I use simple Python parallelism to solve the problem on my laptop? Sometimes the data isn't that big but you just need to run more intense computations on the smaller data, multiprocessing can help.
  4. Can I use a big data framework on my laptop? For some tasks, even with a 25GB dataset, frameworks like Spark and Dask can work on a single laptop.
  5. Need to build a cluster: Take time to think about which distribution of Hadoop to use, which vendors to use, whether you will put the cluster on the cloud or on-premise. You will need input of IT people for this one.

Some questions to ask yourself before you jump to the big guns

37 of 37

Further Reading on Parallelism

  1. Python multiprocessing library
  2. This is an excellent book-High Performance Python, I have the hard copy.
  3. This site has good summary on parallelism in Python