Functional Programming Basics

​

Big Data Analytics with Python, AIMS-Rwanda 2023

Dunstan Matekenya, PhD

Contents

1. Big Data problem solving
2. Big Data package stack in Python
3. Functional programming basics
1. What is functional programming?
2. Advantages of functional programming
3. Functional programming and Big Data processing
4. Lambda functions
5. Higher order functions
6. Functional programming in Python
7. Data structures for functional programming in Python
4. Further reading on functional programming
5. Functional programming tutorial in Python

Advice on Tackling a Big Data Problem

1. Can I optimize pandas to solve the problem?: If you are using Pandas for data munging, you can optimize pandas to load large datasets depending on the nature of your problem
2. How about drawing a sample from the large dataset? Depending on your use case, drawing a sample out of a large dataset may or may not work. Just be careful that you sample correctly.
3. Can I use simple Python parallelism to solve the problem on my laptop? Sometimes the data isn't that big but you just need to run more intense computations on the smaller data, multiprocessing can help.
4. Can I use a big data framework on my laptop? For some tasks, even with a 25GB dataset, frameworks like Spark and Dask can work on a single laptop.
5. Which package should I use?
6. Need to build a cluster: Take time to think about which distribution of Hadoop to use, which vendors to use, whether you will put the cluster on the cloud or on-premise. You will need input of IT people for this one.

Some questions to ask yourself before you jump to the big guns

Big Data Package Ecosystem in Python

1. Apache Spark (Pyspark)
2. Dask
3. Vaex
4. Datatable

There are only 4 packages which are known to handle large datasets in Python. Out of those, Pyspark and Dask are the most stable options for enterprise level data processing

Explore Dask on Your Own

• One of the best features about Dask is that it uses existing Python APIs and data structures, its easy to switch between NumPy, pandas, scikit-learn to their Dask-powered equivalents.
• At the same time, you can also run it on compute clusters such as those powered by Hadoop framework.
• Learn all about Dask here

In this course, we use Pyspark but I encourage you to explore Dask

​

What is Functional Programming?

​

“In computer science, functional programming is a programming paradigm where programs are constructed by applying and composing functions. It is a declarative programming paradigm in which function definitions are trees of expressions that map values to other values, rather than a sequence of imperative statements which update the running state of the program.”

Wikipedia

A pure function is a function whose output value follows solely from its input values and cannot be affected by any mutable state or other side effects. In functional programming, a program consists entirely of evaluation of pure functions.

What is Functional Programming

• A mathematical function programming style
• Follows declarative programming model
• Emphasizes the “what” of the solution instead of “how to” get to the solution
• Uses expressions instead of statements
• LISt Processing Language, known as LISP, was the first functional programming language, starting in the 1950s.
• Haskell and Scala is the most recent representative in this family of programming languages. Apache Spark is written mainly in Scala
• Other languages (e.g., Python, R, Java) also provide rudimentary support for functional programming

Functional programming style in Scala

Procedural programming style in Python

Traditional Vs. Functional Program in Javascript

Source: Wikipedia

Advantages of Functional Programming

• Elegant code: Code is elegant and concise because of higher order function abstractions.
• High level: You’re describing the result you want rather than explicitly specifying the steps required to get there.
• Transparent: The behavior of a pure function depends only on its inputs and outputs, without intermediary values. That eliminates the possibility of side effects, which facilitates debugging and also reduces introduction of bugs
• Parallelizable: Turning FP code to run into parallel requires no changes to the function definition which is different from traditional procedural code.
• Programs are deterministic.

Disadvantages of Functional Programming

• Potential performance losses because of the amount of garbage- collection that needs to happen when we end up creating new variables as we can’t mutate existing ones.
• File I/O is difficult because it typically requires interaction with state.
• Programmers who’re used to imperative programming could find this paradigm harder to grasp.

Advantages of Functional Programming-Parallelization

Input

Function

Output

Increase compute nodes as input size increase

Compute node-1

Compute node-2

Lambda Functions

• In computer programming, an anonymous function (function literal, lambda abstraction, lambda function, lambda expression or block) is a function definition that is not bound to an identifier.
• Anonymous functions are often arguments being passed to higher-order functions or used for constructing the result of a higher-order function that needs to return a function.
• Anonymous functions are ubiquitous in functional programming languages and other languages with first-class functions, where they fulfil the same role for the function type as literals do for other data types.

Source: Wikipedia

Data types such as numbers, strings, booleans etc. don’t need to be bound to a variable. The same can be done for functions!

Lambda Functions in Scala vs. Python

Functional Programming and Big Data processing

• Functional programming lends itself amenable to Big Data processing because of ease of parallelization
• For instance, Spark parallelizes computations using the lambda calculus
• All functional Spark programs are inherently parallelizable-which means when you increase your input data from 1MB to 1 PB during analysis, all you have to do is add more compute resources, no need to change the code

Functional Programming in Python

• Functions in Python are first class citizens-That means functions have the same characteristics as values like strings and numbers.
• Functions have two abilities which are crucial for functional programming as follows:
• They can take another function as an argument
• They can return functions as values
• Storing them in variables just like other data-types
• Anonymous functions are easy to define with lambda
• Therefore, Python provides good support functional programming

Data Structures for Functional Programming in Python

• Mutable data structures such as Dictionaries and Lists are not ideal for functional programs because they can be changed while the program is running
• Instead, immutable data structures are better where you are forced to make a copy of the object before you change it
• In Python “namedtuple” and “tuple” can be used instead of Lists and Dictionaries

Why Does Data Immutability Matter in FP

• In pure functional languages, all data is immutable and the program state cannot change
• What are the implications of this property ?
• Functions are deterministic-the same input will always yield the same output. This makes it easier to re-use functions elsewhere.
• The order of execution of multiple functions does not affect the final outcome of the program.
• Programs don’t contain any side effects.
• We will see that in Apache Spark, all data structures are immutable, you have to make a copy or perform some action/transformations to change it

​

Defining Lambda Functions in Python

How Lambda Functions Fit Together with Python Function Definition and Lambda Calculus

Higher Order Functions

A function is a higher-order function if it either takes in one or more functions as parameters and/or returns a function.

Higher Order Functions in Action

Common Higher Order Functions

• Map-is a higher order function with the following specifications
• Inputs: a function f and a list of elements L
• Outputs: a new list of elements, with f applied to each of the elements of L
• ​
• Reduce-reduces a list of elements to one element using a binary function to successively combine the elements.
• Inputs: a function f , a list of elements L and an accumulator, acc, the parameter that collects the return value. You can think of acc as the initial value
• Outputs: The value of f sequentially applied and tracked in ‘acc’
• Filter

Similar idea to the map and reduce functions in MapReduce programming model

Map Example

Reduce Example

Further Reading in Functional Programming

1. Wikipedia has good content on the topic
2. The Lambda calculus background is also interesting to read
3. These slides provide good introductory information on FP

Now, we already for a tutorial on Functional Programming in Python