1 of 109

Functional programming

For Data Scientists

2 of 109

About this talk

Expectation

beginners

Reality

experienced

3 of 109

Why FP for Data Scientists?

Uniformity

Micro

Code

ETL

=

4 of 109

Functional Programming

The Theory of Everything

5 of 109

Define we must...

What is Functional Programming?

6 of 109

History first

Inspired by

Lambda Calculus

7 of 109

History first

First FP language

LISP

​

Designed by

John McCarthy

8 of 109

History first

In 1977, while introducing FP

John Backus

Defined…

Functional Programs

as...

9 of 109

History first

Being built up in a hierarchical way by means of "combining forms" that allow an "algebra of programs”.

10 of 109

History first

We even had a

Lisp Machine

​

Precursor of

Garbage collector, LAN

11 of 109

And about that definition...

12 of 109

13 of 109

Let’s give it a try...

14 of 109

FP Definition (our attempt)...

A style of programming based on writing declarative code (in contrast to imperative) that favors:

​

  • Immutability (no side effects)
  • Function composition (through the usage of High Order Functions)

15 of 109

Breaking down the definition

Our view

Programming style

16 of 109

Breaking down the definition

Are you a functional programmer?

This is the best functional code I’ve seen…

​

Let’s keep it flexible...

17 of 109

Breaking down the definition

Important

Immutability

18 of 109

Immutability

Usually, a decision to make when solving a problem...

19 of 109

Immutability

l = ['a', 'b']��new_l = l + ['c']

l = ['a', 'b']��l.append('c')

Mutable

Immutable

20 of 109

Immutability

Benefits:

​

  • A sane and known state of the code
  • Testability
  • Concurrency
  • Robustness

21 of 109

> s[0]�=> 'h'

> s�=> 'hello

> s = "hello"�> s + " world"�=> 'hello world'

Python favors immutability

22 of 109

Pandas favors immutability

df.replace(inplace=True)�df.drop(inplace=True)�df.dropna(inplace=True)�df.rename(inplace=True)�df.fillna(inplace=True)�df.set_index(inplace=True)

23 of 109

Numpy favors immutability

How can you append an element to a numpy array?

24 of 109

Numpy favors immutability

25 of 109

Breaking down the definition

This is bad

Side Effects

Really. Bad.

26 of 109

Side Effects involve

  • Accessing any type of IO
    • Network
    • File System
    • Screen, peripherals
  • Raising exceptions
  • Changing object state
  • Accessing global state

27 of 109

Side Effects example: the wrong way

db_conn = connect()��def compute_earnings():� year = input("What year? ")� df = db_conn.select(� from='earnings', year=year)� # do your computations� results = compute(df, year)� db_conn.write(results, year)� print("All done!")

28 of 109

Side Effects example: the wrong way

db_conn = connect()��def compute_earnings():� year = input("What year? ")� df = db_conn.select(� from='earnings', year=year)� # do your computations� results = compute(df, year)� db_conn.write(results, year)� print("All done!")

29 of 109

Side Effects example: the wrong way

db_conn = connect()��def compute_earnings():� year = input("What year? ")� df = db_conn.select(� from='earnings', year=year)� # do your computations� results = compute(df, year)� db_conn.write(results, year)� print("All done!")

Who to blame if something fails?

30 of 109

Side Effects example: the correct way

def compute_earnings(data, year):� # Only our computations� return compute(data, year)�

def main():� year = sys.argv[0]� db_conn = connect()� df = db_conn.select(� from='earnings', year=year)�� results = compute_earnings(df, year�� db_conn.write(results, year)

31 of 109

assert compute_earnings(data, 2016) == X�assert compute_earnings(data, 2015) == Y

Benefits: testability

32 of 109

assert compute_earnings(data, 2016) == X�assert compute_earnings(data, 2015) == Y

Benefits: testability

Known

Input and Output

33 of 109

def scheduled_job():� # ...� results = compute_earnings(df, year)

�def main():� # ...� results = compute_earnings(df, year)

�def task_queue_job():� # ...� results = compute_earnings(df, year)

Benefits: reusability & modularization

34 of 109

The main benefit: reduced complexity

35 of 109

The main benefit: reduced complexity

36 of 109

Therac-25, a radiation therapy machine used in the 80s, contained a bug that costed the lives of at least 3 patients.

​

The bug was caused by a deadly side effect. An incorrect combinations of keys could alter the global state of the machine, leading to race conditions.

​

Therac-25 biggest problem: it was impossible to test.

Side effects are bad...

37 of 109

How to avoid side effects?

By using...

Pure Functions

38 of 109

A pure function is...

  • A function that returns the same value, for the same set of arguments.

​

  • Has NO side effects

​

  • Equivalent of a mathematical function

​

39 of 109

A pure function...

Let us reason about our program better. They’re intuitive and predictable. We can combine them, test them, cache them, parallelize them without problems.

​

They can also be processed lazily (more on this later).

40 of 109

Pure functions are: testable (and reasonable).

assert add(2, 3) == 5�assert add(-1, 3) == 2

41 of 109

Pure functions are: composable

def subtract(x, y):� return add(x, y * -1)���assert subtract(5, 2) == 3

​

42 of 109

Pure functions are: cacheable.

from functools import lru_cache��@lru_cache�def add(x, y):� return x + y��add(2, 3) # computed�add(2, 3) # cached�add(2, 3) # cached

43 of 109

def square_list(a_list):� return parallel(� a_list[:int(len(l) / 2)]� ) + parallel(� a_list[int(len(l) / 2):]� )

Pure functions are: parallelizable.

44 of 109

And pure functions...

Lead us to

Referential Transparency

45 of 109

The ability of swapping a function invocation with its corresponding value, without affecting the program's behavior.

Referential transparency is just...

46 of 109

Referential transparency

msg = "The result is: %s" % add(2, 3)��

msg = "The result is: %s" % 5

47 of 109

Back to the definition

48 of 109

FP Definition (our attempt)...

A style of programming based on writing declarative code (in contrast to imperative) that favors:

​

  • Immutability (no side effects)
  • Function composition (through the usage of High Order Functions)

49 of 109

Back to the definition

Favor

Function composition

50 of 109

The concept is simple, can be seen as nested function definitions:

Function composition

f(g(h(x)))

51 of 109

But thinking in this way it’ll make us:

​

  • Think carefully about the interface of our functions
  • Think about modularization and reuse
  • Increase robustness, a new function that uses an old, tested function, is “safer”.

Function composition

52 of 109

Other important concepts are:

​

  • High order functions
  • Functions as first class objects

​

They’re related, but there are differences.

Function composition

53 of 109

Functions are first class objects (in Python)

def add(x, y):� return x + y��my_sum = add��def compute(x, y, op):� return op(x, y)��compute(1, 2, my_sum)

54 of 109

Functions are first class objects (in Python)

def add(x, y):� return x + y��my_sum = add��def compute(x, y, op):� return op(x, y)��compute(1, 2, my_sum)

55 of 109

Functions are first class objects (in Python)

def add(x, y):� return x + y��my_sum = add��def compute(x, y, op):� return op(x, y)��compute(1, 2, my_sum)

add

add

56 of 109

Functions are first class objects (in Python)

def add(x, y):� return x + y��my_sum = add��def compute(x, y, op):� return op(x, y)��compute(1, 2, my_sum)

57 of 109

Finally: Lambdas

def add(x, y):� return x + y���add = lambda x, y: x + y

58 of 109

Finally: Lambdas

def add(x, y):� return x + y���add = lambda x, y: x + y

  • Expression oriented (last expression is returned)

​

  • Also known as anonymous functions

​

  • Keep them simple!

59 of 109

Back to the definition

60 of 109

FP Definition (our attempt)...

A style of programming based on writing declarative code (in contrast to imperative) that favors:

​

  • Immutability (no side effects)
  • Function composition (through the usage of High Order Functions)

61 of 109

Back to the definition

Favor

Declarative style

62 of 109

Imperative: When you explicitly specify the steps required to achieve a given result.

​

​

Declarative: When you declare (or express) what you want to be achieved.

Declarative vs Imperative

63 of 109

  • Open the white big box (fridge)
  • Grab 4 eggs
  • Open the shelf
  • Get the white powder (flour) and chocolate.
  • Mix them
    • Keep mixing (have you mixed for 5 minutes already?)
  • Put in the oven
  • ...

Example: our imperative robot cook

64 of 109

Example: our imperative robot cook

fridge = open_thing(x, y)�eggs = get_eggs(fridge, 4)

�shelve = open_thing(x, y)�flour = get_flour(shelf, 200)

�while time < 5:� mixture = mix(flour, eggs)� time += 1

�bake(mixture)

65 of 109

Example: our imperative robot cook

fridge = open_thing(x, y)�eggs = get_eggs(fridge, 4)

�shelve = open_thing(x, y)�flour = get_flour(shelf, 200)

�while time < 5:� mixture = mix(flour, eggs)� time += 1

�bake(mixture)

What is the code doing?

66 of 109

  • Was conceived as a pragmatic solution to instruct machines on how to do things.

​

  • Feels a lot like programming micromanagement.

​

  • It’s the standard C code you see everywhere.

Imperative code

67 of 109

  • Bake a chocolate cake, made out of 4 eggs and whole flour.

Example: our declarative robot cook

68 of 109

Example: our declarative robot cook

bake(� prepare_mixture(� cake(� flavor='chocolate', eggs=4)))

69 of 109

  • Expressive code is always better. Easier to read and reason about.

​

  • The implementation is independent from the definition. Abstractions.

Declarative code

This is IMPORTANT

70 of 109

A = np.array([� [1, -1, 2],� [0, -3, 1]�])��x = np.array([2, 1, 0])

Abstraction example: dot product

1

-1

2

0

-3

1

1

2

0

⋅

71 of 109

result = []�row_count = 0��for row in A:� partial_result = 0� for i in range(len(row)):� a_elem = row[i]� x_elem = x[i]� partial_result += (a_elem * x_elem)� result.append(partial_result)� row_count += 1

Abstraction example: imperative solution

72 of 109

A = np.array([� [1, -1, 2],� [0, -3, 1]�])��x = np.array([2, 1, 0])��A @ x # np.dot(A, x)�# np.array([1, -3])

Abstraction example: declarative solution

73 of 109

How is this implemented internally?

A = np.array([� [1, -1, 2],� [0, -3, 1]�])��x = np.array([2, 1, 0])��A @ x # np.dot(A, x)�# np.array([1, -3])

Abstraction example: declarative solution

74 of 109

But, doesn’t it feel like cheating?

result = []�row_count = 0��for row in A:� partial_result = 0� for i in range(len(row)):� a_elem = row[i]� x_elem = x[i]� partial_result += (a_elem * x_elem)� result.append(partial_result)� row_count += 1

A @ x

vs.

75 of 109

The key of declarative programming is

A higher level of

Abstraction

76 of 109

Our robot understands primitives like bake or prepare_mixture. Which are high level primitives.

​

The key is learning this language of high level abstractions.

High level of abstraction

77 of 109

Finding the balance between something too high level that can’t be interpreted.

​

And something too low level that becomes imperative.

High level of abstraction

78 of 109

Functional programming building blocks

The most popular

FP abstractions

79 of 109

Functional programming building blocks

map, filter & reduce

80 of 109

Functional programming building blocks

Take this

map

Give me that

81 of 109

map: let the code explain it

map(lambda x: x ** 2, [1, 2, 3, 4])��>>> [1, 4, 9, 16]

82 of 109

map: let the code explain it

2

1

3

f(x) => x²

4

4

1

9

16

83 of 109

map: let the code explain it

2

1

3

f(x) => x²

4

4

1

9

16

f(x) => x²

f(x) => x²

f(x) => x²

84 of 109

map: visually

Original elements

Transformation

Result elements

85 of 109

map: does it look familiar?

?

86 of 109

map: does it look familiar?

[x ** 2 for x in (1, 2, 3, 4)]��>>> [1, 4, 9, 16]

87 of 109

map: does it look familiar?

arr = np.array([1, 2, 3, 4])��arr ** 2��# > array([ 1, 4, 9, 16])

88 of 109

Functional programming building blocks

Take this

filter

if...

89 of 109

filter: let the code explain it

filter(lambda x: x > 2, [1, 2, 3, 4])��>>> [3, 4]

90 of 109

filter: let the code explain it

2

1

3

f(x) => x > 2

4

3

4

91 of 109

filter: let the code explain it

2

1

3

f(x) => x > 2

4

3

4

f(x) => x > 2

f(x) => x > 2

f(x) => x > 2

92 of 109

filter: visually

Original elements

Result elements

blue-ish

elements

93 of 109

filter: does it look familiar?

?

94 of 109

filter: does it look familiar?

[x for x in (1, 2, 3, 4) if x > 2]��>>> [3, 4]

95 of 109

filter: does it look familiar?

arr = np.array([1, 2, 3, 4])��arr > 2�# > array([False, False, True, True])��arr[arr > 2]�# > array([3, 4])

96 of 109

(also known as fold)

reduce

Squash these things...

Functional programming building blocks

97 of 109

reduce: let the code explain it

reduce(lambda acc, y: acc + y, [1, 2, 3, 4], 0)��>>> 10

98 of 109

combination

reduce: visually

99 of 109

reduce: let the code explain it

reduce(lambda acc, y: acc * y, [1, 2, 3, 4], 1)��>>> 24

100 of 109

reduce: let the code explain it

Important!

No longer part of stdlib

from functools import reduce�reduce(lambda acc, y: acc * y, [1, 2, 3, 4], 1)��>>> 24

101 of 109

Almost every time I see a reduce() call (...), I need to grab pen and paper to diagram what's [that function] supposed to do.

reduce: why was removed from stdilib

Guido van van Rossum

Creator of Python

102 of 109

what about fold “fold”?

# reduce: initializer is 1st element�reduce(lambda acc, y: acc * y, [1, 2, 3, 4])���# fold: initializer is passed as argument�reduce(lambda acc, y: acc * y, [1, 2, 3, 4], 1)

103 of 109

Does “map” and “reduce” sound familiar?

104 of 109

Does “map” and “reduce” sound familiar?

105 of 109

The theory of everything

106 of 109

The theory of everything

words = (� (w.lower(), 1)� for w in text.split() if w)��dict([� (word, len(list(group)))� for word, group in� groupby(� sorted(words), key=lambda d: d[0]� )�])

107 of 109

What’s the difference between

map, filter

And

List comprehensions

You might wonder...

108 of 109

The difference is: laziness

from operator import pow�square = lambda x: pow(x, 2)��map(square, [1, 2, 3, 4])�# => <map object at 0x7f2300fc7198>��[square(x) for x in [1, 2, 3, 4]]�# => [1, 4, 9, 16]

109 of 109

Want to learn more about Data Science?

https://ine.com/pages/data-science