Machine Learning

Prof. Seungchul Lee

Industrial AI Lab.

Contents

• Regression

​

• Classification
• Linear Classification
• Perceptron
• Logistic Regression
• Non-linear Classification

​

2

Regression

3

Linear Model

4

Linear Model

5

Linear Regression as Optimization

6

Linear Regression as Optimization

7

Re-cast Problem as Least Squares

8

Optimization

9

Optimization: Orthogonal Projection

10

the same principle in a higher dimension

Solve using Linear Algebra

• known as least square

​

11

Solve using Linear Algebra

• known as least square

​

12

Scikit-Learn

• Machine Learning in Python
• Simple and efficient tools for data mining and data analysis
• Accessible to everybody, and reusable in various contexts
• Built on NumPy, SciPy, and matplotlib
• Open source, commercially usable - BSD license
• https://scikit-learn.org/stable/index.html#

​

13

Scikit-Learn: Regression

14

Classification: Perceptron

Classification

• Categorizing data into predefined classes or categories based on input features

​

• Examples
• Spam detection: Classifying emails as either spam or not spam
• Image recognition: Identifying objects, animals, or faces in images
• Medical diagnosis: Predicting whether a patient has a particular condition based on diagnostic data

​

16

Classification

17

Classification

• We will learn
• Perceptron
• Logistic regression

​

​

• To find a classification boundary

​

​

18

Classification

• We will learn
• Perceptron
• Logistic regression

​

​

• To find a classification boundary

​

​

19

Classification

• We will learn
• Perceptron
• Logistic regression

​

​

• To find a classification boundary

​

​

20

Perceptron

21

Perceptron

22

Classification Boundary

23

Learning a Hyperplane for Classification

24

Learning a Hyperplane for Classification

25

Learning a Hyperplane for Classification

26

Perceptron Algorithm

• The perceptron implements

​

• Given the training set

​

1. pick a misclassified point

​

• and update the weight vector

27

Perceptron Algorithm: Illustration

28

Perceptron Algorithm: Illustration

29

Perceptron Algorithm: Illustration

30

Perceptron Algorithm: Illustration

31

Perceptron Algorithm: Illustration

32

Perceptron Algorithm: Illustration

33

Why Perceptron Updates Work ?

34

Diagram of Perceptron

• Perceptron can be viewed as a simple neuron in an Artificial Neural Network (ANN)

​

​

​

​

​

​

​

​

​

• Perceptron Update Rule (discrete version):

​

• Gradient Descent Update Rule (continuous version):

​

35

Perceptron in Python

36

Perceptron in Python

37

Perceptron in Python

38

Perceptron in Python

39

Scikit-Learn for Perceptron

40

The Best Hyperplane Separator?

• Limitations
• The Perceptron identifies a separating hyperplane if the data is linearly separable
• However, it merely finds one of the possible solutions

​

41

The Best Hyperplane Separator?

• Improvements
• Identifying the best hyperplane requires an optimization framework

​

• Utilize distance information
• Support Vector Machines (SVM) and Logistic Regression

​

42

Classification: Logistic Regression

Linear Classification: Logistic Regression

• Logistic regression is a classification algorithm
• don't be confused

​

• Perceptron: make use of sign of data

​

• We want to use distance information of all the data points
• logistic regression

44

Using Distances

45

Using Distances

46

Using Distances

47

Using Distances

48

Using all Distances

49

Using all Distances

50

Sigmoid Function

51

Perceptron

Logistic regression

Sigmoid Function

52

Distance

53

Distance

• Perceptron
• Distance is now measured in the mapped feature space

54

Distance

• Logistic Regression
• Distance is now measured in the mapped feature space

​

55

Logistic Regression using Scikit-Learn

56

Non-linear Classification

57

Classifying Non-linearly Separable Data

58

Classifying Non-linearly Separable Data

59

Classifying Non-linearly Separable Data

60

Nonlinear Classification

61

https://www.youtube.com/watch?v=3liCbRZPrZA

Kernel

• Motivation
• We often need to capture nonlinear patterns in the data
• Nonlinear regression: the input-output relationship may not be linear
• Nonlinear classification: the classes may not be separable by a linear boundary
• Linear models (e.g., linear regression, logistic regression) are simply not rich enough

​

• Idea
• Map the data into a higher-dimensional space where it exhibits linear patterns
• Apply a linear model in the new feature space
• Mapping = changing the feature representation

​

• Kernel
• A mapping function that takes nonlinearly distributed data into a higher-dimensional space where it becomes linearly separable
• With the right kernel, a linear classifier can handle nonlinearly distributed data

62

Non-linear Classification

63

Explicit Kernel

64

Non-linear Classification

65

A Remark on Kernel Selection

• So far we have assumed that the kernel function is given

​

• Choosing a suitable kernel for a given dataset is a nontrivial problem, but we will not pursue it further

​

• The reason is that deep learning learns the feature transformation directly from the data.

​

• Unlike kernel methods, a neural network discovers the mapping automatically, so no kernel has to be designed by hand

66