Machine Learning
Prof. Seungtaek Choi
Last Time
Today
Neural Networks
Neural Networks
The brain adapts its function to the input it receives.
The brain flexibly adapts to incoming sensory channels and can even learn entirely new senses. – A general-purpose, multimodal learning machine.
Neuron = input integration 🡪 threshold 🡪 output.
Neurons form networks: from sensory input to motor output.
Perceptron: Binary Linear Classifier
20
Perceptron: Geometric Interpretation
21
Perceptron for New Data
22
Binary Linear Classifier in 2D
23
Idea: Nonlinear Curve Approximated by Multiple Lines
27
AND Problem
28
| | |
0 | 0 | 0 |
0 | 1 | 0 |
1 | 0 | 0 |
1 | 1 | 1 |
| | | |
0 | 0 | -30 | 0 |
0 | 1 | -10 | 0 |
1 | 0 | -10 | 0 |
1 | 1 | 10 | 1 |
1
-30
20
20
OR Problem
29
| | |
0 | 0 | 0 |
0 | 1 | 1 |
1 | 0 | 1 |
1 | 1 | 1 |
| | | |
0 | 0 | -10 | 0 |
0 | 1 | 10 | 1 |
1 | 0 | 10 | 1 |
1 | 1 | 30 | 1 |
1
-10
20
20
How about XOR Problem?
30
| | |
0 | 0 | 0 |
0 | 1 | 1 |
1 | 0 | 1 |
1 | 1 | 0 |
XOR Problem
31
XOR Problem
32
XOR Problem
33
Multiple Perceptrons
34
Multiple Perceptrons
35
Multiple Perceptrons
36
Multiple Perceptrons
37
Multiple Perceptrons
38
Multiple Perceptrons
39
First layer
with neurons
Multiple Perceptrons
40
First layer
Second layer
Another Interpretation
Another Interpretation
1
-10
20
20
1
-30
20
20
Another Interpretation
1
-10
20
20
-30
20
20
-30
20
-20
| | | | |
0 | 0 | 0 | 0 | 0 |
0 | 1 | 1 | 0 | 1 |
1 | 0 | 1 | 0 | 1 |
1 | 1 | 1 | 1 | 0 |
Looks a lot like logistic regression
The only difference is, instead of input a feature vector, the features are just values calculated by the hidden layer
Feature Learning
Looks a lot like logistic regression
The only difference is, instead of input a feature vector, the features are just values calculated by the hidden layer
Another Perspective:�Hidden Layers as Kernel Learning
53
Nonlinear Classification
54
https://www.youtube.com/watch?v=3liCbRZPrZA
Neuron
55
Second Way of Looking at Multiple Perceptrons
56
Common Activation Functions
57
Source: 6.S191 Intro. to Deep Learning at MIT
Discuss later
XOR Problem in Perceptron
58
Nonlinear Mapping
59
Source: Dr. Francois Fleuret at EPFL
Nonlinear Mapping
60
Source: Dr. Francois Fleuret at EPFL
Nonlinear Mapping
61
Source: Dr. Francois Fleuret at EPFL
Neuron
62
Kernel + Neuron
63
Neuron + Neuron
64
Multi Layer Perceptron (MLP)
65
Multi Layer Perceptron (MLP) = Artificial Neural Networks
66
Nonlinear mapping
Multi Layer Perceptron (MLP) = Artificial Neural Networks
67
Nonlinear mapping
Multi Layer Perceptron (MLP) = Artificial Neural Networks
68
…
Nonlinear mappings
Linearly separable
Multi Layer Perceptron (MLP) = Artificial Neural Networks
69
…
Nonlinear mappings
Multiple Linear classifiers
Linearly separable
Multi Layer Perceptron (MLP) = Artificial Neural Networks
70
…
Linear classification
Feature Learning
Nonlinear mappings
Linearly separable
Two Ways of Looking at Artificial Neural Networks
71
Two Ways of Looking at Artificial Neural Networks
72
(1)
(2)
Backpropagation
Training Neural Networks: Optimization
77
Training Neural Networks: Loss Function
78
Training Neural Networks: Gradient Descent
79
Gradients in ANN
81
Gradients in ANN
82
Training Neural Networks: Backpropagation Learning
83
Backpropagation
84
Backpropagation
85
These are what we need for GD
Backpropagation
86
These are what we need for GD
Backpropagation
87
These are what we need for GD
Backpropagation
88
These are what we need for GD
Backpropagation
89
These are what we need for GD
Backpropagation
90
These are what we need for GD
Training Neural Networks with PyTorch
127
Activation Function
Artificial Neural Networks with PyTorch
MNIST database
Our Neural Network (Model)
Input layer
(784)
Hidden layer
(100)
Output layer
(10)
Input image
(28 X 28)
Flattened
Digit prediction
in one-hot-encoding
Iterative Optimization
Implementation in Python
Implementation in Python
Implementation in Python
Implementation in Python
Implementation in Python
Implementation in Python
Next