Closing
Ceremony
Project: Baseline Model
Presentations
Project work
Project: Advanced Model (Upgrade from Baseline)
Week 6
Week 7
Week 8
Week 9
Week 10
Convolutional Neural Networks (CNNs)
Tools for Improving CNNs: Regularization and Transfer Learning
Image Classification with CNNs
Ethics in AI
Advanced Topics in Image Classification: Using VGG16
Project work
Project work
Tuning Neural Networks (Classification)
Project: Start Projects with EDA!
Tuning NNs, Using NNs for classification, Validation Sets
Overfitting & Underfitting
Quick Review
A
B
C
Train MSE: 7.54
Test MSE: 7.78
Train MSE: 0
Test MSE: 3.45
Train MSE: 2.81
Test MSE: 2.94
Best Predictions!
A
B
C
If model is too simple, we underfit.
If model is too complex, we overfit to the training data.
Model is well-tuned. Complex enough to capture the trend, but simple enough to prevent overfitting.
Neural Networks
Quick Review
How to train a neural network
Step 1: Set weights randomly
Price
-25
80
10
120
…
4
Size: 1800 ft^2
Number of Bedrooms: 2
Rating of Local Schools): 75
Repairs Needed ($): $2,000
…
…
Bias Term
How to train a neural network
Step 2: Forward pass to get prediction
Price
-25
80
10
120
…
4
Size: 1800 ft^2
Number of Bedrooms: 2
Rating of Local Schools): 75
Repairs Needed ($): $2,000
…
…
Bias Term
1800(-25) + 2(80) + 75(10) + 2000(120) + 4 = $195,914
How to train a neural network
Step 3: Calculate loss by comparing prediction vs. actual
Price
-25
80
10
120
…
4
Size: 1800 ft^2
Number of Bedrooms: 2
Rating of Local Schools): 75
Repairs Needed ($): $2,000
…
…
Bias Term
Terrible prediction!
Residual = Actual - Predicted
Residual = 780,000 - 195,914
Residual = 584,086
1800(-25) + 2(80) + 75(10) + 2000(120) + 4 = $195,914
How to train a neural network: Gradient Descent
Step 4: Backpropagate to adjust weights to lower loss
Price
-25
80
10
120
…
4
Size: 1800 ft^2
Number of Bedrooms: 2
Rating of Local Schools): 75
Repairs Needed ($): $2,000
…
…
Bias Term
“If I had increased this weight, I would have had less error (lower MSE).”
How to train a neural network:
Step 5: Repeat steps 2-4 for your whole dataset, many times. Repeat until reach desired accuracy.
Remember:
From many small dumb things → one big smart thing
Tuning Neural Networks (& using them for classifications
Week 6
Using NN’s for classifications
Part 1
So far, we’ve used Neural Networks for Regression:
We can also use Neural Networks for Classification:
3 Outcomes
Win
Draw
Lose
Neural Network for Classification
Inputs
Hidden Layers
Outputs
Team Record
Player Ratings
Field Weather
Last Match Result
Home/Away
Chat Waterfall: Why 3 nodes in output layer?
Neural Network for Classification
Inputs
Hidden Layers
Outputs
Team Record
Player Ratings
Field Weather
Last Match Result
Home/Away
Probability of each outcome
→ P(Win)
→ P(Draw)
→ P(Lose)
Neural Network for Classification
Outputs
Probability of each outcome
→
→
→
Use softmax activation function to bound outputs between 0 - 1.
Note: For regression tasks (e.g. predicting price of home), only 1 output node
Predicted price
Tuning Neural Networks
Part 2
Ok, so you’ve built a neural network!
Now, how do we make it great?
Ok, so you’ve built a neural network!
Now, how do we make it great?
We make it well-tuned! But instead of adjusting polynomial terms (like above), we’ll adjust…
Things to tune on a neural network…
Note: There are many more things we could tune, but we’re focusing on the above for this course
Depth (Number of Layers)
Width (Number of Nodes per Layer)
Different choices → Different model complexity
VS.
Chat Waterfall: What is the danger of making our networks extremely complex?
Problem: Can’t use just one test set. May want to make many adjustments after trials on multiple test sets. Might run out of data for a true test…
Solution: Validation Set!
We need to add a validation set! (explanation on next slide)
While tuning …
Fit model on this data
Test model on this data
Repeat until find best model architecture.
Try out:
1 layer, 20 nodes
2 layers, 20 nodes
3 layers, 20 nodes
1 layer, 30 nodes
2 layers, 30 nodes …
Evaluate model predictions on unseen test data (final prediction)
Fit optimal model architecture all this data (final model)
Once you’ve decided the optimal architecture (e.g. 3 layers, 20 nodes)...