1 of 42

ME 5990�Machine Learning for ME

�

Bayes Classification�Cross Validation

2 of 42

Outline

  • Bayes’ classifier with Gaussian Kernel
    • Naïve Bayes classifier
    • Log-likelihood
  • Evaluation methods
    • Cross-validation method
    • Separation of train-validation method

​

​

3 of 42

Modeling the likelihood: Training

  • Now we have conducted almost everything other than validation
  • We “trained” a model to predict one’s gender based on height, weight, and foot size.
  • A “model” is a transfer function that maps input features to label
  • With a specific input (e.g. height = 6, weight = 165, foot size = 10), the model shall produce a unique output

​

Model�function

Label, “output”

Feature, “input”

Feature

Estimated label

4 of 42

Model

  •  

5 of 42

Model the likelihood

  •  

6 of 42

Model the likelihood

  •  

7 of 42

Likelihood Modeling using Naïve Bayes

  • Why we use Multi-variate Gaussian for modeling the likelihood?
  • Can we use an alternative method to avoid multi-variate Gaussian?
  • How about we assume:
    • Height, weight, and foot size individually follow Gaussian distribution for female and male,
    • The overall likelihood is the product of three individual likelihood?

​

8 of 42

Likelihood Modeling using Naïve Bayes

  •  

9 of 42

Likelihood Modeling using Naïve Bayes

  •  

10 of 42

Train Result

  •  

11 of 42

Likelihood

  •  
  •  

12 of 42

Posterior

  •  

13 of 42

Naïve Bayes Classifier

  • For each dimension of feature, we can “train” a likelihood model
    • While Gaussian is common, we don’t have to use Gaussian model
    • Deep learning model will also generate a “likelihood”
  • Treat each dimension of feature as if they were independent, the overall likelihood is the product of each feature dimension
  • The posterior is the normalization of likelihood * prior for all classes
  • We make decision by using the largest posterior (expectation if you are more serious)

​

​

14 of 42

Naïve Bayes Classifier

  •  

15 of 42

Outline

  • Bayes’ classifier with Gaussian Kernel
    • Naïve Bayes classifier
    • Log-likelihood
  • Evaluation methods
    • Cross-validation method
    • Separation of train-validation method

​

​

16 of 42

Log-likelihood

  •  

17 of 42

Log-likelihood

  •  

18 of 42

Log-likelihood

  •  

19 of 42

Log-likelihood

  •  

20 of 42

Log-likelihood

  •  

21 of 42

Log-likelihood

  •  

22 of 42

Log-likelihood

  •  

23 of 42

Outline

  • Bayes’ classifier with Gaussian Kernel
    • Naïve Bayes classifier
    • Log-likelihood
  • Evaluation methods
    • Cross-validation method
    • Separation of train-validation method

​

​

24 of 42

How good is our model?

  • Validate from a separated set, called validation set (some one call it “test set”)
    • Sample 1: (13.3, 1.22, 2.32) (Male)
    • Sample 2: (15.3, 1.12, 1.72) (Female)
    • Sample 3: …

�

​

 

25 of 42

Evaluation/validation of the model

  •  

26 of 42

Evaluation/validation of the model

  • Can we do self-validation?
  • For example, We trained on the following set
  • We use the same set for prediction and compare to the ground truth label

Naïve Bayes Estimation

N-d Gaussian Estimation

Male

Female

Male

Male

Male

Female

Female

Male

Female

Female

Female

Female

Female

Female

Female

Male

Accuracy by Naïve Bayes

Accuracy by N-d Gaussian

87.5%

62.5%

Can we conclude?

27 of 42

Evaluation/validation of the model

  • What we get from self-validation is called “training error”
  • The training error is a reference indicator for the model
  • The training error shall NOT be reported as an evaluation metric
    • Self-evaluation is not trustworthy… Look at Boeing…

Naïve Bayes Estimation

N-d Gaussian Estimation

Male

Female

Male

Male

Male

Female

Female

Male

Female

Female

Female

Female

Female

Female

Female

Male

Training error Naïve Bayes

Training error N-D Gaussian

12.5%

37.5%

We can find training error

​

Training error�is not a metric�for model evaluation

28 of 42

Compare different learning models

  • Instead, another scientific evaluation method is to do k-fold cross validation
  • We want to know is Naïve Bayes for likelihood is better, or Bayes with N-d gaussian likelihood is better?
    • 1. Shuffle the given dataset, and split it into k folds

​

​

    • 2. Select one-fold as validation, others as training,

​

​

​

​

​

​

​

​

​

​

​

​

For this iteration:�Navie Byaes: �Correct: 10, Incorrect 10

Accuracy: 50%

N-D Gaussian Bayes:

Correct: 15, Incorrect 5

Accuracy: 75%

​

​

​

29 of 42

Compare different learning models

    • 3. Repeat this validation process for each folder

​

​

​

​

​

​

​

​

    • Continue until every fold is used as the validation set once

​

​

​

​

​

​

​

​

​

For this iteration:

Navie Byaes: �Correct: 13, Incorrect 7

Accuracy: 65%

N-D Gaussian Bayes:

Correct: 16, Incorrect 4

Accuracy: 80%

​

…

​

​

​

​

​

​

​

30 of 42

Compare different learning models

    • 4. Report mean accuracy for each learning model

​

​

​

​

​

​

​

​

​

​

​

​

Navie Byaes:

Validation 1�Correct: 10, Incorrect 10

Accuracy: 50%

Validation 2

Correct: 13, Incorrect 7

Accuracy: 65%

…

Validation N

Correct 9, Incorrect 11

Accuracy 45%

​

Mean accuracy 52%

Navie Byaes:

Validation 1�Correct: 15, Incorrect 5

Accuracy: 75%

Validation 2

Correct: 16, Incorrect 4

Accuracy: 80%

…

Validation N

Correct 12, Incorrect 8

Accuracy 60%

​

Mean accuracy 67%

31 of 42

Cross Validation

  • Randomly shuffle your dataset into k-fold. (I didn’t shuffle here…)

32 of 42

Cross Validation

  • Use 1 folder for validation, others for training
  • Report accuracy Accuracy_1

33 of 42

Cross Validation

  • Then, repeat the process, select another folder for validating, the rest for training. Get Accuracy_2

34 of 42

Cross Validation

  • Just repeat, till all folders serve as validation set

Accuracy_3

35 of 42

Cross Validation

  • Just repeat, till all folders serve as validation set

Accuracy_4

36 of 42

Cross Validation: calculate mean accuracy

Report the mean accuracy on the validation folds as the model accuracy

37 of 42

Dataset role in machine learning challenges

  • There are many machine learning challenges,
  • The goal is to find the “best model”
  • Some organizers will provide

​

​

​

​

  • You can use cross-validation method to train and validate your various models, and pick the best
  • Submit your accuracy of each model on test data

​

​

Train Data

Test Data

38 of 42

Dataset role in machine learning challenges

  • You shall never use test data for any type of training!

https://www.cvlibs.net/datasets/kitti/user_login.php

​

39 of 42

Separation of train and validation set

  • There are many machine learning challenges,
    • ImageNet Challenge
  • Some organizers will provide

​

​

​

​

​

​

​

Training

Test

​

Validation

40 of 42

Separation of train and validation set

  • You can train various models on the training set
  • Validate each of your model on the validation set to find the “best model”
  • You may mix training and validation if you wish
  • Report accuracy on the testing set as your final evaluation result
  • Never mix any testing data for training!

​

​

41 of 42

Separation of train and validation set

  • Why not using k-fold cross validation?
  • Some models are too large to do k-fold training
    • For the female and male problem
    • If we use N-d Gaussian model, we have 24 parameters to train
    • If we use naïve Bayes model with 1D Gaussian, we have 12 parameters to train
    • We can train fairly fast

​

    • What if we have 10000 parameters to train??

​

42 of 42

Summary

  • Likelihood model is a model. The way to model likelihood varied
  • Naïve Bayes can be used to simplify the training
  • Log-likelihood can avoid boundary-exceeding values caused by product of sequence terms
  • K-fold cross validation is commonly used for small models
  • Large models usually do train-validation separation

​

​

​

​

​

​

​