Lecture #4: ML Quality Assurance
Modern machine learning for data analysis
Common Pitfalls
Avoidance Strategies
http://chakkrit.com/teaching/quantitative-research-methods
LAST UPDATED
27/06/2022
LAST UPDATED
27/06/2022
Regression in Python | An Example
# First, prepare the data for model building�import pandas�df = pandas.read_csv('diabetes.csv')�df.head()��#split dataset in features and target variable�feature_cols = df.columns.to_list()�feature_cols.remove('Outcome')��X = df[feature_cols] # Features�y = df.Outcome # Target variable��from sklearn.model_selection import train_test_split�X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=16)
# Check the distribution of Y-value (~35% positive)
y_train.value_counts()
0 375
1 201
Name: Outcome, dtype: int64
y_test.value_counts()
0 125
1 67
Name: Outcome, dtype: int64
Preparing the dataset: https://www.kaggle.com/datasets/uciml/pima-indians-diabetes-database
LAST UPDATED
27/06/2022
Classification in Python | An Example
# Building the model�from sklearn.linear_model import LogisticRegression�logreg = LogisticRegression(random_state=16) # using the default parameters�logreg.fit(X_train, y_train) # fit the model with data�y_pred = logreg.predict(X_test)
# Build confusion matrix�from sklearn import metrics�cnf_matrix = metrics.confusion_matrix(y_test, y_pred)�
# calculate evaluation metrics�from sklearn.metrics import classification_report�target_names = ['without diabetes', 'with diabetes']�print(classification_report(y_test, y_pred, target_names=target_names))
# calculate AUC�import matplotlib.pyplot as plt�plt.cla()�y_pred_proba = logreg.predict_proba(X_test)[::,1]�fpr, tpr, _ = metrics.roc_curve(y_test, y_pred_proba)�auc = metrics.roc_auc_score(y_test, y_pred_proba)�plt.plot(fpr,tpr,label="data 1, auc="+str(auc))�plt.legend(loc=4)�plt.show()
array([[116, 9],
[ 26, 41]])
precision recall f1-score
without diabetes 0.82 0.93 0.87� with diabetes 0.82 0.61 0.70� accuracy 0.82
macro avg 0.82 0.77 0.78� weighted avg 0.82 0.82 0.81
LAST UPDATED
27/06/2022
Decision Trees (for regression or classification)
Root node
Leaf node
Decision nodes
- Root Node: It represents the entire population or sample.
- Decision Node: A sub-node that split from other node
- Leaf Node: A Node that cannot be further split.
- Pruning: Decision trees can easily overfit. Pruning (i.e., removing sub-nodes) can be performed to avoid overfitting.
Predicting a response variable using a tree-like structure decisioning. To construct a tree, recursively splitting the dataset into leaf-nodes until every leaf-node cannot be split anymore.��The features and rules used in splitting is determined by maximizing the information gain.
Higher
information gain
Less
information gain
However, decision tree can be unstable because small variations in the data might result in a completely different tree being generated.
LAST UPDATED
27/06/2022
Random Forest and XGBoost
Random Forest combines the outputs of multiple decision trees using majority votes (classification) or averaging (regression).
* Robust to noises
XGBoost build multiple trees where weights of the wrong predictions are passed to the next tree to focus on improving the wrong predictions.
* Performs best
LAST UPDATED
27/06/2022
Random Forest and XGBoost
# Random Forest�from sklearn.ensemble import RandomForestClassifier�randomForest = RandomForestClassifier(random_state=16)�randomForest.fit(X_train, y_train)��# calculate AUC�import matplotlib.pyplot as plt�plt.cla()�y_pred_proba = randomForest.predict_proba(X_test)[::,1]�fpr, tpr, _ = metrics.roc_curve(y_test, y_pred_proba)�auc = metrics.roc_auc_score(y_test, y_pred_proba)�plt.plot(fpr,tpr,label="data 1, auc="+str(auc))�plt.legend(loc=4)�plt.show()
�# XG Boost�from xgboost import XGBClassifier�xgboost = XGBClassifier()�xgboost = xgboost.fit(X_train, y_train)��# calculate AUC�import matplotlib.pyplot as plt�plt.cla()�y_pred_proba = xgboost.predict_proba(X_test)[::,1]�fpr, tpr, _ = metrics.roc_curve(y_test, y_pred_proba)�auc = metrics.roc_auc_score(y_test, y_pred_proba)�plt.plot(fpr,tpr,label="data 1, auc="+str(auc))�plt.legend(loc=4)�plt.show()
LAST UPDATED
27/06/2022
Overfitting and Underfitting
How to:�- Estimate the performance using cross validation technique
- Reduce the number of features�(Dimensional reduction)
Over-fitting
Under-fitting
A model describes noises instead of the underlying relationship (overly complex).
The model have poor performance, as it overreacts to minor fluctuations in the training data
A model cannot capture the data trend. Underfitting would occur, for example, when fitting a linear model to non-linear data.
The model would have poor predictive performance as it cannot capture the data trend.
How to:�- Estimate the performance using cross validation technique�- Choose a more advanced algorithm
LAST UPDATED
27/06/2022
Regression analysis is a statistical process for estimating the relationships between a dependent variables (or response variables) and one or more independent variables (or explanatory variables).
Use cases
There are three types of regression analysis (or regression model):
Regression Analysis
Non-Linear Regression�
Linear Regression�
Logistic Regression
Predicting a numerical variable,
assuming a linear relationship
between X and Y
Predicting a binary variable
Predicting a numerical variable,
assuming a non-linear relationship between X and Y
LAST UPDATED
27/06/2022
Machine Learning Quality Assurance | 12 Checklists
An experience report on defect modelling in practice: pitfalls and challenges Chakkrit Tantithamthavorn, and Ahmed E. Hassan In Proceedings of the 40th International Conference on Software Engineering: Software Engineering in Practice, ICSE (SEIP) 2018, Gothenburg, Sweden, May 27 - June 03, 2018, 2018
LAST UPDATED
27/06/2022
#1: Assure no spurious correlations
If your dataset contains anything other than the signal (i.e. noise) that is correlated with your labels, your model will learn to exploit it for prediction.
How to avoid:
Check the correlation between each features and target prediction
For features with high correlation, check
If not, we may exclude the feature out.
We may also consider mixed effects models that can consider the features as random effects, e.g., mixed effects logistic regression�
m <- glmer(remission ~ featureA + featureB + featureC + featureD + (1 | confoundingFeature), data = df, …)
LAST UPDATED
27/06/2022
#2: Assure no data mismatch between testing and use case
Mismatch between testing dataset and the real use case may lead to a model with poor performance.
How to avoid:
Check if the testing data has a similar characteristics as the real usage scenario.
ImageNet�(simple and straight images of objects)
ObjectNet�(Objects with different rotations, backgrounds, and viewpoints)
For example, a model trained using ImageNet dataset
may not performed well with ObjectNet dataset
LAST UPDATED
27/06/2022
#3: Assure no testing data leakage
Using testing data to train the model will mislead the model performance evaluation.
How to avoid:
Apply cross-validation technique properly.
Clearly splitting training and testing dataset
If required, split the validation data from the training dataset
LAST UPDATED
27/06/2022
#4: Assure no testing data leakage in time-series problems
In time-series prediction problem (e.g., predicting stock price), training the model using future data and testing using the past data is strictly prohibited.
How to avoid:
Information leakage when perform random splitting
Expanding window cross-validation
Sliding window cross-validation
LAST UPDATED
27/06/2022
#5: Using proper performance metrics for an imbalanced dataset
In an imbalanced prediction problem (e.g., identifying cancer patient), using metrics like accuracy may mislead the benefit and usefulness of the model in real usage scenario.
How to avoid: By using
AUC = 1 indicate that the model can perfectly predict the outcome
LAST UPDATED
27/06/2022
#6: Assure no duplicate data rows between training and testing
Having duplicate data rows in both training and testing dataset can be considered as data leakage.
How to avoid:
- Include other dimensions (features) to provide more information for the model training
- Check and remove the duplicated data rows to avoid leakage.
ID | Size | Suburb |
1 | 200 | Carlton |
2 | 150 | Melbourne |
3 | 170 | Melbourne |
4 | 340 | Fitzroy |
ID | Size | Suburb |
5 | 200 | Carlton |
6 | 180 | Mlebourne |
7 | 280 | Fitzroy |
8 | 320 | Fitzroy |
Training Dataset
Testing Dataset
LAST UPDATED
27/06/2022
#7: Avoid training a model using too many features
The model that trained with too many features (no. features > no. rows) may predict a single data points using single feature, or making special case just for a single data point.
How to avoid:
Apply Dimensional reduction
Do not use one-hot encoding
Use label encoding instead
*also make sure you treat this as an ordinal variables
LAST UPDATED
27/06/2022
#8: Avoid training a model using highly-correlated features
Having too many features or highly correlated features may interfere the model’s prediction.
How to avoid:
- Perform dimensional reduction to reduce the number of features (e.g., Principal component analysis, correlation analysis)
Principal Component Analysis (PCA)
Correlation analysis based on Spearman Rho
The Impact of Correlated Metrics on the Interpretation of Defect Models Jirayus Jiarpakdee, Chakkrit Tantithamthavorn, and Ahmed E. Hassan IEEE Trans. Software Eng., 2021
LAST UPDATED
27/06/2022
#9: Avoid training a model using severely imbalanced columns (non-explanatory features)
Training a model using imbalanced columns may lead to overfitting (i.e., having too many features that does not provide predictive performance).
How to avoid:
- Exclude imbalanced columns
ID | Size | Suburb | Type (Y) |
1 | 400 | Collingwood | Large |
2 | 150 | Melbourne | Small |
3 | 170 | Melbourne | Small |
4 | 240 | Melbourne | Small |
5 | 220 | Melbourne | Small |
6 | 400 | Melbourne | Large |
7 | 450 | Melbourne | Large |
… | … | … | … |
if suburb = Collingwood:
Type is Large
else:
if size >= 400:
Type is large
else:
Type is small
LAST UPDATED
27/06/2022
#10: Avoid re-balancing on the testing dataset.
Testing dataset should be treated as unseen data. We should preserve the characteristics of the dataset.
How to avoid:
Over-sampling
Apply class re-balancing only on training dataset��(after splitting the test dataset)
Perform class-rebalancing on the training dataset
The Impact of Class Rebalancing Techniques on the Performance and Interpretation of Defect Prediction Models Chakkrit Tantithamthavorn, Ahmed E. Hassan, and Kenichi Matsumoto IEEE Trans. Software Eng., 2020
LAST UPDATED
27/06/2022
#11: Avoid hyperparameter tuning on the testing dataset.
Testing dataset should be treated as unseen data. We should preserve the characteristics of the dataset.
How to avoid:
Hyper parameter tuning
Apply class re-balancing only on training dataset��(after splitting the test dataset)
Automated parameter optimization of classification techniques for defect prediction models Chakkrit Tantithamthavorn, Shane McIntosh, Ahmed E. Hassan, and Kenichi Matsumoto In Proceedings of the 38th International Conference on Software Engineering, ICSE 2016, Austin, TX, USA, May 14-22, 2016, 2016
LAST UPDATED
27/06/2022
#12: Avoid using a naïve cross-validation method
Measuring naïve cross-validation split methods may not accurately reflect the model prediction performance.
How to avoid:
Single random split
K-fold cross validation
Original dataset
Train
Sampling with replacement until reaching the size of theoriginal dataset
Test
Bootstrap sampling validation
* and repeat this process multiple times
An Empirical Comparison of Model Validation Techniques for Defect Prediction Models, Chakkrit Tantithamthavorn, Shane McIntosh, Ahmed E. Hassan, and Kenichi Matsumoto IEEE Trans. Software Eng., 2017
LAST UPDATED
27/06/2022
Dimensionality Reduction
There are two common methods for dimensionality reduction
1) Principal component analysis (PCA)
2) Correlation analysis
Having too many features or highly correlated features may impact the model’s performance
PITFALL: Avoid training the model using too many or highly-correlated features
1) Convert features into matrix that summarizes how the variables in the dataset relate to one another (Covariance matrix).
2) Break the matrix down into two separate components, i.e., directions (i.e., Eigenvectors) and importance (i.e., magnitude or Eigenvalues)�
3) Transform original data to align with the important directions
4) Drop eigenvectors that are relatively unimportant
However, this method is not recommended in empirical research as it minimize the explainability of the prediction model
WITH CASE STUDY EXAMPLE (10m features e.g., protein prediction) – เน้น reduction in explainability - dataset ประมาณนี้จะเยอะมาก (feature เยอะมาก) พอpredict แล้วจะงงๆ ว่าอันไหน truly useful – PCA จะ reduce dimension
Axes become principal components
PITFALL: Avoid training the model using too many or highly-correlated features
Principal Component Analysis (PCA)
Correlation Analysis
Remove highly-correlated variables based on Spearman’s correlation.��1) Find a pair of metric that are highly correlated (Spearman’s rho > 0.7)
2) From a pair, only keep the metric that is the least correlated with other metrics that are not in the pair.��3) Do step 1-2 again to recheck��(see Workshop 2, “Perform correlation analysis”)
PITFALL: Avoid training the model using too many or highly-correlated features
Class-Imbalanced Dataset
In classification problem, the distribution of examples may be skewed toward one class, which which can affect the model’s performance.
Examples:
- identifying patient with cancer � (e.g., only 0.5% of the samples are cancer positive).
- Identifying fake or spam tweets in Twitter
Next, we discussing the approaches to mitigate the effects of imbalanced dataset in the prediction model.
may impact the interpretation of the model
Mitigating Imbalanced Dataset Issues
How to fix:�
1) Choose proper evaluation metrics
1.1) use threshold-dependent metric like AUC, i.e., the Area Under the Receiver Operating Characteristic curve
1.2) use metrics that focus on positive classes, e.g., Recall, F1-score, false alarm rate, balanced accuracy
1.3) find an optimal threshold (based on training dataset)
2) Using ensemble-learning (e.g., random forest) that can focus on minority class
3) Perform class-rebalancing (see next slide)
Class Re-Balancing Techniques
Under-sampling
Over-sampling
Re-sampling the minority class to help the model learns the characteristics of the rare occurrences.
1) Random over-sampling – no loss of information, but prone to overfitting as same information is copied�
2) SMOTE – re-sampling while creating synthetic data using k-nearest neighbor algorithm.
Under-sampling the majority class to equalizes the model learning on both classes. However, this introduces important information loss.
1) Randomly deletes the data from majority classes.
Nevertheless, the re-balanced dataset may not represent the distribution of the original dataset �(i.e., concept drift), which may impact the interpretation of the models
PITFALL: Avoid re-balancing on the whole dataset or testing dataset.
PITFALL ขึ้นก่อน
Solution
Class Re-Balancing Techniques
df = pandas.read_csv('diabetes.csv’)
X = df[feature_cols] # Features�y = df.Outcome # Target variable
from sklearn.model_selection import train_test_split�X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=16)
y_train.value_counts() # proportion before SMOTE
0 375
1 201
Name: Outcome, dtype: int64
�from imblearn.over_sampling import SMOTE�oversample = SMOTE()�X_train, y_train = oversample.fit_resample(X_train, y_train)
y_train.value_counts() # proportion afterSMOTE�0 375
1 375
Name: Outcome, dtype: int64
# Recheck that we did not re-balancing the testing dataset
y_test.value_counts()
# 0 125
# 1 67
PITFALL: Avoid re-balancing on the whole dataset or testing dataset.
Before SMOTE
After SMOTE
Data Leakage
Using the testing or future data when training the model is considered as data leakage.
Clearly separate training/testing dataset
In time sensitive domains, not using future data to train the model
PITFALL: Avoid training the model based on testing data (chorological)_.
Data Duplication
Even though the training and testing data are clearly separated, there might be some data points that contains similar data appeared in both datasets (and might cause overfitting).�
We may have to include other dimensions (features) to provide more information to the model, e.g., number of bedroom and bathroom.��If applicable, we may also exclude the duplicated rows from either training or testing dataset.
PITFALL: Avoid training the model while having duplicate data in training and testing dataset.
ID | Size | Suburb |
1 | 200 | Carlton |
2 | 150 | Melbourne |
3 | 170 | Melbourne |
4 | 340 | Fitzroy |
ID | Size | Suburb |
5 | 200 | Carlton |
6 | 180 | Mlebourne |
7 | 280 | Fitzroy |
8 | 320 | Fitzroy |
Training Dataset
Testing Dataset
Hyperparameter Tuning
Tuning the model’s hyper parameters and build your solutions based on the best performing models. ��The common approaches are Grid Search and Random Search (only one parameter at a time) or using Differential Evolution (multiple parameters).
For examples, tuning number of trees of Random Forest, choosing the solver and penalty of Logistic Regression, tuning gamma and penalty of Support Vector Machines.
PITFALL: Avoid tuning hyperparameters using the testing dataset.
Differential Evolution
Optimizing no. of trees in Random Forest
Hyperparameter Tuning
# Build Random Forest with fixed number of trees = 2��randomForest = RandomForestClassifier(random_state=16, n_estimators=2)�randomForest.fit(X_train, y_train)�y_pred_proba = randomForest.predict_proba(X_test)[::,1]�auc = metrics.roc_auc_score(y_test, y_pred_proba)�
# Using n_estimators = 2 gives AUC = 0.7235820895522388�
PITFALL: Avoid tuning hyperparameters using the testing dataset.
�# Build Random Forest with optimal number of trees��# First, generate validating dataset from TRAINING dataset�X_train_new, X_validate, y_train_new, y_validate = train_test_split(X_train, y_train, test_size=0.20, random_state=16)
�# perform the differential evolution search for optimal #trees
# in this method, we build RF given the #trees and return AUC�def buildAndMeasureAUC(x):� randomForest = RandomForestClassifier(random_state=16, n_estimators=int(x[0]))� randomForest.fit(X_train_new, y_train_new)� y_pred_proba = randomForest.predict_proba(X_validate)[::, 1]� fpr, tpr, _ = metrics.roc_curve(y_validate, y_pred_proba)� return metrics.roc_auc_score(y_validate, y_pred_proba)��bounds = Bounds([1], [200])�differential_evolution(buildAndMeasureAUC, bounds)
# fun: 0.7982113821138211 (AUC)
# message: 'Optimization terminated successfully.’
# nfev: 32
# nit: 1
# success: True
# x: array([12.19969592]) (optimal #trees = 12)
# further check AUC based on testing dataset
# Preparing dataset (Train & Test)�
from sklearn.ensemble import RandomForestClassifier�from sklearn import metrics
from scipy.optimize import differential_evolution, Bounds�
df = pandas.read_csv('diabetes.csv')�feature_cols = df.columns.to_list()�feature_cols.remove('Outcome')�X = df[feature_cols] # Features�y = df.Outcome # Target variable�from sklearn.model_selection import train_test_split�X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=16)