Module 4 : Machine Learning
Session 4E - Boosting Algorithms and Trees for Regression
Sammi Rosser
“Greased Lightnin’”
11th June 2024
What are we covering now?
We’ve got our heads around decision trees and random forests now.
But we know that both of these models are quite prone to overfitting.
Are there any more models we might want to try?
High-performing Models
Image credit: https://www.c-sharpcorner.com/article/xgboost-the-choice-of-champions/
What are we covering now?
XGBoost (Extreme Gradient Boosting)
and LightGBM (light gradient boosting machine)
belong to a family of classifiers known as �gradient boosting classifiers
There are also a range of other variants, like AdaBoost and CatBoost - with slight differences in how they operate, but they all fundamentally use the same underlying principle.
Why these classifiers?
XGBoost and other boosting classifiers work very well on tabular data - which we tend to have a lot of in health, policing, social care and public health.
They can be used for classification or regression problems.
They also often cope reasonably well with smaller datasets.
In comparison, neural networks (which we cover next week) often work very well on unstructured data and benefit from very large datasets.
Boosting
�
rather
than
Sometimes these will be ‘stumps’ with just one split - but they can be deeper depending on the algorithm and parameters
+
+
+
By focussing more on the data points that the previous models misclassified, you can make a really powerful model!
The key thing that connects all of these models are
Difference Between Random Forests and Boosted Trees
Both methods use multiple trees.
But random forests train multiple models independently and then take the class that most of the models predict to be the final prediction.
XGBoost trains multiple models one after the other to make one final good prediction.
+
+
‘Has disease’
‘Has disease’
‘Doesn’t have disease’
Majority vote = ‘Has disease’
Final output = 0.784, so predict ‘has disease’
How does XGBoost work?
We train a weak learner and see how it performs �using a loss function (a way to measure the error)
Key concept
Bonus
Minimising error = minimising loss function
The loss function we use depends on the type of problem - we can adjust this in XGBoost’s parameters
Higher values of the loss function �= prediction was worse
XGBoost also incorporates a regularization term to avoid the model becoming too complex
Calculate error
How does XGBoost work?
We then train a second tree that uses the original features but instead of trying to predict the class, it is trying to predict the errors
Adding the prediction from the first tree and this tree should give us a better result (smaller loss) than just using the first tree.�
The errors are a number, not a class label - but remember we mentioned before that trees can be used to predict a continuous variable too?
Key concept
Bonus
Calculate error
Train tree to predict errors
The gradients of the loss function help indicate the way to adjust the model
The predictions from the subsequent trees are added to the original predictions, but scaled by a learning rate
Learning rates are quite small - say 0.01
Each subsequent tree makes a small contribution to the final prediction
How does XGBoost work?
We repeat this process until we reach a certain number of trees
Key concept
Bonus
Calculate error
We can also use an early stopping criteria �
(e.g. if the performance stops improving
for lots of steps in a row, we stop -
this can help avoid overfitting)
Train tree to predict errors
Calculate error
How does XGBoost work?
The final prediction takes into account the predictions of all of the trees
e.g. if the predictions sum to 0.6, we’ll predict the positive class (1).
If they sum to 0.3, we’ll predict the negative class (0).
We can change the threshold point
(e.g. we could say that anything above 0.4 should count as the positive class (1).
(think back to ROC curves)
Key concept
How does XGBoost work?
Summary
Key concept
Code for Boosted Models
Let’s start with the code for an XGBoost model.
XGBoost is very popular and is a good choice for a wide range of tabular datasets.
And I’ve got great news…
It’s really easy to implement.
XGBoost: The Key Steps
It’s a lot like the previous models!
1. Divide the data into features (inputs) and labels (outputs)
2. Divide the data into training and test sets
3. Apply feature scaling, so that the feature data values are all on a similar scale
We don’t need to apply feature scaling with xgboost! :)
4. Create a logistic regression decision tree random forest XGBoost model
5. Fit the model on our training data
6. Predict y from x in our
7. Assess the performance of the model �(accuracy, precision, recall (sensitivity), F1, ROC curve, AUROC, confusion matrix)
A Code Example
Read in the data
Check the dataset
Split into X (features) and y (labels)
A Code Example
Split into training and testing data
Create a random forest classifier and fit using the training/test data
Importing our Data
Predict the labels for our training and testing data
View the accuracy
Variants of Boosted Models
XGBoost is a very solid choice.
(hot off the press - it’s now incorporating aspects of LightGBM too)
However, there are a range of other boosted models you may want to consider using, like
But let’s start by talking a bit more about XGBoost and when you might want to use it.
XGBoost
Zella King and the team at UCL used XGBoost to train a model on historic emergency department data so they could provide predictions of upcoming bed capacity requirements based on the people currently in the emergency department
A Real-World XGBoost Example
Slide: Zella King, NHS-R webinar 19/06/2024
Downsides of XGBoost
https://blog.cambridgespark.com/getting-started-with-xgboost-3ba1488bb7d4
Tuning XGBoost Models
Parameter | Notes | Default |
eta | Also called ‘step size’ or ‘learning rate’. Lower rates make the model ‘learn’ slower (which can prevent overfitting). | 0.3 |
max_depth | Maximum depth of a single tree. Higher = more likely to overfit (and use lots of memory!) Lower = more likely to underfit | 6 |
min_child_weight | Larger values make the trees less complex (avoids creating very overfitted models) Note that lambda relates to L2 regularization, and alpha to L1 regularization | 1 |
gamma | 0 | |
lambda | 1 | |
alpha | 0 | |
subsample | When set to less than 1, sample this proportion of the data for the next boosting round. Consider using in conjunction with sampling_method | 1 |
Tuning XGBoost Models
That seems like a lot of parameters to try out.
How can you work out the best combination?
In session 4J we’ll talk about grid search and the optuna optimization framework to answer that question.
For now - try out some different combinations!
https://blog.cambridgespark.com/getting-started-with-xgboost-3ba1488bb7d4
Other Alternatives…
There are lots of variants on boosted trees.
XGBoost remains very popular - but it’s worth knowing about the other options as they may work better on your particular dataset.
Like XGBoost did, most of them slot neatly into our existing code!
ADABoost
https://dataheadhunters.com/academy/gradient-boosting-vs-adaboost-battle-of-the-algorithms/
ADABoost: Downsides
https://dataheadhunters.com/academy/gradient-boosting-vs-adaboost-battle-of-the-algorithms/
ADABoost: A Code Example
ADABoost is included in sklearn
CatBoost
Turning categorical data into a series of columns with 1s and 0s instead - more on this later!
CatBoost
Unlike decision trees, random forests, XGBoost and AdaBoost, when using CatBoost we don’t have to manually preprocess the data to one-hot encode our categorical datasets.
CatBoost: A Code Example
Histogram-Based Gradient Boosting Models
Histogram-Based Gradient Boosting Models
Client 1 | 1 |
Client 2 | 3 |
Client 3 | 12 |
Client 4 | 36 |
Client 5 | 13 |
Client 6 | 5 |
Client 7 | 6 |
Client 8 | 25 |
Client 1 | 1-9 |
Client 2 | 1-9 |
Client 3 | 10-19 |
Client 4 | 30-39 |
Client 5 | 10-19 |
Client 6 | 1-9 |
Client 7 | 1-9 |
Client 8 | 20-29 |
Histogram-Based Gradient Boosting Models: A Code Example
LightGBM
LightGBM
LightGBM: A Code Example
Model Reproducibility
Other Classification Algorithms to Be Aware Of
K-Nearest Neighbours
Based on the 2 nearest = red triangle
Based on the 3 nearest = red triangle (but less sure)
Based on the 5 nearest = blue square
Image: https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm
Scaling/normlising of features is required (like in logistic regression)
“Weighted” variant can help overcome issues with data imbalance
Techniques exist to help select what k should be
k too large = underfit (high variance)
k too small = overfit (high bias)
In binary (two-class) classification problems, make k odd to avoid ties!
And more…
Naive Bayes
Classifier built on Bayes’ theorem
Hypothesis = class label
Evidence = data features
Assumes all features in the dataset are independent (probably not true, but works well regardless!)
Like having a big table of probabilities from past data - for a new datapoint, checks against this table and assigns a class
from sklearn.naive_bayes import GaussianNB
model = GaussianNB()
Support Vector Machines
Tries to find the best boundary (hyperplane) so separate data into different classes
Trying to maximise distance from nearest point to boundary of any class (those nearest points = support vectors)
The boundary is then used to classify new datapoints
Can be linear or non-linear
from sklearn import svm
model = svm.SVC()
Summary: Classification Models
Logistic Regression | from sklearn.linear_model import LogisticRegression model = LogisticRegression() | Good baseline model Don’t forget to standardise or scale your data! |
Decision Trees | from sklearn.tree import DecisionTreeClassifier model = DecisionTreeClassifier() | Interpretable, but prone to overfitting |
Random Forest | from sklearn.ensemble import RandomForestClassifier model = RandomForestClassifier() | Less interpretable, but a bit less prone to overfitting |
XGBoost | from xgboost.sklearn import XGBClassifier model = XGBClassifier() | Powerful - used commonly in industry and a good one to try. Not interpretable without the help of SHAP or LIME. |
AdaBoost | from sklearn.ensemble import AdaBoostClassifier model = AdaBoostClassifier() | Older model - not used much now (Other boosting algorithms tend to work better). Same problems as XGBoost with interpretability. |
CatBoost | from catboost import CatBoostClassifier model = CatBoostClassifier() | Great for data with lots of categorical features. Same problems as XGBoost with interpretability. |
LightGBM | from lightgbm import LGBMClassifier model = LGBMClassifier() | Another powerful option! Same problems as XGBoost with interpretability. |
Histogram Gradient Boosting | from sklearn.ensemble import HistGradientBoostingClassifier model = HistGradientBoostingClassifier() | Another powerful option! Same problems as XGBoost with interpretability. |
Let’s have a quick look at the performance of these models on our datasets from earlier.
Open up your copy of random_forest_stroke_exercise.ipynb from the previous session.
Or load boosting_exercise.ipynb if your previous notebook is feeling a bit bloated - this notebook has just got the data from before imported.
Make some additional cells and try out
Try tweaking the parameters and see what performance you can achieve.
If you haven’t already, try writing a function to reduce how much code you are having to copy and paste each time you assess a model.
Exercise 1
Take a 10 minute break, then we’ll work on this for 30 minutes
Train/Validate/Test
Train
Teach the model the patterns
Validate
Used when refining model/ tweaking parameters
Test
Only used at the very end to simulate real-world data and potential performance
A Warning - Variable Names!
from sklearn.metrics import f1_score
f1_score = f1_score(y_test, y_pred_test)
ERROR
from sklearn.metrics import f1_score
f1_score_calculated = f1_score(y_test, y_pred_test)
:)
Regression
Numerical Predictions with Machine Learning Models
What happens if I want to predict something numerical from my data?
For example
Regression
When we are trying to make a machine learning model learn a numerical pattern of data, we call this a regression problem.
And you’ll be pleased to know that
Regression
Like with classification, parameters such as the maximum depth of the tree can have an impact on overfitting of the model - so it will still need to be tuned in a similar way.
max_depth = 2
max_depth = 5
Code Example: �Regression with Decision Trees
from sklearn.tree import DecisionTreeRegressor
regr_dt = DecisionTreeRegressor()
# Train the model using the training sets
regr_dt.fit(diabetes_X_train, diabetes_y_train)
# Make predictions using the testing set
diabetes_y_pred = regr_dt.predict(diabetes_X_test)
Code Example: �Regression with XGBoost
from xgboost import XGBRegressor
regr_xg = XGBRegressor(random_state=42)
# Train the model using the training sets
regr_xg.fit(diabetes_X_train, diabetes_y_train)
# Make predictions using the testing set
diabetes_y_pred = regr_xg.predict(diabetes_X_test)
Metrics with Regression Models
The biggest difference about regression problems is how we measure the performance of the model.
Metrics like accuracy, precision and recall don’t make sense for regression problems.
Instead, we use a range of aggregate metrics relating to how wrong each of the predictions the model is on our training and test sets.
If the true value is 12 and the model predicts 14, that might not be too bad!
If it is 12 and the model predicts 45, that’s definitely worse!
But even for the first model, is that a big error with regards to the actual measure? If the measure only ranges between 11 and 15, it’s a pretty rubbish model…
MAE
MAE = Mean Absolute Error
Work out all of the errors (the difference between the prediction and the real value)
Get the absolute value of the error�(negative values become positive�Positive values stay positive!)
Add all of the absolute errors together.
Divide by the total number of predictions.
A sense of how big the gap between reality and predictions are
Lower = Better
MAPE
MAE = Mean Absolute Percentage Error
The MAE - but as a percentage of the total error.
(we divide each error by the actual value, multiply by 100, and sum these before dividing by the number of datapoint)
E.g. a MAPE of 10% means
A sense of how big the gap between reality and predictions are - but more comparable across datasets
Lower = Better
There are some significant drawbacks and limitations to MAPE - use with caution!
RMSE
RMSE = Root Mean Squared Error
Work out the square of all of the errors (the difference between the prediction and the real value)
Add all of the squared errors together.
Divide by the total number of predictions.
Take the square root of this result.
Another way to get a sense of how big the gap between reality and predictions are
RMSE is more sensitive to outliers (extreme points) than MAE - which is a good thing if very wrong points are more important to you than slightly wrong points
Lower = Better
R2
R2 = Coefficient of Determination
Pronounced ‘R squared’
“The proportion of the variation in the dependent variable that is predictable from the independent variable(s). ”
Higher = Better�
Maximum of 1
Generally calculated as �R² = 1 - (sum of squared residuals / total sum of squares)
Can be inflated by adding more features (even if it doesn’t actually help the predictive power)
Regression Models: Evaluation
Calculating Metrics
from sklearn.tree import DecisionTreeRegressor
from sklearn.metrics import mean_absolute_error, mean_absolute_percentage_error, \
r2_score, root_mean_squared_error
regr_dt = DecisionTreeRegressor()
# Train the model using the training sets
regr_dt.fit(diabetes_X_train, diabetes_y_train)
# Make predictions using the testing set
diabetes_y_pred = regr_dt.predict(diabetes_X_test)
print(f"Mean absolute error: {mean_absolute_error(diabetes_y_test, diabetes_y_pred):.2f}")
print(f"Mean absolute percentage error: {mean_absolute_percentage_error(diabetes_y_test, diabetes_y_pred):.2%}" )
print(f"Root Mean squared error: {root_mean_squared_error(diabetes_y_test, diabetes_y_pred):.2f}")
print(f"Coefficient of determination: {r2_score(diabetes_y_test, diabetes_y_pred):.2f}")
#1 is perfect prediction
Regression Models: Evaluation
Plotting Actual Values vs Predictions
def plot_actual_vs_predicted(actual, predicted):
fig, ax = plt.subplots(figsize=(6, 6))
ax.scatter(actual, predicted, color="black")
ax.axline((1, 1), slope=1)
plt.xlabel('True Values')
plt.ylabel('Predicted Values')
plt.title('True vs Predicted Values')
plt.show()
plot_actual_vs_predicted(�diabetes_y_test, �diabetes_y_pred�)
Regression Models: Evaluation
Plotting Residuals (Errors)
def plot_residuals(actual, predicted):
residuals = actual - predicted
plt.figure(figsize=(10, 5))
plt.hist(residuals, bins=20)
plt.axvline(x = 0, color = 'r')
plt.xlabel('Residual')
plt.ylabel('Frequency')
plt.title('Distribution of Residuals')
plt.show()
plot_residuals(diabetes_y_test, diabetes_y_pred)
Residuals = the size of the gap between the predicted value and the actual value
E.g. if the actual value is 7 but the model predicts 11, the residual is +4
�If the model predicted 5, the residual would be -2
Residuals should be normally distributed
Other Regression Approaches
from lightgbm import LGBMRegressor
from catboost import CatBoostRegressor
from sklearn.ensemble import HistGradientBoostingRegressor
from sklearn.linear_model
import LinearRegression
Data Prep
OneHot Encoding
The titanic and stroke datasets we’ve worked with so far have been pre-processed to make them suitable for the machine learning algorithms we’ve been using.
One big part of this is OneHot encoding.
OneHot Encoding
PassengerId | Cabin�Letter_A | Cabin�Letter_B | Cabin�Letter_C | Cabin�Letter_D | Cabin�Letter_E | Cabin�Letter_�missing |
1 | 0 | 0 | 0 | 0 | 0 | 1 |
2 | 0 | 0 | 1 | 0 | 0 | 0 |
3 | 0 | 0 | 0 | 0 | 0 | 1 |
4 | 0 | 0 | 1 | 0 | 0 | 0 |
5 | 0 | 0 | 0 | 0 | 0 | 1 |
6 | 0 | 0 | 0 | 0 | 0 | 1 |
7 | 0 | 0 | 0 | 0 | 1 | 0 |
PassengerId | CabinLetter |
1 | |
2 | C |
3 | |
4 | C |
5 | |
6 | |
7 | E |
OneHot Encoding
It’s not too difficult to do this in your own datasets!
While there is a method for this available in Sklearn, I think the easiest method is included in Pandas.
Let’s return to the dataset of my music taste from earlier…
OneHot Encoding
Let’s take a look at the genre column.
Song | Release Year | Genre | Does Sammi Like it? |
Sultans of Swing | 1978 | Rock | Yes |
Genie in a Bottle | 1999 | Pop | No |
The Room Where It Happens | 2015 | Musical | Yes |
Boys of Summer | 1984 | Rock | Yes |
I Wish I Had Duck Feet | 1994 | House | No |
Gecko (overdrive) | 2014 | House | Yes |
Unknown Caller | 2009 | Rock | No |
Time Warp | 1975 | Musical | Yes |
Defying Gravity | 2004 | Musical | Yes |
Stickwitu | 2005 | Pop | No |
Never Ever | 1997 | Pop | No |
OneHot Encoding
Song | Release Year | Rock | Pop | Musical | House | Does Sammi Like it? |
Sultans of Swing | 1978 | 1 | 0 | 0 | 0 | Yes |
Genie in a Bottle | 1999 | 0 | 1 | 0 | 0 | No |
The Room Where It Happens | 2015 | 0 | 0 | 1 | 0 | Yes |
Boys of Summer | 1984 | 1 | 0 | 0 | 0 | Yes |
I Wish I Had Duck Feet | 1994 | 0 | 0 | 0 | 1 | No |
Gecko (overdrive) | 2014 | 0 | 0 | 0 | 1 | Yes |
Unknown Caller | 2009 | 1 | 0 | 0 | 0 | No |
Time Warp | 1975 | 0 | 0 | 1 | 0 | Yes |
Defying Gravity | 2004 | 0 | 0 | 1 | 0 | Yes |
Stickwitu | 2005 | 0 | 1 | 0 | 0 | No |
Never Ever | 1997 | 0 | 1 | 0 | 0 | No |
One-hot encoding spreads our single genre column into as many columns as there are categories, and puts a 1 in the matching category for that row and 0 in all other columns.
OneHot Encoding: A Code Example
We can use the get_dummies() function in pandas.
The one_hot object looks like this →
We then drop the genre column from our original dataframe, and join the one_hot dataframe to our original dataframe (using the index to join the two on).
OneHot Encoding: A Code Example
‘True’ and ‘False’ are considered to be equivalent to 1 and 0 respectively.
We don’t have to explicitly change the columns from a boolean to a float or integer - the algorithms can cope with that for us.
However, we can use .astype() to do this for us.
E.g. pd.get_dummies(data[‘Genre’]).astype(‘int’)
There are some other alternatives to one-hot encoding for dealing with categorical data, but this is a good option.
OneHot Encoding: A Code Example
In some instances, the resulting column names may not be very descriptive!
You can use the ‘prefix’ argument to add a string to the start of each resulting column name.
E.g. here
pd.get_dummies(data[‘Genre’], prefix=’Genre’)
Would mean our outputs are instead
Genre_House, Genre_Musical, etc.
Dichotomous Columns
For categories where only two options are present, we can choose a simpler option.
For example, let’s assume we have a category for ‘age_group’ that just has the values ‘child’ or ‘adult’.
Here, instead of one-hot encoding, we could keep it as a single column but set
Child = 0 Adult = 1
This is commonly done when gender categories only provide binary options too, as in the titanic dataset.
There are a couple of ways we could go about this in Python- here’s one!
data[‘age_group’].replace(‘child’, 0, inplace=True)
data[‘age_group’].replace(‘adult’, 1, inplace=True)
Missing Data and More…
We’ll cover more about data prep in later sessions (4j) - OneHot encoding and dealing with dichotomous variables are the key ones for today’s exercise.
But down the line, it will be important to think about
Data quality and prep can have more impact than anything else!
(and often is more of the project time than you’d think…)
The Next Exercise
The Dataset
The Dataset
eid | Integer | Unique Id of the hospital admission | 98 |
vdate | String | Visit date | 8/29/2012 |
rcount | String | Number of readmissions within last 180 days | 0, 1, 2, 3, 4, 5+ |
gender | String | Gender of the patient | M, F |
dialysisrenalendstage | Integer | Flag for renal disease during encounter | 0 or 1 |
asthma | Integer | Flag for asthma during encounter | |
irondef | Integer | Flag for iron deficiency during encounter | |
pneum | Integer | Flag for pneumonia during encounter | |
substancedependence | Integer | Flag for substance dependence during encounter | |
psychologicaldisordermajor | Integer | Flag for major psychological disorder during encounter | |
depress | Integer | Flag for depression during encounter | |
psychother | Integer | Flag for other psychological disorder during encounter | |
fibrosisandother | Integer | Flag for fibrosis during encounter | |
malnutrition | Integer | Flag for malnutrituion during encounter | |
hemo | Integer | Flag for blood disorder during encounter |
The Dataset
secondarydiagnosisnonicd9 | Integer | Flag for whether a non ICD 9 formatted diagnosis was coded as a secondary diagnosis | Integer from 1 to …? |
facid | String | Facility ID at which the encounter occurred | A, B, C, D, E |
lengthofstay | Integer | Length of stay for the encounter |
hematocritic | Float | Average hematocritic value during encounter (g/dL) | Numeric value - sometimes integer, sometimes float. Varying ranges. |
neutrophils | Float | Average neutrophils value during encounter (cells/microL) | |
sodium | Float | Average sodium value during encounter (mmol/L) | |
glucose | Float | Average sodium value during encounter (mmol/L) | |
bloodureanitro | Float | Average blood urea nitrogen value during encounter (mg/dL) | |
creatinine | Float | Average creatinine value during encounter (mg/dL) | |
bmi | Float | Average BMI during encounter (kg/m2) | |
pulse | Float | Average pulse during encounter (beats/m) | |
respiration | Float | Average respiration during encounter (breaths/m) |
The things you’ll wish you could do…
When you start playing around with this dataset, you’ll probably find yourself wishing you could do a few things:
Open up the notebook regression_tree_exercise.ipynb
This will help guide you through the initial data cleaning steps for this dataset.
After that - it’s up to you!
You are trying to build the best-performing model to predict the length of stay from the available dataset.�
Make sure to try out
Then calculate at least one metric for each, like MAE.
Explore the documentation and additional resources to find out more about the hyperparameters you can tune for the regression models.
Exercise 2
We’ll work on this until the end of the day
Further Watching
Further Watching
Further Watching
Further Watching