Ensemble Learning

• Understanding Ensembles
• K-fold cross validation
• Boosting,
• Stumping
• XGBoost
• Bagging
• Subagging
• Random Forest
• Comparison with Boosting
• Different ways to combine classifiers

*

1

Dr. S. M. Patil, Computer Engineering Department , SIGCE

Ensemble Learning

• Ensemble learning helps improve machine learning results by combining several models.
• This approach allows the production of better predictive performance compared to a single model.
• Basic idea is to learn a set of classifiers (experts) and to allow them to vote.

*

2

Dr. S. M. Patil, Computer Engineering Department , SIGCE

Ensemble Learning

*

3

Dr. S. M. Patil, Computer Engineering Department , SIGCE

Ensemble Learning

Why do ensembles work?

• It Overcomes three problems
• Statistical Problem –�The Statistical Problem arises when the hypothesis space is too large for the amount of available data. Hence, there are many hypotheses with the same accuracy on the data and the learning algorithm chooses only one of them! There is a risk that the accuracy of the chosen hypothesis is low on unseen data!
• Computational Problem –�The Computational Problem arises when the learning algorithm cannot guarantees finding the best hypothesis.
• Representational Problem –�The Representational Problem arises when the hypothesis space does not contain any good approximation of the target class(es).

​

*

4

Dr. S. M. Patil, Computer Engineering Department , SIGCE

Ensemble Learning

Methods for Independently Constructing Ensembles –

• Majority Vote
• Bagging and Random Forest
• Randomness Injection
• Feature-Selection Ensembles
• Error-Correcting Output Coding

​

*

5

Dr. S. M. Patil, Computer Engineering Department , SIGCE

Ensemble Learning

Methods for Independently Constructing Ensembles –

• Majority Vote
• Bagging and Random Forest
• Randomness Injection
• Feature-Selection Ensembles
• Error-Correcting Output Coding

Methods for Coordinated Construction of Ensembles –

• Boosting
• Stacking

​

​

*

6

Dr. S. M. Patil, Computer Engineering Department , SIGCE

Ensemble Learning

k-Fold Cross-Validation

• Cross-validation is a resampling procedure used to evaluate machine learning models on a limited data sample.
• The procedure has a single parameter called k that refers to the number of groups that a given data sample is to be split into. As such, the procedure is often called k-fold cross-validation. When a specific value for k is chosen, it may be used in place of k in the reference to the model, such as k=10 becoming 10-fold cross-validation.
• Cross-validation is primarily used in applied machine learning to estimate the skill of a machine learning model on unseen data. 
• That is, to use a limited sample in order to estimate how the model is expected to perform in general when used to make predictions on data not used during the training of the model.

​

​

​

*

7

Dr. S. M. Patil, Computer Engineering Department , SIGCE

Ensemble Learning

k-Fold Cross-Validation

It is a popular method because it is simple to understand and because it generally results in a less biased or less optimistic estimate of the model skill than other methods, such as a simple train/test split.

​

​

*

8

Dr. S. M. Patil, Computer Engineering Department , SIGCE

Ensemble Learning

k-Fold Cross-Validation

• The general procedure is as follows:
• Shuffle the dataset randomly.
• Split the dataset into k groups
• For each unique group:
1. Take the group as a hold out or test data set
2. Take the remaining groups as a training data set
3. Fit a model on the training set and evaluate it on the test set
4. Retain the evaluation score and discard the model
• Summarize the skill of the model using the sample of model evaluation scores

​

​

*

9

Dr. S. M. Patil, Computer Engineering Department , SIGCE

Ensemble Learning

​

• Importantly, each observation in the data sample is assigned to an individual group and stays in that group for the duration of the procedure. This means that each sample is given the opportunity to be used in the hold out set 1 time and used to train the model k-1 times.

*

10

Dr. S. M. Patil, Computer Engineering Department , SIGCE

Ensemble Learning

Configuration of k

• The k value must be chosen carefully for your data sample.
• A poorly chosen value for k may result in a mis-representative idea of the skill of the model, such as a score with a high variance (such as an overestimate of the skill of the model).
• Three common tactics for choosing a value for k are as follows:
• Representative: The value for k is chosen such that each train/test group of data samples is large enough to be statistically representative of the broader dataset.
• k=10: The value for k is fixed to 10, a value that has been found through experimentation to generally result in a model skill estimate with low bias a modest variance.
• k=n: The value for k is fixed to n, where n is the size of the dataset to give each test sample an opportunity to be used in the hold out dataset. This approach is called leave-one-out cross-validation.

​

​

​

*

11

Dr. S. M. Patil, Computer Engineering Department , SIGCE

Ensemble Learning

• Example
• To make the cross-validation procedure concrete, let’s look at a worked example.
• Imagine we have a data sample with 6 observations:

[0.1, 0.2, 0.3, 0.4, 0.5, 0.6]

• The first step is to pick a value for k in order to determine the number of folds used to split the data.
• Here, we will use a value of k=3. That means we will shuffle the data and then split the data into 3 groups.
• Because we have 6 observations, each group will have an equal number of 2 observations.

​

*

12

Dr. S. M. Patil, Computer Engineering Department , SIGCE

Ensemble Learning

• For example:

Fold1: [0.5, 0.2]

Fold2: [0.1, 0.3]

Fold3: [0.4, 0.6]

• We can then make use of the sample, such as to evaluate the skill of a machine learning algorithm.
• Three models are trained and evaluated with each fold given a chance to be the held out test set.
• For example:
• Model1: Trained on Fold1 + Fold2, Tested on Fold3
• Model2: Trained on Fold2 + Fold3, Tested on Fold1
• Model3: Trained on Fold1 + Fold3, Tested on Fold2

​

​

*

13

Dr. S. M. Patil, Computer Engineering Department , SIGCE

Ensemble Learning

• The models are then discarded after they are evaluated as they have served their purpose.
• The skill scores are collected for each model and summarized for use
• https://machinelearningmastery.com/k-fold-cross-validation/

*

14

Dr. S. M. Patil, Computer Engineering Department , SIGCE

Ensemble Learning

Cross-Validation API

• We do not have to implement k-fold cross-validation manually. The scikit-learn library provides an implementation that will split a given data sample up.
• The KFold() scikit-learn class can be used. It takes as arguments the number of splits, whether or not to shuffle the sample, and the seed for the pseudorandom number generator used prior to the shuffle.
• For example, we can create an instance that splits a dataset into 3 folds, shuffles prior to the split, and uses a value of 1 for the pseudorandom number generator.

​

​

​

*

15

Dr. S. M. Patil, Computer Engineering Department , SIGCE

Ensemble Learning

kfold = KFold(3, True, 1)

• The split() function can then be called on the class where the data sample is provided as an argument. Called repeatedly, the split will return each group of train and test sets. Specifically, arrays are returned containing the indexes into the original data sample of observations to use for train and test sets on each iteration.
• For example, we can enumerate the splits of the indices for a data sample using the created KFold instance as follows:

# enumerate splits

for train, test in kfold.split(data):

print('train: %s, test: %s' % (train, test))

​

​

*

16

Dr. S. M. Patil, Computer Engineering Department , SIGCE

Ensemble Learning

• We can tie all of this together with our small dataset used in the worked example 

# scikit-learn k-fold cross-validation

from numpy import array

from sklearn.model_selection import KFold

# data sample

data = array([0.1, 0.2, 0.3, 0.4, 0.5, 0.6])

# prepare cross validation

kfold = KFold(3, True, 1)

# enumerate splits

for train, test in kfold.split(data):

print('train: %s, test: %s' % (data[train], data[test]))

​

​

*

17

Dr. S. M. Patil, Computer Engineering Department , SIGCE

Ensemble Learning

• Running the example prints the specific observations chosen for each train and test set. The indices are used directly on the original data array to retrieve the observation values.

train: [0.1 0.4 0.5 0.6], test: [0.2 0.3]

train: [0.2 0.3 0.4 0.6], test: [0.1 0.5]

train: [0.1 0.2 0.3 0.5], test: [0.4 0.6]

• Usefully, the k-fold cross validation implementation in scikit-learn is provided as a component operation within broader methods, such as grid-searching model hyperparameters and scoring a model on a dataset.
• Nevertheless, the KFold class can be used directly in order to split up a dataset prior to modeling such that all models will use the same data splits. This is especially helpful if you are working with very large data samples. The use of the same splits across algorithms can have benefits for statistical tests that you may wish to perform on the data later.

​

*

18

Dr. S. M. Patil, Computer Engineering Department , SIGCE

Ensemble Learning

Variations on Cross-Validation

• commonly used variations are as follows:
• Train/Test Split: Taken to one extreme, k may be set to 2 (not 1) such that a single train/test split is created to evaluate the model.
• LOOCV: Taken to another extreme, k may be set to the total number of observations in the dataset such that each observation is given a chance to be the held out of the dataset. This is called leave-one-out cross-validation, or LOOCV for short.
• Stratified: The splitting of data into folds may be governed by criteria such as ensuring that each fold has the same proportion of observations with a given categorical value, such as the class outcome value. This is called stratified cross-validation.
• Repeated: This is where the k-fold cross-validation procedure is repeated n times, where importantly, the data sample is shuffled prior to each repetition, which results in a different split of the sample.
• Nested: This is where k-fold cross-validation is performed within each fold of cross-validation, often to perform hyperparameter tuning during model evaluation. This is called nested cross-validation or double cross-validation.

​

*

19

Dr. S. M. Patil, Computer Engineering Department , SIGCE

Ensemble Learning

• Boosting is an ensemble modeling technique that attempts to build a strong classifier from the number of weak classifiers.
• How to create a model using boosting?
• It is done by building a model by using weak models in series. Firstly, a model is built from the training data. Then the second model is built which tries to correct the errors present in the first model. This procedure is continued and models are added until either the complete training data set is predicted correctly or the maximum number of models are added. 
• AdaBoost was the first really successful boosting algorithm developed for the purpose of binary classification. AdaBoost is short for Adaptive Boosting and is a very popular boosting technique that combines multiple “weak classifiers” into a single “strong classifier”.

*

20

Dr. S. M. Patil, Computer Engineering Department , SIGCE

Ensemble Learning

Algorithm: 

1. Initialize the dataset and assign equal weight to each of the data point.
2. Provide this as input to the model and identify the wrongly classified data points.
3. Increase the weight of the wrongly classified data points.
4. if (got required results) �  Goto step 5 �else �  Goto step 2 � 
5. End

​

*

21

Dr. S. M. Patil, Computer Engineering Department , SIGCE

Ensemble Learning

• B1 consists of 10 data points which consist of two types namely plus(+) and minus(-) and 5 of which are plus(+) and the other 5 are minus(-) and each one has been assigned equal weight initially. The first model tries to classify the data points and generates a vertical separator line but it wrongly classifies 3 plus(+) as minus(-).
• B2 consists of the 10 data points from the previous model in which the 3 wrongly classified plus(+) are weighted more so that the current model tries more to classify these pluses(+) correctly. This model generates a vertical separator line that correctly classifies the previously wrongly classified pluses(+) but in this attempt, it wrongly classifies three minuses(-).

​

​

*

22

Dr. S. M. Patil, Computer Engineering Department , SIGCE

Ensemble Learning

• B3 consists of the 10 data points from the previous model in which the 3 wrongly classified minus(-) are weighted more so that the current model tries more to classify these minuses(-) correctly. This model generates a horizontal separator line that correctly classifies the previously wrongly classified minuses(-).

​

• B4 combines together B1, B2, and B3 in order to build a strong prediction model which is much better than any individual model used.
• �

*

23

Dr. S. M. Patil, Computer Engineering Department , SIGCE

Ensemble Learning

*

24

Dr. S. M. Patil, Computer Engineering Department , SIGCE

Steps in adaptive Boosting

Performing Adaptive Boosting, we have to iteratively go through each of the steps

Sum of all recursive classifiers

Ensemble Learning

• Here, we use the following decision stumps
• X= [2, 4, 6]
• Y=[2, 2.5, 3]

*

25

Dr. S. M. Patil, Computer Engineering Department , SIGCE

Ensemble Learning

*

26

Dr. S. M. Patil, Computer Engineering Department , SIGCE

Step 1: 1/N

Ensemble Learning

*

27

Dr. S. M. Patil, Computer Engineering Department , SIGCE

Ensemble Learning

*

28

Dr. S. M. Patil, Computer Engineering Department , SIGCE

Ensemble Learning

*

29

Dr. S. M. Patil, Computer Engineering Department , SIGCE

Ensemble Learning

*

30

Dr. S. M. Patil, Computer Engineering Department , SIGCE

Ensemble Learning

*

31

Dr. S. M. Patil, Computer Engineering Department , SIGCE

Ensemble Learning

XGBoost 

• XGBoost is an implementation of Gradient Boosted decision trees. This library was written in C++.
• It supports various interfaces like CLI , C++,Python(Sklearn), R(Caret package),Julia,Java and JVM language like Scala and platforms like Hadoop
• It is a type of Software library that was designed basically to improve speed and model performance.
• It has recently been dominating in applied machine learning.
• XGBoost models majorly dominate in many Kaggle Competitions.
• In this algorithm, decision trees are created in sequential form. Weights play an important role in XGBoost. Weights are assigned to all the independent variables which are then fed into the decision tree which predicts results. The weight of variables predicted wrong by the tree is increased and the variables are then fed to the second decision tree. These individual classifiers/predictors then ensemble to give a strong and more precise model.
• It can work on regression, classification, ranking, and user-defined prediction problems.

�

​

​

*

32

Dr. S. M. Patil, Computer Engineering Department , SIGCE

Ensemble Learning

​

​

​

*

33

Dr. S. M. Patil, Computer Engineering Department , SIGCE

Ensemble Learning

• XGBoost Features The library is laser-focused on computational speed and model performance, as such, there are few frills. Model Features Three main forms of gradient boosting are supported:
• Gradient Boosting
• Stochastic Gradient Boosting
• Regularized Gradient Boosting

System Features

• For use of a range of computing environments this library provides-
• Parallelization of tree construction
• Distributed Computing for training very large models
• Cache Optimization of data structures and algorithm

​

​

*

34

Dr. S. M. Patil, Computer Engineering Department , SIGCE

Ensemble Learning

XGBoost enhancements/optimizations

• XGBoost features various optimizations built-in to make the training faster when working with large datasets, in addition to its unique method of generating and pruning trees. Here is a handful of the most significant:

​

• Approximate Greedy Algorithm: instead of assessing every candidate split, this algorithm employs weighted quantiles to find the best node split.
• Cash-Aware Access: XGBoost stores data in the CPU’s cache memory.
• Sparsity: Aware Split Finding calculates Gain by putting observations with missing values onto the left leaf when there is some missing data. It then repeats the process by placing them in the appropriate leaf and selecting the scenario with the highest Gain.

�

​

*

35

Dr. S. M. Patil, Computer Engineering Department , SIGCE

Ensemble Learning

Bagging

• It is a homogeneous weak learners’ model that learns from each other independently in parallel and combines them for determining the model average.
• It is also known as Bootstrap Aggregating, is a machine learning ensemble meta-algorithm designed to improve the stability and accuracy of machine learning algorithms used in statistical classification and regression.
• It decreases the variance It decreases the variance and helps to avoid overfitting.
• It is usually applied to decision tree methods. Bagging is a special case of the model averaging approach. 

​

​

​

*

36

Dr. S. M. Patil, Computer Engineering Department , SIGCE

Ensemble Learning

Description of the Technique

• Suppose a set D of d tuples, at each iteration i, a training set D_i of d tuples is selected via row sampling with a replacement method (i.e., there can be repetitive elements from different d tuples) from D (i.e., bootstrap). Then a classifier model Mi is learned for each training set D < i. Each classifier M_i returns its class prediction. The bagged classifier M* counts the votes and assigns the class with the most votes to X (unknown sample).

​

​

​

​

*

37

Dr. S. M. Patil, Computer Engineering Department , SIGCE

Ensemble Learning

Implementation Steps of Bagging

Step 1: Multiple subsets are created from the original data set with equal tuples, selecting observations with replacement.

Step 2: A base model is created on each of these subsets.

Step 3: Each model is learned in parallel with each training set and independent of each other.

Step 4: The final predictions are determined by combining the predictions from all the models.

​

​

​

​

​

*

38

Dr. S. M. Patil, Computer Engineering Department , SIGCE

Ensemble Learning

*

39

Dr. S. M. Patil, Computer Engineering Department , SIGCE

Ensemble Learning

Working of Bagging on training dataset

Since Bagging resamples the original training dataset with replacement, some instance(or data) may be present multiple times while others are left out.

• Original training dataset: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10
• Resampled training set 1: 2, 3, 3, 5, 6, 1, 8, 10, 9, 1�Resampled training set 2: 1, 1, 5, 6, 3, 8, 9, 10, 2, 7�Resampled training set 3: 1, 5, 8, 9, 2, 10, 9, 7, 5, 4

​

​

​

​

​

​

​

*

40

Dr. S. M. Patil, Computer Engineering Department , SIGCE

Ensemble Learning

Algorithm for the Bagging classifier:

Classifier generation:

Let N be the size of the training set.

for each of t iterations:

sample N instances with replacement from the original training set.

apply the learning algorithm to the sample.

store the resulting classifier.

Classification:

for each of the t classifiers:

predict class of instance using classifier.

return class that was predicted most often.

​

*

41

Dr. S. M. Patil, Computer Engineering Department , SIGCE

Ensemble Learning

​

​

​

​

​

​

​

*

42

Dr. S. M. Patil, Computer Engineering Department , SIGCE

Ensemble Learning

Random Forest

• Random Forest is Machine learning Algorithm that belongs to supervisedlearning technique.
• It can be used for both Classification and Regression problems in ML.
• It is based on the concept of ensemble learning, which is a process of combining multiple classifiers to solve a complex problem and to improve the performance of the model.
• Random Forest is a classifier that contains a number of decision trees on various subsets of the given dataset and takes the average to improve the predictive accuracy of that dataset.
• the random forest takes the prediction from each tree and based on the majority votes of predictions, it predicts the final output.
• The greater number of trees in the forest leads to higher accuracy and prevents the problem of overfitting.

​

*

43

Dr. S. M. Patil, Computer Engineering Department , SIGCE

Ensemble Learning

Working of Random Forest

​

​

*

44

Dr. S. M. Patil, Computer Engineering Department , SIGCE

Ensemble Learning

Random Forest

Assumptions for Random Forest :

• There should be some actual values in the feature variable of the dataset so that the classifier can predict accurate results rather than a guessed result.
• The predictions from each tree must have very low correlations.

Why use Random Forest?

• It takes less training time as compared to other algorithms.
• It predicts output with high accuracy, even for the large dataset it runs efficiently.
• It can also maintain accuracy when a large proportion of data is missing.

​

*

45

Dr. S. M. Patil, Computer Engineering Department , SIGCE

Ensemble Learning

Working of Random Forest

Random Forest works in two-phases

1. create the random forest by combining N decision tree

2. to make predictions for each tree created in the first phase.

The Working process

Step-1: Select random K data points from the training set.

Step-2: Build the decision trees associated with the selected data points (Subsets).

Step-3: Choose the number N for decision trees that you want to build.

Step-4: Repeat Step 1 & 2.

Step-5: For new data points, find the predictions of each decision tree, and assign the new data points to the category that wins the majority votes.

​

*

46

Dr. S. M. Patil, Computer Engineering Department , SIGCE

Ensemble Learning

Example: Suppose there is a dataset that contains multiple fruit images. So, this dataset is given to the Random forest classifier. The dataset is divided into subsets and given to each decision tree. During the training phase, each decision tree produces a prediction result, and when a new data point occurs, then based on the majority of results, the Random Forest classifier predicts the final decision.

*

47

Dr. S. M. Patil, Computer Engineering Department , SIGCE

Ensemble Learning

Applications of Random Forest

• Banking: Banking sector mostly uses this algorithm for the identification of loan risk.
• Medicine: With the help of this algorithm, disease trends and risks of the disease can be identified.
• Land Use: We can identify the areas of similar land use by this algorithm.
• Marketing: Marketing trends can be identified using this algorithm.

​

*

48

Dr. S. M. Patil, Computer Engineering Department , SIGCE

Ensemble Learning

Advantages of Random Forest

• Random Forest is capable of performing both Classification and Regression tasks.
• It is capable of handling large datasets with high dimensionality.
• It enhances the accuracy of the model and prevents the overfitting issue.

Disadvantages of Random Forest

• Although random forest can be used for both classification and regression tasks, it is not more suitable for Regression tasks.

*

49

Dr. S. M. Patil, Computer Engineering Department , SIGCE

Ensemble Learning

1. Bagging is a homogeneous weak learners’ model that learns from each other independently in parallel and combines them for determining the model average.
2. Boosting is also a homogeneous weak learners’ model but works differently from Bagging. In this model, learners learn sequentially and adaptively to improve model predictions of a learning algorithm.

​

Some of the factors that cause errors in learning are noise, bias, and variance. The ensemble method is applied to reduce these factors resulting in the stability and accuracy of the result.

*

50

Dr. S. M. Patil, Computer Engineering Department , SIGCE

Ensemble Learning

Bagging

• Bagging is an acronym for ‘Bootstrap Aggregation’ and is used to decrease the variance in the prediction model. Bagging is a parallel method that fits different, considered learners independently from each other, making it possible to train them simultaneously.
• Bagging generates additional data for training from the dataset. This is achieved by random sampling with replacement from the original dataset. Sampling with replacement may repeat some observations in each new training data set. Every element in Bagging is equally probable for appearing in a new dataset. 
• These multi datasets are used to train multiple models in parallel. The average of all the predictions from different ensemble models is calculated. The majority vote gained from the voting mechanism is considered when classification is made. Bagging decreases the variance and tunes the prediction to an expected outcome.

​

*

51

Dr. S. M. Patil, Computer Engineering Department , SIGCE

Ensemble Learning

Bagging

Example of Bagging:

• The Random Forest model uses Bagging, where decision tree models with higher variance are present. It makes random feature selection to grow trees. Several random trees make a Random Forest.

​

​

*

52

Dr. S. M. Patil, Computer Engineering Department , SIGCE

Ensemble Learning

Boosting

• Boosting is a sequential ensemble method that iteratively adjusts the weight of observation as per the last classification. If an observation is incorrectly classified, it increases the weight of that observation. The term ‘Boosting’ in a layman language, refers to algorithms that convert a weak learner to a stronger one. It decreases the bias error and builds strong predictive models.
• Data points mispredicted in each iteration are spotted, and their weights are increased. The Boosting algorithm allocates weights to each resulting model during training. A learner with good training data prediction results will be assigned a higher weight. When evaluating a new learner, Boosting keeps track of learner’s errors. 

Example of Boosting: 

• The AdaBoost uses Boosting techniques, where a 50% less error is required to maintain the model. Here, Boosting can keep or discard a single learner. Otherwise, the iteration is repeated until achieving a better learner.

​

​

​

*

53

Dr. S. M. Patil, Computer Engineering Department , SIGCE

Ensemble Learning

Bagging and Boosting: Similarities

1. Bagging and Boosting are ensemble methods focused on getting N learners from a single learner.
2. Bagging and Boosting make random sampling and generate several training data sets 
3. Bagging and Boosting arrive upon the end decision by making an average of N learners or taking the voting rank done by most of them.
4. Bagging and Boosting reduce variance and provide higher stability with minimizing errors.

​

​

​

​

*

54

Dr. S. M. Patil, Computer Engineering Department , SIGCE

Ensemble Learning

Bagging and Boosting: Differences

• Bagging is a method of merging the same type of predictions. 

Boosting is a method of merging different types of predictions.

• Bagging decreases variance, not bias, and solves over-fitting issues in a model.

Boosting decreases bias, not variance.

• In Bagging, each model receives an equal weight.

In Boosting, models are weighed based on their performance.

• Models are built independently in Bagging.

New models are affected by a previously built model’s performance in Boosting.

• In Bagging, training data subsets are drawn randomly with a replacement for the training dataset.

In Boosting, every new subset comprises the elements that were misclassified by previous models

• Bagging is usually applied where the classifier is unstable and has a high variance. Boosting is usually applied where the classifier is stable and simple and has high bias.

​

​

​

​

​

*

55

Dr. S. M. Patil, Computer Engineering Department , SIGCE

Different ways to combine classifiers

Multiclassifiers

• When there are several classifiers with a common objective it is called a multiclassifier.
• In Machine Learning multiclassifiers are sets of different classifiers which make estimates and are fused together, obtaining a result that is a combination of them.
• Lots of terms are used to refer to multiclassifiers: multi-models, multiple classifier systems, combining classifiers, decision committe, etc.

They can be divided into two big groups:

• Ensemble methods: Refers to sets of systems that combine to create a new system using the same learning technique. Bagging and Boosting are the most extended ones.
• Hybrid methods: Takes a set of different learners and combines them using new learning techniques. Stacking (or Stacked Generalization) is one of the main hybrid multiclassifiers.

​

​

*

56

Dr. S. M. Patil, Computer Engineering Department , SIGCE

Different ways to combine classifiers

• Stacking is one of the most popular ensemble machine learning techniques used to predict multiple nodes to build a new model and improve model performance.
• Stacking enables us to train multiple models to solve similar problems, and based on their combined output, it builds a new model with improved performance.
• Various weak learners are ensembled in a parallel manner in such a way that by combining them with Meta learners, we can predict better predictions for the future.
• This ensemble technique works by applying input of combined multiple weak learners' predictions and Meta learners so that a better output prediction model can be achieved.
• In stacking, an algorithm takes the outputs of sub-models as input and attempts to learn how to best combine the input predictions to make a better output prediction.
• Stacking is also known as a stacked generalization and is an extended form of the Model Averaging Ensemble technique in which all sub-models equally participate as per their performance weights and build a new model with better predictions. This new model is stacked up on top of the others; this is the reason why it is named stacking.

​

*

57

Dr. S. M. Patil, Computer Engineering Department , SIGCE

Different ways to combine classifiers

Architecture of Stacking

• It consists of two or more base/learner's models and a meta-model that combines the predictions of the base models.
• These base models are called level 0 models, and the meta-model is known as the level 1 model.
• So, the Stacking ensemble method includes original (training) data, primary level models, primary level prediction, secondary level model, and final prediction.

​

​

*

58

Dr. S. M. Patil, Computer Engineering Department , SIGCE

Different ways to combine classifiers

Architecture of Stacking

​

​

​

*

59

Dr. S. M. Patil, Computer Engineering Department , SIGCE

Different ways to combine classifiers

Architecture of Stacking

• Original data: This data is divided into n-folds and is also considered test data or training data.
• Base models: These models are also referred to as level-0 models. These models use training data and provide compiled predictions (level-0) as an output.
• Level-0 Predictions: Each base model is triggered on some training data and provides different predictions, which are known as level-0 predictions.
• Meta Model: The architecture of the stacking model consists of one meta-model, which helps to best combine the predictions of the base models. The meta-model is also known as the level-1 model.
• Level-1 Prediction: The meta-model learns how to best combine the predictions of the base models and is trained on different predictions made by individual base models, i.e., data not used to train the base models are fed to the meta-model, predictions are made, and these predictions, along with the expected outputs, provide the input and output pairs of the training dataset used to fit the meta-model.

​

​

​

​

​

*

60

Dr. S. M. Patil, Computer Engineering Department , SIGCE

Different ways to combine classifiers

Steps to implement Stacking models:

• Split training data sets into n-folds using the RepeatedStratifiedKFold as this is the most common approach to preparing training datasets for meta-models.
• Now the base model is fitted with the first fold, which is n-1, and it will make predictions for the nth folds.
• The prediction made in the above step is added to the x1_train list.
• Repeat steps 2 & 3 for remaining n-1folds, so it will give x1_train array of size n,
• Now, the model is trained on all the n parts, which will make predictions for the sample data.
• Add this prediction to the y1_test list.
• In the same way, we can find x2_train, y2_test, x3_train, and y3_test by using Model 2 and 3 for training, respectively, to get Level 2 predictions.
• Now train the Meta model on level 1 prediction, where these predictions will be used as features for the model.
• Finally, Meta learners can now be used to make a prediction on test data in the stacking model.

​

*

61

Dr. S. M. Patil, Computer Engineering Department , SIGCE

Different ways to combine classifiers

Ensemble techniques related to stacking.

• Voting ensembles
• Weighted Average Ensemble
• Blending Ensemble
• Super Learner Ensemble

Voting ensembles:

• This is one of the simplest stacking ensemble methods, which uses different algorithms to prepare all members individually. Unlike the stacking method, the voting ensemble uses simple statistics instead of learning how to best combine predictions from base models separately.
• It is significant to solve regression problems where we need to predict the mean or median of the predictions from base models.
• Further, it is also helpful in various classification problems according to the total votes received for prediction. The label with the higher numbers of votes is referred to as hard voting, whereas the label that receives the largest sums of probability or lesser votes is referred to as soft voting.

​

​

*

62

Dr. S. M. Patil, Computer Engineering Department , SIGCE

Different ways to combine classifiers

• The voting ensemble differs from than stacking ensemble in terms of weighing models based on each member's performance because here, all models are considered to have the same skill levels.
• Member Assessment: In the voting ensemble, all members are assumed to have the same skill sets.
• Combine with Model: Instead of using combined prediction from each member, it uses simple statistics to get the final prediction, e.g., mean or median.

​

​

*

63

Dr. S. M. Patil, Computer Engineering Department , SIGCE

Different ways to combine classifiers

Weighted Average Ensemble

• The weighted average ensemble is considered the next level of the voting ensemble, which uses a diverse collection of model types as contributing members.
• This method uses some training datasets to find the average weight of each ensemble member based on their performance. An improvement over this naive approach is to weigh each member based on its performance on a hold-out dataset, such as a validation set or out-of-fold predictions during k-fold cross-validation. Furthermore, it may also involve tuning the coefficient weightings for each model using an optimization algorithm and performance on a holdout dataset.
• Member Assessment: Weighted average ensemble method uses member performance based on the training dataset.
• Combine With Model: It considers the weighted average of prediction from each member separately.

​

​

*

64

Dr. S. M. Patil, Computer Engineering Department , SIGCE

Different ways to combine classifiers

Blending Ensemble:

• Blending is a similar approach to stacking with a specific configuration.
• It is considered a stacking method that uses k-fold cross-validation to prepare out-of-sample predictions for the meta-model.
• In this method, the training dataset is first to split into different training sets and validation sets then we train learner models on the training sets. Further, predictions are made on the validation set and sample set, where validation predictions are used as features to build a new model, which is later used to make final predictions on the test set using the prediction values as features.
• Member Predictions: The blending stacking ensemble uses out-of-sample predictions on a validation set.
• Combine With Model: Linear model (e.g., linear regression or logistic regression).

�

​

​

*

65

Dr. S. M. Patil, Computer Engineering Department , SIGCE

Different ways to combine classifiers

Super Learner Ensemble:

• This method is quite similar to blending, which has a specific configuration of a stacking ensemble. It uses out-of-fold predictions from learner models and prepares a meta-model.
• However, it is considered a modified form of blending, which only differs in the selection of how out-of-sample predictions are prepared for the meta learner.

​

�

​

​

*

66

Dr. S. M. Patil, Computer Engineering Department , SIGCE