An Overview of Ensemble Learning

Ensemble learning combines multiple weak learners to build a strong learner.

Simple techniques

Majority voting

Majority voting is generally used in classification tasks: take the majority of the model predictions as the final prediction.

Averaging

Take the average of predictions from all the models as the final prediction, in regression or classification (the average of probabilities) tasks.

Weighted average

Assign different weights defining the importance of each model for different models.

Advanced ensemble techniques

Stacking

Stacking (a.k.a. Stacked Generalization, or Super Learner) employs a number of first-layer individual learners (model 1-5 / Tier-1 in the below figures) generated from the training data set, followed by a second-level learner (model 6 / Tier-2, a.k.a. meta-learner).

Blending

Similar to stacking, but use only a devset from the training set to make predictions. The devset and the predictions are used to build the model on test set.

Bagging

Bagging(Bootstrap aggregating):

1. Bootstrapped subsampling;
2. Fit the base model on each of these subsets;
3. Models are run in parallel and independent of each other;
4. The final prediction are determined by combining all model predictions.

Random forest

Pros:

1. Robust against outliers and noise;
2. Reduce variance and typically avoids overfitting;
3. Fast run time;

Cons:

1. Can be slow to score as complexity increases;
2. Lack of transparency due to the complexity of multiple trees;

Boosting

Boosting is a sequential process, where each subsequent model attempts to correct the errors of the previous model.

1. Create a subset of all dataset.
2. Initially, all data points are given the same weights.
3. Fit a base model on this subset.
4. Use this base model to predict on the whole dataset.
5. Calculate errors using golden standard and predictions.
6. The wrongly predicted data are given higher weights.
7. Another model is created with step 3-6. (in order to correct the errors from the previous model)
8. Multiple models are created, each correcting the error of the previous model.
9. The final model (strong learner) is the weighted mean of all the models.

AdaBoost

Adaptive Boosting

Pros:

1. Often the best possible model;
2. Directly optimize the cost function;

Cons:

1. Not robust against outliers and noise;
2. Can overfit;
3. Need to find proper stopping point;

Comparing bagging and boosting

Model error arises from noise, bias, and variance.

• Noise is error by the target function;
• Bias is where the algorithm cannot learn the target;
• Variance comes from sampling.

Boosting is recommended on models that have a high bias, not Bagging.
Conversely, Bagging is recommend for cases of high variance, rather than Boosting.

GBM(Gradient Boosted Models)

XGBoost

XGBoost^[1]

Light GBM

It is useful for large-size dataset.

CatBoost

References

---

1. 1.Chen, T., & Guestrin, C. (2016). XGBoost: A Scalable Tree Boosting System. KDD. ↩
2. 2.Zhou Z. (2016) Ensemble learning ↩