Ensemble methods is a class of machine learning models that combines the prediction of multiple models to improve overall performance and generalisation.

It relies on the assumption that decision-making of a larger group of models is better than one model.

They achieve better performance by combining different individual models “weak learners” to produce a stronger learner (bagging or boosting). Aggregating the models together, can average individual mistakes to reduce the risk of over-fitting, while maintaining strong prediction performance.

Bagging involves training multiple instances of the same learning algorithm on different subsets of the training data. Each model is trained independently, often in parallel.

1. Bootstrapping: Bootstrapping is a random sampling method with replacement to generate different subsets of the training set.
2. Parallel Training: The bootstrap samples are then trained independently and in parallel with each other using weak or base learners.
3. Aggregate Results: An average (regression) or majority of the prediction (classification) are taken to compute a more accurate estimates.

The subsets are typically created by random sampling with replacement, a technique known as bootstrapping. This means that some data points may be repeated in a subset, while others may be omitted.

• Diversity of Models: The diverse training sets lead to the creation of diverse models, capturing different aspects of the underlying data distribution.
• Overfitting Mitigation: Model independence helps mitigate overfitting because the individual models are less likely to learn the same noise in the training data.
• Stability and Reliability: If one model makes a prediction error on a specific subset of the data, other models may compensate for it, leading to a more reliable overall prediction.

1. Tree-Based ML algo that leverage the power of multiple decision trees from making decisions. Hence, ‘forest’ of trees.
2. ‘random’ because it is a forest of randomly created decision trees due to bootstrapping.

1. Bootstrap sampling: A random sample is drawn from the original dataset with replacement.
2. Parallel Training: Construct a decision tree for every random subset and perform the training process concurrently.
3. Aggregate: An average (regression) or majority of the prediction (classification) are taken to compute a more accurate estimates.

Random forest deploys bootstrapped sampling and this means each tree in the forest is trained on different randomised subset of datapoints and features. This diversification introduces variability in the training sets for individual trees and thus reduces the risk of overfitting.

Boosting is an ensemble learning method that combines the predictions of multiple weak learners sequentially (typically simple models) to create a strong learner.

The key difference is that bagging algorithms method usually performs parallel training. Whereas, boosting algorithms perform sequential training such that each subsequent model corrects the errors of its predecessor.

1. Initialisation: Boosting starts by training a weak learner on the entire dataset.
2. Instance Weighting: After the initial model is trained, each instance in the dataset is assigned an initial weight. Misclassified instances are given higher weights, while correctly classified instances receive lower weights.
3. Sequential Training & Optimisation: Boosting trains a series of weak learners sequentially. Each subsequent learner focuses on the instances that the previous ones misclassified.
4. Weighted Aggregation: At each step, the weak learner’s prediction is combined with the predictions of the previous models, and the instance weights are adjusted based on the model’s performance. Correctly classified instances have their weights reduced, while misclassified instances have their weights increased.

Boosting reduces bias by iteratively emphasising the correction of errors made by the previous models in the ensemble. The key mechanism that contributes to bias reduction in boosting is the adaptive learning process, which assigns higher weights to instances that are difficult to classify correctly.