Ensemble Methods

Learning Objectives

• ✓Understand Bootstrap and using the Out-of-Bag set as a free test set
• ✓Understand why Random Forest must restrict the features at each split
• ✓Know the four steps of AdaBoost and its weakness on noisy data
• ✓Understand Gradient Boosting as gradient descent in function space
• ✓Know the three XGBoost improvements and when to reach for which model

The Ensemble Idea

Many weak models voting together beat any one of them, but only if they make mistakes in different places — train them identically and they are right and wrong together, which makes the vote meaningless. Three families: Bagging trains in parallel then averages, so it cuts Variance; Boosting trains sequentially with each model correcting the previous one, so it cuts Bias; Stacking uses a meta-learner to combine models of different kinds.

Bootstrap & Bagging

Bootstrap samples with replacement, producing a new set of the same size $n$ but with duplicates and omissions. The probability a given sample is never drawn converges to $e^{-1} \approx 0.368$, so each set holds about 63.2% of the original data while the remaining 36.8% forms the Out-of-Bag set — a free test set. Bagging's limit is the correlation between trees: when one feature dominates, every tree picks it at the root, so variance only falls to a floor.

from sklearn.ensemble import BaggingClassifier
from sklearn.tree import DecisionTreeClassifier

# DEEP trees as base learners: low bias, high variance
bag = BaggingClassifier(DecisionTreeClassifier(),
                        n_estimators=200, oob_score=True,
                        n_jobs=-1, random_state=42)
bag.fit(X_train, y_train)

print(f'OOB : {bag.oob_score_:.3f}')             # free test-error estimate
print(f'Test: {bag.score(X_test, y_test):.3f}')  # usually very close to OOB

Random Forest

Random Forest adds exactly one mechanism to Bagging: each split considers only $m$ randomly drawn features, with $m = \sqrt{p}$ for classification and $m = p/3$ for regression. Deliberately hiding information makes each tree weaker but reduces the correlation $\rho$, so the ensemble comes out stronger. Feature importance comes in two forms: MDI is fast but biased toward high-cardinality features, while Permutation Importance shuffles each column on the test set and so carries no such bias.

from sklearn.ensemble import RandomForestClassifier
from sklearn.inspection import permutation_importance

rf = RandomForestClassifier(
    n_estimators=300,
    max_features='sqrt',   # each node considers only √p features
    oob_score=True, n_jobs=-1, random_state=42
).fit(X_train, y_train)

print("Test:", rf.score(X_test, y_test), "| OOB:", rf.oob_score_)

# More trustworthy than the built-in feature_importances_
# (which is biased toward high cardinality)
result = permutation_importance(rf, X_test, y_test,
                                n_repeats=10, random_state=42)

Boosting & AdaBoost

Boosting uses very shallow base learners — usually stumps exactly one level deep — chained one after another. AdaBoost repeats four steps: initialize $w_i = 1/n$; train a stump and compute its weighted error $\varepsilon_t$; compute the model weight $\alpha_t$ from its accuracy; raise the weight of misclassified samples, lower the weight of correct ones, then renormalize. Its weakness sits inside that mechanism: a mislabeled sample keeps gaining weight, driving the whole system to pour its effort into noise.

from sklearn.ensemble import AdaBoostClassifier
from sklearn.tree import DecisionTreeClassifier

ada = AdaBoostClassifier(
    estimator=DecisionTreeClassifier(max_depth=1),  # stump: exactly 1 question
    n_estimators=200,
    learning_rate=0.05,
    random_state=42
).fit(X_train, y_train)

print("Test:", ada.score(X_test, y_test))
# Noisy data or mislabeled data → prefer Gradient Boosting instead

Gradient Boosting

Gradient Boosting replaces the reweighting mechanism with a more general idea: train each new model directly on the pseudo-residual, that is the negative gradient of the loss. Under MSE this quantity reduces to the residual $y - F$, which is what lets the method apply to any differentiable loss. Each new tree is one gradient descent step in function space. Because it aims straight at the remaining error it overshoots easily: it needs a small shrinkage $\eta$, subsampling and early stopping.

from sklearn.ensemble import GradientBoostingClassifier

# F0 = starting value → fit a tree to the residual → Fm = Fm-1 + η·hm
gb = GradientBoostingClassifier(
    n_estimators=500,
    learning_rate=0.05,    # a small η plus many trees works better
    max_depth=3,           # SHALLOW trees, the opposite of Random Forest
    subsample=0.8,         # stochastic GB, adds diversity
    n_iter_no_change=20, validation_fraction=0.15
).fit(X_train, y_train)

print("Trees actually used:", gb.n_estimators_)
print("Test:", gb.score(X_test, y_test))

XGBoost

XGBoost is a thoroughly optimized Gradient Boosting. Three differences: the complexity penalty $\Omega(f)$ sits inside the optimization objective, so both the number of leaves and the leaf values are charged; the algorithm approximates the loss with a second-order Taylor expansion, using the gradient $g_i$ and the hessian $h_i$ so it needs fewer rounds; and the engineering side includes parallel split finding and learning a default direction for missing values. The $\gamma$ parameter produces a pruning effect while the tree is still being built.

import xgboost as xgb

model = xgb.XGBClassifier(
    n_estimators=1000, max_depth=5, learning_rate=0.05,
    subsample=0.8, colsample_bytree=0.8,
    reg_lambda=1.0,   # L2 on the leaf values
    reg_alpha=0.0,    # L1
    gamma=0.1,        # minimum gain required to accept a split
    early_stopping_rounds=30, eval_metric='logloss'
)
model.fit(X_train, y_train, eval_set=[(X_val, y_val)], verbose=False)
print("Stopped at tree:", model.best_iteration)

Bias–Variance & Comparison

The two schools differ right at the depth of the base learner. Bagging and Random Forest use deep trees (low bias, high variance) then average, so adding trees is always safe. Boosting uses shallow trees (high bias, low variance) then corrects errors sequentially, which in exchange makes a small $\eta$ and early stopping mandatory. Choosing: Random Forest for a fast baseline, XGBoost/LightGBM for maximum accuracy, LightGBM past a million rows, and a single shallow tree when you need explainability.