Session 05🎯

Preprocessing, Regularization & Feature Selection

Missing data, scaling and encoding; the bias-variance trade-off; Ridge, Lasso, Elastic Net; the three families of feature selection.

Learning Objectives

✓Identify the missingness mechanism before choosing an imputation strategy
✓Know which models need scaling, and encode categorical variables by the right type
✓Read the U-shaped bias-variance curve to diagnose a model
✓Apply Ridge (L2), Lasso (L1) and Elastic Net
✓Select features with Filter, Wrapper or Embedded methods depending on the compute budget

Data Preprocessing

How you impute has to follow why the data is missing. MCAR is missing completely at random from a technical fault — the median or the mode is enough. MAR is missing according to a pattern living in another column, so imputing by group beats imputing globally. MNAR is when the missingness itself carries information; imputing the mean here is inventing data, and the right move is to add a flag column. Always fit the imputer on train, then transform test.

Imputing by missingness mechanismPython

from sklearn.impute import SimpleImputer, KNNImputer

# Numeric columns → median (more robust to outliers than the mean)
num = SimpleImputer(strategy="median")
X_num_train = num.fit_transform(X_num_train)   # fit ONLY on train
X_num_test  = num.transform(X_num_test)

# Categorical columns → most frequent value
cat = SimpleImputer(strategy="most_frequent")

# Or infer from nearby samples
X = KNNImputer(n_neighbors=7).fit_transform(X)

# MNAR: keep the "was missing" signal itself
df["income_NA"] = df["income"].isnull().astype(int)

Scaling & Encoding Categorical Variables

Features on different scales let the large-valued ones dominate distance and gradient computations, which affects kNN, SVM, Gradient Descent, Regularization and PCA; decision trees are immune because they only compare against a threshold. Three scaling options: Standardization as the default, Min-Max for images, Robust when outliers are present. For categorical variables: One-Hot for nominal, Ordinal when a real order exists, target encoding for high-cardinality columns — but computed within each fold.

Scaling & encoding in one pipelinePython

from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder, OrdinalEncoder

prep = ColumnTransformer([
    ("num",  StandardScaler(), ["age", "income", "score"]),
    # Color {Red, Green, Yellow}: no order → One-Hot
    ("nom",  OneHotEncoder(handle_unknown="ignore"), ["color"]),
    # Size {S, M, L}: HAS an order → preserve that order
    ("ord",  OrdinalEncoder(categories=[["S", "M", "L"]]), ["size"]),
])

# Keep it in a pipeline so every transform learns only from each fold's train split
from sklearn.pipeline import make_pipeline
pipe = make_pipeline(prep, model)

Overfitting & the Bias-Variance Trade-off

Expected error splits into three parts. Bias is systematic error from a model that is too crude, and it DECREASES as complexity rises; Variance is sensitivity to the particular dataset, and it INCREASES as complexity rises; noise $\sigma^2$ is a floor you cannot get under. Total error is therefore U-shaped. Diagnosis: poor on both train and test means high bias and more data will not help; good on train but poor on test means high variance, and there more data does help.

\mathbb{E}\big[(y - \hat{f}(x))^2\big] = \underbrace{\text{Bias}^2\big[\hat{f}(x)\big]}_{\text{too simple}} + \underbrace{\mathrm{Var}\big[\hat{f}(x)\big]}_{\text{too complex}} + \underbrace{\sigma^2}_{\text{noise}}

Only the first two terms are adjustable;

\sigma^2

is a hard limit of the problem

Approach 1 — Regularization (Ridge, Lasso, Elastic Net)

The signature of overfitting is coefficients that balloon and cancel each other out across correlated columns. Regularization adds a penalty on coefficient magnitude, forcing the model to pay for large coefficients. Ridge penalizes the square so it shrinks evenly and almost never reaches 0; Lasso penalizes the absolute value so it pushes many coefficients to exactly 0, doubling as variable selection, but it is unstable when two columns nearly coincide. Elastic Net combines both; always pick $\lambda$ by cross-validation.

\begin{aligned} J_{\text{Ridge}}(\theta) &= \text{MSE} + \lambda \sum_{j=1}^{n} \theta_j^{2} \\[6pt] J_{\text{Lasso}}(\theta) &= \text{MSE} + \lambda \sum_{j=1}^{n} \big|\theta_j\big| \end{aligned}

L2 shrinks coefficients evenly; L1 drives them to 0 and so doubles as variable selection

Ridge, Lasso & Elastic Net with cross-validationPython

from sklearn.linear_model import RidgeCV, LassoCV, ElasticNetCV

ridge = RidgeCV(alphas=[0.01, 0.1, 1, 10, 100], cv=5).fit(X_train, y_train)
lasso = LassoCV(alphas=[0.001, 0.01, 0.1, 1], cv=5).fit(X_train, y_train)
enet  = ElasticNetCV(l1_ratio=[0.1, 0.5, 0.9], cv=5).fit(X_train, y_train)

print("Ridge alpha:", ridge.alpha_)
print("Lasso alpha:", lasso.alpha_,
      "| features remaining:", (lasso.coef_ != 0).sum())
# Standardize first, or the penalty will unfairly punish
# columns that merely happen to be measured in small units

Approach 2 — Feature Selection

The three families differ in cost and accuracy. Filter methods score each column with a statistic and are very fast — Variance Threshold, Chi-square, ANOVA F-test, and Mutual Information, which catches nonlinear relationships too ( $y = x^2$ gives Pearson $\approx 0$ but $\text{MI} > 0$ ); the weakness is that each column is judged in isolation. Wrapper methods use model performance itself (RFE, RFECV, Boruta) — more accurate but expensive. Embedded methods select during training, via Lasso or Random Forest importance.

Mutual Information & RFEPython

from sklearn.feature_selection import SelectKBest, mutual_info_classif, RFECV

# Filter — Mutual Information (catches nonlinear relationships)
X_mi = SelectKBest(mutual_info_classif, k=20).fit_transform(X, y)

# Wrapper — drop columns one at a time, final count decided by CV
sel = RFECV(estimator, step=1, cv=5).fit(X, y)
print("Optimal number of features:", sel.n_features_)
print(sel.support_)   # keep/drop mask
print(sel.ranking_)   # 1 = selected

←PreviousLogistic Regression →NextDecision Trees