Session 03📈

Linear Regression

Simple and multiple linear models: MSE, the Normal Equation, gradient descent, checking assumptions, evaluating with R², and the polynomial extension.

Learning Objectives

✓Build simple and multiple linear regression models
✓Know when to use the Normal Equation and when to switch to Gradient Descent
✓Check the four assumptions, detect multicollinearity and read a residual plot
✓Read R² and RSE alongside the MAE/MSE/RMSE metrics
✓Extend to polynomial regression while keeping overfitting under control

The Model & the MSE Loss

Regression predicts a continuous numeric value; the univariate model $\hat{y} = wx + b$ fits a straight line through the cloud of points. Least squares minimizes the sum of squared residuals — squaring removes the sign, penalizes large errors heavily, and keeps the function differentiable everywhere. The trade-off is that MSE is very sensitive to outliers. Univariate regression has a closed-form solution for $w$ and $b$ .

J(w, b) = \frac{1}{m}\sum_{i=1}^{m}\left(\hat{y}_i - y_i\right)^2, \qquad \hat{y}_i = w x_i + b

MSE — the mean squared gap between the prediction

\hat{y}

and the true value

y

Multiple Regression & the Normal Equation

With many features the model compresses to $\hat{y} = Xw$ . The Normal Equation gives the optimal solution in a single computation: no iteration, no learning rate to choose. The limitation is inverting $X^{\top}X$ at $O(n^3)$ cost, which becomes infeasible past roughly 10,000 features; on top of that, if two columns carry nearly the same information then $X^{\top}X$ is singular and the solution destabilizes.

w = (X^{\top} X)^{-1} X^{\top} y

The analytic solution —

O(n^3)

cost from having to invert

X^{\top}X

Gradient Descent

Gradient Descent solves the same problem iteratively, moving the weights against the gradient with step size $\alpha$ . Advantages: no matrix inversion, so it scales to very large feature counts, and it applies to models with no closed form such as logistic regression or neural networks. SGD updates on a single sample; Mini-batch uses 32–256 and exploits the GPU. Standardize first, or the algorithm will oscillate.

w := w - \alpha \, \nabla J(w)

\alpha

too small → slow convergence;

\alpha

too large → oscillation, no convergence

Linear regression via Gradient Descent (NumPy)Python

import numpy as np

def linear_regression_gd(X, y, lr=0.01, epochs=1000):
    m, n = X.shape
    w = np.zeros(n)
    b = 0.0
    for _ in range(epochs):
        y_pred = X @ w + b
        error = y_pred - y
        w -= lr * (2 / m) * (X.T @ error)
        b -= lr * (2 / m) * np.sum(error)
    return w, b

# X must be standardized first, or the loss will diverge to nan

Assumptions & Model Diagnostics

Four assumptions to check: a linear relationship, independent residuals, constant residual variance, and residuals that are approximately normal. The cheapest tool is a plot of residuals against fitted values — randomly scattered around 0 passes, a U shape means the relationship is nonlinear, a funnel shape means the variance is not constant. Multicollinearity makes coefficients unstable and can flip their sign; detect it with VIF, and above 10 consider dropping a column.

Model Evaluation

$R^2$ says how much of the variance in $y$ the model explains compared to a model that always predicts $\bar{y}$ . Remember that $R^2$ never decreases when you add a feature, even a meaningless one, so comparing models with different column counts requires adjusted $R^2$ . Report MAE (easy to interpret) and RMSE (same units as $y$ ) alongside it — an RMSE much larger than the MAE is a sign of a handful of very large errors.

R^2 = 1 - \frac{\mathrm{RSS}}{\mathrm{TSS}} = 1 - \frac{\sum_i (y_i - \hat{y}_i)^2}{\sum_i (y_i - \bar{y})^2}

R^2 = 1

: perfect fit ·

R^2 = 0

: no better than always predicting

\bar{y}

Regression metrics in scikit-learnPython

from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error, mean_absolute_error, r2_score

model = LinearRegression().fit(X_train, y_train)
y_pred = model.predict(X_test)

print("R2  :", r2_score(y_test, y_pred))
print("MAE :", mean_absolute_error(y_test, y_pred))
print("RMSE:", mean_squared_error(y_test, y_pred, squared=False))

Extension: Polynomial Regression

When the real relationship is a curve, add the powers $x^2, x^3, \dots$ to the feature table and run linear regression on the new set of columns. The model is still linear in the parameters $w$ — that is what "linear" actually means — and only nonlinear in $x$ . The higher the degree the more the curve bends to touch each point, so choose the degree by cross-validation rather than by $R^2$ on the training set, which always favors the highest degree.

\hat{y} = w_0 + w_1 x + w_2 x^{2} + \cdots + w_d x^{d}

The larger the degree

d

→ the more the curve wiggles → the easier it overfits

Choosing the polynomial degree by cross-validationPython

from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import cross_val_score

for d in [1, 2, 3, 5, 9]:
    pipe = make_pipeline(PolynomialFeatures(degree=d), LinearRegression())
    cv = cross_val_score(pipe, X, y, cv=5, scoring='r2')
    print(f'degree {d}: CV R2 = {cv.mean():.3f}')
# Pick the degree with the highest CV R2, not the highest train-set R2

←PreviousML, DL & GenAI Overview →NextLogistic Regression