Logistic Regression

Learning Objectives

• ✓Explain why a linear model cannot be used directly for classification
• ✓Read the sigmoid and adjust the decision threshold to the problem
• ✓Know why the loss must be cross-entropy rather than MSE
• ✓Handle multiclass classification with Softmax or One-vs-Rest
• ✓Evaluate with ROC-AUC and PR curves, and handle class imbalance

From Linear Regression to Logistic

Applying linear regression directly to classification breaks in two ways: the output runs outside $[0, 1]$ so it cannot be read as a probability, and a few extreme points are enough to rotate the whole line. Logistic Regression keeps the linear part $z = w^{\top}x + b$ but passes it through a sigmoid before returning a result, so the output always lands in $(0, 1)$ and reads as $P(y = 1 \mid x)$.

The Sigmoid & the Decision Threshold

The sigmoid is S-shaped: $z \to +\infty$ gives $\sigma \to 1$, $z \to -\infty$ gives $\sigma \to 0$, and $z = 0$ gives exactly 0.5. It is steepest in the middle and nearly flat at both ends — that detail is the origin of the vanishing gradient. The 0.5 threshold is only a default: lowering it raises Recall and lowers Precision, raising it does the reverse; choose it from the real cost of each kind of error.

The Loss — Binary Cross-Entropy

MSE does not work here: paired with a sigmoid, the loss surface loses convexity and picks up multiple local minima. Binary Cross-Entropy fixes this completely — the function is convex, it follows from the maximum likelihood principle, and the gradient reduces to exactly $\hat{y} - y$. The penalty climbs steeply with confident mistakes: predicting 0.02 for a sample labeled 1 costs $-\log(0.02) \approx 3.9$, while a hesitant 0.5 costs only 0.69.

Multiclass Classification: Softmax & One-vs-Rest

With $K$ classes, Softmax takes the place of the sigmoid: exponentiate every score then normalize by the sum, producing a probability distribution that adds to 1; the exponential amplifies the gaps. One-vs-Rest trains $K$ binary classifiers and takes the most confident one. Softmax fits when the classes are mutually exclusive; One-vs-Rest is required when a single sample carries several labels at once.

Evaluating a Classifier

Beyond the metrics from session 2, two curves matter. ROC plots TPR against FPR across the full threshold range, and its AUC reads as the probability that the model scores a random positive above a random negative. The PR curve is more honest on heavily skewed data because it never looks at the number of negatives. Handle class imbalance in three steps: adjust the threshold first, then class_weight, and only then SMOTE — applied to the training set alone.

from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report, roc_auc_score

# class_weight='balanced' to compensate for the minority class
model = LogisticRegression(class_weight='balanced', max_iter=1000)
model.fit(X_train, y_train)
y_pred = model.predict(X_test)
y_proba = model.predict_proba(X_test)[:, 1]

print(classification_report(y_test, y_pred))
print("AUC:", roc_auc_score(y_test, y_proba))

# Change the default 0.5 threshold when a miss costs more than a false alarm:
y_pred_03 = (y_proba >= 0.3).astype(int)