Mathematical Foundations for AI

Learning Objectives

• ✓Manipulate vectors and matrices; understand the role of eigenvalues in PCA and SVD
• ✓Read gradients and the three gradient descent variants
• ✓Apply Bayes theorem; read covariance and the correlation coefficient
• ✓Compute entropy and Information Gain; read the result of a hypothesis test
• ✓Distinguish cost functions, convexity, and the two kinds of L1/L2 regularization

Linear Algebra: Vectors & Matrices

A vector $x \in \mathbb{R}^{n}$ is one data point; a matrix $X \in \mathbb{R}^{m \times n}$ is $m$ points stacked into a table. Four operations to know cold: addition, scalar multiplication, the dot product $x \cdot y$ (measures similarity) and the norm $\|x\|_2$ (measures length). A $28 \times 28$ MNIST image flattens into a $784$-dimensional vector; the product $Wx$ is exactly one forward pass.

import numpy as np

# 3x3 grayscale image → matrix
image = np.array([[210,  45,  90],
                  [ 15, 175, 240],
                  [130,  60,  25]])

# Flatten the image into a vector, then multiply by a weight matrix
W = np.random.randn(4, 9)   # 4 neurons, 9 inputs
x = image.flatten()          # shape (9,)
output = W @ x               # matrix-vector product → shape (4,)

Eigenvalues, PCA & SVD

An eigenvector $v$ satisfying $Av = \lambda v$ is a direction that does not rotate; the eigenvalue $\lambda$ measures how far the data spreads along it. PCA ranks the $\lambda$ values and keeps the leading directions, reducing dimensionality with little loss of information; SVD generalizes this through $A = U\Sigma V^\top$, used in compression, LSA and recommender systems. Standardize first — otherwise a column with a large scale takes over the first component.

from sklearn.decomposition import PCA
import numpy as np

X = np.array([[3.2, 3.4], [1.1, 1.5], [2.8, 2.6],
              [4.1, 4.4], [1.8, 2.1], [3.6, 3.2]])

pca = PCA(n_components=1)
X_reduced = pca.fit_transform(X)
print(pca.explained_variance_ratio_)
# [0.980] → 1 component retains 98% of the variance

Calculus: Derivatives & Gradient Descent

The partial derivative $\partial L / \partial w_i$ measures the effect of $w_i$ alone while the other variables stay fixed; the gradient $\nabla$ collects all of them. Gradient descent repeatedly shifts the weights against the gradient. The three variants differ in how many samples each step uses: Batch uses all of them (stable, slow), Stochastic uses one (fast, noisy), and Mini-batch uses 32–256 and is the practical default.

# Loss L(w) = (w − 4)², minimum at w* = 4
# Derivative: dL/dw = 2(w − 4)

w = 0                     # initial weight
lr = 0.25                 # learning rate
gradient = 2 * (w - 4)    # = -8 at w=0
w_new = w - lr * gradient # = 0 - 0.25*(-8) = 2.0
# → half the distance to w* = 4 covered

Probability & Bayes Theorem

A probability distribution assigns confidence to each outcome of a random variable; Normal and Binomial come up most often. Bayes theorem reverses the conditioning, turning a quantity that is easy to count into the one you actually need. With $P(\text{Spam}) = 0.40$, $P(\text{"free"} \mid \text{Spam}) = 0.90$, $P(\text{"free"}) = 0.48$, you get $P(\text{Spam} \mid \text{"free"}) = 0.75$ — this is exactly the mechanism behind Naïve Bayes.

# Bayes theorem applied directly
P_spam_given_free = (0.90 * 0.40) / 0.48   # = 0.75

# Implementation with scikit-learn
from sklearn.naive_bayes import MultinomialNB
model = MultinomialNB()
model.fit(X_train_counts, y_train)   # X = word occurrence counts
predictions = model.predict(X_test_counts)

Statistics: Descriptives, Covariance & Correlation

Descriptive statistics summarize a numeric column through center (mean, median, mode), spread (variance, standard deviation) and shape (skewness, kurtosis). Covariance tells you whether two variables move together or in opposite directions, but its magnitude depends on the units. The Pearson coefficient $\rho$ divides by both standard deviations to cancel the units: $|r| > 0.9$ is very strong, and a p-value $< 0.05$ confirms the result is not chance.

import numpy as np
import scipy.stats as stats

X = np.array([1, 3, 4, 6, 7, 9])            # hours studied
Y = np.array([48, 52, 66, 68, 83, 86])      # exam score

cov = np.cov(X, Y)[0, 1]                    # = 43.2 → same direction
r, p_value = stats.pearsonr(X, Y)
print(f'r = {r:.3f}')       # r = 0.960 → strong correlation
print(f'p = {p_value:.5f}') # p = 0.00234 → statistically significant

Entropy & Information Theory

Entropy measures uncertainty: it is 0 when only one outcome remains and peaks at $\log_2 n$ when every outcome is equally likely (4 choices → 2.0 bits maximum). Information Gain is the drop in entropy after a split, and it is exactly the criterion a decision tree applies at each node. Child entropies must be averaged with weights proportional to branch size.

import numpy as np

def entropy(probs):
    return -sum(p * np.log2(p) for p in probs if p > 0)

# Root node: 20 emails, 10 Spam / 10 Not-Spam
H_root = entropy([0.5, 0.5])              # = 1.0 bit (maximum uncertainty)

# After splitting on the keyword 'unsubscribe':
# Left (12 emails: 10 Spam, 2 Not) and Right (8 emails: 0 Spam, 8 Not)
H_child = 12/20 * entropy([10/12, 2/12]) + 8/20 * entropy([0, 1])  # = 0.390
info_gain = H_root - H_child              # = 0.610 bit → a very good split

Hypothesis Testing

Because proving the alternative hypothesis $H_1$ directly is hard, the standard procedure is to look for evidence against the null hypothesis $H_0$. The p-value answers: if $H_0$ were true, how likely is a difference at least as large as the one observed? Below $\alpha$ (usually 0.05), reject $H_0$. Three commonly used tests: Pearson correlation, the T-test for two groups, and the F-test (ANOVA) for three or more.

from scipy import stats

# Accuracy over 8 independent runs
A = [0.79, 0.81, 0.77, 0.84, 0.75, 0.82, 0.73, 0.80]   # mean 0.789
B = [0.74, 0.76, 0.71, 0.79, 0.72, 0.75, 0.70, 0.78]   # mean 0.744

t_stat, p_value = stats.ttest_ind(A, B)
print(f'p-value = {p_value:.4f}')   # p = 0.0213

# p = 0.0213 < 0.05 → Reject H0: the difference is statistically significant
# F-Test (≥3 groups):  stats.f_oneway(x, y, z)
# One-sample T-Test:   stats.ttest_1samp(A, 0.75)

Optimization: Cost Functions, Convexity & Regularization

The cost function is the quantity the algorithm tries to minimize: MSE for regression, Cross-Entropy for classification. On a convex function every local minimum is also the global one, so wherever gradient descent stops is the best solution. Regularization adds a penalty on coefficient magnitude: L2 adds $\sum w_j^2$ and shrinks evenly, L1 adds $\sum |w_j|$ and drives some coefficients all the way to 0.

import numpy as np

# MSE (Regression)
y_true = np.array([4.0, 6.0, 8.0])
y_pred = np.array([4.3, 5.6, 8.5])
mse = np.mean((y_true - y_pred) ** 2)   # = 0.167 (smaller = better)

# Binary Cross-Entropy (Classification), true label y = 1
# pred=0.95 (confident, correct) → L = -log(0.95) = 0.051  (small)
# pred=0.20 (wrong)              → L = -log(0.20) = 1.609
# pred=0.02 (confident, wrong)   → L = -log(0.02) = 3.912  (very large)