Learn From Data

Learning Objectives

✓Explain why one linear boundary is not enough, through the XOR problem
✓Know MLP structure and the forward pass, and use matrix shapes to catch bugs
✓Choose an activation function and understand the nonlinearity it provides
✓Read the four backpropagation equations via the chain rule
✓Handle vanishing/exploding gradients with ReLU, He/Xavier initialization, Batch Norm

Why Neural Networks?

Logistic Regression can only draw one linear boundary, so it fails on XOR — four points no straight line can separate. This is a hard limit of the model family, not a shortage of data. The way out is to use several boundaries at once: each neuron is a miniature Logistic Regression $a = \sigma(w^{\top}x + b)$ , and the next layer learns how to combine them. Without a nonlinear activation, a hundred layers still amount to a single linear map.

W^{[2]}\big(W^{[1]}x + b^{[1]}\big) + b^{[2]} \;=\; \underbrace{W^{[2]}W^{[1]}}_{W'}\,x + \underbrace{W^{[2]}b^{[1]} + b^{[2]}}_{b'} \;=\; W'x + b'

Drop

\sigma

and two layers collapse into one — this is why a nonlinear activation is mandatory

MLP Structure & the Forward Pass

An MLP stacks layers, each repeating exactly two operations: a matrix product, then an activation. The Universal Approximation Theorem says one sufficiently wide hidden layer can approximate any continuous function, but "sufficiently wide" can be impractically large — a deep network reaches the same capability with far fewer parameters. Checking matrix shapes is debugging skill number one: $W^{[l]}$ has shape $(n^{[l]}, n^{[l-1]})$ . Remember to cache $(z^{[l]}, a^{[l]})$ for backprop.

\begin{aligned} z^{[l]} &= W^{[l]} a^{[l-1]} + b^{[l]} \\[4pt] a^{[l]} &= \sigma\big(z^{[l]}\big) \end{aligned}

Iterate from

l = 1

to

L

, with

a^{[0]} = x

and

\hat{y} = a^{[L]}

Forward pass — a [2, 8, 1] networkPython

import numpy as np

def forward(X, params):
    Z1 = params['W1'] @ X + params['b1']   # (8,2)@(2,m) = (8,m)
    A1 = np.tanh(Z1)                        # hidden: tanh
    Z2 = params['W2'] @ A1 + params['b2']  # (1,8)@(8,m) = (1,m)
    A2 = 1 / (1 + np.exp(-Z2))             # output: sigmoid
    return A2, (Z1, A1, Z2, A2)            # cache for backprop

# Write the shape beside each line — most hand-rolled network bugs are shape mismatches

Activation Functions

The sigmoid maps to $(0,1)$ so it suits a binary output layer, but placed in a hidden layer it causes vanishing gradients because $\sigma' \leq 0.25$ compounds across layers. Tanh is zero-centered and therefore steadier, though it still saturates at both ends. ReLU has a derivative of exactly 1 on the positive side so gradients do not shrink, it is cheap, and it is the default for hidden layers; its drawback is dying ReLU, fixed with Leaky ReLU. Softmax belongs only in a multiclass output layer.

\sigma(z) = \frac{1}{1 + e^{-z}} \qquad \tanh(z) = \frac{e^{z} - e^{-z}}{e^{z} + e^{-z}} \qquad \mathrm{ReLU}(z) = \max(0, z)

Derivatives:

\sigma' = \sigma(1-\sigma) \leq 0.25

·

\tanh' = 1 - \tanh^2

·

\mathrm{ReLU}' = 1

for

z>0

,

0

for

z<0

Loss Functions & Gradient Descent

The loss has to match the output form: MSE for regression, Binary Cross-Entropy with Sigmoid, Categorical Cross-Entropy with Softmax. These pairings are not arbitrary — matching them correctly simplifies the gradient and makes training markedly more stable. The learning rate $\eta$ is the hyperparameter most likely to ruin a run: too small converges slowly, too large blows the weights up to NaN; the usual range is $[0.0001,\, 0.1]$ . Mini-batches of 32–256 are the practical standard.

\theta \leftarrow \theta - \eta \, \nabla_{\theta} L

Every

W

and

b

in every layer is updated at once, after backprop has finished computing the gradients

Advanced Optimizers

Plain SGD oscillates badly when the loss surface is a long narrow valley. Momentum accumulates inertia, so a consistent direction accelerates while a direction that keeps flipping sign cancels itself out. RMSprop gives each parameter its own learning rate, based on a moving average of squared gradients. Adam combines both and is a reasonable default because it works well straight out of the box, though it sometimes generalizes worse than SGD with momentum.

\begin{aligned} \textbf{Momentum:}\quad v &\leftarrow \beta v + (1-\beta)\nabla L, & \theta &\leftarrow \theta - \eta \, v \\[6pt] \textbf{RMSprop:}\quad s &\leftarrow \beta s + (1-\beta)(\nabla L)^{2}, & \theta &\leftarrow \theta - \eta \, \frac{\nabla L}{\sqrt{s + \epsilon}} \end{aligned}

Momentum handles the direction, RMSprop handles the per-parameter step size — Adam is the sum of the two ideas

Backpropagation — the 4 Core Equations

Updating $W^{[l]}$ requires $\partial L / \partial W^{[l]}$ , but a whole chain of operations sits between the two; the Chain Rule lets you walk that chain backwards one link at a time. Define the error signal $\delta^{[l]} = \partial L / \partial z^{[l]}$ and the procedure condenses into four equations: BP1 starts at the output layer, BP2 pushes the error back to the previous layer, and BP3 and BP4 turn the error into gradients for $W$ and $b$ . As a result, one forward pass and one backward pass suffice.

\begin{aligned} \textbf{(BP1)} && \delta^{[L]} &= \nabla_{a} L \odot \sigma'\big(z^{[L]}\big) && \text{error at the output} \\[6pt] \textbf{(BP2)} && \delta^{[l]} &= \big(W^{[l+1]}\big)^{\top} \delta^{[l+1]} \odot \sigma'\big(z^{[l]}\big) && \text{propagate backward} \\[6pt] \textbf{(BP3)} && \frac{\partial L}{\partial W^{[l]}} &= \delta^{[l]} \big(a^{[l-1]}\big)^{\top} && \text{gradient of } W \\[6pt] \textbf{(BP4)} && \frac{\partial L}{\partial b^{[l]}} &= \delta^{[l]} && \text{gradient of } b \end{aligned}

\odot

= elementwise (Hadamard) product. BP2 is the heart of the algorithm: each backward step multiplies in another

(W^{[l+1]})^{\top}

and

\sigma'(z^{[l]})

Backprop — the 4 equations (NumPy)Python

def backward(X, y, params, cache):
    m = X.shape[1]
    Z1, A1, Z2, A2 = cache

    dZ2 = A2 - y                              # BP1: BCE + Sigmoid reduces to this
    dW2 = (1/m) * dZ2 @ A1.T                  # BP3
    db2 = (1/m) * np.sum(dZ2, axis=1, keepdims=True)  # BP4

    dZ1 = params['W2'].T @ dZ2 * (1 - A1**2)  # BP2: tanh derivative
    dW1 = (1/m) * dZ1 @ X.T                    # BP3
    db1 = (1/m) * np.sum(dZ1, axis=1, keepdims=True)  # BP4

    return {'dW1':dW1, 'db1':db1, 'dW2':dW2, 'db2':db2}

Vanishing Gradients, Initialization & Batch Norm

Because $\sigma'(z) \leq 0.25$ and backprop multiplies through every layer, after $L$ layers the gradient is down to roughly $(0.25)^{L}$ — past 10 layers that is essentially 0, leaving the early layers frozen. Fixes in order of effectiveness: ReLU, Batch Normalization, residual connections, and correct initialization (Xavier for Sigmoid/Tanh, He for ReLU). The opposite failure is exploding gradients, handled with Gradient Clipping. Never initialize the weights to 0.

\textbf{Xavier/Glorot:}\;\; W \sim \mathcal{N}\!\left(0,\, \frac{2}{n^{[l-1]} + n^{[l]}}\right) \qquad\qquad \textbf{He:}\;\; W \sim \mathcal{N}\!\left(0,\, \frac{2}{n^{[l-1]}}\right)

Xavier for Sigmoid/Tanh · He for ReLU — keeps signal variance stable across layers from the very first epoch

He initialization + BatchNorm (Keras)Python

from tensorflow.keras.layers import Dense, BatchNormalization

model.add(Dense(64, kernel_initializer='he_normal', use_bias=False))
model.add(BatchNormalization())          # place before the activation
model.add(keras.layers.Activation('relu'))

# Practical routine:
#   (1) Build a network large enough to overfit the training set — proves capacity
#   (2) Only then add L2, Dropout and Early Stopping to pull it back