Deep Learning Foundations & CNNs

Learning Objectives

• ✓Explain how deep learning differs from feature-engineered machine learning
• ✓Compute a neuron forward pass and describe why nonlinear activation is necessary
• ✓Connect loss, backpropagation, gradient descent, and the learning rate
• ✓Calculate the output size of a convolution from kernel, padding, stride, and dilation
• ✓Build a small LeNet-style CNN and track every tensor shape

From Machine Learning to Representation Learning

A conventional machine-learning pipeline often asks a person to design features before classification. Deep learning learns the representation and the decision rule together: early layers capture simple patterns, intermediate layers combine them, and deep layers form task-specific concepts. The value is not merely "more layers"; it is the ability to learn semantic representations, synthesize data, and transfer knowledge from a large source task to a smaller target task. Generalization still has to be measured on unseen data from the same intended distribution, with train data used to fit parameters, validation data to choose configurations, and test data reserved for the final estimate.

The Neuron: Aggregate, Transform, Predict

A neuron first aggregates its inputs into $z = w^{\top}x + b$, then transforms that value with an activation $a = g(z)$. The weights and bias are learnable; the activation is chosen. Without a nonlinear activation, stacking linear layers collapses into one linear map and cannot express a nonlinear boundary. Sigmoid maps to $(0,1)$ and is useful for a binary output, while ReLU is the practical default in hidden layers because it is simple and does not saturate on positive inputs. Softmax converts a vector of class scores into a probability distribution.

Learning: Loss, Backpropagation & Gradient Descent

The loss measures the discrepancy between prediction and target. Backpropagation applies the chain rule from the loss back through every layer, producing one partial derivative for each parameter. Gradient descent then moves parameters in the opposite direction of steepest increase. Batch gradient descent averages the full dataset and is stable but expensive; stochastic gradient descent updates from one example and is noisy; mini-batch training is the usual compromise. A learning rate that is too small wastes computation, while one that is too large can oscillate or diverge.

# PyTorch: forward -> loss -> backward -> update
optimizer.zero_grad()
logits = model(images)
loss = criterion(logits, labels)
loss.backward()
optimizer.step()

Why Images Need Convolution

Flattening an image for a fully connected network destroys the explicit two-dimensional neighborhood and creates a separate weight for every input-output pair. A convolution instead connects a neuron to a local patch and reuses the same kernel at every spatial position. This gives two important inductive biases: nearby pixels interact locally, and a learned pattern can be recognized wherever it appears. Multiple kernels create multiple activation maps; deeper layers combine edges and textures into parts and objects.

Kernel, Padding, Stride & Dilation

The kernel size controls the local window; odd sizes such as $3\times3$ are symmetric around a center pixel. Padding adds border pixels and can preserve resolution. Stride controls how far the kernel moves and therefore downsamples when it is greater than one. Dilation inserts gaps inside a kernel, expanding the receptive field without increasing the number of learned weights. For one spatial dimension, use the formula below and apply the same calculation to height and width. Always check that the result is an integer and verify it against the framework tensor shape.

Pooling, Channels & Feature Hierarchies

Pooling reduces spatial resolution and computation. Max pooling keeps the strongest response in each window; average pooling keeps the local mean. A convolutional layer does not produce just one map: each learned filter produces one output channel, so an RGB input can become dozens of activation maps. Early filters often respond to oriented edges or color contrasts, middle filters to textures and parts, and later filters to class-relevant structures. After feature extraction, flattening or global pooling feeds a classifier head.

LeNet as a Shape-Tracking Exercise

LeNet is small enough to understand end to end: convolution extracts local patterns, pooling downsamples, another convolution increases the feature depth, and dense layers perform classification. For MNIST, the final layer has ten logits. The most useful study habit is to write the tensor shape after every layer before running the model; this catches most architecture errors immediately.

from tensorflow import keras
from tensorflow.keras import layers

model = keras.Sequential([
    layers.Input(shape=(28, 28, 1)),
    layers.Conv2D(6, 5, activation='relu'),
    layers.AveragePooling2D(),
    layers.Conv2D(16, 5, activation='relu'),
    layers.AveragePooling2D(),
    layers.Flatten(),
    layers.Dense(120, activation='relu'),
    layers.Dense(84, activation='relu'),
    layers.Dense(10),  # logits; use from_logits=True in the loss
])

model.summary()