Training CNNs: MNIST, VGG & ResNet

Learning Objectives

• ✓Run the full load, preprocess, build, compile, fit, evaluate, and predict workflow
• ✓Choose a loss, optimizer, batch size, epoch budget, and evaluation metric deliberately
• ✓Diagnose underfitting and overfitting from training and validation curves
• ✓Explain L1, L2, dropout, data augmentation, and early stopping
• ✓Contrast VGG plain stacks with ResNet residual blocks

The Keras Training Contract

A training experiment has a fixed contract. First shape and normalize the data; then define a model whose output matches the label encoding; compile it with a loss, optimizer, and metrics; fit only on training data while monitoring validation data; evaluate once on the test split; finally inspect individual predictions rather than trusting a single aggregate score. For integer MNIST labels, ten logits pair naturally with sparse categorical cross-entropy. Divide pixel values by 255 so the optimizer sees a consistent $[0,1]$ input scale.

from tensorflow import keras

(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()
x_train = x_train.astype('float32') / 255.0
x_test = x_test.astype('float32') / 255.0
x_train = x_train[..., None]
x_test = x_test[..., None]

model.compile(
    optimizer='adam',
    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
    metrics=['accuracy'],
)
history = model.fit(
    x_train, y_train,
    validation_split=0.1,
    epochs=10,
    batch_size=64,
)
test_loss, test_accuracy = model.evaluate(x_test, y_test)

Dense Baseline vs. CNN

A dense MNIST baseline flattens each $28\times28$ image into 784 values, so it is a useful control but has no explicit spatial bias. A CNN keeps the image grid, shares its filters, and typically learns more efficiently. A fair comparison uses the same train/test split, preprocessing, epoch budget, and evaluation metric. Compare validation curves as well as final test accuracy; otherwise a larger model or a longer run can make the conclusion misleading.

dense_model = keras.Sequential([
    keras.layers.Input(shape=(28, 28)),
    keras.layers.Flatten(),
    keras.layers.Dense(256, activation='relu'),
    keras.layers.Dense(128, activation='relu'),
    keras.layers.Dense(10),
])

cnn_model = keras.Sequential([
    keras.layers.Input(shape=(28, 28, 1)),
    keras.layers.Conv2D(32, 3, activation='relu'),
    keras.layers.MaxPooling2D(),
    keras.layers.Conv2D(64, 3, activation='relu'),
    keras.layers.MaxPooling2D(),
    keras.layers.Flatten(),
    keras.layers.Dense(128, activation='relu'),
    keras.layers.Dense(10),
])

Loss, Optimizer, Batch Size & Epochs

The loss defines what "wrong" means; the optimizer defines the update rule. SGD is a strong, controllable baseline, Adam adapts step sizes and often works quickly, AdamW decouples weight decay, and RMSprop is historically common for recurrent models. Batch size controls how many samples estimate one gradient: larger batches are efficient but consume memory and can hide useful noise. An epoch is one pass over the training set, not one update. Set a generous maximum and let validation behavior decide when to stop.

Callbacks Turn Training into a Controlled Experiment

Callbacks automate decisions that should depend on evidence. EarlyStopping ends a run when the monitored validation metric stops improving; restore_best_weights keeps the best epoch rather than the last one. ModelCheckpoint makes the result recoverable. ReduceLROnPlateau lowers the learning rate after progress stalls, while TensorBoard records curves and graphs. Monitor validation loss when the goal is generalization, and never monitor the test set during model selection.

callbacks = [
    keras.callbacks.EarlyStopping(
        monitor='val_loss', patience=5, restore_best_weights=True
    ),
    keras.callbacks.ReduceLROnPlateau(
        monitor='val_loss', factor=0.5, patience=2
    ),
    keras.callbacks.ModelCheckpoint(
        'best.keras', monitor='val_loss', save_best_only=True
    ),
]

Underfitting, Overfitting & Regularization

Underfitting means both training and validation performance are poor: add capacity, improve features, train longer, or reduce excessive regularization. Overfitting appears as strong training performance with a widening validation gap. L1 regularization promotes exact zeros, L2 discourages large weights smoothly, and Elastic Net combines both. Dropout trains a changing ensemble of subnetworks by suppressing random activations during training; it is disabled at inference. Data augmentation, early stopping, and more representative data address overfitting from different angles.

VGG: Depth Through Repeated Small Filters

VGG made a simple design choice systematic: stack many $3\times3$ convolutions, periodically downsample, then classify. Repeating small kernels gives multiple nonlinear transformations and an expanding effective receptive field with fewer parameters than one very large kernel. Its clarity made VGG influential, but the fully connected head and wide stacks are computationally expensive. The lesson is architectural regularity, not that VGG is the best default for a new deployment.

ResNet: Learn the Residual

A deeper plain network can have higher training error than a shallower one, showing an optimization problem rather than simple overfitting. A residual block asks its layers to learn $F(x)=H(x)-x$ and returns $F(x)+x$. If the best mapping is close to identity, the residual can be near zero, and the shortcut gives gradients a clean path through depth. When tensor shapes differ, a projection shortcut can align channels or resolution.