Neural Networks in Practice
Lab: build a neural network from scratch in NumPy with manual backprop, then rebuild it in TensorFlow/Keras and track it with TensorBoard.
Learning Objectives
- ✓Set up the environment, check the GPU and load the make_moons and MNIST datasets
- ✓Implement the forward pass, the loss and backpropagation by hand in NumPy
- ✓Work fluently with all three ways of building a model in Keras
- ✓Choose between model.fit() and a custom training loop with GradientTape
- ✓Track and compare experiments with TensorBoard
Environment & Data Setup
The opening step is checking the TensorFlow/Keras version and whether a GPU is available. The two datasets serve two purposes: make_moons is a nonlinear binary problem on a two-dimensional plane, so the decision boundary can be drawn, while MNIST covers the multiclass part at the end of the session. Both need standardizing. Note that the NumPy portion uses the convention that each column is one sample, the opposite of scikit-learn, so the matrices must be transposed first.
import tensorflow as tf
from tensorflow import keras
from sklearn.datasets import make_moons
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
print("TensorFlow:", tf.__version__)
print("GPU available:", tf.config.list_physical_devices("GPU"))
X, y = make_moons(n_samples=800, noise=0.25, random_state=42)
X = StandardScaler().fit_transform(X)
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)
# The hand-rolled NumPy code uses "each COLUMN is one sample" → transpose
X_train_np, y_train_np = X_train.T, y_train.reshape(1, -1)NumPy from Scratch — Manual Backpropagation
Implementing the network in NumPy leaves no part of it a black box. Four blocks: initialize parameters with He for the ReLU layers, where zero biases are fine but a zero weight matrix absolutely is not; the forward pass through the ReLU layers to a Sigmoid output; the binary cross-entropy, remembering to add a tiny constant inside the logarithm; and the backward pass, where each line matches one of the BP1–BP4 equations from session 8.
def init_params(layer_dims):
params = {}
for l in range(1, len(layer_dims)):
# He init for ReLU — do NOT initialize W to 0
params[f"W{l}"] = np.random.randn(layer_dims[l], layer_dims[l-1]) * np.sqrt(2 / layer_dims[l-1])
params[f"b{l}"] = np.zeros((layer_dims[l], 1))
return params
def sigmoid(z): return 1 / (1 + np.exp(-z))
def relu(z): return np.maximum(0, z)
def forward(X, params):
cache, A, L = [], X, len(params) // 2
for l in range(1, L):
W, b = params[f"W{l}"], params[f"b{l}"]
Z = W @ A + b
cache.append((A, W, b, Z))
A = relu(Z)
W, b = params[f"W{L}"], params[f"b{L}"]
Z = W @ A + b
cache.append((A, W, b, Z))
return sigmoid(Z), cache
def compute_loss(A_out, y, eps=1e-8):
m = y.shape[1]
return -(1 / m) * np.sum(y * np.log(A_out + eps) + (1 - y) * np.log(1 - A_out + eps))Three Ways to Build a Model in Keras
The three APIs differ in flexibility. Sequential stacks layers in a straight chain, compact and readable but usable only when the network really is a single line. Functional describes the network as a computation graph, so it handles multiple inputs and outputs, branches and skip connections. Subclassing lets you write the entire forward pass inside call(), giving maximum flexibility at the cost of summary and plot_model no longer working and debugging getting harder.
# 1. Sequential
model_seq = keras.Sequential([
keras.layers.Dense(32, activation='relu', input_shape=(2,)),
keras.layers.Dropout(0.2),
keras.layers.Dense(1, activation='sigmoid'),
])
# 2. Functional API — an arbitrary computation graph
inputs = keras.Input(shape=(2,))
x = keras.layers.Dense(32, activation='relu')(inputs)
outputs = keras.layers.Dense(1, activation='sigmoid')(x)
model_func = keras.Model(inputs, outputs)
# 3. Subclassing — write the forward pass yourself
class MyModel(keras.Model):
def __init__(self):
super().__init__()
self.dense1 = keras.layers.Dense(32, activation='relu')
self.dense2 = keras.layers.Dense(1, activation='sigmoid')
def call(self, x):
return self.dense2(self.dense1(x))
model_sub = MyModel()Two Ways to Train
For most problems calling model.fit() is enough: Keras handles batching, the forward pass, the loss, the backward pass, the weight update and the callbacks. Two callbacks worth enabling are EarlyStopping with restore_best_weights — without that option the model you keep is the one from the final epoch, which has already started degrading — and ReduceLROnPlateau. A custom training loop is only needed when the problem does not fit the mold; there, remember to pass training=True so Dropout and BatchNorm run in the right mode.
# Option 1: model.fit() with callbacks
model_seq.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])
model_seq.fit(
X_train, y_train, validation_data=(X_val, y_val),
epochs=100, batch_size=64,
callbacks=[
keras.callbacks.EarlyStopping(patience=10, restore_best_weights=True),
keras.callbacks.ReduceLROnPlateau(patience=5, factor=0.5),
])
# Option 2: Custom training loop with GradientTape
optimizer = keras.optimizers.Adam()
loss_fn = keras.losses.BinaryCrossentropy()
for epoch in range(30):
with tf.GradientTape() as tape:
y_pred = model_func(X_train, training=True) # training=True is required
loss = loss_fn(y_train, y_pred)
grads = tape.gradient(loss, model_func.trainable_variables)
optimizer.apply_gradients(zip(grads, model_func.trainable_variables))TensorBoard & Hyperparameter Experiments
TensorBoard plots loss, accuracy and model structure in real time; putting each run in a subdirectory named after its configuration lets you compare them directly on one graph. The first thing to look at is the gap between the train and validation curves: two curves tracking each other means there is capacity to spare, while validation bottoming out then turning upward means overfitting. The last step is extending to multiclass MNIST: a 10-neuron softmax output, with the loss switched to categorical cross-entropy.
import time
run = f"dense32_lr1e-3_{int(time.time())}"
tb = keras.callbacks.TensorBoard(log_dir=f"logs/fit/{run}", histogram_freq=1)
model_seq.fit(X_train, y_train, validation_data=(X_val, y_val),
epochs=30, callbacks=[tb])
# Run: tensorboard --logdir logs/fit
# Extend to multiclass MNIST
model_mc = keras.Sequential([
keras.layers.Flatten(input_shape=(28, 28)),
keras.layers.Dense(128, activation='relu'),
keras.layers.Dense(10, activation='softmax'), # 10 neurons, not 1
])
model_mc.compile(optimizer='adam',
loss='sparse_categorical_crossentropy', # labels are integers
metrics=['accuracy'])