One new idea, built on Chapter 4

A small cost
for large weights.

Early stopping asks when should training end? Regularization asks a different question: while training, can we make extremely large weights a little less attractive?

data loss+size penalty=training objective→gradients→updates

Start from the overfitting clue.

In the HW3 experiment, training loss may keep falling after validation loss begins to rise. The model is getting better at its training examples without getting better at new examples.

Early stopping

Keep watching validation loss. Stop training—or restore an earlier model—when validation performance stops improving.

It changes how long we train.

Regularization

Add another preference to the training objective: fit the examples, but avoid using unnecessarily large weights.

It changes what training tries to minimize.

Important: neither method guarantees better generalization. We use validation evidence to choose settings, then preserve the test set for the final check.

Add one term to the loss.

Chapter 4 describes loss as a number we want to make small. A regularized training objective is still one number—it simply combines two concerns.

total training loss= data loss + strength × weight penalty

Data loss: How poorly the predictions match the labels in this mini-batch.
Penalty: A number that grows when weights grow. For L2 regularization, square the weights and add or average them.
Strength: A setting chosen before training. It controls how much the penalty matters. It is often written as λ (lambda).

The penalty is part of training loss, not a new accuracy score. When reporting validation performance, keep reporting the original validation loss and metric so models are compared on the task itself.

See the compromise.

This toy model has just one weight. The data alone prefers weight = 4. Move the slider to make large weights increasingly expensive.

Regularization strength λ: 0.000 means no penalty. Larger values pull the preferred weight closer to zero.

best w = 4.00

0.00data loss at selected best weight

0.00weighted penalty at selected best weight

0.00total training loss

Notice what regularization does not say: it does not claim that small weights are automatically correct. It says that, when two settings fit the training data similarly, we prefer the one that relies less on large weights.

Change the Chapter 4 loop.

The forward pass stays the same. The loss gains one extra calculation before `backward()`.

for xb, yb in dl:
    predictions = model(xb)

    data_loss = mnist_loss(predictions, yb)
    weight_penalty = (weights ** 2).mean()
    loss = data_loss + strength * weight_penalty

    loss.backward()
    weights.data -= weights.grad * lr
    bias.data   -= bias.grad * lr
    weights.grad.zero_()
    bias.grad.zero_()

What receives a gradient?

The total loss depends on both the predictions and the weights. `backward()` therefore includes both influences when it calculates `weights.grad`.

What about bias?

For this first experiment, penalize the weights and leave the bias alone. That keeps the question focused. Larger systems may make different choices.

When using PyTorch’s SGD optimizer, the same idea is available as `weight_decay`:

optimizer = torch.optim.SGD(model.parameters(), lr=lr, weight_decay=strength)

For this Chapter 4 extension, writing the penalty explicitly makes the changed objective easier to inspect. PyTorch documents SGD’s weight_decay option as an L2 penalty. The numerical value of `strength` need not match `weight_decay` exactly because a hand-written penalty may use a sum or a mean.

Run a fair experiment.

A regularizer is useful only if held-out evidence improves. Keep everything else fixed.

Use the same training and validation split for every run.
Use the same model, starting seed, learning rate, batch size, and epoch count.
Try a small set of strengths, including 0 as the no-regularization baseline.
Plot training data loss and validation data loss. Do not add the penalty to validation loss.
Compare the size of the weights as supporting evidence—not as the final goal.
Choose from validation evidence; use the test set only after the choice is fixed.

Method	What changes?	Question it answers
Early stopping	The epoch at which training ends	When did validation performance stop improving?
L2 regularization	The objective minimized during every update	Can a model fit reasonably well without relying on large weights?

Bonus question: another response to overfitting

Starting from your overfitting run, add an L2 penalty to the training loss. Keep the data split, model, learning rate, and epoch count fixed.

Run a baseline with strength = 0 and at least three positive strengths.
Plot training data loss and validation data loss for every run.
Report the best validation epoch and a simple summary of weight size for each run.
Explain which strength you would choose and what evidence supports that choice.
Compare this method with early stopping. What does each one change?