Start with the loss you already know

What should
confidence cost?

Chapter 4 gives a sensible loss for separating 3s from 7s. Binary cross-entropy asks the same basic question—how much probability did the model give the correct answer?—but prices confident mistakes much more heavily.

raw score→sigmoid→probability p→loss→gradient

Recover Chapter 4’s meaning.

Let `p` mean “the model’s probability that this image is a 3.” The target is 1 for a 3 and 0 for a 7.

If the image is a 3

The target is 1. We want `p` close to 1. Chapter 4 uses the distance from 1:

loss = 1 - p

If the image is a 7

The target is 0. We want `p` close to 0. Chapter 4 uses the distance from 0:

loss = p

def mnist_loss(predictions, targets):
    probabilities = predictions.sigmoid()
    return torch.where(
        targets == 1,
        1 - probabilities,  # image is a 3
        probabilities       # image is a 7
    ).mean()

A compact way to say the same thing is: Chapter 4 loss = 1 minus the probability given to the correct class.

Keep the question; change the scale.

Binary cross-entropy first finds the probability assigned to the correct answer. Then it applies `-log`.

target = 1

correct probability = p → BCE = -log(p)

target = 0

correct probability = 1 - p → BCE = -log(1 - p)

A gentle reading of `-log`: giving the correct answer probability 1 costs 0. Giving it less probability costs more. As the correct-answer probability approaches 0, the cost grows sharply.

Why is it called binary cross-entropy? For one outcome, -log(p) is its information or “surprise.” Entropy is not merely one `-log`; it is an average of surprise under a probability distribution. Cross-entropy averages the surprise produced by one distribution’s predictions when the true outcomes come from another distribution. In our binary 3-versus-7 case, that becomes the average BCE across examples.

BINARYThere are two possible labels: 3 or 7, represented by 1 or 0.

CROSSWe compare the true label with the probability distribution predicted by the model.

ENTROPYWe average a -log(probability) surprise for what actually happened.

Folded refresher: what do log(p) and -log(p) look like?

For probabilities between 0 and 1, log(p) is zero or negative. Multiplying by -1 flips the curve upward, giving a nonnegative loss. Near probability 1, the loss is near 0. Near probability 0, the loss becomes very large.

Probability given to the correct answer	Chapter 4 loss 1 − correct probability	Binary cross-entropy −log(correct probability)
0.99	0.01	0.01
0.90	0.10	0.11
0.60	0.40	0.51
0.50	0.50	0.69
0.10	0.90	2.30
0.01	0.99	4.61

The two losses agree about which direction is better. They disagree about how serious a confident mistake should be.

Move one prediction.

Choose the true label, then move the model’s probability that the image is a 3. The vertical marker shows the current prediction.

Model probability that the image is a 3: 0.80Above 0.5 predicts 3; below 0.5 predicts 7.

correct side

0.80probability given to correct answer

0.20Chapter 4 loss

0.22binary cross-entropy

Write BCE in Chapter 4 style.

The long form shows both cases. The PyTorch helper is the safer implementation.

def binary_cross_entropy_from_probabilities(probabilities, targets):
    losses = -(
        targets * torch.log(probabilities)
        + (1 - targets) * torch.log(1 - probabilities)
    )
    return losses.mean()

One practical problem: `log(0)` is not a usable finite number. Directly taking logs of rounded probabilities can therefore cause numerical trouble.

Chapter 4’s `linear1(xb)` returns raw scores before sigmoid. PyTorch calls these logits. Its combined BCE-with-logits function accepts those raw scores, applies the sigmoid idea internally, and calculates BCE in a numerically stable way:

import torch.nn.functional as F

def bce_loss(logits, targets):
    return F.binary_cross_entropy_with_logits(
        logits, targets.float()
    )

predictions = linear1(xb)   # raw scores: do not call sigmoid first
loss = bce_loss(predictions, yb)

Rule to remember: “with logits” means the function expects raw model scores. Do not apply sigmoid first. Apply sigmoid later when you want probabilities for interpretation or a 0.5 decision threshold.

Compare without declaring a winner.

Both losses are smooth after sigmoid, both reward probability on the correct answer, and both can drive SGD. They create different gradient signals.

Question	Chapter 4 `mnist_loss`	Binary cross-entropy
Smallest possible item loss	0	0
Largest possible item loss	Approaches 1	Grows without a fixed upper limit
Confident wrong prediction	Costs almost 1	Costs much more than an uncertain mistake
Main classroom advantage	Its two cases are immediately readable	It strongly distinguishes “slightly wrong” from “confidently wrong”
PyTorch input	Custom function shown in the chapter	Raw logits with `binary_cross_entropy_with_logits`

Binary is important here. This page concerns one yes/no decision: 3 versus 7. The `F.cross_entropy` used near the end of the Chapter 4 notebook for ten digit classes is related, but it handles a different output shape and target format. Do not swap the functions by name alone.

Stretch question: inspect the loss

Use the same 3-versus-7 data, model, split, learning rate, and number of epochs. Compare Chapter 4’s `mnist_loss` with binary cross-entropy from logits.

Before training, calculate both losses for several hand-chosen probabilities, including a confident correct and confident wrong prediction.
Train once with each loss from the same starting seed.
Plot training loss and validation accuracy. Remember that the loss scales are different, so compare patterns—not raw loss heights.
Find one example each model gets wrong. Record its predicted probability and both loss values.
Explain, in plain language, what BCE changes about the cost of confidence.