Overfitting and Dropout in Python

A network with many weights and few examples can learn them by heart: it remembers each training image instead of learning what a 3 looks like. That's overfitting: high accuracy on the training images, much lower on new ones. The starting code trains a big network (19,210 weights) on only 200 digit images, 200 times over. To look like real, messy data, every 4th training label is replaced by a random digit.

The starting code also gives you three functions: train(model, weight_decay=0) trains a model with Adam, accuracy(model, X, y) gives back the share of images it reads right, and report(name, model) prints that share on the 200 training images, then on the 1,597 test images.

Two fixes: nn.Dropout(0.5) is a layer that sets a random half of the numbers passing through it to 0 on every training pass, and doubles the rest so the total stays about the same. The network can't count on any single hidden number to remember an image. After model.eval() it lets everything through unchanged. weight_decay, a setting of the optimizer, pulls every weight a little toward 0 at each step, so big weights cost something.

drop = nn.Dropout(0.5)
print(drop(torch.ones(4)))   # random, for example tensor([2., 0., 0., 2.])
drop.eval()
print(drop(torch.ones(4)))   # tensor([1., 1., 1., 1.])
opt = torch.optim.SGD(model.parameters(), lr=0.1, weight_decay=0.001)

When you're done, the program prints this. The two Dropout lines come from the lesson's check, which runs a Dropout layer on 8 ones:

Plain network:   train 1.0 test 0.713
Dropout + decay: train 0.83 test 0.85
Dropout while training: [0.0, 2.0, 2.0, 2.0, 2.0, 0.0, 2.0, 2.0]
Dropout after eval():   [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]

Your task: build safer, the same layers as big with nn.Dropout(0.5) before the last layer, train it with weight_decay=0.01, and report it as "Dropout + decay:".

Assignment
Follow the Coding Tutorial to build, train and report a network that can't learn the training images by heart.

Hint
Copy the layers of big and add nn.Dropout(0.5) right after nn.ReLU(). The train accuracy goes down, and that's good news here: a quarter of the training labels are random.