Backpropagation in Python

For one neuron, one formula gave the slope of each weight. In a network, a weight of the first layer reaches the loss only through the layer after it. Backpropagation finds every slope in one trip backwards: first the output error, then each hidden neuron's share of the error, and from each error the slopes of the weights that feed that layer.

The trip uses the chain rule: when one thing feeds the next, their slopes multiply. If h = 2 * x and out = 3 * h, nudging x by 0.001 moves h by 0.002 and out by 0.006: a slope of 2 × 3 = 6.

Two numpy tools keep the shapes right. .T gives back the array turned on its side (rows become columns), and .mean(axis=0) averages down the rows and gives back one number per column:

a = np.array([[1, 2, 3], [5, 6, 7]])
print(a.T.shape)        # (3, 2)
print(a.mean(axis=0))   # [3. 4. 5.]

The starting code holds the XOR network: forward(X) gives back hidden, shape (4, 3), and out, shape (4, 1), and backward has only its return. Every gradient must have the shape of the numbers it belongs to: grad_W2 like W2, (3, 1), and grad_W1 like W1, (2, 3). The check then nudges every weight and bias a tiny bit and makes sure the loss moves the way your gradients say. Your program must print (the last two lines come from the check):

loss: 0.3276
grad_W2 shape: (3, 1) W2 shape: (3, 1)
[[0.062 ]
 [0.0539]
 [0.0345]]
grad_W1 shape: (2, 3) W1 shape: (2, 3)
[[ 0.0015 -0.014   0.0207]
 [-0.0005 -0.0121  0.0149]]
grad_b1 shape: (3,) grad_b2 shape: (1,)
every gradient matches nudging its own weight: True

Your task: inside backward, above the return, compute the output error d_out, the gradients grad_W2 and grad_b2, the hidden error d_hidden, and the gradients grad_W1 and grad_b1.

Assignment
Follow the Coding Tutorial to write the backward pass of the 2-3-1 network, from the output error back to the first layer.

Hint
Check every shape before you type: in @, the inner sizes must match and the outer ones give the result, so (3, 4) @ (4, 1) gives (3, 1). Until all six lines are in, the return stops the run with a NameError.