For six weeks I wrote every backward pass by hand. affine_backward. relu_backward. batchnorm_backward. conv_backward. Both directions, every layer, verified against numerical gradients.
Then I opened PyTorch and wrote loss.backward(). One line.
My first reaction was that I had skipped something.
What PyTorch does when you call backward() is not magic. It is the same chain rule I had been applying manually, executed on a computational graph built automatically as the forward pass ran. Every operation on a tensor with requires_grad=True adds a node to that graph. backward() traverses it in reverse, accumulating gradients using the rules stored at each node.
The assignment teaches PyTorch at three abstraction levels, and the progression matters.
Level 1: functional API. You call F.conv2d, F.relu, manage parameters manually in a list, write the update loop yourself. It feels like NumPy with automatic gradients.
Level 2: nn.Module. You declare layers in __init__, describe connectivity in forward(). The module tracks parameters. An optimizer handles updates.
Level 3: nn.Sequential. The entire three-layer ConvNet becomes six lines.
Each level hides something. What it hides is exactly what you need to understand to debug it.
When I implemented batchnorm_backward by hand, I had to differentiate through variance, mean, and the normalized output — in the right order. Getting it wrong gives a gradient that looks plausible. The network trains, just worse. In PyTorch you never write that backward pass. But if you forget to call model.eval() before validation, BatchNorm keeps using batch statistics instead of running statistics, and accuracy drops — and you will not know why unless you understood what BatchNorm does internally.
The framework removes the cost of implementation. It does not remove the cost of understanding.
Code on GitHub in the first comment.
#PyTorch #DeepLearning #MachineLearning #CS231n #ComputerVision #NeuralNetworks