(base concepts and methods from the Fastai22-23P2 activation and initialization nbs)
We start with a baseline training of a CNN on Fashion MNIST, then progressively improve the accuracy and quality of convergence with normalization methods.
Then we reimplement the LSUV - Layer-Wise Sequential Unit-Variance algorithm and compare it to the previous normalization steps.
We also introduce modified ReLU units with leakiness and learnable parameters.