D4 · Deep Learning
Batch Norm and Layer Norm
Apply the right normalization for a given architecture and batch size.
In 2015 Sergey Ioffe and Christian Szegedy added one operation between layers. Their image model reached the same accuracy in 14 times fewer training steps. The operation is arithmetic a beginner could write. It subtracts a mean and divides by a spread.