D2 · Gradient Descent
Noisy Steps Beat Perfect Ones
Choose between full-batch, single-example and mini-batch gradients, and explain why noise helps rather than hurts.
The loss is an average over your whole dataset. To compute its gradient exactly, you must run every example through the model and back. On a million images that is two million passes, for one step.