D2 · Gradient Descent
The Spike That Ruins a Week
Recognize an exploding gradient, and apply norm clipping without distorting the direction of the update.
A run has been healthy for three days. The loss is falling. Nobody is watching. Then one batch holds a strange example. Its gradient is enormous, and one update throws every weight far out of place.