D2 · Gradient Descent
Watch Them Race
Implement SGD, momentum and Adam by hand, race them on the same 2D surface, and explain why the winner changes with the surface.
You now have four methods. They differ in just two choices: the direction of the step, and its length. SGD takes the raw gradient for both. Momentum smooths the direction. RMSProp sets the length per weight. Adam does both. Same surface, same start, same number of steps: now you can see the differences.