D2 · Gradient Descent
Change the Step as You Go
Pick a learning-rate schedule for a run, and say what warmup protects against.
Far from the bottom, big steps are fine. Almost any step downhill helps. Near the bottom, the same big step overshoots. You land on the far wall, then back again.