L4 · Training a Language Model
Warmup, Then Decay
Read a real run's learning-rate schedule, say why Adam needs a warmup, and say why labs stopped committing to a cosine.
You met warmup and cosine decay in the gradient descent chapter: a short climb, then a long slide. Here is what real runs set them to, and why the climb matters so much for Adam.