D2 · Gradient Descent
Adam: Both Ideas at Once
Write out the Adam update, say what bias correction fixes, and explain how AdamW differs.
You have met both halves. Momentum averages the gradient, so the direction holds steady. RMSProp averages its square, so every weight gets a fair step. Adam keeps both averages and uses them together. It rarely needs much tuning, which is why it spread.