D2 · Gradient Descent
Loss Is a Landscape
Read a loss landscape: every setting of the weights is a place, and the loss is how high you stand.
Imagine a hillside at night. Thick fog. You cannot see the valley or the peak. You only feel the ground under your feet. You want the lowest point. What do you do? You feel which way the ground tilts, step downhill, then feel again.