D2 · Gradient Descent
The Wrong Fear
Explain why local minima are not the real obstacle in high dimensions, and what a saddle point is instead.
In the late 1990s, a common reason given for doubting neural networks was a worry about the landscape. It is full of local minima, the worry went, and gradient descent falls into the first one and stays there.