D4 · Deep Learning
Dropout and Weight Decay
Pick a regularizer from the symptom you see in the curves.
On every training step, dropout switches off half the network's activations at random. A different half each time, so the network never sees its whole self. That damaged network does better on new data than the one you left intact.