M5 · Did It Actually Work?
Searching Without Fooling Yourself
Run a hyperparameter search over three piles of data and say how much the winner flatters you.
You tried twenty one settings and kept the best. That best number is too good, and you can say roughly how much too good before you even look.