M5 · Did It Actually Work?
Benchmarks Wear Out
Explain how a shared test set loses its meaning, through reuse and through leaking into training data.
Two thousand models that know nothing were scored on one test set. The best of them scored 57.8 out of a hundred. Every single one was a coin.