D7 · Neural Scaling Laws
Power Laws Everywhere
Recognize a power law and state what it predicts and for how long.
City sizes, earthquake strengths, how often a word gets used, and how wrong a language model is. Four unrelated things, one shape. For cities and words, people have good stories for why. For language models, the why is still argued about.