D7 · Neural Scaling Laws
What the Curve Does Not Predict
List the things loss curves say nothing about, including data running out.
A scaling curve predicts one number: the average loss on held-out text like the text the model trained on. That is all. Everything else people want from these plots needs its own evidence.