L4 · Training a Language Model
Where the Text Comes From
Describe a real pretraining mix and say why the mix is not the pool.
Most of what a model reads is web pages nobody remembers writing. Books and encyclopedias sit on top of that pile as a thin, carefully chosen layer.