L4 · Training a Language Model
Windows, Batches, Packing
Explain how loose documents become a rectangle of tokens the model can read.
A model never reads a document. It reads a rectangle. Each row is a window of fixed length. The whole rectangle is a batch, and a batch is what one training step sees.