Pretraining never hands the model a single fact. It picks facts up because guessing the next word well is impossible without them.
L4 · Training a Language Model
The objective is one sentence long: guess the next token, and pay for your surprise. Everything people call capability grows out of doing that on enough text. This chapter follows a real run from the first batch to the last checkpoint, then through the stages that turn a text predictor into an assistant. You will check a run against its bill, read a loss curve that goes wrong, and finish able to look at a benchmark table without being fooled by it.
- Guess the Next Token
- Grading the Guess
- Where the Text Comes From
- Cleaning the Text
- Windows, Batches, Packing
- The Loop at Scale
- Warmup, Then Decay
- Small Numbers, Big Spikes
- Splitting the Work
- Tokens Against Parameters
- From Predictor to Assistant
- Learning From Preferences
- DPO, and Doing Less
- Reading a Benchmark Table