L4 · Training a Language Model
Guess the Next Token
Say what a language model is trained to do, and why that one task makes it learn facts.
Nobody taught this thing grammar. It was shown text with the next piece hidden, asked to guess, and corrected. A few trillion times.