L4 · Training a Language Model
Grading the Guess
Read a language model's loss as a perplexity, and say when two perplexities cannot be compared.
You met cross-entropy in the probability chapter: a guess pays the surprise of the chance it gave the truth. A language model pays that at every token. Its loss is the average bill.