L2 · Attention
No Peeking Ahead
Apply a causal mask and say why training would be useless without it.
A language model trains on one task: guess the next token from the ones before it. But in training the whole sentence sits in memory, and attention reaches every position. Token three could simply read token four, the answer. It would learn to copy, not to predict.