D6 · Sequences & Memory
Training vs Generating
Explain teacher forcing and the exposure gap it creates at generation time.
While it trains, a sequence model is handed the correct previous word at every step. Its own mistakes never reach it. When it generates, nobody corrects it. It reads its own last word and carries on. It trains where the past is always right, then works where the past is whatever it just said.