Before attention, a model had to squeeze a whole paragraph into one vector.
D6 · Sequences & Memory
Order carries meaning, so a model that reads a sentence needs a memory. Recurrent networks gave it one: a hidden state passed from step to step. This chapter unrolls that loop, trains it through time, and shows the exact point where gradients die on long inputs. Gates fixed part of it, and a single bottleneck vector in translation exposed the rest, which is precisely the gap attention was invented to close.
- Data Where Order Matters
- A Vector That Remembers
- Unrolling the Loop
- Backprop Through Time
- The Same Problem, Now in Time
- Gates That Choose What to Keep
- GRU: Fewer Gates, Similar Result
- Training vs Generating
- A Character-Level Language Model
- One Vector for a Whole Sentence
- The Case for Attention