D6 · Sequences & Memory
A Vector That Remembers
Describe the hidden state as a running summary of everything seen so far.
Read a sentence one word at a time through a slot, so you can never look back. You would keep a note. After each word you rewrite the note, keeping only what you think you will need. That note is the hidden state. In a real model it is a few hundred numbers.