L3 · The Transformer
Where Facts Live
Summarize the evidence that feed-forward layers act as key-value memories.
In 2021, Geva and colleagues read the feed-forward layer as a memory. Each wide unit has a key that matches a pattern in the text, and a value that pushes certain next words. People could name many of the patterns. Lower layers caught surface patterns; upper layers caught meaning.