L2 · Attention
The KV Cache
Explain what is cached during generation and how memory grows with length.
Writing text is a loop: read everything so far, add one token, repeat. Done naively, token 500 recomputes keys and values for all 500 tokens, though 499 of them are exactly as they were. Thanks to the mask, an old token's key and value never change. Compute them once, and keep them.