D8 · Mechanistic Interpretability
Induction Heads
Explain the copy-and-continue pattern and why it matters for in-context learning.
Give a model a string of random tokens, then start repeating it. It finishes the repeat almost perfectly. It never saw that string in training. So something inside looks back and copies.