L2 · Attention
Many Heads, Many Questions
Explain why splitting into heads beats one wide attention.
A token gets one row of weights, so it can give one answer. But a word needs several: which noun is mine, and which word came just before me? Forced into one row, the answers get averaged. And an average of two good answers is usually a bad one.