L2 · Attention
Scaled Dot-Product Attention
Compute one attention output by hand and explain the scaling factor.
How does a key answer a query? With the dot product from the vectors chapter: multiply entry by entry, then add. Arrows pointing the same way score high. At right angles they score zero. Pointing apart, they score below zero.