L2 · Attention
Softmax Over Positions
Read an attention row as a probability distribution over tokens.
Scores can be anything: minus twelve, four hundred. Attention needs shares, so softmax converts them, as you saw in the probability chapter. Every weight comes out positive, and each row adds up to one. Every token gets a budget of exactly one to spend.