L3 · The Transformer
Positions in Practice
Say where rotary positions act in the block and how context length gets extended.
You met rotary positions in the tokens chapter. Inside attention, the query and the key are each turned by an angle set by their position. The token's own vector is left alone. Only the comparison changes.