L1 · Words as Numbers
Where a Token Sits
Explain why order must be injected and compare absolute with rotary positions.
You now have one learned row per token, in order. The next layer, attention, which L2 builds, mixes the rows by adding them up with weights. A sum does not care about order. Shuffle the rows and it gives the same answer.