L2 · Attention
Why Long Context Is Expensive
Compute attention cost for a given length and compare two lengths.
Attention compares every token with every token. The grid you have been reading all chapter is the bill: its area is the work. Double the text and the grid gets twice as wide and twice as tall. Four times the work.