L10 · Smaller, Faster, Cheaper
The Wall Is Memory
Explain why writing one token is limited by memory speed, and put a ceiling on tokens a second.
A data center card can do close to a thousand trillion operations a second. Yet while it writes a reply to one person, it mostly waits. It is waiting for numbers to arrive from its own memory.