R2 · Landmark Papers, Rebuilt
Attention Is All You Need (2017)
Compute one attention head by hand, and say why the block replaced recurrence.
Attention already existed. Since 2015 a decoder could look back at every source word instead of trusting one summary. But it was bolted onto a chain, and a chain has to finish word one before it starts word two. Big machines sat idle.