L3 · The Transformer
Why It Beat the RNN
Explain why a transformer trains in parallel and a recurrent network cannot.
In 2017 the transformer beat the best translation systems, using a small fraction of their training compute. The base model trained in about twelve hours on eight graphics cards. That cheapness changed what people dared to try next.