L3 · The Transformer
Build a Tiny Transformer
Assemble the parts into a working model and train it on this device.
You have met every piece: a token table, norms, attention, the adds, a feed-forward layer, and a matrix that scores every token. The model you are about to train has 3,672 parameters. Growing it into GPT-2 changes five numbers and nothing else.