L3 · The Transformer
Reading a Real Config
Read layers, heads, width and vocabulary, and count a model's parameters by hand.
The GPT-2 paper lists four sizes: 117, 345, 762 and 1542 million parameters. The files OpenAI released are named 124M, 355M, 774M and 1558M. Count the configs yourself and you get the file names, to the last digit.