L1 · Words as Numbers
How Big Should the Vocabulary Be?
Weigh vocabulary size against sequence length, memory and rare words.
This is the trade, measured on the tokenizer from the last lesson. Vocabulary runs along the bottom, tokens up the side. Early merges save a lot. Late ones save very little.