L1 · Words as Numbers
Characters, Words, Subwords
Compare the three tokenization levels on vocabulary size and sequence length.
Here is one sentence under the smallest cut: every character is a chip. The sentence will not change in this lesson. Only the size of the piece will.