L1 · Words as Numbers
What Arabic Costs a Tokenizer
Measure tokens per word for Arabic and English and explain the price gap.
One measurement from the Jais paper: how many tokens a word costs, on four tokenizers. English stays low on every one of them. Before you see Arabic, make a guess.