L10 · Smaller, Faster, Cheaper
How Far Ahead to Guess
Work out the tokens one big step yields from a draft, and pick how many tokens to draft.
Guess one token ahead and you gain a little. Guess eight ahead with a good draft and you go three times faster. Guess too far with a weak draft and the gain starts to shrink again.