L10 · Smaller, Faster, Cheaper
A Small Model Drafts
Explain how a big model checks a small model's draft in one pass, and why the text does not change.
A small model guesses the next five tokens. The big model checks all five in one pass, in about the time of writing one. If the first four pass, that one big step yields five tokens: the four, plus one of its own. This is speculative decoding.