L10 · Smaller, Faster, Cheaper
The Long Conversation Bill
Stack the levers that shrink a long conversation's cache, and say what each one gives up.
Keep every head's notes for one conversation of 32,768 tokens and they fill 16 GiB. The model itself is 13.5. The notes, not the model, decide how many people one card can serve.