A3 · Shipping and Operating AI
Shipping a Change
Decide the size of an experiment before you look, and read two arms honestly.
The new model beat the old one by four points on your test set. That is not a reason to ship it. It is a reason to run an experiment.