R2 · Landmark Papers, Rebuilt
GPT-2 and GPT-3
Say what changed between the two, and what in-context learning does and does not do.
In 2018 the recipe was two steps. Train on plain text, then fine-tune on a few thousand labeled examples of your task. You needed a labeled set, a training run and a saved model for every task you cared about.