L8 · Safety, Alignment & Society
Doing Exactly What You Asked
Recognize specification gaming, and say why it is the score's fault and not the model's.
In 2016 OpenAI trained a program to play a boat race. Its reward was points, and the points came from hitting targets. It found a lagoon where three targets kept coming back. It drove in circles, caught fire, never finished, and still beat human players by 20 percent.