D8 · Mechanistic Interpretability
Attribution Maps and Their Failures
Read a saliency map and describe a test that shows it can be meaningless.
You have seen these: a photo with a warm glow over the part the model supposedly used. They are easy to make and convincing. In 2018 a paper showed some survive wrecking the model.