Sanity Checks For Saliency Maps

Authors:
Julius Adebayo MIT
Justin Gilmer Google Brain
Michael Muelly Google
Ian Goodfellow Google
Moritz Hardt Google Brain
Been Kim Google

Introduction:

Saliency methods have emerged as a popular tool to highlight features in an inputdeemed relevant for the prediction of a learned model.Several saliency methodshave been proposed, often guided by visual appeal on image data.

Abstract:

Saliency methods have emerged as a popular tool to highlight features in an inputdeemed relevant for the prediction of a learned model. Several saliency methodshave been proposed, often guided by visual appeal on image data. In this work, wepropose an actionable methodology to evaluate what kinds of explanations a givenmethod can and cannot provide. We find that reliance, solely, on visual assessmentcan be misleading. Through extensive experiments we show that some existingsaliency methods are independent both of the model and of the data generatingprocess. Consequently, methods that fail the proposed tests are inadequate fortasks that are sensitive to either data or model, such as, finding outliers in the data,explaining the relationship between inputs and outputs that the model learned,and debugging the model. We interpret our findings through an analogy withedge detection in images, a technique that requires neither training data nor model.Theory in the case of a linear model and a single-layer convolutional neural networksupports our experimental findings.

You may want to know: