Sanity Checks for Saliency Maps Julius Adebayo PhD Student, MIT. - - PowerPoint PPT Presentation

Sanity Checks for ‘Saliency’ Maps

1

Julius Adebayo PhD Student, MIT.

Joint work with

• Developer/Researcher: Model Debugging.
• Safety concerns.
• Ethical concerns.
• Trust: Satiate ‘societal’ need for reasoning to trust an automated system

learned from data.

Some Motivation

2

[Challenges for Transparency, Weller 2017, & Doshi-Velez & Kim, 2017 ]

Goals: Model Debugging

3

• Model Debugging: reveal spurious correlations or the kinds of

inputs that a model is most likely to have undesirable performance. [Ribeiro+ 2016]

Promise of Explanations

4

• Model Debugging: reveal spurious correlations or the kinds of

inputs that a model is most likely to have undesirable performance.

Husky

Promise of Explanations

5

• Model Debugging: reveal spurious correlations or the kinds of

inputs that a model is most likely to have undesirable performance.

Explanation

Husky

Promise of Explanations

6

• Model Debugging: reveal spurious correlations or the kinds of

inputs that a model is most likely to have undesirable performance.

Explanation

Husky Fix

Agenda

• Overview of attribution methods
• This talk will mostly focus on post-hoc explanation methods

for deep neural networks.

• The selection conundrum
• Sanity checks & results
• Theoretical justification by Nie. et. al. 2018.
• Passing sanity checks & recent results
• Conclusion

Saliency/Attribution Maps

8

Predictions

Explanation

Corn

Saliency/Attribution Maps

9

Predictions

Explanation

Corn

Attribution maps provide ‘relevance’ scores for each dimension of the input.

Saliency/Attribution Maps

10

Predictions

Explanation

Corn

Attribution maps provide ‘relevance’ scores for each dimension of the input.

S : Rd → RC

E : Rd → Rd

How to compute attribution?

11

Predictions

Attribution

Corn

Egrad(x) = ∂Si ∂x

[SVZ’13]

Some Issues with the Gradient

12

Predictions Corn

‘Visually noisy’, and can violate sensitivity w.r.t. a baseline input [Sundararajan et. al., Shrikumar et. al., and Smilkov et. al.]

Integrated Gradients

13

Predictions Corn

Sum of ‘interior’ gradients.

[STY’17]

SmoothGrad

14

Predictions Corn

Average attribution of ‘noisy’ inputs.

[STKVW’17]

Gradient-Input

15

Predictions Corn

Element-wise product of gradient and input.

Guided BackProp

16

Predictions Corn

Zero out ‘negative’ gradients and ‘activations’ while back-propagating.

Other Learned Kinds

17

Predictions

Explanation

Corn

[FV’17]

Formulate an explanation as through learned patch removal.

Non-Image Settings: Molecules

18

The Selection Conundrum

19

Predictions Corn

The Selection Conundrum

For a particular task and model, how should a developer/researcher select which method to use?

Desirable Properties

• Sensitivity to the parameters of a model to be

explained.

• Depend on the labeling of the data, i.e., reflect

the relationship between inputs and outputs.

Sanity Checks

• We will use randomization as a way to test both

requirements.

• Model parameter randomization test: randomize (re-

initialize) the parameters of a model and now compare attribution maps for a trained model to those derived from a randomized model.

• Data randomization test: compare attribution maps for a

model trained with correct labels to those derived from a model trained with random labels.

Model Parameter Randomization

Inception V3

• Cascading randomization from top to bottom layers.
• Independent layer randomization.

Conjecture: If a model captures higher level class concepts, then saliency maps should change as the model is being randomized.

Model Parameter Randomization

24

Gradient Gradient-SG Gradient Input Guided Back-propagation GradCAM Integrated Gradients Integrated Gradients-SG

Original Explanation

Guided GradCAM

Cascading randomization from top to bottom layers

Original Image

Conjecture: If a model captures higher level class concepts, then saliency maps should change as the model is being randomized.

Model Parameter Randomization

25

Gradient Gradient-SG Gradient Input Guided Back-propagation GradCAM Integrated Gradients Integrated Gradients-SG

logits Original Explanation

Guided GradCAM

Cascading randomization from top to bottom layers

Original Image

Conjecture: If a model captures higher level class concepts, then saliency maps should change as the model is being randomized.

Model Parameter Randomization

26

Gradient Gradient-SG Gradient Input Guided Back-propagation GradCAM Integrated Gradients Integrated Gradients-SG

logits mixed_7c mixed_7b mixed_7a mixed_6e mixed_6d mixed_6c mixed_6b Original Explanation

Guided GradCAM

Cascading randomization from top to bottom layers

Original Image

Conjecture: If a model captures higher level class concepts, then saliency maps should change as the model is being randomized.

Model Parameter Randomization

27

Gradient Gradient-SG Gradient Input Guided Back-propagation GradCAM Integrated Gradients Integrated Gradients-SG

logits conv2d_1a_3x3 mixed_7c mixed_7b conv2d_2a_3x3 conv2d_2b_3x3 conv2d_4a_3x3 mixed_7a mixed_6e mixed_6d mixed_6c mixed_6b mixed_6a mixed_5d mixed_5c mixed_5b conv2d_3b_1x1 Original Explanation

Guided GradCAM

Cascading randomization from top to bottom layers

Original Image

Conjecture: If a model captures higher level class concepts, then saliency maps should change as the model is being randomized.

Model Parameter Randomization

28

Gradient Gradient-SG Gradient Input Guided Back-propagation GradCAM Integrated Gradients Integrated Gradients-SG

logits conv2d_1a_3x3 mixed_7c mixed_7b conv2d_2a_3x3 conv2d_2b_3x3 conv2d_4a_3x3 mixed_7a mixed_6e mixed_6d mixed_6c mixed_6b mixed_6a mixed_5d mixed_5c mixed_5b conv2d_3b_1x1 Original Explanation

Guided GradCAM

Cascading randomization from top to bottom layers

Original Image

Metrics

Inception v3 - ImageNet

Mixed

7c 7b 7a 6e 6d 6c 6b 6a 5d 5c 5b 4a 3b 2b 2a 1a

logits

• riginal

Conv2d

Rank Correlation ABS

Mixed

7c 7b 7a 6e 6d 6c 6b 6a 5d 5c 5b 4a 3b 2b 2a 1a

logits

• riginal

Conv2d

Rank Correlation No ABS

See Caption Note

• Rank correlation of attribution from model with trained weights to

those derived from partially randomized models.

• Attribution sign changes. Roughly similar regions are, however, still

attributed.

Model Parameter Randomization

Successive Randomization of Layers

Gradient Gradient-SG Gradient-VG Guided Backpropagation Guided GradCAM Integrated Gradients Integrated Gradients-SG

conv_hidden1 conv_hidden2 fc2

• utput-fc
• riginal explanation

Original Image Explanation conv_hidden1 conv_hidden2

• utput-fc
• riginal explanation

fc2

Independent Randomization of Layers

CNN MNIST

Medical Setting

Guided Backpropagation Skeletal Radiograph

Age

Data Randomization

32

CNN - MNIST

True Labels Random Labels

Gradient Gradient-SG Guided BackProp GradCAM Guided GradCAM Integrated Gradients Integrated Gradients-SG Gradient Input

True Labels Random Labels

Gradient Gradient-SG Guided BackProp GradCAM Guided GradCAM Integrated Gradients Integrated Gradients-SG Gradient Input

Rank Correlation - Abs Rank Correlation - No Abs

Absolute-Value Visualization Diverging Visualization

Data Randomization

33

True Labels Random Labels True Labels Random Labels Rank Correlation - Abs

Rank Correlation - No Abs MLP - MNIST

Gradient Gradient-SG Guided BackProp Integrated Gradients Integrated Gradients-SG Gradient Input Gradient Gradient-SG Guided BackProp Integrated Gradients Integrated Gradients-SG Gradient Input

Absolute-Value Visualization Diverging Visualization

Some Insights

• Nie et. al. (ICML 2018) theoretical showed that Guided back

propagation is doing input reconstruction.

• Observed in Mahendra et. al. 2014 (ECCV) as well.

Figure from Nie et. al, 2018.

Summary

• We focused on gradient-based methods mostly.
• Sanity checks don’t tell if a method is good,

just if it is invariant.

• Sole visual inspection can be deceiving.

What about other methods

LIME-5 LIME-10 LIME-20 LIME-50 SHAP Gradient SmoothGrad Guided BackProp PatternNet Pattern Attribution Input-Gradient Integrated Gradients LRP-Z LRP-EPS LRP-SPAF LRP-SPBF VGrad DeepTaylor

Cascading randomization from top to bottom layers for VGG-16

{

LIME Variants

{

Not Previously considered in literature.

• Gupta et. al. fix this with competition for gradients (CGI).

A Fix for Sanity Checks

37

[Figure from Gupta et. al. 2019.]

Other Assessment Methods

• Hooker et. al. (to appear at

Neurips 2019) propose to remove and retrain.

• Adel et. al. propose FSM to

‘quantify’ information content.

• Yang et. al. introduce a

benchmark (w/ground truth) and other metrics to assess how well a map captures model behavior.

• ‘Adversarial’ attack on explanations by Ghorbani et. al.

Attacks

39

• Mean-shift attack by Kindermans & Hooker et. al.

Conundrum Persists

• For methods that pass sanity checks how do we

choose among these?

• Can end-users (developers) use these methods

to debug?

• What about other explanation classes (concepts

and global methods)?