Attention and its (mis)interpretation Danish Pruthi 1 - - PowerPoint PPT Presentation

Attention and its (mis)interpretation

Danish Pruthi

1

Acknowledgements

2

Mansi Gupta Bhuwan Dhingra Zachary C. Lipton Graham Neubig

Outline

• 1. What is attention mechanism?
• 2. Attention-as-explanations
• 3. Manipulating attention weights
• 4. Results and discussion
• 5. Conclusion

3

Outline

• 1. What is attention mechanism?
• 2. Attention-as-explanations
• 3. Manipulating attention weights
• 4. Results and discussion
• 5. Conclusion

4

Pre-attention era

5

Pre-attention era

यह है एक उदाहरण </s>

5

Pre-attention era

यह है एक उदाहरण </s>

LSTM LSTM LSTM LSTM LSTM

Encoder

5

Pre-attention era

यह है एक उदाहरण </s>

LSTM LSTM LSTM LSTM LSTM

Encoder

This example is an

LSTM LSTM LSTM LSTM LSTM

This example is an </s> <s>

Decoder

5

Pre-attention era

यह है एक उदाहरण </s>

LSTM LSTM LSTM LSTM LSTM

Encoder

This example is an

LSTM LSTM LSTM LSTM LSTM

This example is an </s> <s>

Decoder

5

Sentence Representations

Problem: “You can’t cram the meaning of a whole %&!$ing sentence into a single $&!*ing vector!” — Ray Mooney Solution: Use attention (Bahdanau et al. 2015)

6

Basic Idea


Bahdanau et al. 2015

• Encode each word in the sentence into a vector
• When decoding, perform a linear combination of these

vectors, weighted by “attention weights”

• Use this combination in picking the next word

7

Attention

8

Attention

यह है एक उदाहरण

8

Attention

यह है एक उदाहरण This is an <s>

8

Attention

यह है एक उदाहरण This is an <s>

Key vectors

8

Attention

यह है एक उदाहरण This is an <s>

Key vectors Query vector

8

Attention

यह है एक उदाहरण This is an <s>

Key vectors Query vector

compute attention scores

• 0.1

0.3

• 1.0

2.1

8

Attention

यह है एक उदाहरण This is an <s>

Key vectors Query vector

compute attention scores Softmax

• 0.1

0.3

• 1.0

2.1 0.08 0.13 0.03 0.76

8

Attention

यह है एक उदाहरण This is an <s>

Key vectors Query vector

8

Attention

यह है एक उदाहरण This is an <s>

Key vectors Query vector 0.08 0.13 0.03 0.76 * * * * Context vector

8

Attention

यह है एक उदाहरण This is an <s>

Key vectors Query vector 0.08 0.13 0.03 0.76 * * * *

example

Context vector

8

Score Functions

• Dot-Product 


attention

• Bi-linear 


attention

• MLP 


attention

• Scaled dot-product 


attention
 score(st, hi) = v⊤

atanh(Wa[st; hi])

score(st, hi) = s⊤

t hi

score(st, hi) = s⊤

t hi

n score(st, hi) = s⊤

t Wahi

9

Score Functions

• Dot-Product 


attention

• Bi-linear 


attention

• MLP 


attention

• Scaled dot-product 


attention
 score(st, hi) = v⊤

atanh(Wa[st; hi])

score(st, hi) = s⊤

t hi

score(st, hi) = s⊤

t hi

n score(st, hi) = s⊤

t Wahi

9

Outline

• 1. What is attention mechanism?
• 2. Attention-as-explanations
• 3. Manipulating attention weights
• 4. Results and discussion
• 5. Conclusion

10

Attention as explanation

• Used by model-developers to explain models' predictions

Image captioning

Xu et al, 2015

Entailment

Rocktäschel et al, 2015

11

Attention as explanation

• Used by model-developers to explain models' predictions

Image captioning

Xu et al, 2015

BERTViz

Vig et al, 2019

Document classification

Yang et al, 2016

and many others…

11

Attention as explanation

12

"By inspecting the network’s attention, for instance by visually highlighting attention weights, one could attempt to investigate and understand the outcome of neural networks. Hence, weight visualization is now common practice."

Galassi et al., 2019

• Used by model-developers to explain models' predictions

Attention as explanation

• Used by model-developers to explain models' predictions
• Used by practitioners to audit models for bias, fairness,

accountability, etc

Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting,

De-Arteaga, et al, 2019

13

Attention-as-explanation in FAT* contexts


* Fairness, accountability and transparency

De-Arteaga et al., 2019

• Use attention mechanism to identify gender bias in
• ccupation prediction models used as a part of high-

stakes job recommendation models


14

Attention-as-explanation in FAT* contexts


* Fairness, accountability and transparency

De-Arteaga et al., 2019

• Use attention mechanism to identify gender bias in
• ccupation prediction models used as a part of high-

stakes job recommendation models
 "The attention weights indicate which tokens are the most predictive"

14

Attention-as-explanation in FAT* contexts


* Fairness, accountability and transparency

De-Arteaga et al., 2019

• Use attention mechanism to identify gender bias in
• ccupation prediction models used as a part of high-

stakes job recommendation models
 "The attention weights indicate which tokens are the most predictive" We question this assumption: does attention necessarily indicate most predictive tokens?

14

Outline

• 1. What Is attention mechanism?
• 2. Attention-as-explanations in the FAT* context


* Fairness, accountability and transparency

• 3. Manipulating attention weights
• 4. Results and discussion
• 5. Conclusion

15

• Setup tasks such that we know certain features a-priori to

be useful for prediction

• Measure “attention mass” on these tokens
• Examine if the models can be manipulated
• What is the price to pay?

Setup

16

17

Classification Tasks

17

Task Input Example

Classification Tasks

17

Task Input Example

Occupation Prediction (Physician vs Surgeon)

• Ms. X practices medicine in Memphis, TN. Ms. X

speaks English and Spanish.

Classification Tasks

17

Task Input Example

Occupation Prediction (Physician vs Surgeon)

• Ms. X practices medicine in Memphis, TN. Ms. X

speaks English and Spanish. Gender Identification After that, Austen was educated at home until she went to boarding school early in 1785

Classification Tasks

17

Task Input Example

Occupation Prediction (Physician vs Surgeon)

• Ms. X practices medicine in Memphis, TN. Ms. X

speaks English and Spanish. Gender Identification After that, Austen was educated at home until she went to boarding school early in 1785 Sentiment Analysis 
 (SST + Wikipedia) Good acting, good dialogue, good cinematography. Helen Reddy is an Australian singer and activist.

Classification Tasks

17

Task Input Example

Occupation Prediction (Physician vs Surgeon)

• Ms. X practices medicine in Memphis, TN. Ms. X

speaks English and Spanish. Gender Identification After that, Austen was educated at home until she went to boarding school early in 1785 Sentiment Analysis 
 (SST + Wikipedia) Good acting, good dialogue, good cinematography. Helen Reddy is an Australian singer and activist. Acceptance Prediction (Reference Letters) It is with pleasure that I am writing this letter...I highly recommend her for your institution. Percentile:99.0 Rank:Extraordinary.

Classification Tasks

Need for impermissible tokens

18

Need for impermissible tokens

25 50 75 100 Task Occupation 
 Prediction Gender 
 Identification SST + Wiki Reference 
 Letters

With Without

18

Need for impermissible tokens

25 50 75 100 Task Occupation 
 Prediction Gender 
 Identification SST + Wiki Reference 
 Letters

With Without

93.8 96.4

18

Need for impermissible tokens

25 50 75 100 Task Occupation 
 Prediction Gender 
 Identification SST + Wiki Reference 
 Letters

With Without

72.8 93.8 100 96.4

18

Need for impermissible tokens

25 50 75 100 Task Occupation 
 Prediction Gender 
 Identification SST + Wiki Reference 
 Letters

With Without

50.4 72.8 93.8 90.8 100 96.4

18

Need for impermissible tokens

25 50 75 100 Task Occupation 
 Prediction Gender 
 Identification SST + Wiki Reference 
 Letters

With Without

74.7 50.4 72.8 93.8 77.5 90.8 100 96.4

18

19

Sequence-to-sequence Tasks

19

Task Example

Sequence-to-sequence Tasks

19

Task Example Bigram Flipping {w1, w2 … w2n-1, w2n} → {w2, w1, … w2n, w2n-1}

Sequence-to-sequence Tasks

19

Task Example Bigram Flipping {w1, w2 … w2n-1, w2n} → {w2, w1, … w2n, w2n-1} Sequence Copying {w1,w2, … wn-1, wn} → {w1,w2, … wn, wn-1}

Sequence-to-sequence Tasks

19

Task Example Bigram Flipping {w1, w2 … w2n-1, w2n} → {w2, w1, … w2n, w2n-1} Sequence Copying {w1,w2, … wn-1, wn} → {w1,w2, … wn, wn-1} Sequence Reversal {w1,w2, … wn-1, wn} → {wn,wn-1, … w2, w1}

Sequence-to-sequence Tasks

19

Task Example Bigram Flipping {w1, w2 … w2n-1, w2n} → {w2, w1, … w2n, w2n-1} Sequence Copying {w1,w2, … wn-1, wn} → {w1,w2, … wn, wn-1} Sequence Reversal {w1,w2, … wn-1, wn} → {wn,wn-1, … w2, w1} English - German MT This is an example. → Dieser ist ein Beispiel.

Sequence-to-sequence Tasks

Manipulating Attention

20

• Let be the impermissible tokens, m is the mask


𝖩

Manipulating Attention

20

• Let be the impermissible tokens, m is the mask


𝖩

• For any task-specific loss function, a penalty term is added


Manipulating Attention

20

• Let be the impermissible tokens, m is the mask


𝖩

• For any task-specific loss function, a penalty term is added


• The penalty term penalizes the model for allocating attention to

impermissible tokens
 


Manipulating Attention

20

Manipulating Attention

21

Manipulating Attention

Total attention mass

• n all the "allowed" tokens

21

Manipulating Attention

Total attention mass

• n all the "allowed" tokens

Penalty coefficient that modulates attention on impermissible tokens

21

Manipulating Attention

Total attention mass

• n all the "allowed" tokens

Penalty coefficient that modulates attention on impermissible tokens

21

• Side note: In a parallel work, Wiegreffe and Pinter

(2019) propose a different penalty term

Manipulating Attention

22

Manipulating Attention

• Multiple attention heads

22

Manipulating Attention

• Multiple attention heads
• Optimizing the mean over a set of attention heads


22

Manipulating Attention

• Multiple attention heads
• Optimizing the mean over a set of attention heads


• One of the attention heads can be assigned a large amount
• f attention to impermissible tokens


22

Outline

• 1. What Is attention mechanism?
• 2. Attention-as-explanations
• 3. Manipulating attention weights
• 4. Results and discussion
• 5. Conclusion

23

BiLSTM + Attention

x1

biLSTM biLSTM biLSTM

x2

…..

biLSTM

xn x3

αn α1 α2 α3

y

24

x1 x2 xn x3

αn α1 α2 α3

y

Embedding + Attention

(No recurrent connections)

25

Transformer-based Model

Devlin et. al

26

Transformer-based Model

Devlin et. al

26

Restricted BERT

Good Movie [SEP] [CLS] Good Movie [SEP] [CLS]

L12 L.. L1

Predictions

L0

Original

27

Restricted BERT

Good Movie [SEP] [CLS] Good Movie [SEP] [CLS]

L12 L.. L1

Predictions

L0 L0 L12 L.. L1

Good Movie [SEP]

Capital

Delhi [SEP] [CLS]

Predictions

Impermissible Permissible

Original Restricted

27

Occupation Prediction

28

25 50 75 100 Attention type Original Manipulated 
 (λ = 0.1) Manipulated 
 (λ = 1.0)

Accuracy Attention Mass

Occupation Prediction

28

25 50 75 100 Attention type Original Manipulated 
 (λ = 0.1) Manipulated 
 (λ = 1.0)

Accuracy Attention Mass

99.7 97.2

Occupation Prediction

28

25 50 75 100 Attention type Original Manipulated 
 (λ = 0.1) Manipulated 
 (λ = 1.0)

Accuracy Attention Mass

99.7 97.1 97.2

Occupation Prediction

28

25 50 75 100 Attention type Original Manipulated 
 (λ = 0.1) Manipulated 
 (λ = 1.0)

Accuracy Attention Mass

99.7 97.4 97.1 97.2

Occupation Prediction

28

29

Classification Tasks

29

Classification Tasks

29

Classification Tasks

29

Classification Tasks

29

Classification Tasks

29

Classification Tasks

29

Classification Tasks

Alternate mechanisms

Gender-Identification

30

Alternate mechanisms

Gender-Identification

At inference time, what if we hard set the corresponding attention mass to ZERO?

30

Alternate mechanisms

Gender-Identification

At inference time, what if we hard set the corresponding attention mass to ZERO?

50 % 100%

30

Bigram Flip

31

25 50 75 100 Attention type Original None Uniform Manipulated

Accuracy Attention Mass

Bigram Flip

31

25 50 75 100 Attention type Original None Uniform Manipulated

Accuracy Attention Mass

94.5 100

Bigram Flip

31

25 50 75 100 Attention type Original None Uniform Manipulated

Accuracy Attention Mass

94.5 96.5 100

Bigram Flip

31

25 50 75 100 Attention type Original None Uniform Manipulated

Accuracy Attention Mass

5.2 94.5 97.9 96.5 100

Bigram Flip

31

25 50 75 100 Attention type Original None Uniform Manipulated

Accuracy Attention Mass

0.4 5.2 94.5 99.9 97.9 96.5 100

Bigram Flip

31

Bigram Flip

Original

32

Bigram Flip

Original Manipulated

32

Bigram Flip

Original Manipulated A different seed

32

Sequence Copy

33

25 50 75 100 Attention type Original None Uniform Manipulated

Accuracy Attention Mass

Sequence Copy

33

25 50 75 100 Attention type Original None Uniform Manipulated

Accuracy Attention Mass

98.8 100

Sequence Copy

33

25 50 75 100 Attention type Original None Uniform Manipulated

Accuracy Attention Mass

98.8 84.1 100

Sequence Copy

33

25 50 75 100 Attention type Original None Uniform Manipulated

Accuracy Attention Mass

5.2 98.8 93.8 84.1 100

Sequence Copy

33

25 50 75 100 Attention type Original None Uniform Manipulated

Accuracy Attention Mass

0.01 5.2 98.8 99.9 93.8 84.1 100

Sequence Copy

33

Sequence Copy

Original

34

Sequence Copy

Original Manipulated

34

Sequence Copy

Original Manipulated A different seed

34

Sequence Reverse

35

25 50 75 100 Attention type Original None Uniform Manipulated

Accuracy Attention Mass

Sequence Reverse

35

25 50 75 100 Attention type Original None Uniform Manipulated

Accuracy Attention Mass

94.1 100

Sequence Reverse

35

25 50 75 100 Attention type Original None Uniform Manipulated

Accuracy Attention Mass

94.1 84.1 100

Sequence Reverse

35

25 50 75 100 Attention type Original None Uniform Manipulated

Accuracy Attention Mass

4.7 94.1 88.1 84.1 100

Sequence Reverse

35

25 50 75 100 Attention type Original None Uniform Manipulated

Accuracy Attention Mass

0.02 4.7 94.1 99.8 88.1 84.1 100

Sequence Reverse

35

Sequence Reverse

Original

36

Sequence Reverse

Original Manipulated

36

Sequence Reverse

Original Manipulated A different seed

36

English German MT

37

7.5 15 22.5 30 Attention type Original None Uniform Manipulated
 (λ = 1.0) Manipulated
 (λ = 0.1)

BLEU Attention Mass

English German MT

37

7.5 15 22.5 30 Attention type Original None Uniform Manipulated
 (λ = 1.0) Manipulated
 (λ = 0.1)

BLEU Attention Mass

20.7 24.4

English German MT

37

7.5 15 22.5 30 Attention type Original None Uniform Manipulated
 (λ = 1.0) Manipulated
 (λ = 0.1)

BLEU Attention Mass

20.7 14.9 24.4

English German MT

37

7.5 15 22.5 30 Attention type Original None Uniform Manipulated
 (λ = 1.0) Manipulated
 (λ = 0.1)

BLEU Attention Mass

5.9 20.7 18.5 14.9 24.4

English German MT

37

7.5 15 22.5 30 Attention type Original None Uniform Manipulated
 (λ = 1.0) Manipulated
 (λ = 0.1)

BLEU Attention Mass

1.1 5.9 20.7 20.6 18.5 14.9 24.4

English German MT

37

7.5 15 22.5 30 Attention type Original None Uniform Manipulated
 (λ = 1.0) Manipulated
 (λ = 0.1)

BLEU Attention Mass

7 1.1 5.9 20.7 23.7 20.6 18.5 14.9 24.4

English German MT

37

Alternative workarounds

• Through recurrent connections, if they exist.

• Increase in the magnitude of representations

corresponding to impermissible tokens.

38

Human studies

• We present inputs for the task Occupation Prediction and

the predicted outputs (Physician or Surgeon) by one of the models

• We notify the annotators that the input tokens are

highlighted on the basis of an “explanation method” (attention weights)

• We ask the annotators two rating questions

39

Human studies

• Q1: Do you think that this prediction was influenced by

the gender of the individual?

40

Human studies

• Q1: Do you think that this prediction was influenced by

the gender of the individual?

40

Manipulation type

Input example 


Predicted label - Physician

Percentage of sentences (yes)

Human studies

• Q1: Do you think that this prediction was influenced by

the gender of the individual?

40

Manipulation type

Input example 


Predicted label - Physician

Percentage of sentences (yes) No manipulation

• ms. UNK practices medicine in UNK and specializes

in urological surgery. ms. UNK is affiliated with menorah medical center

66%

Human studies

• Q1: Do you think that this prediction was influenced by

the gender of the individual?

40

Manipulation type

Input example 


Predicted label - Physician

Percentage of sentences (yes) No manipulation

• ms. UNK practices medicine in UNK and specializes

in urological surgery. ms. UNK is affiliated with menorah medical center

66% Ours

• ms. UNK practices medicine in UNK and specializes

in urological surgery. ms. UNK is affiliated with menorah medical center

0%

Human studies

• Q1: Do you think that this prediction was influenced by

the gender of the individual?

40

Manipulation type

Input example 


Predicted label - Physician

Percentage of sentences (yes) No manipulation

• ms. UNK practices medicine in UNK and specializes

in urological surgery. ms. UNK is affiliated with menorah medical center

66% Ours

• ms. UNK practices medicine in UNK and specializes

in urological surgery. ms. UNK is affiliated with menorah medical center

0% Weigraff et al, 2019

• ms. UNK practices medicine in UNK and specializes

in urological surgery. ms. UNK is affiliated with menorah medical center

0%

Human studies

• Q2: Do you believe that highlighted tokens capture the

model’s prediction?

41

Manipulation type

Input example 


Predicted label - Physician

Rating
 (1 to 4) No manipulation

• ms. UNK practices medicine in UNK and specializes

in urological surgery. ms. UNK is affiliated with menorah medical center

3.0 / 4 Ours

• ms. UNK practices medicine in UNK and specializes

in urological surgery. ms. UNK is affiliated with menorah medical center

2.67 / 4 Weigraff et al, 2019

• ms. UNK practices medicine in UNK and specializes

in urological surgery. ms. UNK is affiliated with menorah medical center

1.0 / 4

Outline

• 1. What Is attention mechanism?
• 2. Attention-as-explanations
• 3. Manipulating attention weights
• 4. Results and discussion
• 5. Conclusion

42

Conclusion

43

Conclusion

• In organic cases, typically attention is high for the 'right'
• tokens. Consistent across different seeds.

43

Conclusion

• In organic cases, typically attention is high for the 'right'
• tokens. Consistent across different seeds.
• Often attention is easy to manipulate with negligible drop

in accuracy.

43

Conclusion

• In organic cases, typically attention is high for the 'right'
• tokens. Consistent across different seeds.
• Often attention is easy to manipulate with negligible drop

in accuracy.

• Models with manipulated attention often perform better

compared against models with no or uniform attention.

43

Conclusion

• In organic cases, typically attention is high for the 'right'
• tokens. Consistent across different seeds.
• Often attention is easy to manipulate with negligible drop

in accuracy.

• Models with manipulated attention often perform better

compared against models with no or uniform attention.

• Multiple possible ways to find alternate mechanisms that

are not consistent with one another.

43

THANK YOU FOR 
 YOUR

ATTENTION

Questions?

44

Discussion points

• "maybe we can come up with techniques and metrics to

compute the reliability of attention for an explanation, for a general model"

• "While the paper points out a major problem in the way

attention is conceived, it does not make any effort to offer a solution."

45

Discussion points

• "I would have loved to see some more work on showing

that if [accuracy] scores were retained even after changing the attention weights, then what exactly is the model focussing on for its predictions"

46