Adversarial Examples for Evaluating Reading Comprehension Systems - - PowerPoint PPT Presentation

▶
adversarial examples for evaluating
SMART_READER_LITE
LIVE PREVIEW

Adversarial Examples for Evaluating Reading Comprehension Systems - - PowerPoint PPT Presentation

Feb 24, 2024 338 likes •1.05k views

Adversarial Examples for Evaluating Reading Comprehension Systems Robin Jia and Percy Liang Stanford University Reading Comprehension Task Question: The number of new Huguenot colonists declined after what year? Paragraph: The largest

slide-1
SLIDE 1

Adversarial Examples for Evaluating Reading Comprehension Systems

Robin Jia and Percy Liang Stanford University

slide-2
SLIDE 2

Reading Comprehension Task

Question: “The number of new Huguenot colonists declined after what year?” Paragraph: “The largest portion of the Huguenots to settle in the Cape arrived between 1688 and 1689…but quite a few arrived as late as 1700; thereafter, the numbers declined…” Correct Answer: “1700”

2

Stanford Question Answering Dataset (Rajpurkar et al., 2016)

slide-3
SLIDE 3

Progress on SQuAD

3

SQuAD leaderboard, https://rajpurkar.github.io/SQuAD-explorer/ Human Performance Logistic Regression Baseline

Do these models actually understand language?

slide-4
SLIDE 4

Adversarial Evaluation

Question: “The number of new Huguenot colonists declined after what year?” Paragraph: “The largest portion of the Huguenots to settle in the Cape arrived between 1688 and 1689…but quite a few arrived as late as 1700; thereafter, the numbers declined. The number of old Acadian colonists declined after the year of 1675.” Correct Answer: “1700” Predicted Answer: “1675”

4

Model used: BiDAF Ensemble (Seo et al., 2016)

slide-5
SLIDE 5

Adversarial Evaluation

Question: “The number of new Huguenot colonists declined after what year?” Paragraph: “The largest portion of the Huguenots to settle in the Cape arrived between 1688 and 1689…but quite a few arrived as late as 1700; thereafter, the numbers declined. expected yet later be basis need young only required 1961.” Correct Answer: “1700” Predicted Answer: “1961”

5

Model used: BiDAF Ensemble (Seo et al., 2016)

slide-6
SLIDE 6

Outline

  • Inspiration/Motivation
  • Adding Grammatical Sentences
  • Adding Word Salad
  • Trying to build better systems

6

slide-7
SLIDE 7

Outline

  • Inspiration/Motivation
  • Adding Grammatical Sentences
  • Adding Word Salad
  • Trying to build better systems

7

slide-8
SLIDE 8

Some Inspiration

8

+ .007 * =

Panda

58% confidence

Nematode

8% confidence

Gibbon

99% confidence Goodfellow et al., 2014.

Local perturbations don’t change semantics of image, but models are oversensitive to small differences!

slide-9
SLIDE 9

Local perturbations of text

Question: “The number of new Huguenot colonists declined after what year?” Paragraph: “The largest portion of the Huguenots to settle in the Cape arrived between 1688 and 1689…but quite a few arrived as late as 1700; thereafter, the numbers amount declined decreased…”

Plausible alternative answers not always present Hard to find a lot of perturbations to try

9

Li et al., 2017

slide-10
SLIDE 10

Preserving Semantics

  • For images, most local perturbations preserve

semantics

  • For text, most local perturbations alter semantics
  • Even changing one word by a small amount may

not preserve semantics (e.g. entity names)

10

slide-11
SLIDE 11

Concatenative Adversaries

  • Instead of locally altering the input, append

distracting text to the paragraph

  • Must ensure that added text does not actually

answer the question

11

slide-12
SLIDE 12

Distracting Text

Question: “The number of new Huguenot colonists declined after what year ?” Distracting text: “The number of new Huguenot colonists declined after the year 1675 .” Answer according to text: “1675”

12

slide-13
SLIDE 13

Distracting Text

Question: “The number of new Huguenot colonists declined after what year ?” Distracting text: “The number of new old Huguenot Acadian colonists declined after the year 1675 .” Answer according to text: N/A

13

Local perturbations change semantics of sentence, but models are overly stable/insensitive to these changes!

slide-14
SLIDE 14

Outline

  • Inspiration/Motivation
  • Adding Grammatical Sentences
  • Adding Word Salad
  • Trying to build better systems

14

slide-15
SLIDE 15

Grammatical Distractors

15

What city did Tesla move to in 1880? Prague What city did Tadakatsu move to in 1881? Chicago Tadakatsu moved the city of Chicago to in 1881. Tadakatsu moved to the city of Chicago in 1881.

Change entities, numbers, antonyms Generate fake answer with same NER/POS tag Convert to declarative sentence Have crowdworkers fix errors

slide-16
SLIDE 16

Four “dev” systems

16

SQuAD leaderboard, https://rajpurkar.github.io/SQuAD-explorer/ *Some of our results are on older versions of models than shown here

slide-17
SLIDE 17

Results (4 “dev” systems)

17

System Original AddOneSent BiDAF, ensemble (Seo et al., 2016) 80.0 46.9 BiDAF, single (Seo et al., 2016) 75.5 45.7 Match-LSTM, ensemble (Wang & Jiang, 2016) 75.4 41.8 Match-LSTM, single (Wang & Jiang, 2016) 71.4 39.0 Human Performance 92.6 89.2

slide-18
SLIDE 18

Picking a worst-case sentence

18

Tadakatsu moved the city of Chicago to in 1881. Tadakatsu moved to the city of Chicago in 1881.

Have crowdworkers fix errors

Tadakatsu moved to Chicago in 1881. In 1881, Tadakatsu moved to the city of Chicago. Model failed if distracted by any of these

slide-19
SLIDE 19

Results (4 “dev” systems)

19

System Original AddOneSent AddSent BiDAF, ensemble (Seo et al., 2016) 80.0 46.9 34.2 BiDAF, single (Seo et al., 2016) 75.5 45.7 34.3 Match-LSTM, ensemble (Wang & Jiang, 2016) 75.4 41.8 29.4 Match-LSTM, single (Wang & Jiang, 2016) 71.4 39.0 27.3 Human Performance 92.6 89.2 79.5

slide-20
SLIDE 20

Computers on AddSent

20

Model

What city did Tesla move to in 1880?

Gospić Prague Chicago … Adversarial Paragraph

slide-21
SLIDE 21

Computers on AddSent

21

Model

What city did Tesla move to in 1880?

Gospić Prague Chicago …

Deterministically choose argmax

Adversarial Paragraph

slide-22
SLIDE 22

Humans on AddSent

22

Crowd

What city did Tesla move to in 1880?

Gospić Prague Chicago …

Only get noisy samples!

Adversarial Paragraph

slide-23
SLIDE 23

Humans on AddSent

23

Crowd

What city did Tesla move to in 1880?

Gospić Prague Chicago …

Only get noisy samples!

Adversarial Paragraph

slide-24
SLIDE 24

Humans on AddSent

24

Crowd

What city did Tesla move to in 1880?

Gospić Prague Chicago …

Only get noisy samples!

Adversarial Paragraph #2

slide-25
SLIDE 25

Humans on AddSent

25

Crowd

What city did Tesla move to in 1880?

Gospić Prague Chicago …

Noise augmented when picking worst-case sentence

Adversarial Paragraph #3

slide-26
SLIDE 26

Twelve “test” systems

26

SQuAD leaderboard, https://rajpurkar.github.io/SQuAD-explorer/ *Some of our results are on older versions of models than shown here

slide-27
SLIDE 27

Results (12 “test” systems)

27

System Original AddOneSent AddSent ReasoNet, ensemble (Shen et al., 2017) 81.1 49.8 39.4 SEDT, ensemble (Liu et al., 2017) 80.1 46.5 35.0 Mnemonic Reader, ensemble (Hu et al., 2017) 79.1 55.3 46.2 Ruminating Reader (Gong and Bowman, 2017) 78.8 47.7 37.4 jNet (Zhang et al., 2017) 78.6 47.0 37.9 Mnemonic Reader, single (Hu et al., 2017) 78.5 56.0 46.6 ReasoNet, single (Shen et al., 2017) 78.2 50.3 39.4 MPCM, single (Wang et al., 2016) 77.0 50.0 40.3 SEDT, single (Liu et al., 2017) 76.9 44.8 33.9 RaSOR (Lee et al., 2016) 76.2 49.5 39.5 DCR (Yu et al., 2016) 69.3 45.1 37.8 Logistic Regression (Rajpurkar et al., 2016) 50.4 30.4 23.2

slide-28
SLIDE 28

Results (12 “test” systems)

28

System Original AddOneSent AddSent ReasoNet, ensemble (Shen et al., 2017) 81.1 49.8 39.4 SEDT, ensemble (Liu et al., 2017) 80.1 46.5 35.0 Mnemonic Reader, ensemble (Hu et al., 2017) 79.1 55.3 46.2 Ruminating Reader (Gong and Bowman, 2017) 78.8 47.7 37.4 jNet (Zhang et al., 2017) 78.6 47.0 37.9 Mnemonic Reader, single (Hu et al., 2017) 78.5 56.0 46.6 ReasoNet, single (Shen et al., 2017) 78.2 50.3 39.4 MPCM, single (Wang et al., 2016) 77.0 50.0 40.3 SEDT, single (Liu et al., 2017) 76.9 44.8 33.9 RaSOR (Lee et al., 2016) 76.2 49.5 39.5 DCR (Yu et al., 2016) 69.3 45.1 37.8 Logistic Regression (Rajpurkar et al., 2016) 50.4 30.4 23.2

slide-29
SLIDE 29

Results (12 “test” systems)

29

System Original AddOneSent AddSent ReasoNet, ensemble (Shen et al., 2017) 81.1 49.8 39.4 SEDT, ensemble (Liu et al., 2017) 80.1 46.5 35.0 Mnemonic Reader, ensemble (Hu et al., 2017) 79.1 55.3 46.2 Ruminating Reader (Gong and Bowman, 2017) 78.8 47.7 37.4 jNet (Zhang et al., 2017) 78.6 47.0 37.9 Mnemonic Reader, single (Hu et al., 2017) 78.5 56.0 46.6 ReasoNet, single (Shen et al., 2017) 78.2 50.3 39.4 MPCM, single (Wang et al., 2016) 77.0 50.0 40.3 SEDT, single (Liu et al., 2017) 76.9 44.8 33.9 RaSOR (Lee et al., 2016) 76.2 49.5 39.5 DCR (Yu et al., 2016) 69.3 45.1 37.8 Logistic Regression (Rajpurkar et al., 2016) 50.4 30.4 23.2

slide-30
SLIDE 30

Partial Matches

Question: “The number of new Huguenot colonists declined after what year?” Paragraph: “The largest portion of the Huguenots to settle in the Cape arrived between 1688 and 1689…but quite a few arrived as late as 1700; thereafter, the numbers declined. The number of old Acadian colonists declined after the year of 1675.”

30

All models distracted by sentences with

  • nly partial match with the question
slide-31
SLIDE 31

Partial Matches

Question: “The number of new Huguenot colonists declined after what year?” Paragraph: “The largest portion of the Huguenots to settle in the Cape arrived between 1688 and 1689, in seven ships as part of the organised migration, but quite a few arrived as late as 1700; thereafter, the numbers declined, and only small groups arrived at a time.” Correct Answer: “1700”

31

Stanford Question Answering Dataset (Rajpurkar et al., 2016)

slide-32
SLIDE 32

Outline

  • Inspiration/Motivation
  • Adding Grammatical Sentences
  • Adding Word Salad
  • Trying to build better systems

32

slide-33
SLIDE 33

Adversarial Word Salad

  • So far, only explored tiny fraction of possible

distractors

  • Try adding any ungrammatical sequence of

words

  • Incoherent text cannot provide evidence for an

incorrect answer

33

slide-34
SLIDE 34

AddAny

34

What city did Tesla move to in 1880? In January 1880, two of Tesla's uncles put together enough money to help him leave Gospić for Prague… Model predicts: “Prague”

Model used: BiDAF Ensemble (Seo et al., 2016)

slide-35
SLIDE 35

AddAny

35

What city did Tesla move to in 1880? In January 1880, two of Tesla's uncles put together enough money to help him leave Gospić for Prague…heavy industry art countries applied design theory even medical process. Add random common words

slide-36
SLIDE 36

AddAny

36

What city did Tesla move to in 1880? In January 1880, two of Tesla's uncles put together enough money to help him leave Gospić for Prague…heavy industry art countries applied design theory even medical process. Pick one word at random

slide-37
SLIDE 37

AddAny

37

What city did Tesla move to in 1880? In January 1880, two of Tesla's uncles put together enough money to help him leave Gospić for Prague…heavy industry art countries applied design city even medical process. Replace with another common word or question word, to increase probability that model gives a wrong answer

slide-38
SLIDE 38

AddAny

38

What city did Tesla move to in 1880? In January 1880, two of Tesla's uncles put together enough money to help him leave Gospić for Prague…heavy industry art countries applied design city even medical process. Pick one word at random

slide-39
SLIDE 39

AddAny

39

What city did Tesla move to in 1880? In January 1880, two of Tesla's uncles put together enough money to help him leave Gospić for Prague…heavy industry art countries what design city even medical process. Replace with another common word or question word, to increase probability that model gives a wrong answer

slide-40
SLIDE 40

AddAny

40

What city did Tesla move to in 1880? In January 1880, two of Tesla's uncles put together enough money to help him leave Gospić for Prague…what 30 city 1880 what move city city medical move. Model predicts: “medical”

Model used: BiDAF Ensemble (Seo et al., 2016)

slide-41
SLIDE 41

AddAny Results

41

System Original AddOneSent AddSent AddAny BiDAF, ensemble 80.0 46.9 34.2 2.7 BiDAF, single 75.5 45.7 34.3 4.8 Match-LSTM, ensemble 75.4 41.8 29.4 11.7 Match-LSTM, single 71.4 39.0 27.3 7.6

Models can be fooled on almost any example

slide-42
SLIDE 42

Some Inspiration

42

+ .007 * =

Panda

58% confidence

Nematode

8% confidence

Gibbon

99% confidence Goodfellow et al., 2014.

slide-43
SLIDE 43

AddAny

43

What city did Tesla move to in 1880? In January 1880, two of Tesla's uncles put together enough money to help him leave Gospić for Prague…what 30 city 1880 what move city city medical move. Model predicts: “medical”

Model used: BiDAF Ensemble (Seo et al., 2016)

slide-44
SLIDE 44

AddCommon

44

What city did Tesla move to in 1880? In January 1880, two of Tesla's uncles put together enough money to help him leave Gospić for Prague… Model predicts: “Prague”

Model used: BiDAF Ensemble (Seo et al., 2016)

slide-45
SLIDE 45

AddCommon

45

What city did Tesla move to in 1880? In January 1880, two of Tesla's uncles put together enough money to help him leave Gospić for Prague…heavy industry art countries applied design theory even medical process. Add random common words

slide-46
SLIDE 46

AddCommon

46

What city did Tesla move to in 1880? In January 1880, two of Tesla's uncles put together enough money to help him leave Gospić for Prague…heavy industry art countries applied design theory even medical process. Pick one word at random

slide-47
SLIDE 47

AddCommon

47

What city did Tesla move to in 1880? In January 1880, two of Tesla's uncles put together enough money to help him leave Gospić for Prague…heavy industry art countries applied design around even medical process. Replace with another common word, to increase probability that model gives a wrong answer

slide-48
SLIDE 48

AddCommon

48

What city did Tesla move to in 1880? In January 1880, two of Tesla's uncles put together enough money to help him leave Gospić for Prague…finally back would move york hotel through then immediately later. Model predicts: “york hotel”

Model used: BiDAF Ensemble (Seo et al., 2016)

slide-49
SLIDE 49

AddCommon Results

49

System Original AddOneSent AddSent AddAny AddCommon BiDAF, ensemble 80.0 46.9 34.2 2.7 52.6 BiDAF, single 75.5 45.7 34.3 4.8 41.7 Match-LSTM, ensemble 75.4 41.8 29.4 11.7 51.0 Match-LSTM, single 71.4 39.0 27.3 7.6 38.9

slide-50
SLIDE 50

AddCommon Errors

Question: “What type of markets is the dwelling type below?” Distracting text: “be therefore dark business business

  • ther system type feet above.”

Predicted Answer: “dark business”

50

Model used: BiDAF Ensemble (Seo et al., 2016)

slide-51
SLIDE 51

AddCommon Errors

Question: “After the operators are warned by the escape of the steam, what may they then do?” Distracting text: “came followed after then such then increased hand law may.” Predicted Answer: “increased hand law”

51

Model used: BiDAF Ensemble (Seo et al., 2016)

slide-52
SLIDE 52

AddCommon Errors

Question: “Where did he claim the blueprint was stored?” Distracting text: “doubt was did about carried wasn’t year 1961 near policy.” Predicted Answer: “near policy”

52

Model used: BiDAF Ensemble (Seo et al., 2016)

slide-53
SLIDE 53

AddCommon Errors

Question: “What act sets the term for judging the boundaries of sanity to which individuals wishing to sit

  • n the SP must adhere?”

Distracting text: “english our programs industry religion size ran maybe leave poor.” Predicted Answer: “British Nationality Act 1981”

53

Model used: BiDAF Ensemble (Seo et al., 2016)

slide-54
SLIDE 54

AddCommon Errors

Question: “What act sets the term for judging the boundaries of sanity to which individuals wishing to sit on the SP must adhere?” Paragraph: “As in the House of Commons, a number of qualifications apply to being an MSP. Such qualifications were introduced under the House of Commons Disqualification Act 1975 and the British Nationality Act 1981. Specifically, members must be over the age of 18 and must be a citizen of the United Kingdom, the Republic of Ireland, one of the countries in the Commonwealth of Nations, a citizen of a British overseas territory, or a European Union citizen resident in the UK. Members of the police and the armed forces are disqualified from sitting in the Scottish Parliament as elected MSPs, and similarly, civil servants and members of foreign legislatures are disqualified. An individual may not sit in the Scottish Parliament if he or she is judged to be insane under the terms of the Mental Health (Care and Treatment) (Scotland) Act 2003. english our programs industry religion size ran maybe leave poor.” Correct Answer: “Mental Health (Care and Treatment) (Scotland) Act 2003” Predicted Answer: “British Nationality Act 1981”

54

slide-55
SLIDE 55

Outline

  • Inspiration/Motivation
  • Adding Grammatical Sentences
  • Adding Word Salad
  • Trying to build better systems

55

slide-56
SLIDE 56

What can we do?

  • We’ve identified weaknesses in existing

models—how can we fix them?

56

slide-57
SLIDE 57

Adversarial Training

  • What if we train on these adversarial examples?
  • Run AddSent without crowdsourcing on training

data

57

What city did Tesla move to in 1880? Prague What city did Tadakatsu move to in 1881? Chicago Tadakatsu moved the city of Chicago to in 1881.

Change entities, numbers, antonyms Generate fake answer with same NER/POS tag Convert to declarative sentence

slide-58
SLIDE 58

Adversarial Training

58

10 20 30 40 50 60 70 80 Standard Training Data Adversarial Training Data Original AddSent AddSentMod Model used: BiDAF Single (Seo et al., 2016)

slide-59
SLIDE 59

Adversarial Training

  • Has the model really learned the right thing?
  • Create AddSentMod, similar to AddSent
  • Add sentences to beginning instead of end
  • Use different set of fake answers

59

Prague Chicago

Generate fake answer with same NER/POS tag

slide-60
SLIDE 60

Stockholm

Adversarial Training

  • Has the model really learned the right thing?
  • Create AddSentMod, similar to AddSent
  • Add sentences to beginning instead of end
  • Use different set of fake answers

60

Prague

Generate fake answer with same NER/POS tag

slide-61
SLIDE 61

Adversarial Training

  • Easy to overfit to a given adversary
  • Similar patterns observed with adversarial training

in computer vision

61

10 20 30 40 50 60 70 80 Standard Training Data Adversarial Training Data Original AddSent AddSentMod

slide-62
SLIDE 62

Future Work

  • Iteratively collect data that’s hard for the model

as it trains

  • Adversary must be general enough so that
  • verfitting not an issue

62

slide-63
SLIDE 63

Thank you!

All code, data, and experiments available on http://tiny.cc/adversarial-squad-codalab Thanks to our funding sources!

63

slide-64
SLIDE 64

How good are today’s systems?

64

System SQuAD F1 Score Interactive AoA Reader, ensemble (HIT + iFLYTEK) 85.3 r-net, ensemble (Microsoft Research Asia) 84.7 r-net, single (Micorosft Research Asia) 83.5 smarnet, ensemble (Eigen Technology & Zhejiang Univ.) 83.5 DCN+, single (Salesforce Research) 82.8 MEMEN, ensemble (Eigen Technology & Zhejiang Univ.) 82.7 ReasoNet, ensemble (Microsoft Research Redmond) 82.6 Mnemonic Reader, ensemble (NUDT & Fudan Univ.) 82.4 Human Performance 91.2

SQuAD leaderboard, https://rajpurkar.github.io/SQuAD-explorer/

slide-65
SLIDE 65

Errors due to distracting text

65

20 40 60 80 100 MatchLSTM single MatchLSTM ensemble BiDAF single BiDAF ensemble Human Percent with wrong answer in distractor sentence

slide-66
SLIDE 66

Adversary Generalization

  • Do adversarial examples generated to fool one

system also fool other systems?

66

slide-67
SLIDE 67

AddSent Generalization

10 20 30 40 50 60 70 80

Match, single Match, ensemble BiDAF, single BiDAF, ensemble F1 Score Match Single Data Match Ensemble Data BiDAF Single Data BiDAF Ensemble Data AddOneSent Data

67

Border: data targeting current model

slide-68
SLIDE 68

AddAny Generalization

10 20 30 40 50 60 70 80

Match, single Match, ensemble BiDAF, single BiDAF, ensemble F1 Score Match Single Data Match Ensemble Data

68

Border: data targeting current model

slide-69
SLIDE 69

Conclusion

  • Evaluation metrics are important!
  • Existing models are deficient in many ways
  • Some errors can be explained; others are more

unintuitive

69