It Takes Two to Tango: Towards Theory of AIs Mind It Takes Two to - - PowerPoint PPT Presentation

▶
it takes two to tango towards theory of ai s mind it
SMART_READER_LITE
LIVE PREVIEW

It Takes Two to Tango: Towards Theory of AIs Mind It Takes Two to - - PowerPoint PPT Presentation

Feb 26, 2024 47 likes •1.95k views

It Takes Two to Tango: Towards Theory of AIs Mind It Takes Two to Tango: Towards Theory of AIs Mind * Arjun Deshraj Prithvijit Viraj Prabhu Devi Parikh * * * Chandrasekaran Yadav Chattopadhyay Georgia Tech Georgia Tech

slide-1
SLIDE 1
slide-2
SLIDE 2

It Takes Two to Tango: Towards Theory of AI’s Mind

slide-3
SLIDE 3

It Takes Two to Tango: Towards Theory of AI’s Mind

Viraj Prabhu

Georgia Tech

Devi Parikh

Georgia Tech

Arjun Chandrasekaran

Georgia Tech

Deshraj Yadav

Georgia Tech

Prithvijit Chattopadhyay

Georgia Tech

* * * *

slide-4
SLIDE 4

Slide: 2

slide-5
SLIDE 5

Outline

Slide: 2

Theory of AI’s mind (ToAIM): Motivation Theory of AI’s mind (ToAIM): Experimental Setup and Results Theory of AI’s mind (ToAIM): Take-away messages Theory of AI’s mind (ToAIM): Ongoing work - Human-AI Games

slide-6
SLIDE 6

Outline

Slide: 2

Theory of AI’s mind (ToAIM): Motivation Theory of AI’s mind (ToAIM): Experimental Setup and Results Theory of AI’s mind (ToAIM): Take-away messages Theory of AI’s mind (ToAIM): Ongoing work - Human-AI Games

slide-7
SLIDE 7

Theory of Mind (ToM)

Slide: 3

slide-8
SLIDE 8

Theory of Mind (ToM)

  • Ability to attribute mental states to others

Slide: 3

slide-9
SLIDE 9

Theory of Mind (ToM)

  • Ability to attribute mental states to others

Humans attribute mental states to fellow humans

Slide: 3

slide-10
SLIDE 10

Theory of Mind (ToM)

  • Ability to attribute mental states to others

Humans attribute mental states to fellow humans

Slide: 3

slide-11
SLIDE 11

Theory of Mind (ToM)

  • Ability to attribute mental states to others

Humans attribute mental states to fellow humans Make reasonable inferences about their behavior

Slide: 3

slide-12
SLIDE 12

Theory of Mind (ToM)

  • Ability to attribute mental states to others

Humans attribute mental states to fellow humans Make reasonable inferences about their behavior

  • Crucial for collaborative team-performance

Slide: 3

slide-13
SLIDE 13

Motivation

Slide: 4

slide-14
SLIDE 14

Motivation

  • Traditional AI Research

Slide: 4

slide-15
SLIDE 15

Motivation

  • Traditional AI Research

AI more accurate AI more human-like

Slide: 4

slide-16
SLIDE 16

Motivation

  • Traditional AI Research

AI more accurate AI more human-like

Slide: 4

slide-17
SLIDE 17

Motivation

Slide: 5

slide-18
SLIDE 18

Motivation

  • As AI progresses rise of collaborative work with AI agents

Slide: 5

slide-19
SLIDE 19

Motivation

  • As AI progresses rise of collaborative work with AI agents

Siri

Slide: 5

slide-20
SLIDE 20

Motivation

  • As AI progresses rise of collaborative work with AI agents

Siri Cortana

Slide: 5

slide-21
SLIDE 21

Motivation

  • As AI progresses rise of collaborative work with AI agents

Siri Cortana Google Assistant

Slide: 5

slide-22
SLIDE 22

Motivation

  • As AI progresses rise of collaborative work with AI agents

Siri Cortana Google Assistant

Slide: 5

Personal Assistants

slide-23
SLIDE 23

Motivation

  • As AI progresses rise of collaborative work with AI agents

Siri Cortana Google Assistant IBM Watson

Slide: 5

Personal Assistants

slide-24
SLIDE 24

Motivation

  • As AI progresses rise of collaborative work with AI agents

Siri Cortana Google Assistant IBM Watson

Slide: 5

Self-driving Cars Personal Assistants

slide-25
SLIDE 25

Motivation

  • As AI progresses rise of collaborative work with AI agents

Siri Cortana Google Assistant IBM Watson

Slide: 5

Self-driving Cars Personal Assistants Sensitive Applications

slide-26
SLIDE 26

Motivation

Slide: 6 Slide Credits: Devi Parikh

slide-27
SLIDE 27

Motivation

  • Aiding visually impaired users

Slide: 6 Slide Credits: Devi Parikh

slide-28
SLIDE 28

Motivation

  • Aiding visually impaired users

Slide: 6 Peter just uploaded a picture from his vacation in Hawaii Slide Credits: Devi Parikh

slide-29
SLIDE 29

Motivation

  • Aiding visually impaired users

Slide: 6 Peter just uploaded a picture from his vacation in Hawaii Great, is he at the beach? Slide Credits: Devi Parikh

slide-30
SLIDE 30

Motivation

  • Aiding visually impaired users

Slide: 6 Peter just uploaded a picture from his vacation in Hawaii Great, is he at the beach? No, on a mountain Slide Credits: Devi Parikh

slide-31
SLIDE 31

Motivation

  • Aiding visually impaired users

Slide: 6 Peter just uploaded a picture from his vacation in Hawaii Great, is he at the beach? No, on a mountain ………… Slide Credits: Devi Parikh

slide-32
SLIDE 32

Motivation

Slide: 7

slide-33
SLIDE 33

Motivation

  • Human-AI teams in healthcare

Slide: 7

slide-34
SLIDE 34

Motivation

  • Human-AI teams in healthcare

Slide: 7

slide-35
SLIDE 35

Motivation

  • Human-AI teams in healthcare

Slide: 7

slide-36
SLIDE 36

Motivation

  • Human-AI teams in healthcare

Slide: 7

Report-1

slide-37
SLIDE 37

Motivation

  • Human-AI teams in healthcare

Slide: 7

Diagnosis-1 Report-1

slide-38
SLIDE 38

Motivation

  • Human-AI teams in healthcare

Slide: 7

Diagnosis-1 Report-1 Report-2

slide-39
SLIDE 39

Motivation

  • Human-AI teams in healthcare

Slide: 7

Diagnosis-1 Diagnosis-2 Report-1 Report-2

slide-40
SLIDE 40

Motivation

  • Human-AI teams in healthcare

Slide: 7

Diagnosis-1 Diagnosis-2 Which instance to trust? Report-1 Report-2

slide-41
SLIDE 41

Motivation

  • Human-AI teams in healthcare

Slide: 7

Diagnosis-1 Diagnosis-2 Which instance to trust? Report-1 Report-2

slide-42
SLIDE 42

Motivation

  • Human-AI teams in healthcare

Slide: 7

Diagnosis-1 Diagnosis-2 Which instance to trust?

  • Critical for human to have a sense of AI’s

Report-1 Report-2

slide-43
SLIDE 43

Motivation

  • Human-AI teams in healthcare

Slide: 7

Diagnosis-1 Diagnosis-2 Which instance to trust?

  • Critical for human to have a sense of AI’s

Failure Modes Behavior

Report-1 Report-2

slide-44
SLIDE 44

Theory of AI’s Mind (ToAIM)

Slide: 8

slide-45
SLIDE 45

Theory of AI’s Mind (ToAIM)

  • For human-AI teams to be effective, humans must also develop a

theory of AI’s mind

Slide: 8

slide-46
SLIDE 46

Theory of AI’s Mind (ToAIM)

  • For human-AI teams to be effective, humans must also develop a

theory of AI’s mind

Slide: 8

slide-47
SLIDE 47

Theory of AI’s Mind (ToAIM)

  • For human-AI teams to be effective, humans must also develop a

theory of AI’s mind

Slide: 8

slide-48
SLIDE 48

Theory of AI’s Mind (ToAIM)

  • For human-AI teams to be effective, humans must also develop a

theory of AI’s mind

Slide: 8

slide-49
SLIDE 49

Theory of AI’s Mind (ToAIM)

  • For human-AI teams to be effective, humans must also develop a

theory of AI’s mind

  • Predict success, failure and responses
  • Approximate a neural network!

Slide: 8

slide-50
SLIDE 50

Outline

Slide: 9

slide-51
SLIDE 51

Outline

Slide: 9

Theory of AI’s mind (ToAIM): Motivation Theory of AI’s mind (ToAIM): Experimental Setup and Results Theory of AI’s mind (ToAIM): Take-away messages Theory of AI’s mind (ToAIM): Ongoing work - Human-AI Games

slide-52
SLIDE 52

Outline

Slide: 9

Theory of AI’s mind (ToAIM): Motivation Theory of AI’s mind (ToAIM): Experimental Setup and Results Theory of AI’s mind (ToAIM): Take-away messages Theory of AI’s mind (ToAIM): Ongoing work - Human-AI Games

slide-53
SLIDE 53

AI Agent

Slide: 10

slide-54
SLIDE 54

AI Agent

  • Vicki : A VQA Model

Initial Scope: Visual Question Answering (VQA)

Slide: 10

slide-55
SLIDE 55

AI Agent

  • Vicki : A VQA Model

Initial Scope: Visual Question Answering (VQA)

Slide: 10

slide-56
SLIDE 56

AI Agent

  • Vicki : A VQA Model

Initial Scope: Visual Question Answering (VQA)

What is the child doing?

Slide: 10

slide-57
SLIDE 57

AI Agent

  • Vicki : A VQA Model

Initial Scope: Visual Question Answering (VQA)

What is the child doing?

Vicki Vision Language

Slide: 10

slide-58
SLIDE 58

AI Agent

  • Vicki : A VQA Model

Initial Scope: Visual Question Answering (VQA)

What is the child doing?

Vicki

playing baseball

Vision Language

Slide: 10

slide-59
SLIDE 59

AI Agent

  • Vicki : A VQA Model

Initial Scope: Visual Question Answering (VQA)

What is the child doing?

Vicki

playing baseball

Vision Language Dataset

Slide: 10

slide-60
SLIDE 60

AI Agent

  • Vicki : A VQA Model

Initial Scope: Visual Question Answering (VQA)

What is the child doing?

Vicki

playing baseball

Vision Language 248349 QI-pairs in train-split Dataset

Slide: 10

slide-61
SLIDE 61

AI Agent

  • Vicki : A VQA Model

Initial Scope: Visual Question Answering (VQA)

What is the child doing?

Vicki

playing baseball

Vision Language 248349 QI-pairs in train-split Vicki can answer

  • nly from the

top-1k answers Dataset

Slide: 10

slide-62
SLIDE 62

AI Agent

Slide: 11

slide-63
SLIDE 63

AI Agent

  • VQA model by Lu, Yang et al. NIPS 2016

Jiasen Lu Jianwei Yang

Slide: 11

slide-64
SLIDE 64

AI Agent

  • VQA model by Lu, Yang et al. NIPS 2016
  • Hierarchical co-Attention model

Jiasen Lu Jianwei Yang

Slide: 11

slide-65
SLIDE 65

AI Agent

  • VQA model by Lu, Yang et al. NIPS 2016
  • Hierarchical co-Attention model

Jiasen Lu Jianwei Yang

Slide: 11

slide-66
SLIDE 66

Vicki’s Quirks

Slide: 12

slide-67
SLIDE 67

Vicki’s Quirks

  • Imperfect vision

Slide: 12

slide-68
SLIDE 68

Vicki’s Quirks

  • Imperfect vision
  • Limited capability to understand language

Slide: 12

slide-69
SLIDE 69

Vicki’s Quirks

  • Imperfect vision
  • Limited capability to understand language
  • Can’t reason about common-sense

Slide: 12

slide-70
SLIDE 70

Vicki’s Quirks

  • Imperfect vision
  • Limited capability to understand language
  • Can’t reason about common-sense
  • Limited vocabulary

Slide: 12

slide-71
SLIDE 71

Vicki’s Quirks

  • Imperfect vision
  • Limited capability to understand language
  • Can’t reason about common-sense
  • Limited vocabulary
  • Doesn’t understand question-image relevance

Slide: 12

slide-72
SLIDE 72

Vicki’s Quirks

  • Imperfect vision
  • Limited capability to understand language
  • Can’t reason about common-sense
  • Limited vocabulary
  • Doesn’t understand question-image relevance
  • Heavily influenced by dataset biases

Slide: 12

slide-73
SLIDE 73

Vicki’s Quirks

  • Imperfect vision
  • Limited capability to understand language
  • Can’t reason about common-sense
  • Limited vocabulary
  • Doesn’t understand question-image relevance
  • Heavily influenced by dataset biases

Vicki

Slide: 12

slide-74
SLIDE 74

Vicki’s Quirks

  • Imperfect vision
  • Limited capability to understand language
  • Can’t reason about common-sense
  • Limited vocabulary
  • Doesn’t understand question-image relevance
  • Heavily influenced by dataset biases

Vicki Q1, I1 A1

Slide: 12

slide-75
SLIDE 75

Vicki’s Quirks

  • Imperfect vision
  • Limited capability to understand language
  • Can’t reason about common-sense
  • Limited vocabulary
  • Doesn’t understand question-image relevance
  • Heavily influenced by dataset biases

Vicki Q1, I1 Q2, I2 A1 A2

Slide: 12

slide-76
SLIDE 76

Vicki’s Quirks

  • Imperfect vision
  • Limited capability to understand language
  • Can’t reason about common-sense
  • Limited vocabulary
  • Doesn’t understand question-image relevance
  • Heavily influenced by dataset biases

Vicki Q1, I1 Q2, I2 A1 A2

Slide: 12

slide-77
SLIDE 77

Vicki’s Quirks

  • Imperfect vision
  • Limited capability to understand language
  • Can’t reason about common-sense
  • Limited vocabulary
  • Doesn’t understand question-image relevance
  • Heavily influenced by dataset biases

Vicki Q1, I1 Q2, I2 Qn, In A1 A2 An

Slide: 12

slide-78
SLIDE 78

Vicki’s Quirks

  • Imperfect vision
  • Limited capability to understand language
  • Can’t reason about common-sense
  • Limited vocabulary
  • Doesn’t understand question-image relevance
  • Heavily influenced by dataset biases

Vicki Q1, I1 Q2, I2 Qn, In A1 A2 An Helps us pick on Vicki’s quirks

Slide: 12

slide-79
SLIDE 79

Vicki’s Quirks

Slide: 13

slide-80
SLIDE 80

Vicki’s Quirks

What color is the grass? Blue

Slide: 13

slide-81
SLIDE 81

Vicki’s Quirks

What color is the grass? Blue What are the people doing? Eating

Slide: 13

slide-82
SLIDE 82

Vicki’s Quirks

What color is the grass? Blue What are the people doing? Eating How many people are there? 4 What is the man holding? Fire Hydrant

Slide: 13

slide-83
SLIDE 83

ToAIM

Slide: 14

slide-84
SLIDE 84

ToAIM

  • To study/evaluate ToAIM Large-scale experiments on MTurk

Slide: 14

slide-85
SLIDE 85

ToAIM

  • To study/evaluate ToAIM Large-scale experiments on MTurk

Subjects on AMT Vicki

Slide: 14

slide-86
SLIDE 86

ToAIM

  • To study/evaluate ToAIM Large-scale experiments on MTurk

Task Interface Subjects on AMT Vicki

Slide: 14

slide-87
SLIDE 87

ToAIM

  • To study/evaluate ToAIM Large-scale experiments on MTurk

Task Interface Subjects on AMT Vicki Failure Prediction Knowledge Prediction

Slide: 14

slide-88
SLIDE 88

ToAIM

Slide: 15

slide-89
SLIDE 89

ToAIM

  • Failure Prediction

Slide: 15

slide-90
SLIDE 90

ToAIM

  • Failure Prediction

How many people are there?

Slide: 15

slide-91
SLIDE 91

ToAIM

  • Failure Prediction

How many people are there? Subject thinks Vicki will answer correctly

Slide: 15

slide-92
SLIDE 92

ToAIM

  • Failure Prediction

How many people are there? Subject thinks Vicki will answer correctly Correctly

Slide: 15

slide-93
SLIDE 93

ToAIM

  • Failure Prediction

Slide: 16

slide-94
SLIDE 94

ToAIM

Slide: 17

slide-95
SLIDE 95

ToAIM

  • Knowledge Prediction

Slide: 17

slide-96
SLIDE 96

ToAIM

  • Knowledge Prediction

How many people are there?

Slide: 17

slide-97
SLIDE 97

ToAIM

  • Knowledge Prediction

How many people are there? Subject thinks Vicki will answer 4 4

Slide: 17

slide-98
SLIDE 98

ToAIM

  • Knowledge Prediction

How many people are there? Subject thinks Vicki will answer 4

4

4

Slide: 17

slide-99
SLIDE 99

ToAIM

  • Knowledge Prediction

Slide: 18

slide-100
SLIDE 100

ToAIM

Slide: 19

slide-101
SLIDE 101

ToAIM

  • We evaluate the role of

Slide: 19

slide-102
SLIDE 102

ToAIM

  • We evaluate the role of

Training Explanation Modalities

Slide: 19

slide-103
SLIDE 103

ToAIM

Slide: 20

slide-104
SLIDE 104

ToAIM

  • Training via Instant Feedback

Slide: 20

slide-105
SLIDE 105

ToAIM

  • Training via Instant Feedback

How many people are there?

Slide: 20

slide-106
SLIDE 106

ToAIM

  • Training via Instant Feedback

How many people are there? FP

Slide: 20

slide-107
SLIDE 107

ToAIM

  • Training via Instant Feedback

How many people are there? FP

Slide: 20

slide-108
SLIDE 108

ToAIM

  • Training via Instant Feedback

How many people are there? Feedback FP

Slide: 20

slide-109
SLIDE 109

ToAIM

  • Training via Instant Feedback

How many people are there? Feedback 4

4

FP KP

Slide: 20

slide-110
SLIDE 110

ToAIM

  • Training via Instant Feedback

How many people are there? Feedback 4

4 3

FP KP

Slide: 20

slide-111
SLIDE 111

ToAIM

  • Training via Instant Feedback

How many people are there? Feedback 4

4 3 3

FP KP

Slide: 20

slide-112
SLIDE 112

ToAIM

Slide: 21

slide-113
SLIDE 113

ToAIM

  • Explanation Modalities

Slide: 21

slide-114
SLIDE 114

ToAIM

  • Explanation Modalities

Slide: 21

Vicki

What is the child doing?

slide-115
SLIDE 115

ToAIM

  • Explanation Modalities

Slide: 21

  • Access to internal states of a model for a prediction

Vicki

What is the child doing? playing baseball Access to internal states of a model

slide-116
SLIDE 116

ToAIM

  • Explanation Modalities

Slide: 22

slide-117
SLIDE 117

ToAIM

  • Explanation Modalities

Slide: 22

QI-Attention

  • Explicit question-image attention maps from HieCoAtt
slide-118
SLIDE 118

ToAIM

  • Explanation Modalities

Slide: 22

QI-Attention

  • Explicit question-image attention maps from HieCoAtt

Which words in the question to listen to?

slide-119
SLIDE 119

ToAIM

  • Explanation Modalities

Slide: 22

QI-Attention

  • Explicit question-image attention maps from HieCoAtt

Which words in the question to listen to? Which regions in the image are important?

slide-120
SLIDE 120

ToAIM

  • Explanation Modalities

Slide: 23

slide-121
SLIDE 121

ToAIM

  • Explanation Modalities

Slide: 23

Grad-CAM

  • Grad-CAM: implicit attention mechanism
slide-122
SLIDE 122

ToAIM

  • Explanation Modalities

Slide: 23

Grad-CAM

  • Grad-CAM: implicit attention mechanism

Attention visualization corresponding to Vicki’s most confident answer

slide-123
SLIDE 123

ToAIM

  • Explanation Modalities

Slide: 24

slide-124
SLIDE 124

ToAIM

  • Explanation Modalities

Slide: 24

How many people are there?

  • Vicki’s confidence in top-5 answers

Top-5 answer confidence

slide-125
SLIDE 125

ToAIM

  • Explanation Modalities

Slide: 24

How many people are there?

  • Vicki’s confidence in top-5 answers

Vicki’s confidence in the top-5 answers without revealing the answers Top-5 answer confidence

slide-126
SLIDE 126

ToAIM

Slide: 25

slide-127
SLIDE 127

ToAIM

  • Training + Explanation Modalities

Slide: 25

slide-128
SLIDE 128

ToAIM

  • Training + Explanation Modalities

How many people are there?

+

How many people are there? Slide: 25

slide-129
SLIDE 129

ToAIM

  • Training + Explanation Modalities

How many people are there?

+

How many people are there? Slide: 25

slide-130
SLIDE 130

ToAIM

  • Training + Explanation Modalities

How many people are there?

+

How many people are there? Slide: 25

slide-131
SLIDE 131

ToAIM

  • Training + Explanation Modalities

Feedback

How many people are there?

+

How many people are there? Slide: 25

slide-132
SLIDE 132

ToAIM

  • Training + Explanation Modalities

Feedback 4

How many people are there?

+

How many people are there? Slide: 25

slide-133
SLIDE 133

ToAIM

  • Training + Explanation Modalities

Feedback 4

4

How many people are there?

+

How many people are there?

3

Slide: 25

slide-134
SLIDE 134

ToAIM

  • Training + Explanation Modalities

Feedback 4

4 3

How many people are there?

+

How many people are there?

3

Slide: 25

slide-135
SLIDE 135

Experimental Results

Slide: 26

slide-136
SLIDE 136

Experimental Results

Failure Prediction

Slide: 26

slide-137
SLIDE 137

Experimental Results

IF+QI-Att IF+Top-5 IF+GCAM IF FP No Train Accuracy

Failure Prediction

Slide: 26 100 90 80 70 60 50 40 30 20 10

slide-138
SLIDE 138

Experimental Results

IF+QI-Att IF+Top-5 IF+GCAM IF FP No Train Accuracy

Failure Prediction

Training Helps Slide: 26 100 90 80 70 60 50 40 30 20 10

slide-139
SLIDE 139

Experimental Results

IF+QI-Att IF+Top-5 IF+GCAM IF FP No Train Accuracy

Failure Prediction

Training Helps

  • Exp. Mod. don’t help

Slide: 26 100 90 80 70 60 50 40 30 20 10

slide-140
SLIDE 140

Experimental Results

Slide: 27

slide-141
SLIDE 141

Experimental Results

Knowledge Prediction

Slide: 27

slide-142
SLIDE 142

Experimental Results

IF+QI-Att IF+Top-5 IF+GCAM IF KP No Train Accuracy

Knowledge Prediction

Slide: 27 100 75 50 25

slide-143
SLIDE 143

Experimental Results

IF+QI-Att IF+Top-5 IF+GCAM IF KP No Train Accuracy

Knowledge Prediction

Training Helps Slide: 27 100 75 50 25

slide-144
SLIDE 144

Experimental Results

IF+QI-Att IF+Top-5 IF+GCAM IF KP No Train Accuracy

Knowledge Prediction

Training Helps

  • Exp. Mod. don’t help

Slide: 27 100 75 50 25

slide-145
SLIDE 145

Outline

Slide: 28

slide-146
SLIDE 146

Outline

Slide: 28

Theory of AI’s mind (ToAIM): Motivation Theory of AI’s mind (ToAIM): Experimental Setup and Results Theory of AI’s mind (ToAIM): Take-away messages Theory of AI’s mind (ToAIM): Ongoing work - Human-AI Games

slide-147
SLIDE 147

Outline

Slide: 28

Theory of AI’s mind (ToAIM): Motivation Theory of AI’s mind (ToAIM): Experimental Setup and Results Theory of AI’s mind (ToAIM): Take-away messages Theory of AI’s mind (ToAIM): Ongoing work - Human-AI Games

slide-148
SLIDE 148

Take-away messages

Slide: 29

slide-149
SLIDE 149

Take-away messages

  • Advocate a research agenda towards ToAIM

Slide: 29

slide-150
SLIDE 150

Take-away messages

  • Advocate a research agenda towards ToAIM
  • Lay people successfully form ToM with a few (50) examples

Slide: 29

slide-151
SLIDE 151

Take-away messages

  • Advocate a research agenda towards ToAIM
  • Lay people successfully form ToM with a few (50) examples
  • Existing explanation modalities don’t help in predicting

what AI will do

Slide: 29

slide-152
SLIDE 152

Take-away messages

  • Advocate a research agenda towards ToAIM
  • Lay people successfully form ToM with a few (50) examples
  • Existing explanation modalities don’t help in predicting

what AI will do

  • Novel evaluation protocol for explanation modalities

Slide: 29

slide-153
SLIDE 153

Take-away messages

  • Advocate a research agenda towards ToAIM
  • Lay people successfully form ToM with a few (50) examples
  • Existing explanation modalities don’t help in predicting

what AI will do

  • Novel evaluation protocol for explanation modalities

Slide: 29

slide-154
SLIDE 154

ArXiv

Slide: 30 Slide Credits: Devi Parikh

slide-155
SLIDE 155

Outline

Slide: 31

slide-156
SLIDE 156

Outline

Slide: 31

Theory of AI’s mind (ToAIM): Motivation Theory of AI’s mind (ToAIM): Experimental Setup and Results Theory of AI’s mind (ToAIM): Take-away messages Theory of AI’s mind (ToAIM): Ongoing work - Human-AI Games

slide-157
SLIDE 157

Outline

Slide: 31

Theory of AI’s mind (ToAIM): Motivation Theory of AI’s mind (ToAIM): Experimental Setup and Results Theory of AI’s mind (ToAIM): Take-away messages Theory of AI’s mind (ToAIM): Ongoing work - Human-AI Games

slide-158
SLIDE 158

Ongoing Work

Slide: 32

slide-159
SLIDE 159

Ongoing Work

  • Evaluate human-AI collaborative performance

Slide: 32

slide-160
SLIDE 160

Ongoing Work

  • Evaluate human-AI collaborative performance
  • Goal-driven tasks (cooperative human-AI games)

Slide: 32

slide-161
SLIDE 161

Ongoing Work

  • Evaluate human-AI collaborative performance
  • Goal-driven tasks (cooperative human-AI games)

Visual 20 Questions GuessWhich

Slide: 32

slide-162
SLIDE 162

Ongoing Work

  • Evaluate human-AI collaborative performance
  • Goal-driven tasks (cooperative human-AI games)

Visual 20 Questions GuessWhich

Slide: 32

slide-163
SLIDE 163

GuessWhich

Slide: 33

slide-164
SLIDE 164

GuessWhich

  • Players: 2 Agents - Questioner - and Answerer -

Slide: 33

Q A

slide-165
SLIDE 165

GuessWhich

  • Players: 2 Agents - Questioner - and Answerer -

Slide: 33

  • Parameters:

Q A

slide-166
SLIDE 166

GuessWhich

  • Players: 2 Agents - Questioner - and Answerer -

Slide: 33

  • Parameters:
  • Fixed pool of images

Q A

slide-167
SLIDE 167

GuessWhich

  • Players: 2 Agents - Questioner - and Answerer -

Slide: 33

  • Parameters:
  • Fixed pool of images
  • Fixed number of rounds of dialog (10)

Q A

slide-168
SLIDE 168

GuessWhich

  • Players: 2 Agents - Questioner - and Answerer -

Slide: 33

  • Parameters:
  • Fixed pool of images
  • Fixed number of rounds of dialog (10)
  • In our implementation, Human (Questioner) and AI (Answerer)

Q A

slide-169
SLIDE 169

GuessWhich

Slide: 34

slide-170
SLIDE 170

GuessWhich

Slide: 34

A selects an image from

slide-171
SLIDE 171

GuessWhich

Slide: 34

A selects an image from

slide-172
SLIDE 172

GuessWhich

Slide: 34

A selects an image from

slide-173
SLIDE 173

GuessWhich

Q

Slide: 34

A selects an image from

unknown to

slide-174
SLIDE 174

GuessWhich

Q

Slide: 34

A selects an image from

unknown to

A andQ are given a caption to get a idea about

slide-175
SLIDE 175

GuessWhich

Q

Slide: 34

A selects an image from

unknown to

A andQ are given a caption to get a idea about

makes a guess about

Q

slide-176
SLIDE 176

GuessWhich

Q

Slide: 34

A selects an image from

unknown to

Q asks questions so as to locate A andQ are given a caption to get a idea about

makes a guess about

Q

slide-177
SLIDE 177

GuessWhich

Q

Slide: 34

A selects an image from

unknown to

Q asks questions so as to locate

from

A andQ are given a caption to get a idea about

makes a guess about

Q

slide-178
SLIDE 178

GuessWhich

Q

Slide: 34

A selects an image from

unknown to

Q asks questions so as to locate

from

A answersQ’s questions according to A andQ are given a caption to get a idea about

makes a guess about

Q

slide-179
SLIDE 179

GuessWhich

Q

Slide: 34

A selects an image from

unknown to

Q asks questions so as to locate

from

A answersQ’s questions according to

makes a guess about

Q A andQ are given a caption to get a idea about

makes a guess about

Q

slide-180
SLIDE 180

GuessWhich

Q

Slide: 34

A selects an image from

unknown to

Q asks questions so as to locate

from

A answersQ’s questions according to

makes a guess about

Q

after every round of dialog

A andQ are given a caption to get a idea about

makes a guess about

Q

slide-181
SLIDE 181

GuessWhich

Slide: 35

slide-182
SLIDE 182

GuessWhich

Slide: 35

  • Human-AI Team: Human (Questioner) and AI (Answerer)
slide-183
SLIDE 183

GuessWhich

Slide: 35

  • Human-AI Team: Human (Questioner) and AI (Answerer)
  • Image retrieval setting. Metrics: mean-rank & mean-reciprocal

rank

slide-184
SLIDE 184

GuessWhich

Slide: 35

  • Human-AI Team: Human (Questioner) and AI (Answerer)
  • Image retrieval setting. Metrics: mean-rank & mean-reciprocal

rank

  • Our AI Agent Visual Conversation Agents
slide-185
SLIDE 185

GuessWhich

Slide: 35

  • Human-AI Team: Human (Questioner) and AI (Answerer)
  • Image retrieval setting. Metrics: mean-rank & mean-reciprocal

rank

  • Our AI Agent Visual Conversation Agents
  • Das & Kottur et al. ICCV 2017

Abhishek Das Satwik Kottur

slide-186
SLIDE 186

GuessWhich

Slide: 35

  • Human-AI Team: Human (Questioner) and AI (Answerer)
  • Image retrieval setting. Metrics: mean-rank & mean-reciprocal

rank

  • Our AI Agent Visual Conversation Agents
  • Das & Kottur et al. ICCV 2017

Abhishek Das Questioner Answerer Satwik Kottur

slide-187
SLIDE 187

Slide: 37

slide-188
SLIDE 188

Slide: 37

slide-189
SLIDE 189

Slide: 38

slide-190
SLIDE 190

That’s all folks! Questions?

Slide: 38

  • Interface videos: https://deshraj.github.io/TOAIM/
  • Interfaces: https://github.com/deshraj/TOAIM/tree/master/Assets/Interfaces
  • Guess-Which: http://gw.cloudcv.org/