Teaching Categories to Human Learners with Visual Explanations - - PowerPoint PPT Presentation

▶
teaching categories to human learners with visual
SMART_READER_LITE
LIVE PREVIEW

Teaching Categories to Human Learners with Visual Explanations - - PowerPoint PPT Presentation

Mar 18, 2024 151 likes •947 views

Teaching Categories to Human Learners with Visual Explanations Oisin Mac Aodha Can we design teaching algorithms that will enable humans to become better at visual categorization? Why Visual Expertise? What species? Why Visual Expertise?

slide-1
SLIDE 1

Teaching Categories to Human Learners with Visual Explanations

Oisin Mac Aodha

slide-2
SLIDE 2

Can we design teaching algorithms that will enable humans to become better at visual categorization?

slide-3
SLIDE 3

Why Visual Expertise? What species?

slide-4
SLIDE 4

Why Visual Expertise? Cancerous?

slide-5
SLIDE 5

Why Visual Expertise? Poisonous?

slide-6
SLIDE 6

Why Visual Expertise? Forgery?

slide-7
SLIDE 7

https://en.wikipedia.org/wiki/Grey_heron https://ebird.org/species/cocher1

Grey heron Cocoi heron

Challenges - 1 Visual Similarity

slide-8
SLIDE 8

Challenges - 2 Within Class Variation

slide-9
SLIDE 9

https://en.wikipedia.org/wiki/Grey_heron

Challenges - 3 “Attribution”

Which pixels “explain” the class label?

slide-10
SLIDE 10
slide-11
SLIDE 11

ht

hypothesis

Machine Teacher Student/Learner

h*

hypothesis

data & label feedback

slide-12
SLIDE 12

Teaching Visual Expertise

Set of images with class labels

...

slide-13
SLIDE 13

Teaching Visual Expertise

Teaching algorithm & student model Set of images with class labels

...

slide-14
SLIDE 14

Teaching Visual Expertise

, ,

Sequence of teaching images Teaching algorithm & student model Set of images with class labels

...

Class 2 Class 1 Class 1

slide-15
SLIDE 15

Spaced Repetition Leitner 1972 Settles & Meeder 2016 Hunziker et al. 2019 Choffin et al. 2019 ... Theoretical Goldman & Kearns 1995 Zhu 2013 Chen et al. 2018 ... Visual Categories Singla et al. 2014 Johns et al. 2015 Chen et al. 2018 ...

Machine Teaching Landscape

Decision Making Bak et al. 2016 ...

slide-16
SLIDE 16

https://www.inaturalist.org/observations/9869215

Connecticut Warbler

  • r MacGillivray's Warbler
slide-17
SLIDE 17

https://www.inaturalist.org/observations/9869215

Connecticut Warbler

  • r MacGillivray's Warbler
slide-18
SLIDE 18

https://www.inaturalist.org/observations/9869215 https://www.inaturalist.org/observations/3949369

Connecticut Warbler MacGillivray's Warbler

slide-19
SLIDE 19

https://www.inaturalist.org/observations/9869215 https://www.inaturalist.org/observations/3949369

Connecticut Warbler MacGillivray's Warbler

slide-20
SLIDE 20

Teaching Categories to Human Learners with Visual Explanations

CVPR 2018

Yuxin Chen

  • Uni. of Chicago

Yisong Yue Caltech Pietro Perona Caltech Shihan Su Caltech

slide-21
SLIDE 21

x is an image

slide-22
SLIDE 22

e is an associated explanation

slide-23
SLIDE 23

Visual “Explanations”

Monarch Viceroy Queen Red Admiral Cabbage White

slide-24
SLIDE 24

Visual “Explanations”

Monarch Viceroy Queen Red Admiral Cabbage White

Learning Deep Features for Discriminative Localization CVPR 2016

slide-25
SLIDE 25
slide-26
SLIDE 26

h1 h2 h3 h*

h is a hypothesis

slide-27
SLIDE 27

length of bill body color “roundness” “eye whiteness”

slide-28
SLIDE 28
slide-29
SLIDE 29

How to Choose Teaching Set T to Teach h*? h*

slide-30
SLIDE 30

Student Model

Singla et al. Near-Optimally Teaching the Crowd to Classify ICML 2014

slide-31
SLIDE 31

Student Model

Singla et al. Near-Optimally Teaching the Crowd to Classify ICML 2014

“win stay, lose switch”

slide-32
SLIDE 32

Student Model

“win stay, lose switch”

slide-33
SLIDE 33

Student Model - With Explanations

“Good” “Bad”

slide-34
SLIDE 34

Student Model - With Explanations

“Good” “Bad”

slide-35
SLIDE 35

Student Model - With Explanations

slide-36
SLIDE 36

Selecting the Teaching Set T

Select for largest reduction in expected error

slide-37
SLIDE 37

h1 h2 h3 h*

slide-38
SLIDE 38

h1 h2 h3 h*

h* h1 h2 h3

P(h) =

slide-39
SLIDE 39

h1 h2 h3 h*

h* h1 h2 h3

P(h) = Select Teaching Example 1

slide-40
SLIDE 40

h1 h2 h3 h*

h* h1 h2 h3

P(h|x1) = Update Model

slide-41
SLIDE 41

h1 h2 h3 h*

h* h1 h2 h3

P(h|x1) = Select Teaching Example 2

slide-42
SLIDE 42

h1 h2 h3 h*

h* h1 h2 h3

P(h|x1, x2) = Update Model

slide-43
SLIDE 43

h1 h2 h3 h*

h* h1 h2 h3

P(h|x1, x2) = Repeat …

slide-44
SLIDE 44

Multiclass Teaching

Independent posterior per class

slide-45
SLIDE 45
slide-46
SLIDE 46

Tutorial Teaching Testing

Experimental Setup

Familiarize participants with interface Teach for 20 iterations Test for 20 iterations (to measure performance)

slide-47
SLIDE 47

A) A) Viceroy B) B) Monarch C) C) Queen D) D) Red Admiral

Step 1 - Query Learner

Which Species is Present?

slide-48
SLIDE 48

A) A) Viceroy B) B) Monarch C) C) Queen D) D) Red Admiral

Step 2 - Get Learner Response

Which Species is Present?

slide-49
SLIDE 49

A) A) Viceroy B) B) Monarch C) C) Queen D) D) Red Admiral

Step 3 - Provide Feedback

Which Species is Present?

slide-50
SLIDE 50
slide-51
SLIDE 51

Retina Images

Macular Edema Normal Subretinal Fluid

1125 images, 3 classes

~ 40 participants per dataset per teaching algorithm

slide-52
SLIDE 52

Subretinal fluid

image “explanation”

slide-53
SLIDE 53

Macular Edema

image “explanation”

slide-54
SLIDE 54

Results for Retina Images

slide-55
SLIDE 55

Results for Retina Images

slide-56
SLIDE 56

Results for Retina Images

slide-57
SLIDE 57

Chinese Characters

Grass Mound Stem

717 images, 3 classes

slide-58
SLIDE 58

Results for Chinese Characters

slide-59
SLIDE 59

Results for Chinese Characters

slide-60
SLIDE 60

Results for Chinese Characters

slide-61
SLIDE 61

Test Accuracy Test Accuracy Number of Participants Explain (Ours) “CNN Features” Explain (Ours) “Crowd Features”

slide-62
SLIDE 62

“CNN Features” “Crowd Features”

Grass Mound Stem

slide-63
SLIDE 63

Butterflies 2,224 images, 5 classes

Monarch Viceroy Queen Red Admiral Cabbage White

slide-64
SLIDE 64

Results for Butterflies

slide-65
SLIDE 65
slide-66
SLIDE 66

Next steps for teaching visual knowledge ….

slide-67
SLIDE 67

Becoming the Expert: Interactive Multi-Class Machine Teaching CVPR 2015 Johns, Mac Aodha, Brostow Understanding the Role of Adaptivity in Machine Teaching: The Case of Version Space Learners NeurIPS 2018 Chen, Singla, Mac Aodha, Perona, Yue

Interactive Teaching

slide-68
SLIDE 68

Teaching Multiple Concepts to Forgetful Learners NeurIPS 2019 Hunziker, Chen, Mac Aodha, Gomez Rodriguez, Krause, Perona, Yue, Singla

Memory decays over time Spaced repetition model Estimate learner recall

Modelling Learner Memory Decay

slide-69
SLIDE 69

Scaling Up Visual Teaching - ebird.org/quiz

slide-70
SLIDE 70

Teaching Fine-Grained Detail

Learning explanations through teaching

slide-71
SLIDE 71

Closing the Loop

Teaching super human image understanding

Prediction of cardiovascular risk factors from retinal fundus photographs via deep learning Poplin et al. Nature Biomedical Engineering 2018

slide-72
SLIDE 72
slide-73
SLIDE 73

Questions

Teaching GUI, model code, and data: https://github.com/macaodha/explain_teach

slide-74
SLIDE 74
slide-75
SLIDE 75
slide-76
SLIDE 76
slide-77
SLIDE 77

iNaturalist Dataset

8,142 classes >400K images

Learning How to Perform Low Shot Learning

The iNaturalist Species Classification and Detection Dataset CVPR 2018 Van Horn, Mac Aodha, Song, Cui, Sun, Shepard, Adam, Perona, Belongie