SLIDE 1
BubbleView: an interface for
crowdsourcing image importance maps and tracking visual attention
Nam Wook Kim* Zoya Bylinskii* Michelle Borkin Krzysztof Gajos Aude Oliva Fredo Durand Hanspeter Pfister
Eye-tracking for capturing human visual attention Eye-tracking for - - PowerPoint PPT Presentation
Eye-tracking for capturing human visual attention Eye-tracking for - - PowerPoint PPT Presentation
BubbleView : an interface for crowdsourcing image importance maps and tracking visual attention Nam Wook Zoya Michelle Krzysztof Aude Fredo Hanspeter Kim* Bylinskii* Borkin Gajos Oliva Durand Pfister Eye-tracking for capturing human
SLIDE 2
SLIDE 3
in-lab experiment tedious calibration specialized hardware
Eye-tracking for capturing human visual attention
SLIDE 4
in-lab experiment tedious calibration specialized hardware
Difficult to scale up data collection to more than a few participants
Eye-tracking for capturing human visual attention
SLIDE 5
Bubble View
An alternative for eye tracking using discrete mouse clicks to measure which information people consciously choose to examine.
2x
SLIDE 6
Inspiration: Bubbles [Gosselin & Schyns, 2001]
Face stimuli Punctured by bubbles Gender categorization The eyes and mouth
SLIDE 7
Face stimuli Punctured by bubbles Gender categorization The eyes and mouth
Inspiration: Bubbles [Gosselin & Schyns, 2001]
BubbleView generalizes this idea to allow users to control where they want to look.
SLIDE 8
Cursor-Based Attention Tracking
[Schulte- Mecklenbeck et al. 2011] [Jiang et al. 2015]
SLIDE 9
Cursor-Based Attention Tracking
[Schulte- Mecklenbeck et al. 2011] [Jiang et al. 2015]
Discrete clicks instead of continuous movements to explicitly record points of interest
SLIDE 10
Cursor-Based Attention Tracking
[Schulte- Mecklenbeck et al. 2011] [Jiang et al. 2015]
We systematically evaluate cursor-based tracking under different parameters and task settings.
SLIDE 11
Evaluated on Various Image Types
Static Webpages Natural Scene Images Information Visualizations Graphic Designs
MASSVIS [Borkin et al. 2016] OSIE [Xu et al. 2014] FIWI [Shen and Zhao 2014] GDI [O’Donovan et al. 2014]
SLIDE 12
Evaluation Configuration
Static Webpages Natural Scene Images Information Visualizations Describe
(unlimited time)
Bubble radius
(16,24,32,40)
Graphic Designs
MASSVIS [Borkin et al. 2016]
vs Fixations
SLIDE 13
Description Task
Click and describe the image
clicks vs fixations
Unlimited time + 150 minimum characters
SLIDE 14
Varied Bubble Sizes
How does bubble radius size affect performance?
16 pixels 24 pixels 32 pixels 40 pixels
SLIDE 15
Collected Data
Clicks & Description changes over time.
SLIDE 16
Clicks & Description changes over time.
Collected Data
MASSVIS [Borkin et al. 2016]
Filtered malicious data & Compared clicks to eye-fixations
SLIDE 17
Evaluation Configuration
GDI [O’Donovan et al. 2014]
Static Webpages Natural Scene Images Information Visualizations
Graphic Designs Free-view
(10 sec)
vs Annotations
Much less informational content in graphic design images
SLIDE 18
Free-Viewing Task
10 seconds of viewing No description required
SLIDE 19
Evaluation Configuration
Natural Scene Images Information Visualizations
Static Webpages Free-view
(10 sec, 30 sec)
Describe
(unlimited time)
Bubble radius
(30,50,60) FIWI [Shen and Zhao 2014]
vs Fixations Graphic Designs
SLIDE 20
Evaluation Configuration
Natural Scene Images Free-view
- 1. clicks
(10 sec)
- 2. movements
(5 sec)
Static Webpages
Information Visualizations
Graphic Designs vs Fixations
OSIE [Xu et al. 2014]
SLIDE 21
Evaluation Configuration
Static Webpages
Information Visualizations
Graphic Designs
OSIE [Xu et al. 2014]
Mouse Clicks Mouse Movement Eye Fixations Natural Scene Images Free-view
- 1. clicks
(10 sec)
- 2. movements
(5 sec)
vs Fixations
SLIDE 22
MASSVIS [Borkin et al. 2016] OSIE [Xu et al. 2014] FIWI [Shen and Zhao 2014] GDI [O’Donovan et al. 2014]
Evaluation Configuration
Information Visualizations Describe
(unlimited time)
Bubble radius
(16,24,32,40)
Graphic Designs Free-view
(10 sec)
Natural Scene Images Free-view
- 1. clicks
(10 sec)
- 2. movements
(5 sec)
vs Fixations Static Webpages Free-view
(10 sec, 30 sec)
Describe
(unlimited time)
Bubble radius
(30,50,60)
vs Fixations vs Annotations vs Fixations
10 experiments with 28 different parameter combinations
SLIDE 23
Evaluation Tools Experimental Results Future Applications
SLIDE 24
Evaluation Tools Experimental Results Future Applications
SLIDE 25
Clicks
Computing CC score
Fixations
SLIDE 26
Clicks Fixations
Computing CC score
SLIDE 27
Computing CC score
- 1
+1
SLIDE 28
Computing NSS score
Clicks Fixations
SLIDE 29
Computing NSS score
Normalized by eye fixation consistency
SLIDE 30
Computing NSS score
Report % of fixations that clicks can explain
Normalized by eye fixation consistency
SLIDE 31
Evaluation Tools Experimental Results Future Applications
SLIDE 32
Clicks are more effective than mouse movements for measuring observer behavior.
Take-away #1:
SLIDE 33
Clicks vs Movements
Clicks Movements
SLIDE 34
Clicks are conscious decisions
- f importance
<
Clicks Movements Intentionality
SLIDE 35
Clicks are conscious decisions
- f importance
<
Clicks Movements Intentionality
Clicks are a better approximation to eye fixations
SLIDE 36
Clicks are predictive of eye fixations across a variety of image types and tasks.
Take-away #2:
SLIDE 37
Clicks predict fixations on visualizations
Fixations Clicks
clicks of 10 participants explain 90% of fixations
SLIDE 38
Clicks predict fixations on natural images
Fixations Clicks
clicks of 10 participants explain 78% of fixations
SLIDE 39
Clicks predict fixations on webpages
Fixations Clicks
clicks of 10 participants explain 78% of fixations
SLIDE 40
Click patterns match fixation patterns
500 300 100 300 1000 600 200
Horizontal (x) coordinate
eye fixations bubble clicks Visualizations Natural images Webpages Heatmap intensity
SLIDE 41
Click patterns match fixation patterns
500 300 100 300 1000 600 200
Horizontal (x) coordinate
eye fixations bubble clicks Visualizations Natural images Webpages Heatmap intensity title header
SLIDE 42
Click patterns match fixation patterns
500 300 100 300 1000 600 200
Horizontal (x) coordinate
eye fixations bubble clicks Visualizations Natural images Webpages Heatmap intensity center bias
SLIDE 43
0.5 0.6 0.7 0.8 0.9 1 1.1 1.2 1.3 1.4 1.5
2 4 6 8 10 12 14
# BubbleView participants Similarity between clicks and fixations
More involved tasks lead to better clicks
SLIDE 44
0.5 0.6 0.7 0.8 0.9 1 1.1 1.2 1.3 1.4 1.5
2 4 6 8 10 12 14
10 sec 30 sec describe } free-view Similarity between clicks and fixations
More involved tasks lead to better clicks
# BubbleView participants
SLIDE 45
0.5 0.6 0.7 0.8 0.9 1 1.1 1.2 1.3 1.4 1.5
2 4 6 8 10 12 14
10 sec 30 sec describe engagement Similarity between clicks and fixations
More involved tasks lead to better clicks
# BubbleView participants
SLIDE 46
Task time and bubble size interact to affect clicks.
Take-away #3:
SLIDE 47
Performance is stable across bubble sizes
16 pix 24 pix 32 pix Effort x 1.5 clicks
SLIDE 48
Blur affects clicks more than bubble size
30 50 70 30 50 70
Bubble radius (pixels) Blur sigma (pixels)
SLIDE 49
Blur affects clicks more than bubble size
30 50 70 30 50 70
Bubble radius (pixels) Blur sigma (pixels)
Largest blur and bubble sizes reduce exploration (1-2 deg. of visual angle best).
SLIDE 50
0.5 0.6 0.7 0.8 0.9 1 1.1 1.2 1.3 1.4 1.5
2 4 6 8 10 12 14
0.5 0.6 0.7 0.8 0.9 1 1.1 1.2 1.3 1.4 1.5
2 4 6 8 10 12 14
# BubbleView participants # BubbleView participants
Task time and bubble size interact
10 sec task 30 sec task Similarity between clicks and fixations 70 pix 50 pix 30 pix
SLIDE 51
BubbleView can be used to rank image elements by importance.
Take-away #4:
SLIDE 52
Ranking elements by importance
.8 .6 .4
SLIDE 53
.8 .6 .4
title label paragraph axis legend annotation
- bject
axis label source graphical element axis (time) text header row annotation (arrow) 0.0 0.2 0.4 0.6 0.8
Eye fixations .0 .2 .4 .6 .8
Ranking elements by importance
SLIDE 54
title label paragraph axis legend annotation
- bject
axis label source graphical element axis (time) text header row annotation (arrow) 0.0 0.2 0.4 0.6 0.8
Eye fixations BubbleView clicks .0 .2 .4 .6 .8
Ranking elements by importance
SLIDE 55
title label paragraph axis legend annotation
- bject
axis label source graphical element axis (time) text header row annotation (arrow) 0.0 0.2 0.4 0.6 0.8
Eye fixations BubbleView clicks .0 .2 .4 .6 .8
Spearman r = 0.96
Ranking elements by importance
SLIDE 56
Ranking elements by importance
0.98 1.00 0.94 0.31
.58 .74 1.0 .86 .52 .33 .98 1.0 .94 .31
Spearman r = 0.60
SLIDE 57
Another measurement of importance
Graphic design Crowd annotations
- Avg. annotation
O’Donovan et al. [TVCG’14]
SLIDE 58
Design choice: collecting importance
Clicks Fixations Annotations “unconscious” conscious conscious explorative explorative constrained
SLIDE 59
BubbleView measures importance
Clicks Fixations with free-viewing Clicks with description with free-viewing Intentionality Effort, task time, consistency “Saliency” “Importance”
SLIDE 60
Evaluation Tools Experimental Results Future Applications
SLIDE 61
Retargeting & Thumbnailing
[Bylinskii et al. UIST 2017]
Input image Importance map
SLIDE 62
Retargeting & Thumbnailing
[Bylinskii et al. UIST 2017]
Input image Thumbnail
Carving out more important regions
Importance map
SLIDE 63
Prediction of Visual Importance
[visimportance.csail.mit.edu]
Providing real time feedback based on importance predictions
SLIDE 64
Task-specific attentional data
Which city is ranked first (find an extremum)?
Visual Question Answering
SLIDE 65
Other image domains & tasks
Visualization Natural scene Webpage Graphic design Tasks Free-viewing Description
SLIDE 66
Other image domains & tasks
Medical images Satelite images Posters Mobile Visualization Natural scene Webpage Graphic design Visual search Q & A Analysis tasks Tasks Free-viewing Description
SLIDE 67
Code Paper Demo
http://bubbleview.namwkim.org
2x
SLIDE 68
SLIDE 69