Learning Visual Importance for Graphic Designs and Data - - PowerPoint PPT Presentation

▶
learning visual importance for graphic designs and data
SMART_READER_LITE
LIVE PREVIEW

Learning Visual Importance for Graphic Designs and Data - - PowerPoint PPT Presentation

Nov 05, 2023 217 likes •763 views

Learning Visual Importance for Graphic Designs and Data Visualizations Zoya Bylinskii , Nam Wook Kim, Peter ODonovan, Sami Alsheikh, Spandan Madan, Hanspeter Pfister, Fredo Durand, Bryan Russell, Aaron Hertzmann Today, were on the verge

slide-1
SLIDE 1

Learning Visual Importance for Graphic Designs and Data Visualizations

Zoya Bylinskii, Nam Wook Kim, Peter O’Donovan, Sami Alsheikh, Spandan Madan, Hanspeter Pfister, Fredo Durand, Bryan Russell, Aaron Hertzmann

slide-2
SLIDE 2

“Today, we’re on the verge of another revolution, as artificial intelligence and machine learning turn the graphic design field on its head again.”

https://www.wired.com/story/when-websites-design-themselves Sept 20, 2017

slide-3
SLIDE 3
slide-4
SLIDE 4
slide-5
SLIDE 5

Learning Visual Importance

slide-6
SLIDE 6
  • bottom-up pop-out

fonts, colors, styles

slide-7
SLIDE 7
  • bottom-up pop-out

fonts, colors, styles

slide-8
SLIDE 8
  • bottom-up pop-out

fonts, colors, styles

slide-9
SLIDE 9
  • bottom-up pop-out

fonts, colors, styles

  • design elements

title, annotation, visual

slide-10
SLIDE 10
  • bottom-up pop-out

fonts, colors, styles

  • design elements

title, annotation, visual

  • element locations

layout priors

slide-11
SLIDE 11

Retargeting Thumbnailing Design feedback

slide-12
SLIDE 12

Retargeting Thumbnailing Design feedback

slide-13
SLIDE 13

O’Donovan, Agarwala, Hertzmann [CHI’15] O’Donovan, Agarwala, Hertzmann [TVCG’14]

related work

Graphic Design Importance (GDI) dataset

slide-14
SLIDE 14

Pang, Cao, Lau, Chan [Siggraph Asia’16] Rosenholtz, Dorai, Freeman [ACM 2011]

related work

slide-15
SLIDE 15

How to define and measure importance?

  • Eye fixations
  • Mouse clicks
  • Explicit importance annotations

data collection

slide-16
SLIDE 16

What Makes a Visualization Memorable?[InfoVis 2013] Beyond Memorability: Visualization Recognition and Recall [InfoVis 2015]

Memory Eye-tracking Comprehension

massvis.mit.edu Recognized

eye fixations data collection data collection

eye fixations data collection

slide-17
SLIDE 17

Memory Eye-tracking Comprehension

massvis.mit.edu Recognized

eye fixations data collection data collection

eye fixations data collection

Eye fixations can give us important clues about how people perceive visualizations

What Makes a Visualization Memorable?[InfoVis 2013] Beyond Memorability: Visualization Recognition and Recall [InfoVis 2015]

slide-18
SLIDE 18

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Relative Importance Score

What design elements are most important?

eye fixations data collection data collection

eye fixations data collection

What Makes a Visualization Memorable?[InfoVis 2013] Beyond Memorability: Visualization Recognition and Recall [InfoVis 2015]

slide-19
SLIDE 19

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Relative Importance Score

What design elements are most important?

eye fixations data collection data collection

eye fixations data collection

What Makes a Visualization Memorable?[InfoVis 2013] Beyond Memorability: Visualization Recognition and Recall [InfoVis 2015]

0.8 0.6 0.4

slide-20
SLIDE 20

eye fixations data collection experimenter head stabilization infrared camera specialized hardware

slide-21
SLIDE 21

BubbleView: an interface for crowdsourcing image importance maps and tracking visual attention. [TOCHI, in press]

bubble clicks data collection

slide-22
SLIDE 22

Computing importance maps

bubble clicks data collection

BubbleView: an interface for crowdsourcing image importance maps and tracking visual attention. [TOCHI, in press]

slide-23
SLIDE 23

Fixations Clicks bubble clicks data collection

slide-24
SLIDE 24

bubble clicks data collection Fixations Clicks

slide-25
SLIDE 25

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Relative Importance Score eye gaze bubble clicks

Spearman’s r = 0.96

bubble clicks data collection

slide-26
SLIDE 26

Input Average Annotation Crowd Annotations annotations data collection

Graphic Design Importance (GDI) dataset

slide-27
SLIDE 27

Choosing an importance representation

Annotations Clicks annotations data collection

slide-28
SLIDE 28

data visualizations graphic designs

We create importance models for:

model details

slide-29
SLIDE 29

data visualizations graphic designs

We create importance models for:

model details

GDI Dataset 1078 designs MASSVIS Dataset 1411 visualizations

slide-30
SLIDE 30

Training our importance model

FCN-16s network

  • fully-automatic prediction
  • real-time performance

model details

slide-31
SLIDE 31

FCN adapted from semantic segmentation

FCN-32 FCN-16 skip connection REFINEMENT

FCN-16s network

model details

slide-32
SLIDE 32

FCN-16s

Bitmap design in, importance out

model details

slide-33
SLIDE 33

Ground truth Ground truth Prediction Prediction

We make importance predictions for:

graphic designs data visualizations results

slide-34
SLIDE 34

Ground truth Our model Judd DeepGaze SalNet SALICON

visualizations results

slide-35
SLIDE 35

Ground truth Our model Judd DeepGaze SalNet SALICON

visualizations results

CC↑ KL↓ Judd 0.11 0.49 SalNet 0.24 0.77 SALICON 0.54 0.76 DeepGaze2 0.54 0.47 DeepGaze 0.57 3.48 Our model 0.69 0.33

slide-36
SLIDE 36

visualizations results

slide-37
SLIDE 37

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Relative Importance Score eye gaze bubble clicks predictions

Ground truth Prediction Spearman’s r = 0.96

Is element importance preserved?

visualizations results

slide-38
SLIDE 38

Limitations

Ground truth Prediction

visualizations results

slide-39
SLIDE 39

graphic designs results

slide-40
SLIDE 40

Model Input

graphic designs results

slide-41
SLIDE 41

Model Input People Faces Text

O’Donovan, Agarwala, Hertzmann [TVCG’14]

graphic designs results

slide-42
SLIDE 42

graphic designs results

Ground truth OD-Full OD-Automatic Our model

slide-43
SLIDE 43

graphic designs results

Ground truth OD-Full OD-Automatic Our model

RMSE↓ R2↑ Saliency 0.229 0.462 OD-Automatic 0.212 0.539 Our model 0.203 0.576 OD-Full 0.155 0.754

slide-44
SLIDE 44

applications

Retargeting Thumbnailing Design feedback

slide-45
SLIDE 45

retargeting applications

Original design Importance heatmap Our model Edge-energy Judd DeepGaze

slide-46
SLIDE 46

retargeting

MTurk evaluation

  • better than: edge energy Judd saliency random crops
  • similar to: DeepGaze (deep natural image saliency)

Predicted importance performed:

≈

applications

slide-47
SLIDE 47

thumbnailing applications

Input Importance heatmap Thumbnail

slide-48
SLIDE 48

thumbnailing applications

slide-49
SLIDE 49

thumbnailing applications

slide-50
SLIDE 50

thumbnailing applications

Can retrieve visualizations more efficiently:

  • 1.96 clicks with importance-based thumbnails
  • 3.25 clicks with resized visualizations
slide-51
SLIDE 51

interactive applications

Design Improvement Dataset

slide-52
SLIDE 52

interactive applications

Prediction Ground truth

slide-53
SLIDE 53

interactive

visimportance.csail.mit.edu

applications

slide-54
SLIDE 54

website: visimportance.csail.mit.edu code: github.com/cvzoya/visimportance