UNDERSTANDING AND PREDICTING IMAGE MEMORABILITY AT A LARGE SCALE - - PowerPoint PPT Presentation

▶
understanding and predicting image memorability at a
SMART_READER_LITE
LIVE PREVIEW

UNDERSTANDING AND PREDICTING IMAGE MEMORABILITY AT A LARGE SCALE - - PowerPoint PPT Presentation

Sep 26, 2023 414 likes •661 views

UNDERSTANDING AND PREDICTING IMAGE MEMORABILITY AT A LARGE SCALE A. Khosla, A. S. Raju, A. Torralba and A. Oliva Experiments by: Tyler Folkman QUICK DEMO LaMem Demo POPULARITY DATA Random sample of scene categories from SUN dataset.

slide-1
SLIDE 1

UNDERSTANDING AND PREDICTING IMAGE MEMORABILITY AT A LARGE SCALE

  • A. Khosla, A. S. Raju, A. Torralba and A. Oliva

Experiments by: Tyler Folkman

slide-2
SLIDE 2

QUICK DEMO

  • LaMem Demo
slide-3
SLIDE 3

POPULARITY DATA

  • Random sample of scene categories from SUN dataset.
  • Task was to press the space bar whenever they saw an

identical repeat of an image at any time in the sequence.

  • Memorability score defined as percentage of correct

detections.

  • 2,222 target images.
slide-4
SLIDE 4

RANK CORRELATION

Human Performance State of the Art MemNet Popularity Data 0.75 0.54* 0.52

* Isola, P ., Xiao, J., Torralba, A., Oliva, A. What makes an image memorable? IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2011. Pages 145-152.

slide-5
SLIDE 5

T-SNE EMBEDDINGS

slide-6
SLIDE 6
slide-7
SLIDE 7
slide-8
SLIDE 8
slide-9
SLIDE 9
slide-10
SLIDE 10

DO PEOPLE MAKE IMAGES MEMORABLE?

  • 17.8% of the data have pedestrians detected in them.
  • 2.4% have faces detected.
  • Pedestrians detected using HOG features and faces

using Haar feature-based cascade classifiers.

  • What if these people were blurred?
slide-11
SLIDE 11

BASELINES

0.73 0.73 0.72

slide-12
SLIDE 12

BLURRING PEOPLE

0.80 0.70 Actual Memorability: 0.90

slide-13
SLIDE 13

All Images Images with People Images without People

Number of Images Change in memorability (normal - blurred)

slide-14
SLIDE 14

Blurring detected pedestrians or faces doesn’t seem to consistently decrease memorability

slide-15
SLIDE 15

REMOVING PEOPLE

0.80 0.75 Actual Memorability: 0.90

slide-16
SLIDE 16

All Images Images with People Images without People

Number of Images Change in memorability (normal - removed)

slide-17
SLIDE 17

Removing people or faces from images shows stronger signs of decreasing memorability, but still not very conclusive.

slide-18
SLIDE 18

ADDING FACE TO IMAGES

  • What happens if we paste a face into all of our

images?

Actual: 0.61 Predicted: 0.60 Predicted with Face: 0.62

slide-19
SLIDE 19

All Images Images with People Images without People

Number of Images Change in memorability (added faces - normal)

slide-20
SLIDE 20

Adding a face to images seems to increase memorability

slide-21
SLIDE 21

SUMMARY

  • MemNet generalizes to the popularity dataset - approaching state-of-the-art results

(without fine-tuning).

  • t-SNE embeddings suggest people might improve memorability while landscapes and

structures are not very memorable.

  • Inconclusive results when blurring/removing people in images and its effects on
  • MemNet. Perhaps stronger results if hand blur all people.
  • Adding a single 27x27 face to images looks to boost predicted memorability especially

for images with no people.

  • Adding or removing people from images may be changing predicted memorability for
  • ther reasons.
slide-22
SLIDE 22

REFERENCES

  • http://web.mit.edu/phillipi/Public/

WhatMakesAnImageMemorable/

  • http://memorability.csail.mit.edu/