We Are Humor Beings: Understanding and Predicting Visual Humor - - PowerPoint PPT Presentation

We Are Humor Beings: Understanding and Predicting Visual Humor

Shuai Wang

University of Toronto

March 29, 2016

1 / 31

Intro

◮ An integral part but not understood in detail

2 / 31

Intro

◮ An integral part but not understood in detail ◮ An adult laughs 18 times a day

3 / 31

Intro

◮ An integral part but not understood in detail ◮ An adult laughs 18 times a day ◮ A good sense humor

◮ is related to communication competence ◮ helps raise an individual’s social status & popularity ◮ even helps attract compatible mates ◮ makes yourself happier :) 4 / 31

What makes an image funny?

5 / 31

Humor Techniques

◮ Animal doing something unusual ◮ Person doing something unusual ◮ Somebody getting hurt ◮ Somebody getting scared

6 / 31

Animal doing something unusual

7 / 31

Person doing something unusual

8 / 31

Somebody getting hurt

9 / 31

Somebody getting scared

10 / 31

Changing objects can alter the funniness of a scene

11 / 31

Removing Incongruities

An elderly person kicking a football while skateboarding is incongruous, but a young girl doing so is not

12 / 31

Adding Incongruities

Add incongruities (and humor) by replacing the expected with the unexpected

13 / 31

Two Tasks to Understand Visual Humor

◮ Predicting how funny a given scene is (scene-level) ◮ Changing the funniness of a scene (object-level)

14 / 31

Object-level Features

◮ Object embedding (150-d): captures the context in which

an object usually occurs

◮ Local embedding (150-d): weighted sum of object

embeddings of all other instances

15 / 31

Scene-level Features

◮ Cardinality (150-d): bag-of-words representation of how

many instances of each object are in the scene

◮ Location (300-d): horizontal and vertical coordinates of every

• bject (closest to the center if multiple instance)

◮ Scene embedding (150-d): sum of object embeddings of all

• bjects in the scene

16 / 31

Predicting Funniness Score

◮ Dataset: 6,400 scenes, with funny score from 1-5 labelled by

workers from Amazon Mechanical Turk

17 / 31

Predicting Funniness Score

◮ Dataset: 6,400 scenes, with funny score from 1-5 labelled by

workers from Amazon Mechanical Turk

◮ Support Vector Regressor (SVR) on scene-level features

18 / 31

Predicting Funniness Score

◮ Dataset: 6,400 scenes, with funny score from 1-5 labelled by

workers from Amazon Mechanical Turk

◮ Support Vector Regressor (SVR) on scene-level features ◮ Metric: average relative error

19 / 31

Predicting Funniness Score: Ablation Analysis

Different feature subsets perform about the same: slightly better than baseline (average score of the training scenes)

20 / 31

Alter Funniness of a Scene

◮ Detect the objects that do (or do not) contribute to humor ◮ Identify which objects should replace the objects from step 1

21 / 31

Predicting Objects to be Replaced

◮ On average, the model replaces 3.67 objects (2.54 ground

truth) → this bias towards replace ensures a large ‘margin’

22 / 31

Predicting Objects to be Replaced

◮ On average, the model replaces 3.67 objects (2.54 ground

truth) → this bias towards replace ensures a large ‘margin’

◮ Animate objects like humans and animals are more likely

sources of humor → tends to replace these objects

23 / 31

Funny → Unfunny

Old man dancing → young boy dancing Hawk stealing meat → baseball

24 / 31

Funny → Unfunny

Cute puppy → Insect Watermelon → Ax

25 / 31

Unfunny → Funny

Couple having dinner at the table → Puppies having dinner at the table

26 / 31

Unfunny → Funny

Cating playing around → Racoon driving motorcycle

27 / 31

Discussion

◮ Style/genre of an image or painting can make a difference

28 / 31

Discussion

◮ Style/genre of an image or painting can make a difference ◮ Dataset is small: 6,400 images

29 / 31

Discussion

◮ Style/genre of an image or painting can make a difference ◮ Dataset is small: 6,400 images ◮ Feature representation can be improved

30 / 31

31 / 31