CSM-EE727 6 Seconds of Sound and Vision: Creativity in - - PowerPoint PPT Presentation

CSM-EE727

6 Seconds of Sound and Vision: Creativity in Micro-Videos

Authors

Miriam Redi, Neil O’Hare, Rossano Schifanella, Michele Trevisiol, Alejandro Jaimes

Published

CVPR14

Lausanne - May 11th 2020

* https://www.thedrum.com/news/2016/10/28/where-twitter-went-wrong-with-vine

• Study the notion of “creativity”.
• Use short/micro-videos (eg: Vine, Instagram, Facebook).
• Vine emergence (google trends)

○

1st peak: July 2013.

○

CVPR14 submission: November 2013.

2

Introduction

6 SECONDS OF SOUND AND VISION: CREATIVITY IN MICRO-VIDEOS

• Research questions:

○

Can we understand/generate features that are linked to a creative video ?

○

Can we automatically detect it ?

• Contributions:

○

Microvideo labelled dataset.

○

New set of features (audio-visual) to describe novelty and aesthetic value of videos.

○

Correlation between generated features and creativity.

3

Introduction

6 SECONDS OF SOUND AND VISION: CREATIVITY IN MICRO-VIDEOS

• What is creativity ?

“All who study creativity agree that for something to be creative, it is not enough for it to

be novel: it must have value, or be appropriate to the cognitive demands of the situation.”

• In this paper:

4

Definition

6 SECONDS OF SOUND AND VISION: CREATIVITY IN MICRO-VIDEOS

• Only a small fraction of video are creative (1.9%).
• 4’000 videos

○

a) 1’000: contains #vineart, #artwork, #vineartist, #vineartgallery

○

b) 200: mentioned in online article about vine creativity

○

c) 2’300: video authored by creators identified in previous criteria.

○

d) 500: random video from streamline

Christian Abbet 5

Dataset

6 SECONDS OF SOUND AND VISION: CREATIVITY IN MICRO-VIDEOS

• Labelled with Crowdflower (now Figure Eight).
• Quality checked with “Gold Standard”.
• 5 labels (yes, no, idk) per video.

Christian Abbet 6

Dataset

6 SECONDS OF SOUND AND VISION: CREATIVITY IN MICRO-VIDEOS

Christian Abbet 7

Features Modelling

6 SECONDS OF SOUND AND VISION: CREATIVITY IN MICRO-VIDEOS

Christian Abbet 8

Experimental Results - Features

6 SECONDS OF SOUND AND VISION: CREATIVITY IN MICRO-VIDEOS

• Analysis on set D-100 (clean version)
• Use 7 sets as described in feature

modelling

• Use Multiple Correlation Coefficient

(MPC) that link correlation between multidimensional variable and unidimensional one.

• Both aesthetic ( ) and novelty ( ) are

important.

• Emotional less correlated than

sensory.

• Audio not so important.

Christian Abbet 9

Experimental Results - Features

6 SECONDS OF SOUND AND VISION: CREATIVITY IN MICRO-VIDEOS

Correlation (pearson) between individual features and labels.

• Homogeneity ( )
• Video warm and bright colors ( )
• Uniqueness ( )
• No presence of people, skin color ( )
• Pleasant emotions ( )
• No loop and shake ( )

Christian Abbet 10

Experimental Results - Classification

6 SECONDS OF SOUND AND VISION: CREATIVITY IN MICRO-VIDEOS

• Classification on 3 sets (D-60, D-80, D-100).
• ⅔ Training, ⅓ Testing and balanced sets
• Use of SVM (RBF kernel) on all (7) feature sets.
• Data augmentation with feature that rely on one frame.
• Use 1000 unlabeled video to compute KMeans clustering.

Christian Abbet 11

Experimental Results

6 SECONDS OF SOUND AND VISION: CREATIVITY IN MICRO-VIDEOS

Christian Abbet 12

Conclusion

6 SECONDS OF SOUND AND VISION: CREATIVITY IN MICRO-VIDEOS

• Defining creative videos as videos that are novel and have aesthetic

value.

• If sampling is random, only 1.9% f video are considered as creative
• Dataset of > 3800 micro-videos.
• New features (aesthetic and visual) proposed.
• Positive correlation for homogeneity, warm colors, pleasant

emotions.

• Negative correlation for presence of people, loop and shaky video.
• 80% accuracy on test D-100 set.
• Future work:

○

Feature to model intellectual aspect.

○

Include meta-data (non audiovisual), such as tags, tweet, user profile.

○

Include other video platforms.

Christian Abbet 13

Thank you :)

6 SECONDS OF SOUND AND VISION: CREATIVITY IN MICRO-VIDEOS