Knowlywood: Mining Activity Knowledge From Hollywood Narratives - - PowerPoint PPT Presentation

Knowlywood: Mining Activity Knowledge From Hollywood Narratives

Date:2016/08/30 Author:Nilet Tandon, Gerard de Melo, Abir De, Gerhard Wrikum Source:CIKM’15 Advisor:Jia-Ling Koh Speaker:Pei-Hao Wu

1

Outline

• Introduction
• Method
• Experiment
• Conclusion

2

Introduction

• Motivation
• Major knowledge graphs focus on factual knowledge
• Ex: songs and awards of an artist, CEOs and products of companies, etc.
• With ground-breaking new products like Google Now, Apple’s Siri, there is need for

commonsense knowledge

• enabling smart interpretation of queries relating to everyday human activities

3

Introduction

• Goal
• Automatically compiling large amounts of knowledge about human activities from

narrative text

4

Introduction

• Flow chart

5

Outline

• Introduction
• Method
• Experiment
• Conclusion

6

Method

• Input sentence: The man began to shoot a video in the moving bus
• ClausIE:
• (“the man”, ”began to shoot”, “a video”), (“the man”, ”began to shoot”, ”in the

moving bus”)…etc.

• OpenNLP:
• (“the man”), (“began to shoot”), (“a video”), (“in”), (“the moving bus”)

7

Method

• Sense Analysis - VerbNet
• Providing the syntactic frame
• Ex: Agent.animate V Patient.animate PP Instrument.solid
• Selectional restriction
• Ex: Patient.animate requires this patient be a lining being
• Sense Analysis - WordNet
• Mapping the word to disambiguated WordNet senses
• Ex: shoot -> shoot#1、shoot#2…etc.

8

Method

• WSD system It-Makes-Sense(IMS)
• For an initial disambiguation of word
• Notation
• Sw: the set of candidate WordNet sense
• Sv: the set of candidate VerbNet sense
• i: word
• j: sense

9

Method

• Most-frequent-sense rank
• Additional feature used in the ILP
• Notation
• Sw: the set of candidate WordNet sense
• Sv: the set of candidate VerbNet sense
• i: word
• j: sense

10

Method

• 𝑡𝑧𝑜𝑗𝑘
• Frame match score for word i and VerbNet sense j
• 𝑡𝑓𝑛𝑗𝑘
• Selectional restriction score of the roles in a VerbNet frame j for word i

11

Method

• ILP Model
• 𝑌𝑗𝑘 = 1 if word i is mapped to sense j
• At most one sense is chosen for each word

12

Method

• Graph inference
• Connection between different activity frames
• parent type(T)、semantic similarity edges(S)、temporal order(P)
• Using PSL framework for relational learning and inference
• Edge Priors
• An activity as a (verb-sense, noun-sense) pair
• Using WordNet’s taxonomic hierarchy to estimate T and S
• Using GSP to find P edge

13

Method

• T edge
• The prior between two pairs(v1,n1),(v2,n2) is calculate as a score t(v1,v2)*t(n1,n2)
• For noun sense: using WordNet hypernymy
• For verb sense: WordNet hypernymy and VerbNet verb hierarchy
• The score is 1 if parent and child are connected and 0 otherwise
• Ex: “go up an elevation” is the parent type of “hike up a hill”

14

Method

• S edge
• The prior between two pairs(v1,n1),(v2,n2) is calculate as a score sim(v1,v2)*sim(n1,n2)
• For noun sense: using WordNet path similarity measure
• For verb sense: using WordNet groups and VerbNet class membership
• Ex: “climb up a mountain” is semantic similarity to “hike up a hill”

15

Method

• P edge
• Using GSP to efficiently determine P edge
• maximum gap=4、minimum support=3
• Ex
• Input: “wake up”->“drink water”->“brush teeth”->“eat breakfast”->“go out”
• Output: “wake up”->”go out”

16

support = 𝑔𝑠𝑓𝑟(𝑏1 𝑞𝑠𝑓𝑤 𝑏2) 𝑔𝑠𝑓𝑟 𝑏1 𝑔𝑠𝑓𝑟(𝑏2)

Method

• PSL
• Computing a cleaner graph of T, S, and P edges with scores
• PSL model with the following soft first-order logic rules
• Parents often inherit prev. (P) edges from their children
• P(a,b) Λ T(a,a’) Λ T(b,b’) => P(a’,b’)
• Similar activities are likely to share parent types
• S(a,b) Λ T(b, 𝑐0) => T(a, 𝑐0)

17

Method

• Taxonomy construction
• Synsets
• The previous steps may produce overly specific activities
• Ex: “embrace spouse”, “hug wife”, “hug partner”, etc.
• Grouping similar activities together into a single frame
• Pruning S from the previous step for activity merging
• WordNet path similarity as a measure of semantic distance

18

Method

• Taxonomy construction
• Hierarchy
• Some of activities may subsume others
• Ex: “divorce husband” is subsumed by “break up with a partner”
• “break up with a partner” is more general than “divorce husband”
• Pruning T from the previous step for activity hierarchy induction
• An activity taxonomy is a directed acyclic graph (DAG)
• Using WordNet path similarity but only consider hypernym

19

Outline

• Introduction
• Method
• Experiment
• Conclusion

20

Experiment

• System components
• Data processing
• 1.89 million scenes from several sources
• 560 movie scripts, scripts of 290 TV series, and scripts of 179 sitcoms form wikia.com and

dailyscript.com

• 103 novels for Project Gutenberg
• Textual descriptions of videos about cooking

21

Experiment

• System components
• Had human judges annotate at least 250 random samples

22

Experiment

• Knowlywood KB evaluation
• Compiled a random sample of 119 activities from the KB
• Expert human annotators to judge each attribute, and we compute the precision as

𝑑 𝑑+𝑗

• c: counts of correct, i: counts of incorrect
• Comparison with ConceptNet
• Mapped CN’s relations to our notion of activity attribute

23

Experiment

• Knowlywood KB evaluation
• Comparison with ReVerb
• ReVerb: a system which aims at mining all possible subject-predicate-object triple from text
• Using MovieClips tag to map word to tag
• Two datasets as input to ReVerb
• ReVerbMCS: script data that we used for our system
• ReVerbClue: ClueWeb09

24

Experiment

• Knowlywood KB evaluation
• Movie scene tagging
• Selecting 1000 clips from Movieclips.com as gold data
• Giving [participant, location, time] and then assess top-k activity recommendations

25

Outline

• Introduction
• Method
• Experiment
• Conclusion

26

Conclusion

• This paper presented Knowlywood, the first comprehensive KB of human

activities

• The one million activity frames is an important asset for a variety of

applications such as image and video understanding

27