Multimodal Knowledge Graphs Generation Methods, Applications, and - - PowerPoint PPT Presentation
Multimodal Knowledge Graphs Generation Methods, Applications, and Challenges ShihFu Chang Alireza Zareian, Hassan Akbari, Brian Chen, Svebor Karaman, Zhecan James Wang, and Haoxuan You Columbia University Prof. Heng Ji, Manling Li, Di Lu,
1
Alireza Zareian, Hassan Akbari, Brian Chen, Svebor Karaman, Zhecan James Wang, and Haoxuan You Columbia University
Manling Li, Di Lu, and Spencer Whitehead University of Illinois, Urbana‐Champaign
E
ntitie s, e ve nts, re la tio ns, e tc . 2
T e xt IE
Visit Isr ae l Princ e Willia m
The first-ever official visit by a British royal to Israel is underway. Prince William the 36-year-old Duke of Cambridge and second in line to the throne will meet with both Israeli and Palestinian leaders over the next three days.
E
ntitie s, e ve nts, re la tio ns, e tc .
E
ve nts de sc rib e wha t ha ppe ns
E
ntitie s a re c ha ra c te rize d b y the a rg ume nt ro le the y pla y in e ve nts
3
T e xt IE
Visit Isr ae l Princ e Willia m
The first-ever official visit by a British royal to Israel is underway Prince William the 36-year-old Duke of Cambridge and second in line to the throne will meet with both Israeli and Palestinian leaders over the next three days.
Age nt De stina tion
Applic a tio n: Que stio n Answe ring , Re a so ning , Hypo the sis Ve rific a tio n a nd Disc o ve ry
4
T e xt IE
Visit Isr ae l Princ e Willia m
F ind re c e nt visits o f po litic ia ns to I sra e l. Answe rs:
The first-ever official visit by a British royal to Israel is underway Prince William the 36-year-old Duke of Cambridge and second in line to the throne will meet with both Israeli and Palestinian leaders over the next three days.
Age nt De stina tion
TransportPerson_Instrument = stretcher
Stretcher Fire
5
Visua l da ta c o nta ins c o mple me nta ry da ta use d fo r:
Visua l I
llustra tio n
Disa mb ig ua tio n Additio na l De ta ils
6
Attac k Pr
s Bus Age nt T ar ge t Instr ume nt Stone T r anspor t Instr ume nt T r anspor t Wounde d pr
Age nt Pe r son Suppor te r s Pe r son De stina tion Ra lly
Cha lle ng e s:
Pa rsing ima g e s/ vide o s to struc ture s Gro unding e ve nt/ e ntitie s a c ro ss mo da litie s E
xtra c ting c o mple me nta ry multimo da l a rg ume nts
7
T e xt IE Visua l IE
?
Applic ation
Sc e ne g ra ph T e xt gr aph Multi- Moda l Knowle dg e Gra ph
E
xtra c t struc ture d re pre se nta tio n o f a sc e ne
E
ntitie s a nd the ir se ma ntic re la tio nships
8
Object Detection
E
xisting me tho d
E
xtra c t o b je c t pro po sa ls
Co nte xtua lize fe a ture s b y
RNN (o r me ssa g e pa ssing )
Cla ssify a ll no de s a nd
pa irs o f no de s
L
imitations
Co mputa tio na lly e xha ustive
𝑃 𝑜
fo r 𝑜 100 pro po sa ls
Diffic ult to mo de l hig he r
“girl e ating c ake with fo rk”
Re q uire s full supe rvisio n
9
(Xu et. al, CVPR 2017)
Neural Motifs (Zellers, Yatskar, Thomson, Choi, CVPR 2018) One of the SOTA methods for scene graph generation
Our wo rk: Visual Se mantic Parsing Ne twork (Zare ian e t al. CVPR19) Ge ne ra lize d fo rmula tio n o f sc e ne g ra ph g e ne ra tio n
E
ntity-c e ntric b ipa rtite re pre se nta tio n o f pre dic a te s & e ntitie s
Re duc e c o mputa tio na l c o mple xity fro m 𝑃 𝑜
to sub -q ua dra tic
Mo de l a rg ume nt ro le re la tio ns b e yo nd (sub je c t, o b je c t), (a g e nt, pa tie nt) re la tio ns
10
e ating holding be longs age nt patie nt Gir l Cake Hand F
instrume nt
Our wo rk: Visual Se mantic Parsing Ne twork (Zare ian e t al. CVPR20) Ge ne ra lize d fo rmula tio n o f sc e ne g ra ph g e ne ra tio n
E
ntity-c e ntric b ipa rtite re pre se nta tio n o f pre dic a te s & e ntitie s
Re duc e c o mputa tio na l c o mple xity fro m 𝑃 𝑜
to sub -q ua dra tic
Mo de l a rg ume nt ro le re la tio ns b e yo nd (sub je c t, o b je c t), (a g e nt, pa tie nt) re la tio ns
11
e ating holding be long age nt patie nt Gir l Cake Hand F
instrume nt
12
𝐼
1
𝐼
2
𝐼
𝑜
𝐼
1
𝐼
2
𝐼
3
𝐼
𝑜
… …
RPN Ro I Alig n T ra ina b le Pre dic a te E mb e dding Ba nk
I
nitia lize e ntity a nd pre dic a te no de s
Co mpute ro le -spe c ific a tte ntio n sc o re s
I
nput: e ntity-pre dic a te fe a ture pa irs
Output: sc a la r fo r e a c h the ma tic ro le
13
𝐼
1
𝐼
2
𝐼
𝑜
𝐼
1
𝐼
2
𝐼
3
𝐼
𝑜
… …
FC
age nt patie nt instrume nt
Bi- dir
e c tional Me ssage passing
E
ntitie s Role s Pr e dic ate s
14
𝐼
1
𝐼
2
𝐼
𝑜
𝐼
1
𝐼
2
𝐼
3
𝐼
𝑜
… … …
age nt patie nt instrume nt
… … … …
FC_
→ .
FC_
→ .
FC_
→ .
FC_
→ .
→
FC_
→
FC_
→ .
…
FC_
→
Bi- dir
e c tional Me ssage passing
E
ntitie s Role s Pr e dic ate s
15
𝐼
1
𝐼
2
𝐼
𝑜
𝐼
1
𝐼
2
𝐼
3
𝐼
𝑜
… … …
age nt patie nt instrume nt
… … … …
FC_
→
FC_
→
FC_
→
→
FC_
→
FC_
→.
…
FC_
→
Bi-dire c tio na l Me ssa g e pa ssing Re pe a t fo r 𝑣 ite ra tio ns Cla ssify no de s a nd e dg e s 16
𝐼
1
𝐼
2
𝐼
𝑜
𝐼
1
𝐼
2
𝐼
3
𝐼
𝑜
… … … … … … … …
age nt patie nt instrume nt
… … … …
e ating holding be long Gir l Cake Hand F
… …
FC
We akly supe r
vise d tr aining
Unkno wn a lig nme nt b e twe e n o utput a nd g ro und truth g ra phs 17
𝐼
1
𝐼
2
𝐼
𝑜
𝐼
1
𝐼
2
𝐼
3
𝐼
𝑜
… … … … … … … …
age nt patie nt instrume nt
… … … …
e ating holding be long Gir l Cake Hand F
… … Gr
uth
𝓜𝑭 𝓜𝑸 𝓜𝑺
Gir l | 𝐷1 Cake | 𝐷2 Hand| 𝐷3 F
e ating| 𝐷1 be long| 𝐷𝑜 holding| 𝐷2
18
CCV20)
L
ink c o nc e pts in sc e ne g ra phs to e xte rna l kno wle dg e b a se s suc h a s Co nc e ptNe t
Pa ss me ssa g e s o ve r b ridg e s b e twe e n
sc e ne g ra phs a nd e xte rna l g ra phs
Re fine b ridg e s b e twe e n g ra phs
19
20
Ours (GB- Ne t) Base line (KE RN) Ours (GB- Ne t) Base line (KE RN)
21 L
Bridg e visua l a nd te xt kno wle dg e g ra phs Witho ut using pre de fine d c la ssifie rs
Cha lle ng e s
Se nsitive to do ma in va ria tio ns Ab stra c t c o nc e pt no t g ro unda b le
Cha lle ng e s:
Pa rsing ima g e s/ vide o s to struc ture s Gro unding e ntitie s a c ro ss mo da litie s Jo int e xtra c tio n o f multimo da l
a rg ume nt
22
T e xt IE Visua l IE
?
Applic ation
Sc e ne g ra ph T e xt gr aph Multi- Moda l Knowle dg e Gra ph
23
Attac k Pr
s Bus Age nt T ar ge t Instr ume nt Stone T r anspor t Instr ume nt T r anspor t Wounde d pr
Age nt Pe r son Suppor te r s Pe r son De stina tion Ra lly
Event Movement.TransportPerson deploy Arguments Transporter United States Destination
Passenger soldiers Vehicle land vehicle Vehicle land vehicle Last week , U.S . Secretary of State Rex Tillerson visited Ankara, the first senior administration official to visit Turkey, to try to seal a deal about the battle for Raqqa and to
to Washington's backing of the Kurdish Democratic Union Party (PYD) militias. Turkish forces have attacked SDF forces in the past around Manbij, west of Raqqa, forcing the United States to deploy dozens of soldiers on the outskirts
risk derailing an assault on Raqqa.
Input: News article text and image
Output: Image‐related Events & Visual Argument Roles
land vehicle land vehicle
24
Event Conflict.Attack airstrikes Arguments Attacker U.S.-led coalition forces Target airplane Target vehicle
Output: Image‐related Events & Visual Argument Roles
Input: News article text and image
In March , Turkish forces escalated attacks on the YPG in northern Syria , forcing U.S. to deploy a small number of forces in and around the town of Manbij to the northwest
focus remains on Islamic State. Meanwhile, Raqqa is being pummeled by airstrikes mounted by U.S.-led coalition forces and Syrian warplanes. Local anti-IS activists say the air raids fail to distinguish between military and non-military targets …
airplane vehicle
25
place means
26
Linguistic Structure (Abstract Meaning Representation (AMR) / Dependency Tree) Visual Semantic Graph [Zareian et al. CVPR20]
27
28
(Li, Zareian, et al, ACL20)
Cross-Attention X – Loss
29
Cross-Attention X – Loss
30
Event Type Argument Roles Life.Die Agent, Victim, Instrument, Place, Time Transaction.TransferMoney Giver, Recipient, Beneficiary, Money, Instrument, Place, Time Conflict.Attack Attacker, Instrument, Place, Target, Time Conflict.Demonstrate Demonstrator, Instrument, Police, Place, Time Contact.Phone-Write Participant, Instrument, Place, Time Contact.Meet Participant, Place, Time Justice.ArrestJail Agent, Person, Instrument, Place, Time Movement.Transport Agent, Artifact/Person, Instrument, Destination, Origin, Time
31
(Li, Zareian, et al, ACL20)
32
Training with MM Multimodal Task
33
Missed by text-only model Misclassified by image-only model as “Demonstration”
Unde rsta nd se ma ntic s in ima g e s a nd la ng ua g e , e xplo re c o mmo nse nse Pro vide to -the -po int a nswe r 34
Ze lle rs e t al. CVPR 2019
E
Atte ntio n tra nsfo rme rs limite d to spa rse c o nne c tio ns in sc e ne g ra phs
35
[CL S] Why … ? [SE P] … Graph- base d Global- L
ransforme rs (GL AT , E CCV’20) … … pe r son1
e ntity pr e dic ate
subje c t c or e fe re nc e
ma sking Ima g e - te xt ma tc hing
e lation r e c ognition QA
pr e dic ate
1 2 3 4 5
CCV20)
36
1 2 3 4
laye r 2 laye r L … c onc at + line ar loc al he ads global he ads
1111
1
5555
5
laye r 1 Node Classifie r E dge Classifie r de c ode r
pe r son r iding be hind mountain ?
5 1 2 3 4 5 1 2 3 4 5
…
pe r son r iding be hind mountain bike
1 2 3 4 5
e ntity pr e dic ate
subje c t
pe r son r iding be hind mountain hor se
ground truth Node & E dge L
Model Type (Entity #, Predicate #) Q -> A LXMERT Initial Graph (36,18) 65.09 (baseline) Relevance Sel. (8, x) 74.04 (+8.95) GLAT
(LXMERT)
Initial Graph (36, 18) 65.24 (baseline) Relevance Sel. (26, x) 69.57 (+4.33) Relevance Sel. (18, x) 72.33 (+7.09) Relevance Sel. (8, x) 74.45 (+9.21)
Q: Why is sheep near the construction ? A: Sheep is near its natural habitat as well.
Initial Graph man, vest, pants, building, rock, sky, window, shirt
(sorted by confidence score from SG)
Relevance, Question building, door, man, men, window, rock, ground, animal
(sorted by relevance score against question)
Relevance, Question + Answer Candidate man, building, animal, dirt, rock, gate, ground, plant
(sorted by relevance score against question + answer candidate)
39
Figure 1. FDA approved drugs of most interest for repurposing as potential Ebola virus treatments.
KG from caption text FDA Drugs Ebola approve repurpose PDF images extraction, segmentation, and recognition Multimedia Knowledge Graph Construction Treatment
Multimo da l K
no wle dg e Gra phs
Unde rsta nding se ma ntic struc ture s in b o th la ng ua g e a nd visio n Jo int re pre se nta tio n a nd mo de ls
Applic a tio ns
Re a so ning (VCR) Disc o ve ry (COVI
D-19) Cha lle ng e s
Ope n-vo c a b ula ry a nd Se lf-Supe rvise d mo de ls K
no wle dg e g ra phs fo r vide o
Co mmo nse nse E
xtra c tio n fro m MM K G physic s, b e ha vio r, c a usa l/ te mpo ra l 40
T e xt IE Visua l IE
?
Applic ation
Sc e ne g ra ph T e xt gr aph Multi- Moda l Knowle dg e Gra ph
Za re ia n, Alire za , Sve b o r K
a ra ma n, a nd Shih-F u Cha ng . "We a kly Supe rvise d Visua l Se ma ntic Pa rsing ." Pro c e e ding s o f the I E E E / CVF Co nfe re nc e o n Co mpute r Visio n a nd Pa tte rn Re c o g nitio n. CVPR 2020.
Za re ia n, Alire za , Sve b o r K
a ra ma n, a nd Shih-F u Cha ng . "Bridg ing kno wle dg e g ra phs to g e ne ra te sc e ne g ra phs." a rXiv pre print a rXiv:2001.02314 (2020). E CCV 2020.
Akb a ri, Ha ssa n, Sve b o r K
a ra ma n, Sura b hi Bha rg a va , Bria n Che n, Ca rl Vo ndric k, a nd Shih-F u Cha ng . "Multi-le ve l multimo da l c o mmo n se ma ntic spa c e fo r ima g e -phra se g ro unding ." I n Pro c e e dings o f the I
E E E Co nfe re nc e o n Co mpute r Visio n and Patte rn Re c o gnitio n. 2019.
L
i, Ma nling , Alire za Za re ia n, Qi Ze ng , Spe nc e r White he a d, Di L u, He ng Ji, a nd Shih-F u Cha ng . "Cro ss-me dia Struc ture d Co mmo n Spa c e fo r Multime dia E ve nt E xtra c tio n." arXiv pre print arXiv:2005.02472 (2020). ACL 2020.
Za re ia n, Alire za , Ha o xua n Yo u, Zhe c a n Wa ng , a nd Shih-F
u Cha ng . "L e a rning Visua l Co mmo nse nse fo r Ro b ust Sc e ne Gra ph Ge ne ra tio n." arXiv pre print
arXiv:2006.09623 (2020). E
CCV 2020. 41