SPICE: Semantic Propositional Image Caption Evaluation Presented to - - PowerPoint PPT Presentation
SPICE: Semantic Propositional Image Caption Evaluation Presented to the COCO Consortium, Sept 2016 Peter Anderson 1 , Basura Fernando 1 , Mark Johnson 2 and Stephen Gould 1 1 Australian National University 2 Macquarie University ARC Centre of
ARC Centre of Excellence for Robotic Vision ARC Centre of Excellence for Robotic Vision 1
Peter Anderson1, Basura Fernando1, Mark Johnson2 and Stephen Gould1
1 Australian National University 2 Macquarie University
www.roboticvisi
ARC Centre of Excellence for Robotic Vision 2
Source: MS COCO Captions dataset Source: http://aipoly.com/
www.roboticvisi
ARC Centre of Excellence for Robotic Vision 3
www.roboticvisi
ARC Centre of Excellence for Robotic Vision 4
Source: Lin Cui, Large-scale Scene UNderstanding Workshop, CVPR 2015
www.roboticvisi
ARC Centre of Excellence for Robotic Vision 5
www.roboticvisi
ARC Centre of Excellence for Robotic Vision 6
(High n-gram similarity)
(Low n-gram similarity) …n-gram overlap is not necessary or suffjcient for two sentences to mean the same A young girl standing on top
court. A giraffe standing on top
A shiny metal pot filled with some diced veggies. The pan on the stove has chopped vegetables in it.
Source: MS COCO Captions dataset
…SPICE primarily addresses false positives
www.roboticvisi
ARC Centre of Excellence for Robotic Vision 7
www.roboticvisi
ARC Centre of Excellence for Robotic Vision 8
www.roboticvisi
ARC Centre of Excellence for Robotic Vision 9
1 Johnson et. al. Image Retrieval Using Scene Graphs, CVPR 2015 2 Klein & Manning: Accurate Unlexicalized Parsing, ACL 2003 3 Schuster et. al: Generating semantically precise scene graphs from textual descriptions for
improved image retrieval, EMNLP 2015
(court) (girl, young) (girl, standing) (court, tennis) (girl, on-top-of, court)
uples
www.roboticvisi
ARC Centre of Excellence for Robotic Vision 10
www.roboticvisi
ARC Centre of Excellence for Robotic Vision 11
www.roboticvisi
ARC Centre of Excellence for Robotic Vision 12
www.roboticvisi
ARC Centre of Excellence for Robotic Vision 13
www.roboticvisi
ARC Centre of Excellence for Robotic Vision 14
www.roboticvisi
ARC Centre of Excellence for Robotic Vision 15
Pearson ρ correlation between evaluation metrics and human judgments for the 15 competition entries plus human captions in the 2015 COCO Captioning Challenge, using 40 reference captions.
Source: Our thanks to the COCO Consortium for performing this evaluation using MS COCO Captions C40.
www.roboticvisi
ARC Centre of Excellence for Robotic Vision 16
SPICE picks the same top-5 as human evaluators.
Absolute scores are lower with 40 reference captions (compared to 5 reference captions)
Source: Our thanks to the COCO Consortium for performing this evaluation using MS COCO Captions C40.
www.roboticvisi
ARC Centre of Excellence for Robotic Vision 17
*Hale, J: A probabilistic Earley Parser as a Psycholinguistic Model 2001; Levy, R: Expectation-based syntactic comprehension 2008
www.roboticvisi
ARC Centre of Excellence for Robotic Vision 18
Breakdown of SPICE F-score over objects, attributes and relations
www.roboticvisi
ARC Centre of Excellence for Robotic Vision 19
Breakdown of attribute F-score over color, number and size attributes
www.roboticvisi
ARC Centre of Excellence for Robotic Vision 20
ARC Centre of Excellence for Robotic Vision ARC Centre of Excellence for Robotic Vision 21
Link: SPICE Project Page (http://panderson.me/spice) Acknowledgement: We are grateful to the COCO Consortium for re-evaluating the 2015 Captioning Challenge entries using SPICE.