Semantic Spaces for Zero-Shot Behaviour Analysis Xun Xu Computer - PowerPoint PPT Presentation

Semantic Spaces for Zero-Shot Behaviour Analysis Xun Xu Computer Vision and Interactive Media Lab, NUS Singapore 1

Collaborators Prof. Shaogang Gong Dr. Timothy Hospedales 2

Outline • Background • Transductive Zero-Shot Action Recognition • Multi-Task Zero-Shot Embedding • Zero-Shot Crowd Analysis 3

Video Behaviour Defined as Visually Distinguishable Activities • Human Actions • Crowd Behaviour 4

Human Actions • Individual or multiple interactive human activities 5 Soomro, et al. “UCF101 : A Dataset of 101 human actions classes from videos in the wild.” 2012

Human Actions Tasks • Action Recognition Eye Makeup Rafting Swimming Fencing Diving Archery 6

Human Actions Tasks • Action Detection (Retrieval) Given query “Swimming” return ranked videos Lower Ranking …… 7

Crowd Behaviour • A group of people acting collectively 8 Shao, J., et al. “Deeply learned attributes for crowded scene understanding .” CVPR 2015

Crowd Behaviour Tasks • Crowd Behaviour Profiling 9

Crowd Behaviour Tasks • Crowd Anomaly Detection 10 Hassner, T., et al. “Violent flows: Real-time detection of violent crowd behavior .” CVPR 2012

Potential Applications Human Computer Interaction Surveillance Video Sharing 11

Outline • Background • Transductive Zero-Shot Action Recognition • Multi-Task Zero-Shot Embedding • Zero-Shot Crowd Analysis 12

Motivation • Ever Increasing #Categories for action recognition 2004 2005 2010 Weizmann 9 Classes KTH 6 Classes Olympic Sports 16 Classes 2011 2012 2015 203 Classes UCF101 101 Classes HMDB51 51 Classes 13

Motivation • Ever Increasing #Categories Limitations 2004 2005  Expensive to collect training data 2010 Weizmann 9 Classes KTH 6 Classes Olympic Sports 16 Classes  Annotating video is costly 2011 2012 2015 203 Classes UCF101 101 Classes HMDB51 51 Classes 14

Zero-Shot Learning (ZSL) • Can we use videos from known class to help predict videos from unknown classes? Known Classes Unknown Classes Shot-Put Hammer Throw Discus Throw 15

Attribute Semantic Space • Attribute Based Attributes Hammer Throw Throw Away Outdoor Discus Throw Turn Around Ball Bend 16

Attribute Semantic Space • Attribute Based Attributes Hammer Throw Throw Away Outdoor Discus Throw Turn Around Ball Shot-put Bend Known a priori 17

Attribute Semantic Space • Attribute Based Attributes Hammer Throw Throw Away Test video Outdoor Discus Throw Turn Around Ball Shot-put Bend 18

Attribute Semantic Space • Attribute Based Attributes Limitations Discus Throw Throw Away • Ontological problem Outdoor Hammer • Manual label attributes is Throw Turn costly for videos Around • Incompatible with other Ball Shot-put attribute sets Bend 19

Word-Vector Semantic Space Feature Space X Word-Vector Space Z Hammer Discus Throw = [0.2 0.5 0.1 …] Throw  ( ) z f x Discus Throw Hammer Throw = [0.1 0.6 0.1 …] 20

Word-Vector Semantic Space Feature Space X Word-Vector Space Z Hammer Discus Throw = [0.2 0.5 0.1 …] Throw ShotPut = [0.3 0.4 0.2 …] Discus Throw Hammer Throw = [0.1 0.6 0.1 …] 21

Semantic Word-Vector • Skip-gram model predicts adjacent words 1 T   max log p(z | z ) t  j t T { z }      t c j c , j 1 0 T exp(z z )   i j p(z z ) | i j T exp(z z ) i j i Result of this optimization vec (“ball”)=[ -0.004 0.01 0.01 -0.03 0.05] vec (“sword”)=[0.16 0.06 0.09 -0.06 -0.002] vec (“archery”)=[0.02 0.01 0.02 -0.03 -0.03] vec (“boxing”)=[ -0.08 -0.01 0.15 -0.01 0.09] Mikolov, T., et al. "Distributed representations of words and phrases and their compositionality .” NIPS2013 22 Pennington, J., et al. "Glove: Global vectors for word representation." EMNLP 2014.

Benefits • Geometric Meaningful Word-Vector Space ship Far Away Run cat Walk Closer dog 23

Benefits • Unsupervised Semantic Space 24

Benefits • Wide coverage of words Vec (“Apple”) = [0.2 0.3 0.1 …] Vec (“Bear”) = [0.1 0.9 0.1 …] Vec (“Car ”) = [0.6 0.2 0.4 …] Vec (“Desk”) = [0.2 0.8 0.4 …] Vec (“Fish”) = [0.5 0.2 0.3 …] … 25

Benefits • Uniform across datasets Dataset 1 Dataset 2 Discus Throw = [0.2 0.5 …] Discus Throw = [0.2 0.5 …] HammerThrow = [0.1 0.2 …] HammerThrow = [0.1 0.2 …] 26

Challenges • Domain Shift Feature Space X Semantic Vector Space Y Discus Throw Hammer Throw HammerThrow Sword Exercise Discus Throw Play Guitar 27

Challenges • Domain Shift Feature Space X Semantic Vector Space Y Discus Throw Hammer Throw HammerThrow Sword Exercise Discus Throw Confusion Play Guitar 28

Our Solution 29 Xu, X., et al. “ Transductive Zero-Shot Action Recognition by Word-Vector Embedding .” IJCV 2017

Our Solution 30 Xu, X., et al. “ Transductive Zero-Shot Action Recognition by Word-Vector Embedding .” IJCV 2017

Low-Level Visual Feature • Improved Trajectory Feature for x 31 Wang, H. and Schmid , C., et al. “Action recognition with improved trajectories,” ICCV13

Our Solution 32 Xu, X., et al. “ Transductive Zero-Shot Action Recognition by Word- Vector Embedding.” IJCV 2017

Combinations of Multi Words • A phrase is constructed from single word vectors Additive Composition vec (“Apply Eye Makeup”) = vec (“Apply”) + vec (“Eye”) + vec (“Makeup”) vec (“Brushing Teeth”) = vec (“Brushing”) + vec (“Teeth”) vec (“Playing Guitar”) = vec (“Playing”) + vec (“Guitar”) 33

Our Solution 34 Xu, X., et al. “ Transductive Zero-Shot Action Recognition by Word- Vector Embedding.” IJCV 2017

Visual to Semantic Mapping by Regularized Linear Regression • Multi-Dimensional Regularized Linear Regression N  2 2    min z Wx W i i 2 2 W  i 1 W x is N Dimension z is D Dimension x 1 z 1 Feature Space Semantic Space x 2 z 2 …… x 3 … 35

Domain Shift – Semi Supervised (Manifold Regularized) Regression • Semi-supervised regression is applied to tackle domain shift which takes test data distribution into consideration trg X Target Train Data tr X trg Target Test Data te Train and Test Data in Feature Space X  X trg tr tr  trg X X te te KNN Graph KNN Graph to model Manifold weight     2       f x f x : x [ X ;X ] Manifold Regularizor ij i j tr te 2 36

Domain Shift – Semi Supervised (Manifold Regularized) Regression • Semi-supervised regression is applied to tackle domain shift which takes test data distribution into consideration trg X Target Train Data tr X trg Target Test Data te KNN Graph to model Manifold N   2 2 2        min z Wx W Wx Wx i i ij i j 2 2 2 W  i ij 1 37

Our Solution Additional datasets are available 38 Xu, X., et al. “ Transductive Zero-Shot Action Recognition by Word- Vector Embedding.” IJCV 2017

Data Augmentation • Use more training data from Auxiliary Dataset to help learn a better regression Augmented Train and Test Data in Feature Space X  [ X trg ; X aux ] tr tr  trg X X te te trg X Target Train Data tr aux Auxiliary Data X trg X Target Test Data te trg X Target Dataset Train Data tr (e.g. HMDB51) Data Augmentation X aux Auxiliary Dataset Data More Data is considered to learn more robust regressor (e.g. UCF101) 39

Semantic Word Vector Approach 40

Zero-Shot Recognition by Nearest Neighbor • Do nearest Neighbor search in word-vector space to predict category of test data HulaHoop Fencing Basketball W Diving TestData Kayaking Minimal distance Rafting TaiChi Category Name Test Video Instance 41

Domain Shift – SelfTraining • Self-training is applied to tackle domain shift Category Name  z f ( x ) Test Video Instance te z z  Z("Taichi") g("Taichi") 2 3 K 1  *  Z ("Taichi") z z z te K 4 Z("Taichi") 1 z  NN( Z("Taichi"),K ) te z , K ) is the KNN function NN( Z proto 5 z 7 z z 4 NN example 8 Z ("Taichi") * 6     Z ("Taichi") * ( z z z z ) 4 5 6 7 8 42

Domain Shift – SelfTraining • Self-training is applied to tackle domain shift Category Name  z f ( x ) Test Video Instance te z z  Z("Taichi") g("Taichi") 2 3 K 1  *  Z ("Taichi") z z z te K 4 Z("Taichi") 1 z  NN( Z("Taichi"),K ) te z , K ) is the KNN function NN( Z proto 5 z 7 z z 4 NN example 8 Z ("Taichi") * 6     Z ("Taichi") * ( z z z z ) 4 5 6 7 8 43