SLIDE 1
Weilong Yang Simon Fraser University
1
Action Recognition and Detection
2
Walking Running Jogging Boxing Waving T X Y
Learning Transferable Distance Functions For Human Action - - PowerPoint PPT Presentation
Learning Transferable Distance Functions For Human Action - - PowerPoint PPT Presentation
Learning Transferable Distance Functions For Human Action Recognition and Detection Weilong Yang Simon Fraser University 1 Action Recognition and Detection Walking Running Jogging Boxing Waving T X Y 2 Applications Action related
SLIDE 2
SLIDE 3
3
Applications
Action related video search
Sports and Dancing video search
Event Detection
Automatic abnormality detection in surveillance
videos
SLIDE 4
Motivation
On KTH & Weizmann action datasets, almost
100% accurancy is achieved. [Jhuang et al. ICCV07,
Fathi & Mori CVPR08 ]
Most of methods rely on a large amout of
training set.
Half-half split or Leave-one-out cross validation
It is unrealistic to collect this many training
samples for some action.
4
One Clip Many Clips
SLIDE 5
Query Action Template Set
Template A Template B Template C Template D Throwing R-dancing M-dancing Kicking Label:
SLIDE 6
Related Works
One shot learning of object categories [Fei-Fei et al.
ICCV03]
Visual Object Identification [Ferencz et al. IJCV07]
Transfer Learning:
The ability of a system to recognize and apply knowledge and skills learned in previous tasks to novel tasks .[Pan & Yang, TKDE 2009]
6
SLIDE 7
7
Query
A B C Hyper- Features Fq
Learning Action Recognition Distance
Templates
Action Detection
SLIDE 8
Patch based Action comparison
Query Template
Frame-to-Frame Distance
8
Motion Descriptor
[Efros et al. ICCV 03]
SLIDE 9
Patch based Action comparison
Query Template
Frame Correspondence Frame-to-Frame Distance
9
Elementary Patch-to-Patch Distance
SLIDE 10
10
Query
A B C Hyper- Features Fq
Learning Action Recognition Distance
Templates
Action Detection
SLIDE 11
Local Distance Function
[Frome et al. NIPS06]
11
SLIDE 12
Local Distance Function
[Frome et al. NIPS06]
Triplet
12
Large Training set required
SLIDE 13
Transferable Distance Function
13
SLIDE 14
Transferable Distance Function
14
SLIDE 15
Transferable Distance Function
Hyper- Feature Transferable
15
SLIDE 16
Max-Margin Formulation
- Triplet
- It is convex and similar to the primal problem of SVM
16
SLIDE 17
Hyper-Features
Codebook representation
Descriptor for each patch
○ HOG + Positions
Obtaining codebook with the size of
○ K-means clustering
Hyper-feature for each patch
○ A dimensional vector
17
SLIDE 18
Summary of Features
18
Motion Cue Shape Cue & Positions
Patch Matching
Hyper- Feature
Patch Weighting
SLIDE 19
19
Query
A B C Hyper- Features Fq
Learning Action Recognition Distance
Templates
Action Detection
SLIDE 20
Recognizing an Action
20
Query Template A Template B Template C
Hyper- Features Fq
SLIDE 21
21
Query
A B C Hyper- Features Fq
Learning Action Recognition Distance
Templates
Action Detection
SLIDE 22
Experiments on Action Recognition
Train the transferable distance function on
Weizmann, and test on KTH.
- The source training set does not contain the actions of the
template set
- Each Action in the testing set has only one clip as template
transfer
22
skip jack jump side bend wave1 pjump
Source Training Set
Weizmann walk run jog clap wave2
Testing set
KTH
SLIDE 23
Visualization
23
Codeword Ranking Learnt Weights on Testing Actions
SLIDE 24
Five Rounds of Experiments
For each round, we randomly select one
actor, then choose one clip per action from this actor as the template.
5% improvement
24
Dc : Direct Comparison (W = 1) Tr : Transferable Distance Function
SLIDE 25
Confusion Matrix of the Round 2
Direct Comparison Avg: 70.9% Transfer Avg: 76.7%
25
Clpping vs. Waving Jogging vs. Running
SLIDE 26
Efficiency
With the learnt distance function, we can sort the
patches on each frame by their saliency.
Instead of using all patches, we can choose the
top N patches with high weights for matching.
10 Patches on Each Frame
26
SLIDE 27
27
Query
A B C Hyper- Features Fq
Learning Action Recognition Distance
Templates
Action Detection
SLIDE 28
28
Human Action Detection
SLIDE 29
29
Cascade Structure
Reject Reject Reject Cascade Stage 1 Cascade Stage 2 Cascade Stage N
SLIDE 30
30
Cascade Structure
All Sub- Windows
Decision
Reject Reject Reject Hyper- Features Fq
SLIDE 31
Efficient Action Detection
31
SLIDE 32
Contributions
Transferable distance function Learning
Hyper-features based on appearance and
positions
Max-margin Learning framework
Action recognition from one clip
Template Matching based on motion
Efficient action detection from one clip
Cascade structure
32
SLIDE 33
33