Learning SPD-matrix-based Representation for Visual Recognition Lei - - PowerPoint PPT Presentation

Lei Wang VILA group School of Computing and Information Technology University of Wollongong, Australia

22-OCT-2018

Learning SPD-matrix-based Representation for Visual Recognition

Introduction

• How to represent an image?

– Scale, rotation, illumination, occlusion, background clutter, deformation, …

Cat:

• Hand-crafted, global features

– Color, texture, shape, structure, etc. – Goal: “Invariant and discriminative”

• Classifier

– K-nearest neighbor, SVMs, Boosting, …

• 1. Before year 2000

• Invariant to view angle, rotation, scale,

illumination, clutter, ...

• 2. Days of the Bag of Features (BoF) model

Local Invariant Features

Interest point detection

• r

Dense sampling An image becomes “A bag of features”

• 3. Era of Deep Learning

Deep Local Descriptors

“Cat”

Depth Height Width

Image(s): a set of points/vectors

Image set classification

6

Action recognition

vs.

Neuroimaging analysis How to pool a set of points/vectors to obtain a global visual representation ? Object detection & classification

Covariance representation

• Max pooling, average (sum) pooling, etc.
• Covariance pooling

A set of local descriptors

x1 x2 . . . xn

How to pool?

Essentially a second-order pooling

• Introduction on Covariance representation
• Our research work

– Discriminatively Learning Covariance Representation – Exploring Sparse Inverse Covariance Representation – Moving to Kernel-matrix-based Representation (KSPD) – Learning KSPD in deep neural networks

• Conclusion

Introduction on Covariance representation

Covariance Matrix

vs.

Introduction on Covariance representation

Use a Covariance matrix as a feature representation

10

Image is from http://www.statsref.com/HTML/index.html?multivariate_distributions.html

Introduction on Covariance representation

11

belongs to Symmetric Positive Definite (SPD) matrix resides on a manifold instead of the whole space

Introduction on Covariance representation

?

12

How to measure the similarity of two SPD matrices?

Introduction on SPD matrix

Similarity measures for SPD matrices

13

SPD matrices Kernel method Geodesic distance Euclidean mapping

Introduction on SPD matrix

14

Geodesic distance

Pennec X, Fillard P, Ayache N. A Riemannian framework for tensor

• computing. IJCV, 2006

Fletcher P T, Principal geodesic analysis

• n symmetric spaces: Statistics of

diffusion tensors. Computer Vision and Mathematical Methods in Medical and Biomedical Image Analysis., 2004 Förstner W, Moonen B. A metric for covariance matrices, Geodesy-The Challenge of the 3rd Millennium, 2003 Lenglet C, Statistics on the manifold of multivariate normal distributions: Theory and application to diffusion tensor MRI processing. Journal of Mathematical Imaging and Vision, 2006

2003 2004 2006

Introduction on SPD matrix

15

Euclidean mapping

Arsigny V, Log-Euclidean metrics for fast and simple calculus on diffusion tensors. Magnetic resonance in medicine, 2006, Veeraraghavan A, Matching shape sequences in video with applications in human movement

• analysis. PAMI, IEEE Transactions on, 2005

Tuzel O, Pedestrian detection via classification on riemannian manifolds. PAMI, IEEE Transactions on, 2008

2005 2006 2008

Introduction on SPD matrix

16

Kernel methods

Vemulapalli R, Pillai J K, Chellappa R. Kernel learning for extrinsic classification

• f manifold features, CVPR, 2013

Harandi M et al. Sparse coding and dictionary learning for SPD matrices: a kernel approach, ECCV, 2012

• S. Jayasumana, et. al., Kernel methods on the Riemannian

manifold of symmetric positive definite matrices, CVPR 2013. Sra S. Positive definite matrices and the S-

• divergence. arXiv preprint arXiv:1110.1773,

2011. Wang R., et. al., Covariance discriminative learning: A natural and efficient approach to image set classification, CVPR, 2012

2011 2012 2013

Quang, Minh Ha, et. Al., Log-Hilbert- Schmidt metric between positive definite

• perators on Hilbert spaces. NIPS. 2014.

2014

Introduction on SPD matrix

17

Integration with deep learning

Li et al., Is Second-order Information Helpful for Large-scale Visual Recognition? ICCV2017 Huang et al., A Riemannian Network for SPD Matrix Learning, AAAI2017 Improved Bilinear Pooling with CNN, Lin and Maji, BMVC2017 Lin et al, Bilinear CNN Models for Fine-grained Visual Recognition, ICCV2015 Ionescu et al, , Matrix Backpropagation for Deep Networks with Structured Layers, ICCV2015

2015 2017

Koniusz et al., A Deeper Look at Power Normalizations,, CVPR 2018

2018

• Introduction on Covariance representation
• Our research work

– Discriminatively Learning Covariance Representation – Exploring Sparse Inverse Covariance Representation – Moving to Kernel-matrix-based Representation (KSPD) – Learning KSPD in deep neural networks

• Conclusion

Motivation

Covariance Matrix

Covariance matrix needs to be estimated from data

Motivation

• Covariance estimate becomes unreliable

– High-dimensional (d) features – Small sample (n)

• Existing work

– Not consider the quality of covariance representation – Especially the estimate of eigenvalues

20

Motivation

Stein Kernel

21

Motivation

• 1. Eigenvalue estimation becomes biased when the

number of samples is inadequate

22

Motivation

• 2. The eigenvalues are not collectively manipulated

toward greater discrimination

Class 1 Class 2

23

Let’s do a data-dependent “eigenvalue massage”

Class 1 Class 2 Class 1 Class 2

Proposed method

24

adjustment

We propose “Discriminative Covariance Representation”

Power-based adjustment Coefficient-based adjustment

Proposed method

25

• adjusted S-Divergence:
• Power-based adjustment
• Coefficient-based adjustment

Discriminative Stein kernel (DSK)

Proposed method

26

How to learn the optimal adjustment parameter ?

• Kernel Alignment based method
• Class Separability based method
• Radius-margin Bound based Framework

Proposed method

27

Discriminative Stein kernel (DSK)

Experimental Result

• Brodatz texture
• ADNI rs-fMRI
• ETH-80 object
• FERET face

Data sets

Experimental Result

29

The most difficult 15 pairs of Brodatz texture data set

Experimental Result

30

The most difficult 15 pairs of Brodatz texture data set

DSK vs. eigenvalue estimation improvement methods

Discussion

[1] X. Mestre, “Improved estimation of eigenvalues and eigenvectors of covariance matrices using their sample estimates,” IEEE Trans. Inf. Theory, vol. 54, pp. 5113–5129, Nov. 2008. [2] B. Efron and C. Morris, “Multivariate empirical Bayes and estimation of covariance matrices,” Ann. Stat., vol. 4, pp. 22–32, 1976. [3] A. Ben-David and C. E. Davidson, “Eigenvalue estimation of hyper-spectral Wishart covariance matrices from limited number of samples,” IEEE Trans. Geosci. Remote Sens., vol. 50, pp. 4384–4396, May 2012.

31

• Introduction on Covariance representation
• Our research work

– Discriminatively Learning Covariance Representation – Exploring Sparse Inverse Covariance Representation – Moving to Kernel-matrix-based Representation (KSPD) – Learning KSPD in deep neural networks

• Conclusion

Introduction

Applications with high dimensions but small sample issue

33

Small sample 10 ~ 300 High dimensions 50 ~ 400

Introduction

34

This results in singular covariance estimate, which adversely affects representation. How to address this situation? Data + Prior knowledge Explore the underlying structure of visual features

Proposed SICE representation

35

Structure sparsity in skeletal human action recognition

• Only a small number of joints are directly linked.
• How to represent such direct links?

Sparse Inverse Covariance Estimation

(SICE)

Proposed SICE representation

36

Proposed SICE representation

37

Properties of SICE representation:

• is guaranteed to be nonsingular
• reduces over-fitting, giving more reliable representation
• Measures the partial correlation, allowing the sparsity

prior to be conveniently imposed

Application to Skeletal Action Recognition

38

Application to Skeletal Action Recognition

Application to other tasks

40

The principle of ``Bet on sparsity''

• Introduction on Covariance representation
• Our research work

– Discriminatively Learning Covariance Representation – Exploring Sparse Inverse Covariance Representation – Moving to Kernel-matrix-based Representation (KSPD) – Learning KSPD in deep neural networks

• Conclusion

Introduction

Again, look into Covariance representation

42

…

Introduction

Again, look into Covariance representation

43

…

i-th feature j-th feature Just a linear kernel function!

Introduction

Covariance representation

44

Resulting issues:

• Only modeling linear correlation of features.
• A single, fixed representation form.
• Unreliable or even singular covariance estimate.

Proposed kernel-matrix representation

Let’s use a kernel matrix instead

45

Advantages:

• Model nonlinear relationship between features;
• For many kernels, M is guaranteed to be nonsingular, no matter

what the feature dimensions and sample size are.

• Maintain the size of covariance representation and the

computational load.

Covariance SPD Matrix!

Application to Skeletal Action Recognition

46

Application to Skeletal Action Recognition

47

Application to Object Recognition

48

Application to Deep Learning Features

49

58 60 62 64 66 68 70 72 74 76 78 80 Alex Net (F7) VGG-19 Net (Conv5) Fisher Vector (CVPR15) Cov-RP Ker-RP (RBF)

Comparison on MIT Indoor Scenes Data Set

(Classification accuracy in percentage)

Discussion

SICE vs. Kernel matrix: which is better?

50

Discussion

51

SICE vs. Kernel matrix representation: which is better?

• Introduction on Covariance representation
• Our research work

– Discriminatively Learning Covariance Representation – Exploring Sparse Inverse Covariance Representation – Moving to Kernel-matrix-based Representation (KSPD) – Learning KSPD in deep neural networks

• Conclusion

Covariance representation

Integration with Deep Learning

Bilinear CNN Models for Fine-grained Visual Recognition, Lin et al, ICCV2015

Covariance representation

Integration with Deep Learning

Matrix Backpropagation for Deep Networks with Structured Layers, Ionescu et al, ICCV2015

Covariance representation

Integration with Deep Learning

Improved Bilinear Pooling with CNN, Lin and Maji, BMVC2017

Covariance representation

Integration with Deep Learning

Is Second-order Information Helpful for Large-scale Visual Recognition?, Li et al., ICCV2017

Proposed DeepKSPD

Motivation

57

 The kernel-matrix-based SPD representation  has not been developed upon deep local descriptors  has not been jointly learned via deep learning  Existing matrix backpropagation for learning covariance-

representation via deep networks

 encounters numerical stability issue

Proposed DeepKSPD

Architecture and layers

58

Proposed DeepKSPD

Matrix backpropagation

59

Proposed DeepKSPD

Matrix backpropagation

60

H = f(K) on the kernel matrix K

~

?

Proposed DeepKSPD

Existing matrix backpropagation

Matrix Backpropagation for Deep Networks with Structured Layers, Ionescu et al, ICCV2015

Proposed DeepKSPD

Result from the literature of Operator Theory (1951)

Proposed DeepKSPD

Existing matrix backpropagation (Ionescu et al, ICCV2015) Proposed matrix backpropagation What is their relationship?

Proposed DeepKSPD

Generalise to matrix α-rooting normalisation

Experimental Result

Fine-grained Image Recognition

Experimental Result

Fine-grained Image Recognition

Experimental Result

Numerical stability of backpropagation

Experimental Result

DeepKSPD vs DeepCOV

Experimental Result

Ablation study

• Learning width θ in the GRBF kernel
• Learning α in matrix α-rooting normalisation

Research trend on learning SPD representation

• Consider higher-order feature relationship

Kernel Pooling for Convolutional Neural Networks, Cui et al, CVPR2017

Research trend on learning SPD representation

• Improve the computational efficiency

(d) Low-rank Bilinear Pooling for Fine-Grained Classification, Kong et al, CVPR2017 (c) Compact Bilinear Pooling, Gao et al, CVPR2016 Statistically-motivated Second-order Pooling, Yu and Salzmann, ECCV2018

Conclusion

• Discriminative Stein kernel to address two issues in

covariance representation

• SICE representation to incorporate structure sparsity
• Kernel matrix representation to move beyond linear, fixed

covariance representation

• End-to-end deep learning of KSPD representation

1.

• J. Zhang, L. Wang, L. Zhou, and W. Li, Learning Discriminative Stein Kernel

for SPD Matrices and Its Applications, IEEE Transactions on Neural Networks and Learning Systems (TNNLS), Vol. 27, Issue 5, pp. 1020-1033, May 2016. 2.

• J. Zhang, L. Wang, L. Zhou, and W. Li, Exploiting Structure Sparsity for

Covariance-based Visual Representation, arXiv:1610.08619 [cs.CV]. 3.

• L. Wang, J. Zhang, L. Zhou, C. Tang and W. Li, Beyond Covariance: Feature

Representation with Nonlinear Kernel Matrices, IEEE International Conference

• n Computer Vision (ICCV), December 2015.

4.

• M. Engin, L. Wang, L. Zhou, and X. Liu, DeepKSPD: Learning Kernel-matrix-

based SPD Representation for Fine-grained Image Recognition, The 15th European Conference on Computer Vision (ECCV), September 2018.

Conclusion

• Better understand SPD-matrix-based representation

– What is it modelling, relationship to other pooling schemes?

• Learn the optimal SPD representation from data

– Optimisation on manifold, kernel learning, prior knowledge?

• Computational issue

– Deal with high-dimensional features and large data set?

• Beyond SPD representation

– Rectangular matrix – Higher order information – Spatial or temporal order

On-going Issues

Other related publications

74

• J. Zhang, L. Zhou and L. Wang, Subject-adaptive Integration of Multiple SICE

Brain Networks with Different Sparsity, Pattern Recognition, 63 642-652, 2017.

• L. Zhou, L. Wang, J. Zhang, Y. Shi and Y. Gao, Revisiting Distance Metric

Learning for SPD Matrix based Visual Representation, IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.

• L. Zhou, L. Wang, L. Liu, P. Ogunbona, and D. Shen, Learning Discriminative

Bayesian Networks from High-dimensional Continuous Neuroimaging Data, IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), Volume: 38 , Issue: 11 , Nov. 1 2016 .

• J. Zhang, L. Zhou, L. Wang, and W. Li, Functional Brain Network Classification

With Compact Representation of SICE Matrices, IEEE Transactions on Biomedical Engineering, 62 (6), 1623-1634, 2015.

• L. Zhou, L. Wang and P. Ogunbona. Discriminative Sparse Inverse Covariance

Matrix: Application in Brain Functional Network Classification, IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), June 2014

• L. Zhou, L. Wang, L. Liu, P. Ogunbona and D. Shen. Max-margin Based

Learning for Discriminative Bayesian Network from Neuroimaging Data, In the 17th International Conference on MICCAI, September 2014.

Q&A

Images Courtesy of Google Image