Characteristic number regression for fiducial facial feature - - PowerPoint PPT Presentation

Characteristic number regression for fiducial facial feature extraction

Presenter : Xin Fan Joint work with Prof. Zhongxuan Luo and Dr. Risheng Liu Published in IEEE TIP’15 and ICME’15

Outline

• Motivations
• Facial structure with geometric invariants
• Facial feature extraction with geometry regressions

Motivations

• Fiducial facial point localization under pose/viewpoint

changes (perspective transformation).

Motivations

• Geometry works parallel (complementary) to multi-view

associations for facial analysis

Motivations

• Geometrical constraints are always important for facial

analysis

• Explicit shape modelling

ASM, AAM, CLM

• Implicit texture/shape regressions

the mapping from texture to shape Fern regressor [CVPR’12] Classifier pruning [ICCV’13] SDM[ICCV’13] Random forest [CVPR’14] Deep networks[ECCV’14] Highly dependent on the availability of training examples

• Explicit shape regressions

the mapping from geometry to geometry

Outline

• Motivations
• Facial structure with geometric invariants
• Facial feature extraction with geometry regressions

Facial geometry

• Human faces are highly structured and present

common geometries across age, gender, and race of individuals.

• Eye corners are collinear [Gee94].
• The lines connecting eye corners, nostrils and mouth corners

are mutually parallel.

• The line through eye corners is perpendicular to the line

connecting the midpoints of nostrils and mouth corners

 The parallelism and perpendicularity involves

more points and vary with viewpoints.

 The characteristic number describe the intrinsic

geometry given by more points, and preserves under viewpoint changes.

Characteristic ratio-Definition [Luo10]

u v

1

p

v b u a p

1 1 1

 

u v

1

p

2

p 

v b u a p

1 1 1

 

v b u a p v b u a p

n n n

    

2 2 2

Characteristic ratio-Definition [Luo10]

Characteristic number

• Definition derived from the characteristic ratio
• Let P1,…, Pr be r distinct points on m-dimensional projective

space, and these points form a loop.

• There are n points lying on the segments PiPi+1
• The characteristic number (CN) is

( ) ( ) ( ) , , 1 1 j j j i i i i i i i

Q P P  

 

 

n({P

i}i1 r ;{Qi ( j)}i1,...,r j1,...,n):

( i,i

( j)

i,i1

( j) j1 n



)

i1 r



Characteristic number-Properties

• Theorem: The characteristic number is a projective

invariant [Luo14].

• The characteristic number extends the cross ratio
• More points are included (r =2, n=2).
• Relaxes the collinear and coplanar constraints.

Q1

(1)  1,1 (1)P 1  1,2 (1)P 2

Q1

(2)  2,2 (2)P 2  2,1 (2)P 1 n(P

1,P 2;Q1 (1),Q1 (2))  1,1 (1)

1,2

(1)  2,2 (2)

2,1

(2)  CR

Intrinsic properties of a curve

• Characteristic number of a line[Luo10]

u w v

P R Q

w b u a P

a a ) ( 1 ) ( 1

  v b w a Q

b b ) ( 1 ) ( 1

 

u b v a R

c c ) ( 1 ) ( 1

 

1

) ( 1 ) ( 1 ) ( 1 ) ( 1 ) ( 1 ) ( 1

   

c c b b a a

a b a b a b

c b a

  ] ; , ][ ; , ][ ; , [

1

R u v Q v w P w u 

Intrinsic properties of a curve

• Characteristic number of a conic[Luo10]

u w v

w b u a p

a a ) ( 1 ) ( 1 1

  v b w a q

b b ) ( 1 ) ( 1 1

  u b v a r

c c ) ( 1 ) ( 1 1

 

1

p

2

r

2

q

1

q

1

r

2

p

w b u a p

a a ) ( 2 ) ( 2 2

  v b w a q

b b ) ( 2 ) ( 2 2

  u b v a r

c c ) ( 2 ) ( 2 2

  c b a

1

) ( 2 ) ( 1 ) ( 2 ) ( 1 ) ( 2 ) ( 1 ) ( 2 ) ( 1 ) ( 2 ) ( 1 ) ( 2 ) ( 1

  

c c c c b b b b a a a a

a a b b a a b b a a b b

] , ; , [ ] , ; , [ ] , ; , [

2 1 2 1 2 1 2

r r u v q q v w p p w u    

Intrinsic properties of a curve

• Characteristic number of an n-order curve [Luo10]

u w v

a

) ( 1 a

p

) (a n

p

) ( 2 a

p

b

) ( 1 b

p

) ( 2 b

p

) (b n

p c

) ( 2 c

p

) ( 1 c

p

) (c n

p

] , , ; , [ ] , , ; , [ ] , , ; , [

) ( ) ( 1 ) ( ) ( 1 ) ( ) ( 1 c n c b n b a n a n

p p u v p p w u p p v w       

For a curve of degree n, the characteristic number

n  (1)

n.

This property does rely on the choice of the three lines, and reflects the intrinsic geometries of a curve.

Extension to hypersurfaces [Luo14]

1 2

, ,...,

r

P P P

1 i i

PP

 

1,..., ( ) 1,..., j n j i i r

Q

 

( 1)rn 

( 1)

( 1) m

n 



i

P

This property is independent

• n the existence of the

hypersurvface and/or curve.

• A hypersurface of degree n in the m-dimensional space

not through any intersects each line precisely in . Then the characteristic number of w.r.t. the points is .

• Conversely, if the characteristic number is ,

then all the points lie on a hypersurface of degree n.

Facial geometry

• Human faces are highly structured and present

common geometries across age, gender, and race of individuals.

• Eye corners are collinear [Gee94].
• The lines connecting eye corners, nostrils and mouth corners

are mutually parallel.

• The line through eye corners is perpendicular to the line

connecting the midpoints of nostrils and mouth corners

 The parallelism and perpendicularity involves

more points and vary with viewpoints.

 The characteristic number describe the intrinsic

geometry given by more points, and preserves under viewpoint changes.

Facial geometry given by CN

(a) (b) (c) (d)

 These invariant priors, reported for the first time

to our best knowledge, reflect common facial geometries similar to the collinearity but on a larger scale involving more points for more facial components.

Facial geometry given by CN

• Verification on LFW (10k+ images)

Facial geometry given by CN

• Verification on our collections

Outline

• Motivations
• Facial structure with geometric invariants
• Facial feature extraction with geometry regressions

Objective

• Fiducial facial point localization under pose/viewpoint

changes (perspective transformation).

Motivations

• Geometrical constraints are always important for facial

analysis

• Explicit shape modelling

ASM, AAM, CLM

• Implicit texture/shape regressions

the mapping from texture to shape Fern regressor [CVPR’12] Classifier pruning [ICCV’13] SDM[ICCV’13] Random forest [CVPR’14] Deep networks[ECCV’14]

• Explicit shape regressions

the mapping from geometry to geometry

Formulation

• Find the points satisfying the CN constraints
• Gradient descent to minimize the energy (TIP’15)
• Build the regression (mapping) from point

configurations to target CN values. (ICME’15)

Fiducial point localization with CN priors

• Landmark errors by using collinearity, all CN

constraints and no shape constraints [TIP’15].

Localization results-PIE

Localization results-children

Comparisons with the state-of-the-art

• Comparisons on LFW

Comparisons with the state-of-the-art

• Comparisons on Helen

Comparisons with the state-of-the-art

• Comparisons on LFPW

Implicit regressions are sensitive to training sets Training: 400*15 faces all poses(15 pose) Test: 140*15 faces, for each pose 140

POSE 1 POSE 2 POE 3 POSE 4 POSE 5 POSE 6 POSE 7 POSE 8 POSE 9 POSE 10 POSE 11 POSE 12 POSE 13 POSE 14 POSE 15

ESR

0.10 56 0.094 0.11 92 0.09 44 0.104 9 0.098 0.093 7 0.109 0.092 9 0.100 8 0.105 1 0.096 8 0.106 5 0.099 0.103 6

SDM

0.08 83 0.079 3 0.07 52 0.07 97 0.112 2 0.083 9 0.069 1 0.067 5 0.075 3 0.094 7 0.080 6 0.073 3 0.072 5 0.079 5 0.090 6

LBF

0.04 16 0.036 1 0.03 39 0.03 66 0.040 7 0.039 5 0.034 2 0.033 8 0.033 4 0.040 2 0.042 8 0.037 1 0.037 1 0.036 8 0.043

Normalized Errors

Implicit regressions are sensitive to training sets Training: 400*3 faces of pose 1, 8 and 15 Test: 140*15 faces, for each pose 140

POSE 1 POSE 2 POE 3 POSE 4 POSE 5 POSE 6 POSE 7 POSE 8 POSE 9 POSE 10 POSE 11 POSE 12 POSE 13 POSE 14 POSE 15

ESR

0.10 87 0.111 3 0.11 09 0.17 60 0.315 2 0.176 5 0.141 7 0.041 9 0.115 5 0.160 9 0.256 7 0.180 1 0.099 5 0.103 0.102

SDM

0.07 67 0.104 1 0.16 77 0.27 80 0.480 4 0.173 1 0.097 0.064 6 0.104 8 0.171 1 0.300 6 0.132 0.082 1 0.090 3 0.079 1

LBF

0.04 29 0.046 7 0.04 29 0.05 05 0.051 4 0.050 8 0.188 2 0.034 3 0.048 1 0.056 7 0.056 7 0.060 3 0.054 6 0.053 8 0.043 1

Normalized Errors

Implicit regressions are sensitive to training sets Training: 400*15 faces all poses(15 pose) Test: 140*15 faces, for each pose 140

POSE 1 POSE 2 POE 3 POSE 4 POSE 5 POSE 6 POSE 7 POSE 8 POSE 9 POSE 10 POSE 11 POSE 12 POSE 13 POSE 14 POSE 15

ESR

0.10 56 0.094 0.11 92 0.09 44 0.104 9 0.098 0.093 7 0.109 0.092 9 0.100 8 0.105 1 0.096 8 0.106 5 0.099 0.103 6

SDM

0.08 83 0.079 3 0.07 52 0.07 97 0.112 2 0.083 9 0.069 1 0.067 5 0.075 3 0.094 7 0.080 6 0.073 3 0.072 5 0.079 5 0.090 6

LBF

0.04 16 0.036 1 0.03 39 0.03 66 0.040 7 0.039 5 0.034 2 0.033 8 0.033 4 0.040 2 0.042 8 0.037 1 0.037 1 0.036 8 0.043

Normalized Errors

More discussions

• Our shape regression (shape to shape) works well with

frontal shapes.

Accuracy increase with different configurations (color) of shape examples.

pose-1：30 R profile images pose-8: 30 frontal faces pose-15: 30 L profile images pose-1-8-15: Mix of above all poses: 2 shapes / pose

More discussions

• Comparisons with the state-of-the-art on our collections

with pose changes

More discussions

• Work as complementary correction to the texture-to-

shape regressions

[CVPR’12] [CVPR’13] [CVPR’13]+ CN regression

Related publications

[Luo10] Luo, Z.; Liu, F.; X., S. On Singularity of Spline Space Over Morgan-Scott’s Type Partition. Journal of Mathematical Research & Exposition 2010, 30, 1–16. [Luo13] Luo, Z.; Luo, D.; Fan, X.; Zhou, X.; Jia, Q. A shape descriptor based on new projective invariants. Proc. ICIP, 2013. [Luo14] Luo, Z.; Zhou, X.; Gu, D.X. From a projective invariant to some new properties of algebraic hypersurfaces. Science China Mathematics 2014. [Fan14] Fan, X.; Luo Z.*; Zhang, J.; Zhou X.; Jia, Q.; Luo, D. Characteristic Number: Theory and Its Application to Shape Analysis. Axioms 2014, 3, 202-221. [Fan14a] Hao Wang, Xin Fan*, and Yuntao Li, "Fiducial Facial Point Extraction with Cross Ratio", Proc. of IEEE International Conference

• n Image Processing (ICIP), 2014.

[Fan15] Fan, X.*; Wang, H.; Luo Z.; Hu, W.; Luo, D.; Li, Y. Fiducial Facial point Extraction Using a Novel Projective Invariant. IEEE Trans.

• n Image Processing (TIP),2015.

Related publications

[Fan15a] Yuntao Li, Xin Fan*, Risheng Liu, Yuyao Feng, Zhongxuan Luo, and Zezhou Li, “Characteristic number regression for facial feature extraction”, IEEE International Conference on Multimedia and Expo (ICME), 2015. (Oral) [Jia14] Jia, Q.; Fan, X.*; Luo, Z.; Liu, Y.; Guo, H. A New Geometric Descriptor for Symbols with Affine Deformations. Pattern Recognition Letters 2014, 40, 128–135. [Hu14] Wenyu Hu, Zhongxuan Luo, and Xin Fan*, "Image Retargeting via Adaptive Scaling with Geometry Preservation", IEEE Journal on Emerging and Selected Topics in Circuits and Systems (JETCAS), 4(1): 70-81, 2014. [Fan14c] Xin Fan*, Zhi Chai, Yuyao Feng, Yi Wang, Shengfa Wang, and Zhongxuan Luo, "An Efficient Mesh-based Face Beautifier on Mobile Devices", Neurocomputing. (Online)

Acknowledgments

• Prof. Xianfeng Gu, Dr. Wenyu Hu, and Dr. Qi Jia

PhD Students：Xinchen Zhou Master students: Hao Wang, Yuntao Li, Yuyao Feng Undergraduate students: Kang Huyan and Zhi Chai Natural Science Foundation of China 61033012,61003177,61272371,11171052 The program for New Century Excellent Talents（NCET- 11-0048）