Multi-modal Face Recognition Hu Han hanhu@ict.ac.cn http: / / - - PowerPoint PPT Presentation

▶
multi modal face recognition
SMART_READER_LITE
LIVE PREVIEW

Multi-modal Face Recognition Hu Han hanhu@ict.ac.cn http: / / - - PowerPoint PPT Presentation

Sep 23, 2023 268 likes •776 views

Multi-modal Face Recognition Hu Han hanhu@ict.ac.cn http: / / vipl.ict.ac.cn/ members/ hhan 2016/ 04/ 06 2 Trend on multi-modal (face) recognition Multi-modal & cross-modal FR Conclusion and discussion hanhu@ict.ac.cn

slide-1
SLIDE 1

Multi-modal Face Recognition

Hu Han

hanhu@ict.ac.cn http: / / vipl.ict.ac.cn/ members/ hhan 2016/ 04/ 06

slide-2
SLIDE 2

Institute of Computing Technology, Chinese Academy of Sciences

Outline

 Background  Related work  Multi-modal & cross-modal FR  Trend on multi-modal (face) recognition  Conclusion and discussion

2 2016/4/6 hanhu@ict.ac.cn

slide-3
SLIDE 3

Institute of Computing Technology, Chinese Academy of Sciences

Background

 Unconstrained sensing & uncooperative

subject scenario poses great challenges to unimodal FR system

3 2016/4/6 hanhu@ict.ac.cn

“It’s our intention to go through every frame of every video.” – Boston Police Commissioner, Ed Davis “We are particularly interested in reviewing video footage captured by bystanders with cell phones

  • r personal cameras near either of the blasts… In

an investigation of this nature, no detail is too small.” – Attorney General, Eric H. Holder Jr.

slide-4
SLIDE 4

Institute of Computing Technology, Chinese Academy of Sciences

Background

 Unimodal FR

 A manually selected probe face image of the

suspect (Tamerlan Tsarnaev) with the best quality is matched with its true mate by a COTS with rank-5000 among a 1M gallery set

4 2016/4/6 hanhu@ict.ac.cn

Probe 1 M

slide-5
SLIDE 5

Institute of Computing Technology, Chinese Academy of Sciences

Background

 Challenges

 Low quality surveillance videos and images  Intentional thwarting of identification (e.g.

sunglasses and hats)

 Daunting amount of data  Videos or images are n/ a  …

 Multi-modal FR is a possible solution

 Advances in computing and imaging tech.

 RGB, depth, NIR, 3D, sketch, etc.

 Multi-modality, multi-view, multi-biometrics

5 2016/4/6 hanhu@ict.ac.cn

slide-6
SLIDE 6

Institute of Computing Technology, Chinese Academy of Sciences

Background

6 2016/4/6 hanhu@ict.ac.cn

Top K Matche s

…

Top K Matche s Top K Matche s

Human operators manually review K*n images (n = # of images in the face media collection)

…

Traditional Forensic Investigation Workflow

slide-7
SLIDE 7

Institute of Computing Technology, Chinese Academy of Sciences

Outline

 Background  Related work  Multi-modal & cross-modal FR  Trend on multi-modal (face) recognition  Conclusion and discussion

7 2016/4/6 hanhu@ict.ac.cn

slide-8
SLIDE 8

Institute of Computing Technology, Chinese Academy of Sciences

Related work

 Multi-modal FR

 2D + 3D

 Beumier and Acheroy, PRL’01  Chang et al., ACM-W’03  …

 2D + depth

 Lu and Jain, TPAMI’06

 2D + 3D + NIR

 Bowyer et al., 2003-2011

 Most are: Per-modal matching + score-level

fusion

8 2016/4/6 hanhu@ict.ac.cn

slide-9
SLIDE 9

Institute of Computing Technology, Chinese Academy of Sciences

Related work

 Cross-modal FR

 Modality transformation

 Wang & Tang, TPAMI’09 (sketch vs. photo)  Gao et al., TCSVT’12 (sketch vs. photo)  3D face modeling, Blanz & Vetter’03 (2D vs. 3D)  …

 Invariant features

 Lei & Li, CVPR’09  VIS-NIR  Klare & Jain, TPAMI’13; Han & Jain TIFS’13; Klum

et al., TIFS’14

 VIS-NIR, forensic sketch, VIS-TIR

9 2016/4/6 hanhu@ict.ac.cn

slide-10
SLIDE 10

Institute of Computing Technology, Chinese Academy of Sciences

Outline

 Background  Related work  Multi-modal & cross-modal FR

 Multi-modal FR

 Trend on multi-modal (face) recognition  Conclusion and discussion

10 2016/4/6 hanhu@ict.ac.cn

slide-11
SLIDE 11

Institute of Computing Technology, Chinese Academy of Sciences

Multi-modal face recognition

11 2016/4/6 hanhu@ict.ac.cn

. . .

Human operators manually review 1*K images

?

Still image Sketch Video 3D

slide-12
SLIDE 12

Institute of Computing Technology, Chinese Academy of Sciences

Multi-modal face recognition

 A hierarchical quality-based fusion

12 2016/4/6 hanhu@ict.ac.cn

Image/video Sketch 3D

30-40岁 男性 白人 …

Q1 Q2 Q3 Q4 1M Mugshot，True mate is matched at rank-112 (vs. rank-5000 in unimodal) Quality measures：

slide-13
SLIDE 13

Institute of Computing Technology, Chinese Academy of Sciences

Multi-modal face recognition

13 2016/4/6 hanhu@ict.ac.cn

Face Track Extraction U  u1,u2,...,ua V  v1,v2,...,vb Multi‐ frame Score‐level Fusion:

  • mean
  • median
  • max
  • min

... ... All Frame Pairs

… …

a  b

Similarity Matrix

... …

s u1,v1

 

... ... s ua,vb

 

s U,V

 

Not Same Same

 t  t

COTS Face Matcher

Matching a Face Track from a Video

√

slide-14
SLIDE 14

Institute of Computing Technology, Chinese Academy of Sciences

Multi-modal face recognition

 Pose Correction via 3D Face Modeling

14 2016/4/6 hanhu@ict.ac.cn

slide-15
SLIDE 15

Institute of Computing Technology, Chinese Academy of Sciences

Multi-modal face recognition

 Close set identification

 4,249 gallery images, 596 probe subjects

15 2016/4/6 hanhu@ict.ac.cn

Images Multi- modal Videos

slide-16
SLIDE 16

Institute of Computing Technology, Chinese Academy of Sciences

Multi-modal face recognition

 Close set identification

 4,249 gallery images + 1M background

mugshots, 596 probe subjects

16 2016/4/6 hanhu@ict.ac.cn

slide-17
SLIDE 17

Institute of Computing Technology, Chinese Academy of Sciences

Multi-modal face recognition

 Open set identification

 The person of interest may not be present in

legacy face databases

 The gallery consists of 596 subjects with at

least two images in the LFW database and at least one video in the YTF database

17 2016/4/6 hanhu@ict.ac.cn

slide-18
SLIDE 18

Institute of Computing Technology, Chinese Academy of Sciences

Multi-modal face recognition

 Quality based fusion

18 2016/4/6 hanhu@ict.ac.cn

slide-19
SLIDE 19

Institute of Computing Technology, Chinese Academy of Sciences

Multi-modal face recognition

 A Case Study on the Boston Bomber

(Gallery of one million mugshots)

19 2016/4/6 hanhu@ict.ac.cn

slide-20
SLIDE 20

Institute of Computing Technology, Chinese Academy of Sciences

Multi-modal face recognition

 Forensic Sketches from Low Quality Video

20 2016/4/6 hanhu@ict.ac.cn

Retrieval ranks without and with demographic filtering are given as #(#)

slide-21
SLIDE 21

Institute of Computing Technology, Chinese Academy of Sciences

Deep multi-modal FR

21 2016/4/6 hanhu@ict.ac.cn

CNNs CNNs CNNs CNNs CNNs CNNs RGB D RGB D

slide-22
SLIDE 22

Institute of Computing Technology, Chinese Academy of Sciences

Deep multi-modal FR

 Deep RGBD face recognition

 900,000 RGBD images of 700 subjects

22 2016/4/6 hanhu@ict.ac.cn

Modality Accuracy RGB 0.93 Depth 0.86 Deep RGB-D 0.98

slide-23
SLIDE 23

Institute of Computing Technology, Chinese Academy of Sciences

Outline

 Background  Related work  Multi-modal & cross-modal FR

 Cross-modal FR

 Trend on multi-modal (face) recognition  Conclusion and discussion

23 2016/4/6 hanhu@ict.ac.cn

slide-24
SLIDE 24

Institute of Computing Technology, Chinese Academy of Sciences

Cross-modal face recognition

 Compatible with huge existing 2D face

images

 RGBD vs. RGB

 Modality is not available

 NIR vs. VIS  Sketch vs. photo

24 2016/4/6 hanhu@ict.ac.cn

slide-25
SLIDE 25

Institute of Computing Technology, Chinese Academy of Sciences

Cross-modal face recognition

 Sketch to mugshot matching

 Viewed

sketch: drawing/ synthesizing a sketch while looking at a subject/ photo

 Forensic

sketch: drawing/ synthesizing a sketch based on verbal description from the victim or eyewitness

 COTS matcher for photo-to-photo matching

can achieve

  • ver

than 85% rank-1 identification rate for viewed sketch in 2013, while its performance for forensic sketch identification is less than 10%

25 2016/4/6 hanhu@ict.ac.cn

slide-26
SLIDE 26

Institute of Computing Technology, Chinese Academy of Sciences

Cross-modal face recognition

 Sketch to mugshot matching

26 2016/4/6 hanhu@ict.ac.cn

slide-27
SLIDE 27

Institute of Computing Technology, Chinese Academy of Sciences

Sketch to mugshot matching

 Sketch leads to arrest of suspects

27 2016/4/6 hanhu@ict.ac.cn

Timothy McVeigh (the Oklahoma City bomber) David Berkowitz (Son of Sam) Ted Kaczynski (the Unabomber)

slide-28
SLIDE 28

Institute of Computing Technology, Chinese Academy of Sciences

Sketch to mugshot matching

 Local and holistic matching

 Local matching: component based rep.

28 2016/4/6 hanhu@ict.ac.cn Hu Han, Brendan Klare, Kathryn Bonnen, and Anil K. Jain. Matching Composite Sketches to Face Photos: A Component Based Approach. IEEE Transactions on Information Forensics and Security (T-IFS), vol. 8, no. 1, pp. 191-204, Jan. 2013.

slide-29
SLIDE 29

Institute of Computing Technology, Chinese Academy of Sciences

Sketch to mugshot matching

 Component based rep. is an inverse

process of sketch composition

29 2016/4/6 hanhu@ict.ac.cn

slide-30
SLIDE 30

Institute of Computing Technology, Chinese Academy of Sciences

Sketch to mugshot matching

 Local and holistic matching

 Holistic matching: dense keypoint features

30 2016/4/6 hanhu@ict.ac.cn

slide-31
SLIDE 31

Institute of Computing Technology, Chinese Academy of Sciences

Sketch to mugshot matching

 Complementarity

31 2016/4/6 hanhu@ict.ac.cn

slide-32
SLIDE 32

Institute of Computing Technology, Chinese Academy of Sciences

Sketch to mugshot matching

 Hand-drawn

and software-generated forensic sketch

32 2016/4/6 hanhu@ict.ac.cn

slide-33
SLIDE 33

Institute of Computing Technology, Chinese Academy of Sciences

Sketch to mugshot matching

 Software-generated viewed sketch

33 2016/4/6 hanhu@ict.ac.cn

slide-34
SLIDE 34

Institute of Computing Technology, Chinese Academy of Sciences

Generalized cross-modal FR

 Cross-distance

and cross-spectral matching in nighttime FR

34 2016/4/6 hanhu@ict.ac.cn

150m NIR at night Enrolled VIS

slide-35
SLIDE 35

Institute of Computing Technology, Chinese Academy of Sciences

Cross-distance and cross- spectral FR

 A

learning based image restoration method to recover a high-quality face image from a low-quality

 Dictionary building

35 2016/4/6 hanhu@ict.ac.cn

slide-36
SLIDE 36

Institute of Computing Technology, Chinese Academy of Sciences

Cross-distance and cross- spectral FR

 Per-patch

recovery using LLE

36 2016/4/6 hanhu@ict.ac.cn

slide-37
SLIDE 37

Institute of Computing Technology, Chinese Academy of Sciences

Cross-distance and cross- spectral FR

 Restored images

37 2016/4/6 hanhu@ict.ac.cn

slide-38
SLIDE 38

Institute of Computing Technology, Chinese Academy of Sciences

Cross-distance and cross- spectral FR

 Cross-distance and intra-spectral test

38 2016/4/6 hanhu@ict.ac.cn

slide-39
SLIDE 39

Institute of Computing Technology, Chinese Academy of Sciences

Cross-distance and cross- spectral FR

 Cross-distance and cross-spectral test

39 2016/4/6 hanhu@ict.ac.cn

slide-40
SLIDE 40

Institute of Computing Technology, Chinese Academy of Sciences

Sketch to mugshot matching

 Licensed

to MorphoTrak,

  • ne
  • f

the world’s leading biometrics companies

 Nighttime FR system

received funding from FBI

40 2016/4/6 hanhu@ict.ac.cn

slide-41
SLIDE 41

Institute of Computing Technology, Chinese Academy of Sciences

Outline

 Background  Related work  Multi-modal & cross-modal FR

 Multi-modal FR

 Trend on multi-modal (face) recognition  Conclusion and discussion

41 2016/4/6 hanhu@ict.ac.cn

slide-42
SLIDE 42

Institute of Computing Technology, Chinese Academy of Sciences

Trend on multi-modal (face) recognition

 The other biometrics or multi-biometrics

 Tattoo, gesture,…  Google Abacus project

 General object

 A. Wang, J. Lu, J. Cai, T.-J. Cham, and G.

Wang. Large-Margin Multi-Modal Deep Learning for RGB-D Object Recognition, IEEE

  • Trans. Multimedia, 17(11): 1887-1898, Nov.

2015.

 RGB-D, 300 common household objects

42 2016/4/6 hanhu@ict.ac.cn

slide-43
SLIDE 43

Institute of Computing Technology, Chinese Academy of Sciences

Trend on multi-modal (face) recognition

 The other biometrics or multi-biometrics

 Tattoo, gesture,…  Google Abacus project

43 2016/4/6 hanhu@ict.ac.cn

[1] http://www.telegraph.co.uk/sport/football/teams/serbia/8061619/Masked-ringleader-of-crowd-trouble- during-Italy-Serbia-clash-identified-by-tattoos.html [2] Hu Han and Anil K. Jain. Tattoo Based Identification: Sketch to Image Matching. ICB, 2013.

Masked ringleader of crowd trouble during Italy-Serbia clash identified by his tattooed arms [1].

slide-44
SLIDE 44

Institute of Computing Technology, Chinese Academy of Sciences

Trend on multi-modal (face) recognition

 The other biometrics or multi-biometrics

 Tattoo, gesture,…  Google Abacus Project (Google I/ O 2015)

44 2016/4/6 hanhu@ict.ac.cn

slide-45
SLIDE 45

Institute of Computing Technology, Chinese Academy of Sciences

Trend on multi-modal (face) recognition

 The other biometrics or multi-biometrics

 Tattoo, gesture,…  Google Abacus Project (Google I/ O 2015)

45 2016/4/6 hanhu@ict.ac.cn

You are your password!

3TB data

slide-46
SLIDE 46

Institute of Computing Technology, Chinese Academy of Sciences

Outline

 Background  Related work  Multi-modal & cross-modal FR

 Multi-modal FR

 Trend on multi-modal (face) recognition  Conclusion and discussion

46 2016/4/6 hanhu@ict.ac.cn

slide-47
SLIDE 47

Institute of Computing Technology, Chinese Academy of Sciences

What I want to convey...

 Multi-modal

FR significantly boosts the face recognition performance, particularly in unconstrained scenarios; but the

  • ptimum

process pipelines of individual modalities and fusing scheme are still not known

 Cross-modality FR, particularly forensic sketch

recognition, has wide applications, but remains an open problem

 Download

 [ Data] Still & video & sketch & 3D face images  [ Data] Cross-distance, cross-spectral face images  [ Data] Computer generated viewed-sketches  [ Protocol] Open-set identification protocol  http: / / biometrics.cse.msu.edu/ pubs/ databases.html

2016/4/6 hanhu@ict.ac.cn 47

slide-48
SLIDE 48

Institute of Computing Technology, Chinese Academy of Sciences

Related papers

[ 1] Lacey Best-Rowden, Hu Han* , Charles Otto, Brendan Klare, and Anil K. Jain. Unconstrained Face Recognition: Identifying a Person of Interest from a Media Collection, IEEE Transactions on Information Forensics and Security (T-IFS), vol. 9, no. 12, pp. 2144-2157, Dec. 2014.

[ 2] Scott Klum, Hu Han* , Brendan Klare, and Anil K. Jain. The FaceSketchID System: Matching Facial Composites to Mugshots. IEEE Transactions on Information Forensics and Security (T-IFS), vol. 9, no. 12, pp. 2248-2263, Dec. 2014.

[ 3] Dongoh Kang, Hu Han, Anil K. Jain, and Seong-Whan Lee. Nighttime Face Recognition at Large Standoff: Cross-Distance and Cross-Spectral Matching, Pattern Recognition (P.R.), vol. 47, no. 12,

  • pp. 3750-3766, Dec. 2014.

[ 4] Hu Han, Brendan Klare, Kathryn Bonnen, and Anil K. Jain. Matching Composite Sketches to Face Photos: A Component Based Approach. IEEE Transactions on Information Forensics and Security (T-IFS), vol. 8, no. 1, pp. 191-204, Jan. 2013.

[ 5] Hu Han and Anil K. Jain. Tattoo Based Identification: Sketch to Image Matching. In Proc. 6th IAPR International Conference

  • n

Biometrics (ICB), Madrid, Spain, June 4-7, 2013. (Oral)

2016/4/6 hanhu@ict.ac.cn 48

slide-49
SLIDE 49

Institute of Computing Technology, Chinese Academy of Sciences

Thank You!

hanhu@ict.ac.cn

http://vipl.ict.ac.cn/members/hhan

2016/4/6 49 hanhu@ict.ac.cn