Constrained Video Face Clustering using 1NN Relations Vicky - - PowerPoint PPT Presentation

▶
constrained video face clustering using 1nn relations
SMART_READER_LITE
LIVE PREVIEW

Constrained Video Face Clustering using 1NN Relations Vicky - - PowerPoint PPT Presentation

Dec 26, 2023 766 likes •1.05k views

Constrained Video Face Clustering using 1NN Relations Vicky Kalogeiton Andrew Zisserman Video face clustering In Input Ou Outp tput Video source: [Tapaswi ICVGIP 2014] 1 Why does it matter? Comfort Fun Access Modern applications

slide-1
SLIDE 1

Constrained Video Face Clustering using 1NN Relations

Vicky Kalogeiton Andrew Zisserman

slide-2
SLIDE 2

Video face clustering

1

In Input Ou Outp tput

Video source: [Tapaswi ICVGIP 2014]

slide-3
SLIDE 3

Why does it matter?

Comfort Fun Access

slide-4
SLIDE 4

Modern applications Automatic story telling

3

Grand entry of the king’s horses and men. AR ARYA, wearing a helm and cloak, pushes her way into a tall wagon for a better look…. Grand entry of the king’s horses and men. ARYA, wearing a helm and cloak, pushes her way into a tall wagon for a better look…. In rides JO JOFFREY, , followed by the HOU HOUND

slide-5
SLIDE 5

C1C: Constrained 1NN Clustering

Overview

4

Mu Must-lin link

t 1st neighbor min-cut cannot- link clust er must-link

Ca Cannot-lin link

1st neighbor min-cut cannot-link cluster must-link

slide-6
SLIDE 6

Friends dataset Contributions C1C: Constrained 1NN Clustering

Overview

5

Mu Must-lin link

t 1st neighbor min-cut cannot- link clust er must-link

Ca Cannot-lin link

ü No training required ü Scalable ü Low computational cost ü Outperforms state of the art ü Friends: challenging

  • season 3 (10h)
  • ~25 episodes
  • 17k head tracks
  • 49 characters

1st neighbor min-cut cannot-link cluster must-link

slide-7
SLIDE 7

Outline

  • FINCH clustering method
  • Self-supervised Constraints
  • C1C pipeline
  • Friends dataset
  • Experimental Results

6

slide-8
SLIDE 8

Related work

7 S-Siam [Sharma T-BIOM 2020] T-Siam [Sharma FG 2019] BCL [Tapaswi ICCV 2019] CCL [Sharma FG 2020] [Everingham BMVC 2006] [Cinbis ICCV 2011] [Bojanowski ICCV 2013] [Wu ICCV 2013,CVPR 2013]

  • Use HAC
  • Require learning
  • High computational cost
slide-9
SLIDE 9

C1C: Constrained 1NN Clustering

8

[Code]

http://www.robots.ox.ac.uk/~vgg/research/c1c/

Hierarchical Clustering method Self-supervised constraints

slide-10
SLIDE 10

C1C – FINCH

9

[Code]

http://www.robots.ox.ac.uk/~vgg/research/c1c/

FINCH

  • Pairwise distances between all instances
  • At every partition, link all first NN
  • Merge instances that are first neighbors
  • r have a common first neighbor
  • Represent a cluster with the average of its

instances

[S. Sarfraz CVPR 19]

Hierarchical Clustering method Self-supervised constraints

slide-11
SLIDE 11

C1C— Self-supervised Constraints

10

[Code]

http://www.robots.ox.ac.uk/~vgg/research/c1c/

Ca Cannot-lin link Must st-Li Link nk

t

Hierarchical Clustering method Self-supervised constraints

slide-12
SLIDE 12

C1C: Constrained 1NN Clustering

11

slide-13
SLIDE 13

C1C: Constrained 1NN Clustering

12

cannot-link must-link

slide-14
SLIDE 14

13 (a)

C1C: Constrained 1NN Clustering

cannot-link must-link

(b)

slide-15
SLIDE 15

14 (a)

C1C: Constrained 1NN Clustering

cannot-link must-link

(b)

slide-16
SLIDE 16

15 (a)

C1C: Constrained 1NN Clustering

1st neighbor cannot-link must-link

(b)

slide-17
SLIDE 17

16 (a)

C1C: Constrained 1NN Clustering

1st neighbor cannot-link must-link

(b) (c)

slide-18
SLIDE 18

17 (a)

C1C: Constrained 1NN Clustering

1st neighbor cannot-link must-link

(b)

slide-19
SLIDE 19

18 (a)

C1C: Constrained 1NN Clustering

1st neighbor min-cut cannot-link must-link

(b)

slide-20
SLIDE 20

C1C: Constrained 1NN Clustering

1st neighbor min-cut cannot-link cluster must-link

slide-21
SLIDE 21

Friends dataset

  • Friends season 3 (~10h, 25 episodes)
  • 17k head tracks, 49 characters (six main, 43 secondary)

20

Ma Main Se Secondary

Gunther Rachel date Janice Kate Rachel Monica Joey Phoebe Ross Carol

slide-22
SLIDE 22

Datasets & Metrics

21

Buffy the Vampire Slayer The Big Bang Theory (BBT)

WCP: Weighted Clustering Purity

# clusters Purity

Tr Trade-of

  • ff

WCP = 1 & '

()* |,|

  • (.(

.(:purity of cluster /

  • (: #samples in cluster /

Implementation

  • Architecture:

ResNet-50

  • Pre-trained

MS-Celeb-1M

  • Fine-tuned:

VGGFace2

slide-23
SLIDE 23

90.8% 82.9% 69.7% 92.9% 86.5%

95.3% 88.1% 77.0%

BBT BBT Bu Buffy Fr Friends %W %WCP CP FINCH [Sarfraz CVPR 19] BCL [Tapaswi ICCV 19] C1C [Ours]

Quantitative Results

22

ü C1 C1C: better FINCH and BCL ü Fri Friends: challenging

slide-24
SLIDE 24

Quantitative Results

23

90.8% 82.9% 69.7% 92.9% 86.5%

95.3% 88.1% 77.0%

BBT BBT Bu Buffy Fr Friends %W %WCP CP FINCH [Sarfraz CVPR 19] BCL [Tapaswi ICCV 19] C1C [Ours]

ü Fri Friends: challenging ü C1 C1C: better FINCH and BCL

slide-25
SLIDE 25

Qualitative results

24

slide-26
SLIDE 26

Conclusions & Future work

  • C1C:

– links instances through 1NN relations – must-link and cannot-link constraints

  • Advantages:

– scalable – no training required – low computational cost

  • Friends dataset
  • State of the art results

25

  • Overcome failures by using more context
  • Automatically estimate #characters
slide-27
SLIDE 27

Thank you