Wit ith Im Image Clu lustering Jianwei Yang Devi Parikh Dhruv - - PowerPoint PPT Presentation

Unsuperv rvised Learning Jointly Wit ith Im Image Clu lustering

Vir irgin inia ia Tech

Jianwei Yang Devi Parikh Dhruv Batra

https://filebox.ece.vt.edu/~jw2yang/

1

2

Huge amount of images!!!

3

Huge amount of images!!! Learning without annotation efforts

4

Huge amount of images!!! Learning without annotation efforts What we need to learn?

5

An open problem

Huge amount of images!!! Learning without annotation efforts What we need to learn?

6

An open problem A hot problem

Huge amount of images!!! Learning without annotation efforts What we need to learn?

7

Various methodologies An open problem A hot problem

Huge amount of images!!! Learning without annotation efforts What we need to learn?

8

Learning distribution (structure)

Jain, Anil K., M. Narasimha Murty, and Patrick J. Flynn. "Data clustering: a review." ACM computing surveys (CSUR) 31.3 (1999): 264-323.

Clustering

9

Learning distribution (structure)

Jain, Anil K., M. Narasimha Murty, and Patrick J. Flynn. "Data clustering: a review." ACM computing surveys (CSUR) 31.3 (1999): 264-323.

Clustering

K-means (Image Credit: Jesse Johnson) 10

Learning distribution (structure)

Jain, Anil K., M. Narasimha Murty, and Patrick J. Flynn. "Data clustering: a review." ACM computing surveys (CSUR) 31.3 (1999): 264-323.

Clustering

K-means (Image Credit: Jesse Johnson) Hierarchical Clustering 11

Learning distribution (structure)

Jain, Anil K., M. Narasimha Murty, and Patrick J. Flynn. "Data clustering: a review." ACM computing surveys (CSUR) 31.3 (1999): 264-323.

Clustering

K-means (Image Credit: Jesse Johnson) Spectral Clustering Manor et al, NIPS’04 Hierarchical Clustering 12

Learning distribution (structure)

Jain, Anil K., M. Narasimha Murty, and Patrick J. Flynn. "Data clustering: a review." ACM computing surveys (CSUR) 31.3 (1999): 264-323.

Clustering

K-means (Image Credit: Jesse Johnson) Spectral Clustering Manor et al, NIPS’04 Hierarchical Clustering Graph Cut Shi et al, TPAMI’00 13

Learning distribution (structure)

Jain, Anil K., M. Narasimha Murty, and Patrick J. Flynn. "Data clustering: a review." ACM computing surveys (CSUR) 31.3 (1999): 264-323.

Clustering

K-means (Image Credit: Jesse Johnson) DBSCAN, Ester et al, KDD’96 (Image Credit: Jesse Johnson) Spectral Clustering Manor et al, NIPS’04 Hierarchical Clustering Graph Cut Shi et al, TPAMI’00 14

Learning distribution (structure)

Jain, Anil K., M. Narasimha Murty, and Patrick J. Flynn. "Data clustering: a review." ACM computing surveys (CSUR) 31.3 (1999): 264-323.

Clustering

K-means (Image Credit: Jesse Johnson) DBSCAN, Ester et al, KDD’96 (Image Credit: Jesse Johnson) Spectral Clustering Manor et al, NIPS’04 Hierarchical Clustering Graph Cut Shi et al, TPAMI’00 EM Algorithm, Dempster et al, JRSS’77 15

Learning distribution (structure)

Jain, Anil K., M. Narasimha Murty, and Patrick J. Flynn. "Data clustering: a review." ACM computing surveys (CSUR) 31.3 (1999): 264-323.

Clustering

K-means (Image Credit: Jesse Johnson) DBSCAN, Ester et al, KDD’96 (Image Credit: Jesse Johnson) Spectral Clustering Manor et al, NIPS’04 Hierarchical Clustering Graph Cut Shi et al, TPAMI’00 EM Algorithm, Dempster et al, JRSS’77 NMF, Xu et al, SIGIR‘03 (Image Credit: Conrad Lee) 16

Learning distribution (structure) Sub-space Analysis

PCA (Image Credit: Jesse Johnson) ICA (Image Credit: Shylaja et al) tSNE, Maaten et al, JMLR’08 Subspace Clustering, Vidal et al. Sparse coding, Olshausen et al. Vision Research’97 17

Learning representation (feature)

Yoshua Bengio, Aaron Courville, and Pierre Vincent. "Representation learning: A review and new perspectives." IEEE Transactions on Pattern Analysis and Machine Intelligence. 35.8 (2013): 1798-1828.

Autoencoder, Hinton et al, Science’06 (Image Credit: Jesse Johnson) DBN, Hinton et al, Science’06 DBM, Salakhutdinov et al, AISTATS’09 Bengio et al, TPAMI’13 18

Learning representation (feature)

VAE, Kingma et al, arXiv’13 (Image Credit: Fast Forward Labs) GAN, Goodfellow et al, NIPS’14 DCGAN, Radford et al, arXiv’15 (Image Credit: Mike Swarbrick Jones) 19

Most Recent CV Works

Spatial context, Doersch et al, ICCV’15 Temporal context, Wang et al, ICCV’15 Solving Jigsaw, Noroozi et al, ECCV’16 Context Encoder, Deepak et al, CVPR’16 Ego-motion, Jayaraman et al, ICCV’15

20

Most Recent CV Works

Visual concept clustering, Huang et al, CVPR’16 Graph constraint, Li et al, ECCV’16 TAGnet, Wang et al, SDM’16 Deep Embedding, Xie et al, ICML’16

21

Our Work

Joint Unsupervised Learning (JULE)

• f Deep Representations and Image Clusters

22

Outline

• Intuition
• Approach
• Experiments
• Extensions

23

Intuition

Meaningful clusters can provide supervisory signals to learn image representations

24

Intuition

Meaningful clusters can provide supervisory signals to learn image representations

Good representations help to get meaningful clusters

25

Intuition

Cluster images first, and then learn representations

26

Intuition

Cluster images first, and then learn representations Learn representations first, and then cluster images

27

Intuition

Cluster images and learn representations progressively Cluster images first, and then learn representations Learn representations first, and then cluster images

28

Intuition

Good cluster Good representations Good clusters Good representations Poor clusters Poor representations

29

Intuition

Good cluster Good representations Good clusters Good representations Poor clusters Poor representations

30

Intuition

Good cluster Good representations Good representations Good clusters Poor clusters Poor representations

31

Intuition

Good cluster Good representations Good representations Good clusters Poor clusters Poor representations

32

Approach

• Framework
• Objective
• Algorithm & Implementation

33

Approach: Framework

Agglomerative Clustering Convolutional Neural Network

Representation Learning Agglomerative Clustering

argmin ( | , )

y

L y I  arg min ( | , ) L y I





,

argmin ( , | )

y

L y I





34

Approach: Framework

Convolutional Neural Network Agglomerative Clustering

arg min ( | , ) L y I



 argmin ( | , )

y

L y I 

35

Approach: Recurrent Framework

36

Approach: Recurrent Framework

37

Approach: Recurrent Framework

38

Approach: Recurrent Framework

39

Approach: Recurrent Framework

40

Approach: Recurrent Framework

41

Approach: Recurrent Framework

Backward at each time-step is time-consuming and prone to over-fitting!

42

Approach: Recurrent Framework

How about updating once for multiple time-steps?

Backward at each time-step is time-consuming and prone to over-fitting!

43

Approach: Recurrent Framework

Partially Unrolling: divide all T time-steps into P periods

In each period, we merge clusters for multiple times and update CNN parameters at the end of period

44

Approach: Recurrent Framework

Partially Unrolling: divide all T time-steps into P periods

In each period, we merge clusters for multiple times and update CNN parameters at the end of period

45

Approach: Recurrent Framework

In each period, we merge clusters for multiple times and update CNN parameters at the end of period

Partially Unrolling: divide all T time-steps into P periods

P is determined by a hyper-parameter will be introduced later

46

Approach: Objective Function

Overall loss:

,

argmin ( , | )

y

L y I





argmin ( | , )

y

L y I  arg min ( | , ) L y I





47

Approach: Objective Function

Loss at time-step t:

Conventional Agg. Clustering Strategy Proposed Agg. Clustering Strategy

48

Approach: Objective Function

Loss at time-step t:

Conventional Agg. Clustering Strategy Proposed Agg. Clustering Strategy Affinity measure

49

Approach: Objective Function

Loss at time-step t:

Conventional Agg. Clustering Strategy Proposed Agg. Clustering Strategy i-th cluster

50

Approach: Objective Function

Loss at time-step t:

Conventional Agg. Clustering Strategy Proposed Agg. Clustering Strategy K_c nearest neighbor clusters of i-th cluster

51

Approach: Objective Function

Loss at time-step t:

Conventional Agg. Clustering Strategy Proposed Agg. Clustering Strategy Affinity between i-th cluster and its NN

52

Approach: Objective Function

Loss at time-step t:

Conventional Agg. Clustering Strategy Proposed Agg. Clustering Strategy Affinity between i-th cluster and its NN Differences between two cluster affinities

53

Approach: Objective Function

Loss at time-step t:

Conventional Agg. Clustering Strategy Proposed Agg. Clustering Strategy Affinity between i-th cluster and its NN Differences between two cluster affinities Merge these two clusters

54

Approach: Objective Function

Loss at time-step t:

Conventional Agg. Clustering Strategy Proposed Agg. Clustering Strategy Affinity between i-th cluster and its NN Differences between two cluster affinities Merge these two clusters

55

Approach: Objective Function

Loss in forward pass in period p (merge clusters): Loss in forward pass in period p (merge clusters):

56

Approach: Objective Function

Loss in forward pass in period p (merge clusters): Loss in forward pass in period p (merge clusters):

57

Approach: Objective Function

Loss in forward pass in period p (merge clusters): Loss in forward pass in period p (merge clusters):

CNN parameters are fixed

58

Approach: Objective Function

Loss in forward pass in period p (merge clusters): Loss in forward pass in period p (merge clusters):

CNN parameters are fixed Cluster labels are fixed

59

Approach: Objective Function

Forward Pass: Simple Greedy Algorithm Merge two clusters which minimize the loss at each time step

60

Approach: Objective Function

Forward Pass: Simple Greedy Algorithm Merge two clusters which minimize the loss at each time step

61

Approach: Objective Function

Forward Pass: Simple Greedy Algorithm Merge two clusters which minimize the loss at each time step

62

Approach: Objective Function

Forward Pass: Simple Greedy Algorithm Merge two clusters which minimize the loss at each time step

63

Approach: Objective

Backward Pass:

64

Approach: Objective

Backward Pass:

Consider all previous periods

65

Approach: Objective

Backward Pass:

Cluster based loss is not proper for batch optimization!!!

Consider all previous periods

66

Approach: Objective

Backward Pass:

Cluster based loss is not proper for batch optimization!!!

Consider all previous periods

Approximation:

67

Approach: Objective

Backward Pass:

Convert to sample-based loss:

Consider all previous periods

Intra-sample affinity Inter-sample affinity

Recall cluster-based loss:

68

Approach: Objective

Backward Pass:

Convert to sample-based loss:

Consider all previous periods

Intra-sample affinity Inter-sample affinity

Recall cluster-based loss:

Weighted triplet loss

69

Approach: Algorithm & Implementation

70

Approach: Algorithm & Implementation

Raw image data

71

Approach: Algorithm & Implementation

Raw image data Assume it is known

72

Approach: Algorithm & Implementation

Raw image data Assume it is known Randomly initialize CNN parameters 4 samples in each cluster in average

73

Approach: Algorithm & Implementation

Raw image data Assume it is known Randomly initialize CNN parameters 4 samples in each cluster in average Train CNN for about 20 epochs

74

Approach: Algorithm & Implementation

Raw image data Assume it is known Randomly initialize CNN parameters 4 samples in each cluster in average Train CNN for about 20 epochs We can go back and retrain the model, but it improve slightly

75

Experiments

• Datasets
• Network Architecture
• Image Clustering
• Representation Learning

76

Experiments: Datasets

MNIST (70000, 10, 28x28) USPS (11000, 10, 16x16) COIL20 (1440, 20, 128x128) COIL100 (7200, 100, 128x128) UMist (575, 20, 112x92) FRGC (2462, 20, 32x32) CMU-PIE (2856, 68, 32x32) Youtube Face (1000, 41, 55x55)

77

Experiments: Settings

Two important parameters Set the layer numbers so that the Output feature map is about 10x10

78

Experiments: Clustering : Performance

+6.43% on NMI to best performance of existing approaches averaged over all datasets

79

Experiments: Clustering : Performance

+12.76% on AC to best performance of existing approaches averaged over all datasets

80

Experiments: Clustering : Performance

Average +21.5% on NMI

81

Experiments: Clustering : Performance

Average +25.7% on NMI

82

Experiments: Clustering : Performance

Our clustering performance vs. that of existing clustering approaches using raw image data.

Clustering performance using our representation fed to existing clustering algorithms.

Experiments: Clustering : Visualization

COIL-20 COIL-100

84

Experiments: Clustering : Visualization

USPS MNIST-test

85

Experiments: Clustering : Ablation study

86

Experiments: Clustering : Verification

87

Experiments: Clustering : Time Cost

88

Experiments: Representation Learning

Testing generalization of our learnt (unsupervised) representation to LFW face verification. Evaluation on CIFAR-10 classification

Representation transfer Representation learning

89

Extensions: Data Visualization

90

Conclusion

• A new unsupervised learning method jointly with image clustering,

cast the problem into a recurrent optimization problem;

• In the recurrent framework, clustering is conducted during forward

pass, and representation learning is conducted during backward pass;

• A unified loss function in the forward pass and backward pass;
• Performance outperforms the state-of-the-art over a number of

datasets;

• It can also learn plausible representations for image recognition.

91

Thanks!

https://github.com/jwyang/joint-unsupervised-learning

92