Linear Manifold Embeddings of Pattern Clusters Robert Haralick - - PowerPoint PPT Presentation

▶
linear manifold embeddings of pattern clusters
SMART_READER_LITE
LIVE PREVIEW

Linear Manifold Embeddings of Pattern Clusters Robert Haralick - - PowerPoint PPT Presentation

Feb 26, 2024 302 likes •598 views

Linear Manifolds The Algorithm Empirical Evaluation Linear Manifold Embeddings of Pattern Clusters Robert Haralick Rave Harpaz Pattern Recognition Laboratory The Graduate Center, City University of New York DIMACS 2005 Linear Manifold

slide-1
SLIDE 1

Linear Manifolds The Algorithm Empirical Evaluation

Linear Manifold Embeddings of Pattern Clusters

Robert Haralick Rave Harpaz

Pattern Recognition Laboratory The Graduate Center, City University of New York

DIMACS 2005

Linear Manifold Embeddings of Pattern Clusters Haralick, Harpaz

slide-2
SLIDE 2

Linear Manifolds The Algorithm Empirical Evaluation

Linear Manifolds

Informally, a linear manifold is a subspace that may have been shifted away from the origin. A subspace is an instance of a linear manifold that contains the origin.

C1 C2 C3 C4

Linear Manifold Embeddings of Pattern Clusters Haralick, Harpaz

slide-3
SLIDE 3

Linear Manifolds The Algorithm Empirical Evaluation

Linear Manifolds

Each point xi in a set of a d-dim points that all lie on an m-dim linear manifold can be modeled as: xi = µ +     . . . . . . b1 · · · bm . . . . . .     λi Each point xi in a set of points that all manifest a shift pattern in the full space can be modeled as: xi = p + 1Li e.g. x1 =   2 6 4   +   1 1 1   2 =   4 8 6  

Linear Manifold Embeddings of Pattern Clusters Haralick, Harpaz

slide-4
SLIDE 4

Linear Manifolds The Algorithm Empirical Evaluation

Linear Manifolds

Each point xi in a set of a d-dim points that all lie on an m-dim linear manifold can be modeled as: xi = µ +     . . . . . . b1 · · · bm . . . . . .     λi Each point xi in a set of points that all manifest a scale pattern in the full space can be modeled as: xi = pLi e.g. x1 =   2 6 4   2 =   4 12 8  

Linear Manifold Embeddings of Pattern Clusters Haralick, Harpaz

slide-5
SLIDE 5

Linear Manifolds The Algorithm Empirical Evaluation

Shift and Scale Patterns as Linear Manifolds

100 200 300 400 100 200 300 400 200 400 600 800

scale scale shift

x y z 50 100 150 200 250 300 350 400

Shift Cluster

x y z 100 200 300 400 500 600 700 Scale Cluster

PC1shift = (0.5774, 0.5774, 0.5774)′ PC1scale = (0.3810, 0.2540, 0.8890)′ R = ✵ ❅ 1 1 1 1 1 1 1 1 1 ✶ ❆ PearsonR = 1 MSRshift = 0 MSRscale = 3236.3

Linear Manifold Embeddings of Pattern Clusters Haralick, Harpaz

slide-6
SLIDE 6

Linear Manifolds The Algorithm Empirical Evaluation

Linear Manifolds - Patterns in Subspaces

Shift pattern that exists only in a subspace: xi = Br(µr + 1rφi) + Bc(µc + λi) = Brµr + Br1rφi + Bcµc + Bcλi (Br|Bc) = I8

X1 X2 X3 X4 X5 X6 X7 X8

The linear manifold embedding: xi = (Br|Bc)

  • µr

µc

  • +
  • Br

1r √r |Bc √rφi λi

  • Linear Manifold Embeddings of Pattern Clusters

Haralick, Harpaz

slide-7
SLIDE 7

Linear Manifolds The Algorithm Empirical Evaluation

Linear Manifolds - Patterns in Subspaces

x1 x2 x3 X1 X2 X3

Linear Manifold Embeddings of Pattern Clusters Haralick, Harpaz

slide-8
SLIDE 8

Linear Manifolds The Algorithm Empirical Evaluation

Linear Manifolds - Adding an Error Term

Definition (The Linear Manifold Cluster Model) Let D be a set of d-dimensional points, C ⊆ D a subset of points that constitute a cluster, xi some point in C, b1, . . . , bd an orthonormal set

  • f vectors that span Rd, (bi, . . . , bj) a matrix whose columns are the

vectors bi, . . . , bj, and µ some point in Rd. Then each xi ∈ C is modeled by,

xi = µ + ✵ ❇ ❇ ❅ . . . . . . b1 · · · bm . . . . . . ✶ ❈ ❈ ❆ λi + ✵ ❇ ❇ ❅ . . . . . . bm+1 · · · bd . . . . . . ✶ ❈ ❈ ❆ ψi

Linear Manifold Embeddings of Pattern Clusters Haralick, Harpaz

slide-9
SLIDE 9

Linear Manifolds The Algorithm Empirical Evaluation

Shift Pattern - Bicluster (Cheng 00), Floc (Yang 02), pCluster (Wang 02)

Definition (Shift Pattern Cluster Model) Let D be a set of d-dimensional points, C ⊆ D the subset of points manifesting a shift pattern in some r-dimensional subspace of the data, and xi some point in C. Then each xi ∈ C can be modeled by, xi = Brµr + Br1rφi + Brψi + Bcµc + Bcλ Proposition Every point xi in a d-dimensional space that fits the shift pattern cluster model, also fits the linear manifold cluster model, where the dimension of the linear manifold is d − r + 1, and the model is given by:

xi = (Br|Bc) ✒ µr µc ✓ + ✒ Br 1r √r |Bc ✓ ✥ √rφi +

1′

r

√r ψi

λ ✦ + Br ✒ Ir − 1r1′

r

r ✓ ψi

Linear Manifold Embeddings of Pattern Clusters Haralick, Harpaz

slide-10
SLIDE 10

Linear Manifolds The Algorithm Empirical Evaluation

Scale Pattern

Definition (Scale Pattern Cluster Model) Let D be a set of d-dimensional points, C ⊆ D the subset of points manifesting a scale pattern in some r-dimensional subspace of the data, and xi some point in C. Then each xi ∈ C can be modeled by, xi = φiBrµr + Brψi + Bcµc + Bcλi Proposition Every point xi in a d-dimensional space that fits the scale pattern cluster model, also fits the linear manifold cluster model, where the dimension of the linear manifold is d − r + 1, and the model is given by:

xi = Bcµc + ✒ Br µr µr |Bc ✓ ✥ µr φi +

µ′

r

µr ψi

λi ✦ + Br ✒ Ir − µrµ′

r

µr 2 ✓ ψi

Linear Manifold Embeddings of Pattern Clusters Haralick, Harpaz

slide-11
SLIDE 11

Linear Manifolds The Algorithm Empirical Evaluation

The Bicluster Model (Cheng et al. 00)

MSRS = H(I, J) =

1 |I||J|

  • i∈I,j∈J(Yij − ¯

Yi − ¯ Yj − ¯ YIJ)2 The Underlying Model - Two Way ANOVA Yij = µ + φi + ψj + ǫij Each point in a bicluster can be modeled by: xi = 1µ + 1φi + ψ + ǫi where φi is a scalar denoting the residual effect of the i-th gene, ψ = (ψ1, . . . ψd)′ a vector containing the residual effects of the conditions, and ǫi ∼ N(0, σ2I)

Linear Manifold Embeddings of Pattern Clusters Haralick, Harpaz

slide-12
SLIDE 12

Linear Manifolds The Algorithm Empirical Evaluation

The Bicluster Model (Cheng et al. 00)

Proposition Every point xi in a d-dim space that fits a bicluster model embedded in an r-dim subspace, also fits the linear manifold cluster model, where the dimension of the linear manifold is d − r + 1, and the model is given by:

xi = (Br|Bc) ✒ 1rµr + ψ µc ✓ + ✒ Br 1r √r |Bc ✓ ✥ √rφi +

1′

r

√r ǫi

λi ✦ +Br ✒ Ir − 1r1′

r

r ✓ ǫi

Linear Manifold Embeddings of Pattern Clusters Haralick, Harpaz

slide-13
SLIDE 13

Linear Manifolds The Algorithm Empirical Evaluation

Subspace Clusters

Consist of a subset of points and a corresponding subset of attributes, such that these points form a dense region in a subspace defined by the set of corresponding attributes.

x y z x y

CLIQUE (Agrawal 98), MAFIA (Nagesh 99),PROCLUS (Aggarwal 99), ORCLUS (Aggarwal 00)

Linear Manifold Embeddings of Pattern Clusters Haralick, Harpaz

slide-14
SLIDE 14

Linear Manifolds The Algorithm Empirical Evaluation

Other Instances of Linear Manifolds - Negative Correlations

x y z 100 200 300 400 500 600 700 800 200 400 600 800 200 400 600 800 300 400 500 600 700 800 x y z

R = ✵ ❅ 1

  • 1

1

  • 1

1

  • 1

1

  • 1

1 ✶ ❆ PearsonR = 0.3181 MSR = 18280 Yip et al. (2004)- HARP , to detect co-regulated genes, create a reflective copy of the data set, cluster and remove the copy.

Linear Manifold Embeddings of Pattern Clusters Haralick, Harpaz

slide-15
SLIDE 15

Linear Manifolds The Algorithm Empirical Evaluation

Other Instances of Linear Manifolds - Linear Combinations of Variables

200 400 600 200 400 600 50 100 150 200 250 300 x y z

z = b0 + b1x + b2y

PearsonR = 0.4509 MSR = 8975 Coefficient of multiple determination: R2 = P(ˆ z − ¯ z)2 P(z − ¯ z)2 = 1 4C, Böhm et al. (2004)

Linear Manifold Embeddings of Pattern Clusters Haralick, Harpaz

slide-16
SLIDE 16

Linear Manifolds The Algorithm Empirical Evaluation

Other Instances of Linear Manifolds - Latent Variables

1 2 3 4 5 6 7 8

R−1 ⇓

1 2 3 4 5 6 7 8

1

xi = R(µ + 1dφi)

2

yi = xi − Rµ = 1dφi

3

φi =

  • x′

i xi

d

4

C = 1

n

n

i=1 yi(1dφi)′

5

[u, s, v] = svd(C)

6

R = uv′

Linear Manifold Embeddings of Pattern Clusters Haralick, Harpaz

slide-17
SLIDE 17

Linear Manifolds The Algorithm Empirical Evaluation

Data Transformations

x1 x2 x3 x4 x5 shift pattern x1 x2 x3 x4 x5 log transformation of a shift pattern x1 x2 x3 x4 x5 row mean subtraction on a shift pattern

x1 x2 x3 x4 x5 mean/var transformation on a shift pattern x1 x2 x3 x4 x5 scale pattern x1 x2 x3 x4 x5 log transformation on a scale pattern x1 x2 x3 x4 x5 row mean subtraction on a scale pattern x1 x2 x3 x4 x5 mean/var transformation on a scale pattern x1 x2 x3 x4 x5 0−Dim manifold x1 x2 x3 x4 x5 log transformation to a 0−Dim manifold 1 2 3 4 5 row mean subtraction on a 0−Dim manifold x1 x2 x3 x4 x5 mean/var transformation on a 0−dim manifold

Linear Manifold Embeddings of Pattern Clusters Haralick, Harpaz

slide-18
SLIDE 18

Linear Manifolds The Algorithm Empirical Evaluation

Data Transformations

x1 x2 x3 x4 x5 x6 x7 x8 x9 x10 Shift pattern before normalization x1 x2 x3 x4 x5 x6 x7 x8 x9 x10 Shift pattern after normalization x1 x2 x3 x4 x5 x6 x7 x8 x9 x10 Scale pattern before normalization x1 x2 x3 x4 x5 x6 x7 x8 x9 x10 Scale pattern after normalization

Linear Manifold Embeddings of Pattern Clusters Haralick, Harpaz

slide-19
SLIDE 19

Linear Manifolds The Algorithm Empirical Evaluation

The Algorithm

Main Idea

1

Sample minimal subsets of points to construct trial linear manifolds of various dimensions.

2

Compute distance histograms of the data to each trial manifold.

3

Of all the manifolds constructed, select the one whose associated histogram shows the best separation between a mode near zero and the rest of the data.

4

Partition the data based on the best separation.

5

Repeat the procedure on each block of the partitioned data.

Linear Manifold Embeddings of Pattern Clusters Haralick, Harpaz

slide-20
SLIDE 20

Linear Manifolds The Algorithm Empirical Evaluation

How are trial manifolds sampled?

To construct an m-dimensional manifold we need to sample m + 1 points. Example- constructing a 2D manifold

x0 x1 x2

Linear Manifold Embeddings of Pattern Clusters Haralick, Harpaz

slide-21
SLIDE 21

Linear Manifolds The Algorithm Empirical Evaluation

How many trial manifolds need to be examined?

−5 5 10 15 P=0.2

−5 5 10 15 20 25 30 −5 5 10 15 20 25 30 −5 5 10 15 20 25 30

Linear Manifold Embeddings of Pattern Clusters Haralick, Harpaz

slide-22
SLIDE 22

Linear Manifolds The Algorithm Empirical Evaluation

How many trial manifolds need to be examined?

Assuming there are ˆ K clusters having approximately the same number of points. Then the probability that a sample of m + 1 points all come from the same cluster is approximately

  • 1

ˆ K

m . The probability that out of n samples of m + 1 points, none come from the same cluster, is approximately (1 − (1/ ˆ K)m)n 1 − (1 − (1/ ˆ K)m)n will be the probability that at least for one of the samples all of its m + 1 points come from the same cluster. Therefore the sample size n required such that this probability is greater than some value 1 − ǫ is given by n ≥ log ǫ log(1 − (1/ ˆ K)m)

Linear Manifold Embeddings of Pattern Clusters Haralick, Harpaz

slide-23
SLIDE 23

Linear Manifolds The Algorithm Empirical Evaluation

Selecting the best trial manifold/best separation

−5 5 10 15 20 25 30 −5 5 10 15 20 25 30 −5 5 10 15 20 25 30

To compute a separation score we first need to find the two classes or distributions involved. This problem is cast into histogram thresholding problem.

Linear Manifold Embeddings of Pattern Clusters Haralick, Harpaz

slide-24
SLIDE 24

Linear Manifolds The Algorithm Empirical Evaluation

Kittler and Illingworth Minimum Error Thresholding (86)

−2 2 4 6 8 10 12 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 x p(x) Classification error for Mixture of Two Gaussians T

t J(t) depth

Minimize: P(error) = ❩

x>T

p(x|c1)P(c1)dx + ❩

x≤T

p(x|c2)P(c2)dx KI86: J(T) = 1+2 (P1(T) log σ1(T) + P2(T) log σ2(T))−2 (P1(T) log P1(T) + P2(T) log P2(T)) Goodness of separation: discriminability = (µ1(T) − µ2(T))2 σ2

1(T) + σ2 2(T)

× depth = J(T ′) − J(T)

Linear Manifold Embeddings of Pattern Clusters Haralick, Harpaz

slide-25
SLIDE 25

Linear Manifolds The Algorithm Empirical Evaluation

Time Series Clustering (UCI KDD Archive)

600 × 60, A-decreasing trend, B-cyclic, C-normal, D-upward shift, E-increasing trend, F-downward shift.

in1 in2 in3 in4 in5 in6 total

  • ut1

57 57

  • ut2

80 1 81

  • ut3

43 99 142

  • ut4

20 98 118

  • ut5

99 99

  • ut6

41 41

  • ut7

23 23

  • ut8

1 36 1 1 39 total 100 100 100 100 100 100 600 Total Correct=533 Accuracy=88.8333

Linear Manifold Embeddings of Pattern Clusters Haralick, Harpaz

slide-26
SLIDE 26

Linear Manifolds The Algorithm Empirical Evaluation

Yeast Data - mitotic cell cycle 2884 × 17

(Cho 1998, Tavazoie 1999, http://arep.med.harvard.edu/biclustering/)

100 200 300 400 500 600 100 200 300 400 500 600 yeast data dims:1,2 100 200 300 400 500 600 100 200 300 400 500 600 yeast data dims: 5,9 100 200 300 400 500 600 200 400 600 100 200 300 400 500 600

yeast data dims:1,2,3 100 200 300 400 500 600 200 400 600 100 200 300 400 500 600 yeast data dims:2,7,11

100 200 300 400 500 600 50 100 150 200 250 300 350 400 450 500 yeast data dims: 6,13 100 200 300 400 500 600 100 200 300 400 500 600 yeast data dims: 7,12 100 200 300 400 500 600 200 400 600 100 200 300 400 500 600

yeast data dims: 4,10,17 100 200 300 400 500 600 200 400 600 100 200 300 400 500 600 yeast data dims:8,12,15

PC1 = (0.82, 0.95, 1.02, 0.95, 1.02, 0.93, 0.99, 0.97, 0.92, 1.17, 1.05, 1.02, 0.92, 1.04, 1.03, 1.09, 1.04)′ 93% Linear Manifold Embeddings of Pattern Clusters Haralick, Harpaz

slide-27
SLIDE 27

Linear Manifolds The Algorithm Empirical Evaluation

Yeast Data Results (regular manifolds)

2 4 6 8 10 12 14 16 18 400 450 500 550 600 size=10 msr1=105.7032 msr2=103.4332 lm dim=2 2 4 6 8 10 12 14 16 18 260 280 300 320 340 360 380 400 size=5 msr1=55.8846 msr2=54.9463 lm dim=2 2 4 6 8 10 12 14 16 18 350 400 450 500 550 600 size=94 msr1=250.8523 msr2=220.0825 lm dim=5 2 4 6 8 10 12 14 16 18 220 240 260 280 300 320 340 size=44 msr1=208.8599 msr2=184.9669 lm dim=7 2 4 6 8 10 12 14 16 18 100 200 300 400 500 600 size=62 msr1=1600.9392 msr2=1600.9392 lm dim=1 2 4 6 8 10 12 14 16 18 100 200 300 400 500 600 size=10 msr1=790.9896 msr2=790.9896 lm dim=1

Linear Manifold Embeddings of Pattern Clusters Haralick, Harpaz

slide-28
SLIDE 28

Linear Manifolds The Algorithm Empirical Evaluation

Yeast Data Results (MSR manifolds)

2 4 6 8 10 12 14 16 18 350 400 450 500 550 600 size=130 msr1=226.2737 msr2=226.2737 lm dim=1 2 4 6 8 10 12 14 16 18 100 150 200 250 300 350 400 size=231 msr1=259.3166 msr2=259.3166 lm dim=1 2 4 6 8 10 12 14 400 450 500 550 600 size=12 msr1=72.4352 msr2=48.5136 lm dim=5

100 200 300 400 500 600 700 800 900 1000 2 4 6 8 10 12 14 16 18 50 100 150 200 250 300

size dim msrs

Biclustering/Linear Manifold Clustering

Linear Manifold Embeddings of Pattern Clusters Haralick, Harpaz