A Quantum- -Statistical Statistical- -Mechanical Mechanical A - - PowerPoint PPT Presentation

September 2007 September 2007 IW IW-

• SMI2007, Kyoto

SMI2007, Kyoto 1 1

A Quantum A Quantum-

• Statistical

Statistical-

• Mechanical

Mechanical Extension of Gaussian Mixture Model Extension of Gaussian Mixture Model

Kazuyuki Tanaka Kazuyuki Tanaka

Graduate School of Information Sciences, Graduate School of Information Sciences, Tohoku University, Sendai, Japan Tohoku University, Sendai, Japan http:// http://www.smapip.is.tohoku.ac.jp/~kazu www.smapip.is.tohoku.ac.jp/~kazu/ /

In c In collaborat

• llaboration with

ion with Koji Koji Tsuda Tsuda Max Planck Institute for Biological Cybernetics, Max Planck Institute for Biological Cybernetics, Germany Germany

September 2007 September 2007 IW IW-

• SMI2007, Kyoto

SMI2007, Kyoto 2 2

Contents Contents

1. 1. Introduction Introduction 2. 2. Conventional Gaussian Mixture Model Conventional Gaussian Mixture Model 3. 3. Quantum Mechanical Extension of Quantum Mechanical Extension of Gaussian Mixture Model Gaussian Mixture Model 4. 4. Quantum Belief Propagation Quantum Belief Propagation 5. 5. Concluding Remarks Concluding Remarks

September 2007 September 2007 IW IW-

• SMI2007, Kyoto

SMI2007, Kyoto 3 3

Information Processing Information Processing by using Quantum Statistical by using Quantum Statistical-

• Mechanics

Mechanics

Quantum Annealing in Optimizations Quantum Annealing in Optimizations Quantum Error Correcting Codes Quantum Error Correcting Codes etc... etc...

Massive Information Processing Massive Information Processing by means of Density Matrix by means of Density Matrix

September 2007 September 2007 IW IW-

• SMI2007, Kyoto

SMI2007, Kyoto 4 4

Motivations Motivations

How can we construct the quantum Gaussian mixture model? How can we construct a data- classification algorithm by using the quantum Gaussian mixture model?

September 2007 September 2007 IW IW-

• SMI2007, Kyoto

SMI2007, Kyoto 5 5

Contents Contents

1. 1. Introduction Introduction 2. 2. Conventional Gaussian Mixture Model Conventional Gaussian Mixture Model 3. 3. Quantum Mechanical Extension of Quantum Mechanical Extension of Gaussian Mixture Model Gaussian Mixture Model 4. 4. Quantum Belief Propagation Quantum Belief Propagation 5. 5. Concluding Remarks Concluding Remarks

September 2007 IW-SMI2007, Kyoto 6

Prior of Gauss Mixture Model

1

) 1 ( α = =

i

X P

2

) 2 ( α = =

i

X P

3

) 3 ( α = =

i

X P

∏

=

= = =

N i i i

x X P x X P

1

) ( ) ( r r

1 3 2

1

α

2

α

3

α

Histogram

Label xi is generated randomly and independently of each node.

3 labels

xi =1 xi =2 xi =3 One of three labels 1,2 and 3 is assigned to each node.

September 2007 IW-SMI2007, Kyoto 7

Date Generating Process

Data yi are generated randomly and independently

• f each node.

⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎝ ⎛ − − = = =

2 2

) ( 2 1 exp 2 1 ) | (

k i k k i i i

y k X y Y P μ σ σ π

10 , 60

1 1

= = σ μ 30 , 150

1 1

= = σ μ 20 , 200

3 3

= = σ μ

xi =1 xi =2 xi =3

September 2007 IW-SMI2007, Kyoto 8

Gauss Mixture Models

i

x i i

x X P α = = ) (

∏ ∑ ∏ ∑

= = = =

= = = = = =

N i k i k k N i k i i i i

y g k X P k X y Y P y Y P

1 3 1 1 3 1

) ( ) ( ) | ( ) , , | ( α α σ μ r r r r r

⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎝ ⎛ − − = = = =

2 2

) ( 2 1 exp 2 1 ) ( ) | (

k i k k i k i i i

y y g k X y Y P μ σ σ π

Prior Probability Data Generating Process

) , , | ( max arg ) ˆ , ˆ , ˆ (

) , , (

α σ μ α σ μ

α σ μ

r r r r r r r

r r r

y P =

Marginal Likelihood for Hyperparameters μ,σ and α

September 2007 IW-SMI2007, Kyoto 9

Conventional Gauss Mixture Models

∑ ∑

− = − =

←

1 1

) ( ) (

N i i k N i i k i k

y y y ρ ρ μ

∑

←

k k k k k i

μ k g μ k g y ) , , | ( ) , , | ( ) ( α σ α α σ α r r r r r r ρ

∑ ∑

− = − =

←

1 1 2 k 2

) ( ) ( )

• (

N i i k N i i k i k

y y y ρ ρ μ σ

∑

=

←

N i i k k

y N

1

) ( 1 ρ α

α, μ, σ ρ(yi)

Data :

2 1

⎟ ⎟ ⎟ ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎜ ⎜ ⎜ ⎝ ⎛ =

N

y y y y M r Parameters :

2 1

⎟ ⎟ ⎟ ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎜ ⎜ ⎜ ⎝ ⎛ =

N

x x x x M r

) , , | ( max arg ) ˆ , ˆ , ˆ (

) , , (

α σ μ α σ μ

α σ μ

r r r r r r r

r r r

y P =

Labels

September 2007 September 2007 IW IW-

• SMI2007, Kyoto

SMI2007, Kyoto 10 10

Contents Contents

1. 1. Introduction Introduction 2. 2. Conventional Gaussian Mixture Model Conventional Gaussian Mixture Model 3. 3. Quantum Mechanical Extension of Quantum Mechanical Extension of Gaussian Mixture Model Gaussian Mixture Model 4. 4. Quantum Belief Propagation Quantum Belief Propagation 5. 5. Concluding Remarks Concluding Remarks

September 2007 IW-SMI2007, Kyoto 11

Quantum Gauss Mixture Models

∏ ∑

= =

= =

N i k i k k

y g y Y P

1 3 1

) ( ) , , | ( α α σ μ r r r r r

⎟ ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎜ ⎝ ⎛ ⎟ ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎜ ⎝ ⎛ + ⎟ ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎜ ⎝ ⎛ = ⎟ ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎜ ⎝ ⎛ = ⎟ ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎜ ⎝ ⎛ =

∑

=

) ( ln ) ( ln ) ( ln ln ln ln exp Tr ) ( ln ) ( ln ) ( ln exp Tr ) ( ln ) ( ln ) ( Tr ) (

3 2 1 3 2 1 3 3 2 2 1 1 3 3 2 2 1 1 3 1 i i i i i i i i i k i k k

y g y g y g y g y g y g y g y g y g y g α α α α α α α α α α

⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎝ ⎛ − − =

2 2

) ( 2 1 exp 2 1 ) (

k i k k i k

y y g μ σ σ π

September 2007 IW-SMI2007, Kyoto 12

Quantum Gauss Mixture Models

⎟ ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎜ ⎝ ⎛ − =

3 2 1

ln ln ln α α α F

⎟ ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎜ ⎝ ⎛ − = ) ( ln ) ( ln ) ( ln ) (

3 2 1 i i i i

y g y g y g y G

∏

=

− − − =

N i i

y y P

1

) exp( Tr )) ( exp( Tr ) , , | ( F G F α σ μ r r r r

⎟ ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎜ ⎝ ⎛ − =

3 2 1

ln ln ln α γ γ γ α γ γ γ α F

∏ ∑

= =

= =

N i k i k k

y g y Y P

1 3 1

) ( ) , , | ( α α σ μ r r r r r Quantum Representation

September 2007 IW-SMI2007, Kyoto 13

Quantum Gauss Mixture Models

∑∑

= =

− = ⎟ ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎜ ⎝ ⎛ − =

3 1 3 1 ) ( 3 3 2 2 1 1

) ( ln ) ( ln ) ( ln ) (

k l kl i kl i i i i

B y g y g y g y X H α γ γ γ α γ γ γ α

∏

=

− − =

N i i

y y P

1

) exp( Tr )) ( exp( Tr ) , , | ( F H α σ μ r r r r

⎩ ⎨ ⎧ ≠ = = ) ( ) ( ) ( ln

) (

l k l k y g B

i k k i kl

γ α

⎟ ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎜ ⎝ ⎛ = ⎟ ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎜ ⎝ ⎛ = ⎟ ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎜ ⎝ ⎛ = 1 1 1

33 12 11

X X X L

September 2007 IW-SMI2007, Kyoto 14

) , , | ( max arg ) ˆ , ˆ , ˆ (

) , , (

α σ μ α σ μ

α σ μ

r r r r r r r

r r r

y P =

) ( ) ( ) ( 1 ) ( ) ( ) 1 ( ) (

Tr Tr e Tr Tr ) e Tr ln(

i i i i i i

y H y H kl y y H kl y H y kl

e e X d e X e B

− − − − − − −

= = ∂ ∂

∫

H H

λ

λ λ

α σ μ r r r and , for Condition Extremun

Linear Response Formulas

∏

=

− − =

N i i

y y P

1

) exp( Tr )) ( exp( Tr ) , , | ( F H α σ μ r r r r

Maxmum Likelihood Estimation in Quantum Gauss Mixture Model

September 2007 IW-SMI2007, Kyoto 15

Quantum Gauss Mixture Models

∑ ∑

= =

←

N i i kk N i i kk i k

y y y

1 1

) ( Tr ) ( Tr ρ X ρ X μ ) ( ) (

Tr ) (

i i

y y i

e e y

H H

ρ

− −

←

∑ ∑

= =

←

N i i kk N i i kk i k

y y y

1 1 2 k 2

) ( Tr ) ( Tr )

• (

ρ X ρ X μ σ

⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎝ ⎛ ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎝ ⎛ ←

∑

= N i i kk k

y N

1

) ( 1 ln Tr exp ρ X α

α, μ, σ ρ(yi)

Data :

2 1

⎟ ⎟ ⎟ ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎜ ⎜ ⎜ ⎝ ⎛ =

N

y y y y M r Parameters :

2 1

⎟ ⎟ ⎟ ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎜ ⎜ ⎜ ⎝ ⎛ =

N

x x x x M r

) , , | ( max arg ) ˆ , ˆ , ˆ (

) , , (

α σ μ α σ μ

α σ μ

r r r r r r r

r r r

y P =

September 2007 September 2007 IW IW-

• SMI2007, Kyoto

SMI2007, Kyoto 16 16

Image Segmentation Image Segmentation

Original Image Histogram Conventional Gauss Mixture Model Quantum Gauss Mixture Model

γ = 0.2 γ = 0.4

255 255 255

) exp( Tr )) ( exp( Tr ) , , | ( F G F − − − =

i i

y y P α σ μ r r r

September 2007 September 2007 IW IW-

• SMI2007, Kyoto

SMI2007, Kyoto 17 17

Image Segmentation Image Segmentation

Original Image Histogram Conventional Gauss Mixture Model Quantum Gauss Mixture Model

γ = 0.5 γ = 1.0

255 255 255

) exp( Tr )) ( exp( Tr ) , , | ( F G F − − − =

i i

y y P α σ μ r r r

September 2007 September 2007 IW IW-

• SMI2007, Kyoto

SMI2007, Kyoto 18 18

Contents Contents

1. 1. Introduction Introduction 2. 2. Conventional Gaussian Mixture Model Conventional Gaussian Mixture Model 3. 3. Quantum Mechanical Extension of Quantum Mechanical Extension of Gaussian Mixture Model Gaussian Mixture Model 4. 4. Quantum Belief Propagation Quantum Belief Propagation 5. 5. Concluding Remarks Concluding Remarks

September 2007 September 2007 IW IW-

• SMI2007, Kyoto

SMI2007, Kyoto 19 19

Image Segmentation by Combining Image Segmentation by Combining Gauss Mixture Model with Potts Model Gauss Mixture Model with Potts Model

( ) ( ) ( ) ( ) ( ) ( ) ( ) ( )

10 4 3 2 1 , 192 4 , 192 3 , 127 2 , 64 1 = = = = = = = = σ σ σ σ μ μ μ μ

Belief Propagation Belief Propagation

( )⎟

⎟ ⎠ ⎞ ⎜ ⎜ ⎝ ⎛ × ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎝ ⎛ = =

∏ ∏

− = Neighbour Nearest : , 1

exp ) (

ij x x N i x

j i i

J x X P δ α r r

= = >

Potts Model

4 labels

September 2007 September 2007 IW IW-

• SMI2007, Kyoto

SMI2007, Kyoto 20 20

Image Segmentation Image Segmentation

Original Image Histogram

Gauss Mixture Model Gauss Mixture Model and Potts Model

Belief Belief Propagation Propagation

September 2007 IW-SMI2007, Kyoto 21

Density Matrix and Reduced Density Matrix

∑

∈

≡

B ij ij

H H ˆ

( )

H ρ − ≡ exp 1 Z

ρ ρ

i i \

tr ≡

ρ ρ

ij ij \

tr ≡

ij i i

ρ ρ

\

tr ≡

Reduced Density Matrix Reducibility Condition

j

i

September 2007 IW-SMI2007, Kyoto 22

Reduced Density Matrix and Effective Fields

⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎝ ⎛ ≅

∑

∈ →

i

B i k i

λ ρ

k i

Z exp 1

⎟ ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎜ ⎝ ⎛ ⊗ + ⊗ + − ≅

∑ ∑

∈ → ∈ → i \ B l j l j \ B k i k ij ij

j i

λ I I λ H ρ exp 1

ij

Z

i

j

i

All effective field are matrices

i

B j Bi \ i B j \

September 2007 IW-SMI2007, Kyoto 23

Belief Propagation for Quantum Statistical Systems

⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎝ ⎛ ⎥ ⎥ ⎦ ⎤ ⎢ ⎢ ⎣ ⎡ ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎝ ⎛ ⊗ + ⊗ + − + − =

∑ ∑ ∑

∈ → ∈ → ∈ → → i \ B l j l j \ B k i k ij j \ B k i k i j

j i i

λ I I λ H λ λ exp tr log

\i ij i

Z Z

Propagating Rule of Effective Fields

ij i i

ρ ρ

\

tr ≡

j

i

Output

September 2007 September 2007 IW IW-

• SMI2007, Kyoto

SMI2007, Kyoto 24 24

Contents Contents

1. 1. Introduction Introduction 2. 2. Conventional Gaussian Mixture Model Conventional Gaussian Mixture Model 3. 3. Quantum Mechanical Extension of Quantum Mechanical Extension of Gaussian Mixture Model Gaussian Mixture Model 4. 4. Quantum Belief Propagation Quantum Belief Propagation 5. 5. Concluding Remarks Concluding Remarks

September 2007 September 2007 IW IW-

• SMI2007, Kyoto

SMI2007, Kyoto 25 25

Summary Summary

An Extension to Quantum Statistical Mechanical An Extension to Quantum Statistical Mechanical Gaussian Mixture Model Gaussian Mixture Model Practical Algorithm Practical Algorithm Linear Response Formula Linear Response Formula Application of Potts Model and Application of Potts Model and Quantum Belief Propagation Quantum Belief Propagation Applications to Data Mining Applications to Data Mining Extension to Quantum Deterministic Annealing Future Problem Future Problem