Spin Glasses and Information Processing Pavithran S Iyer Guide: - - PowerPoint PPT Presentation

Outline Overview Information Theory Disordered spin systems Implications of the correspondence Questions Bibliography

Spin Glasses and Information Processing

Pavithran S Iyer Guide: Prof. V.V Sreedhar

Chennai Mathematical Institute

April 25, 2011

Pavithran S Iyer Guide: Prof. V.V Sreedhar Spin Glasses and Information Processing

Outline Overview Information Theory Disordered spin systems Implications of the correspondence Questions Bibliography

1 Overview 2 Information Theory

Communication problem Error correcting codes Shannon Heartely theorem

3 Disordered spin systems

Introduction Reason for correspondence Spin glass physics

4 Implications of the correspondence

SK Model REM Convolution Codes

5 Questions 6 Bibliography

Pavithran S Iyer Guide: Prof. V.V Sreedhar Spin Glasses and Information Processing

Outline Overview Information Theory Disordered spin systems Implications of the correspondence Questions Bibliography

Outline

Work described - papers by N. Sourlas and a book by Nishimori Looking at: correspondences Error correcting code ⇔ Spin Hamiltonian Signal to noise ⇔ J2

J2

Maximum likelihood Decoding ⇔ Find a ground state Error probability per bit ⇔ Ground state magnetization Sequence of most probable symbols ⇔ magnetization at T = 1 Convolutional Codes ⇔ One dimentional spin glasses Viterbi decoding ⇔ T = 0 Transfer matrix algorithm BCJR decoding ⇔ T = 1 Transfer matrix algorithm Gallager LDPC codes ⇔ Diluted p-spin ferromagnets Turbo Codes ⇔ Coupled spin chains Zero error threshold ⇔ Phase transition point Belief propagation algorithm ⇔ Iterative solution of TAP equations

Pavithran S Iyer Guide: Prof. V.V Sreedhar Spin Glasses and Information Processing

Outline Overview Information Theory Disordered spin systems Implications of the correspondence Questions Bibliography

Outline

correspondences Error correcting code ⇔ Spin Hamiltonian Signal to noise ⇔ J2

J2

Maximum likelihood Decoding ⇔ Find a ground state Error probability per bit ⇔ Ground state magnetization Sequence of most probable symbols ⇔ magnetization at T = 1 Convolutional Codes ⇔ One dimentional spin glasses Viterbi decoding ⇔ T = 0 Transfer matrix algorithm BCJR decoding ⇔ T = 1 Transfer matrix algorithm Gallager LDPC codes ⇔ Diluted p-spin ferromagnets Turbo Codes ⇔ Coupled spin chains Zero error threshold ⇔ Phase transition point Belief propagation algorithm ⇔ Iterative solution of TAP equations

Pavithran S Iyer Guide: Prof. V.V Sreedhar Spin Glasses and Information Processing

Outline Overview Information Theory Disordered spin systems Implications of the correspondence Questions Bibliography Communication problem

Communication Problem

− → usual formulation - message from Alice to Bob Alice transmits encoded input - (gaussian) channel inflicts error - Bob tries to recover from the error Statistical formulation: Bob’s perspective - given an output, maximizes his guess of the input being correct. Maximizing quantity: P

• Jin|Jout
• called the posterior

probability. Maximum Aposteriori Probability or MAP decoding: compute conditional probabilities using baye’s theorem, assign Jin

i

= 1 if P (Ji = 1|Jout) > P (−1|Jout) and Jin

i

= −1

• therwise. Maximum information about the (Alice) input which can be transmitted

across the channel to Bob = channel capacity C. Input (signal) power S & Noise (power) = N, then important quantities S N = signal to noise ratio and for a gaussian channel, channel capacity 1

• S

Pavithran S Iyer Guide: Prof. V.V Sreedhar Spin Glasses and Information Processing

Outline Overview Information Theory Disordered spin systems Implications of the correspondence Questions Bibliography Communication problem

Communication Problem

− → usual formulation - message from Alice to Bob Alice transmits encoded input - (gaussian) channel inflicts error - Bob tries to recover from the error Statistical formulation: Bob’s perspective - given an output, maximizes his guess of the input being correct. Maximizing quantity: P

• Jin|Jout
• called the posterior

probability. Maximum Aposteriori Probability or MAP decoding: compute conditional probabilities using baye’s theorem, assign Jin

i

= 1 if P (Ji = 1|Jout) > P (−1|Jout) and Jin

i

= −1

• therwise. Maximum information about the (Alice) input which can be transmitted

across the channel to Bob = channel capacity C. Input (signal) power S & Noise (power) = N, then important quantities S N = signal to noise ratio and for a gaussian channel, channel capacity 1

• S

Pavithran S Iyer Guide: Prof. V.V Sreedhar Spin Glasses and Information Processing

Outline Overview Information Theory Disordered spin systems Implications of the correspondence Questions Bibliography Communication problem

Communication Problem

Alice transmits encoded input - (gaussian) channel inflicts error - Bob tries to recover from the error Statistical formulation: Bob’s perspective - given an output, maximizes his guess of the input being correct. Maximizing quantity: P

• Jin|Jout
• called the posterior

probability. Maximum Aposteriori Probability or MAP decoding: compute conditional probabilities using baye’s theorem, assign Jin

i

= 1 if P (Ji = 1|Jout) > P (−1|Jout) and Jin

i

= −1

• therwise. Maximum information about the (Alice) input which can be transmitted

across the channel to Bob = channel capacity C. Input (signal) power S & Noise (power) = N, then important quantities S N = signal to noise ratio and for a gaussian channel, channel capacity C = 1 2 log2

• 1 + S

N

• .

Pavithran S Iyer Guide: Prof. V.V Sreedhar Spin Glasses and Information Processing

Outline Overview Information Theory Disordered spin systems Implications of the correspondence Questions Bibliography Communication problem

Communication Problem

Statistical formulation: Bob’s perspective - given an output, maximizes his guess of the input being correct. Maximizing quantity: P

• Jin|Jout
• called the posterior

probability. Maximum Aposteriori Probability or MAP decoding: compute conditional probabilities using baye’s theorem, assign Jin

i

= 1 if P (Ji = 1|Jout) > P (−1|Jout) and Jin

i

= −1

• therwise. Maximum information about the (Alice) input which can be transmitted

across the channel to Bob = channel capacity C. Input (signal) power S & Noise (power) = N, then important quantities S N = signal to noise ratio and for a gaussian channel, channel capacity C = 1 2 log2

• 1 + S

N

• .

Pavithran S Iyer Guide: Prof. V.V Sreedhar Spin Glasses and Information Processing

Outline Overview Information Theory Disordered spin systems Implications of the correspondence Questions Bibliography Communication problem

Communication Problem

Statistical formulation: Bob’s perspective - given an output, maximizes his guess of the input being correct. Maximizing quantity: P

• Jin|Jout
• called the posterior

probability. Maximum Aposteriori Probability or MAP decoding: compute conditional probabilities using baye’s theorem, assign Jin

i

= 1 if P (Ji = 1|Jout) > P (−1|Jout) and Jin

i

= −1

• therwise. Maximum information about the (Alice) input which can be transmitted

across the channel to Bob = channel capacity C. Input (signal) power S & Noise (power) = N, then important quantities S N = signal to noise ratio and for a gaussian channel, channel capacity C = 1 2 log2

• 1 + S

N

• .

Pavithran S Iyer Guide: Prof. V.V Sreedhar Spin Glasses and Information Processing

Outline Overview Information Theory Disordered spin systems Implications of the correspondence Questions Bibliography Error correcting codes

Error correcting code

Not all encodings can assure recovery from error - only certain codes called error correcting codes. Crux: add redundant bits to input message - majority of bits are unaffected by error -

• riginal message can be retrieved.

Redundancy is undesirable - slow rate of information transmission. Rate of transmission Rate of information = # bits for encoding (ignoring error) # bits used for encoding with error

Pavithran S Iyer Guide: Prof. V.V Sreedhar Spin Glasses and Information Processing

Outline Overview Information Theory Disordered spin systems Implications of the correspondence Questions Bibliography Error correcting codes

Error correcting code

Not all encodings can assure recovery from error - only certain codes called error correcting codes. Crux: add redundant bits to input message - majority of bits are unaffected by error -

• riginal message can be retrieved.

Redundancy is undesirable - slow rate of information transmission. Rate of transmission Rate of information = # bits for encoding (ignoring error) # bits used for encoding with error

Pavithran S Iyer Guide: Prof. V.V Sreedhar Spin Glasses and Information Processing

Outline Overview Information Theory Disordered spin systems Implications of the correspondence Questions Bibliography Error correcting codes

Error correcting code

Not all encodings can assure recovery from error - only certain codes called error correcting codes. Crux: add redundant bits to input message - majority of bits are unaffected by error -

• riginal message can be retrieved.

Redundancy is undesirable - slow rate of information transmission. Rate of transmission Rate of information = # bits for encoding (ignoring error) # bits used for encoding with error

Pavithran S Iyer Guide: Prof. V.V Sreedhar Spin Glasses and Information Processing

Outline Overview Information Theory Disordered spin systems Implications of the correspondence Questions Bibliography Shannon Heartely theorem

A theorem

Aim: Maximum rate. upperbound ? Shannon Heartely or Noisy channel coding theorem The rate of an error correcting code cannot exceeed the channel capacity for noiseless

• transmission. R ≤ C.

The aim of every ECC is to go as close to the bound. This bound is a theoretical maximum.

Pavithran S Iyer Guide: Prof. V.V Sreedhar Spin Glasses and Information Processing

Outline Overview Information Theory Disordered spin systems Implications of the correspondence Questions Bibliography Shannon Heartely theorem

A theorem

Aim: Maximum rate. upperbound ? Shannon Heartely or Noisy channel coding theorem The rate of an error correcting code cannot exceeed the channel capacity for noiseless

• transmission. R ≤ C.

The aim of every ECC is to go as close to the bound. This bound is a theoretical maximum.

Pavithran S Iyer Guide: Prof. V.V Sreedhar Spin Glasses and Information Processing

Outline Overview Information Theory Disordered spin systems Implications of the correspondence Questions Bibliography Introduction

A disordered spin system

Ising model H = −J

i<j SiSj . For any choice of J - exactly 2 ground states.

Bad information storage structures - can only store two units of information. Need plenty of ground states. Spin glass - lot of equilibrium states. Naively: put Jij = SiSj. J - local to a pair of sites - link variable:

• +1

: Ferro −1 : Antiferro Finding ground states of a simple spin glass is difficult[1] - main reason - link variables are random - frustration causing many degenerate ground states. Note: would not happen if all Jij = some constant.

Pavithran S Iyer Guide: Prof. V.V Sreedhar Spin Glasses and Information Processing

Outline Overview Information Theory Disordered spin systems Implications of the correspondence Questions Bibliography Introduction

A disordered spin system

Ising model H = −J

i<j SiSj . For any choice of J - exactly 2 ground states.

Bad information storage structures - can only store two units of information. Need plenty of ground states. Spin glass - lot of equilibrium states. Naively: put Jij = SiSj. J - local to a pair of sites - link variable:

• +1

: Ferro −1 : Antiferro Finding ground states of a simple spin glass is difficult[1] - main reason - link variables are random - frustration causing many degenerate ground states. Note: would not happen if all Jij = some constant.

Pavithran S Iyer Guide: Prof. V.V Sreedhar Spin Glasses and Information Processing

Outline Overview Information Theory Disordered spin systems Implications of the correspondence Questions Bibliography Introduction

A disordered spin system

Ising model H = −J

i<j SiSj . For any choice of J - exactly 2 ground states.

Bad information storage structures - can only store two units of information. Need plenty of ground states. Spin glass - lot of equilibrium states. Naively: put Jij = SiSj. J - local to a pair of sites - link variable:

• +1

: Ferro −1 : Antiferro Finding ground states of a simple spin glass is difficult[1] - main reason - link variables are random - frustration causing many degenerate ground states. Note: would not happen if all Jij = some constant.

Pavithran S Iyer Guide: Prof. V.V Sreedhar Spin Glasses and Information Processing

Outline Overview Information Theory Disordered spin systems Implications of the correspondence Questions Bibliography Introduction

A disordered spin system

Ising model H = −J

i<j SiSj . For any choice of J - exactly 2 ground states.

Bad information storage structures - can only store two units of information. Need plenty of ground states. Spin glass - lot of equilibrium states. Naively: put Jij = SiSj. J - local to a pair of sites - link variable:

• +1

: Ferro −1 : Antiferro Finding ground states of a simple spin glass is difficult[1] - main reason - link variables are random - frustration causing many degenerate ground states. Note: would not happen if all Jij = some constant.

Pavithran S Iyer Guide: Prof. V.V Sreedhar Spin Glasses and Information Processing

Outline Overview Information Theory Disordered spin systems Implications of the correspondence Questions Bibliography Introduction

A disordered spin system

Ising model H = −J

i<j SiSj . For any choice of J - exactly 2 ground states.

Bad information storage structures - can only store two units of information. Need plenty of ground states. Spin glass - lot of equilibrium states. Naively: put Jij = SiSj. J - local to a pair of sites - link variable:

• +1

: Ferro −1 : Antiferro Finding ground states of a simple spin glass is difficult[1] - main reason - link variables are random - frustration causing many degenerate ground states. Note: would not happen if all Jij = some constant.

Pavithran S Iyer Guide: Prof. V.V Sreedhar Spin Glasses and Information Processing

Outline Overview Information Theory Disordered spin systems Implications of the correspondence Questions Bibliography Reason for correspondence

Interesting correspondance

Going back - maximizing posterior probability P(Jin|Jout) - using baye’s theorem relates it to P(Jout|Jin). After some algebra, we find: P(Jij|Jout) = exp

• u

n Mn k1...knJin k1 . . . Jin kn − i hiJin i

• →

exponent strikingly similar to the hamiltonian of a spin glass - minimize this ⇒ find a ground state of the spin glass we have a spin glass where

• Jin

i

• play the role of spins and Mk1...kn contain link

variables for n-spin interactions. Instead of finding ground states, demontrating encoding & decoding, we look at more similarities between SG physics and ECC. All the above are only for ising spins.

Pavithran S Iyer Guide: Prof. V.V Sreedhar Spin Glasses and Information Processing

Outline Overview Information Theory Disordered spin systems Implications of the correspondence Questions Bibliography Reason for correspondence

Interesting correspondance

Going back - maximizing posterior probability P(Jin|Jout) - using baye’s theorem relates it to P(Jout|Jin). After some algebra, we find: P(Jij|Jout) = exp

• u

n Mn k1...knJin k1 . . . Jin kn − i hiJin i

• →

exponent strikingly similar to the hamiltonian of a spin glass - minimize this ⇒ find a ground state of the spin glass we have a spin glass where

• Jin

i

• play the role of spins and Mk1...kn contain link

variables for n-spin interactions. Instead of finding ground states, demontrating encoding & decoding, we look at more similarities between SG physics and ECC. All the above are only for ising spins.

Pavithran S Iyer Guide: Prof. V.V Sreedhar Spin Glasses and Information Processing

Outline Overview Information Theory Disordered spin systems Implications of the correspondence Questions Bibliography Reason for correspondence

Interesting correspondance

Going back - maximizing posterior probability P(Jin|Jout) - using baye’s theorem relates it to P(Jout|Jin). After some algebra, we find: P(Jij|Jout) = exp

• u

n Mn k1...knJin k1 . . . Jin kn − i hiJin i

• →

exponent strikingly similar to the hamiltonian of a spin glass - minimize this ⇒ find a ground state of the spin glass we have a spin glass where

• Jin

i

• play the role of spins and Mk1...kn contain link

variables for n-spin interactions. Instead of finding ground states, demontrating encoding & decoding, we look at more similarities between SG physics and ECC. All the above are only for ising spins.

Pavithran S Iyer Guide: Prof. V.V Sreedhar Spin Glasses and Information Processing

Outline Overview Information Theory Disordered spin systems Implications of the correspondence Questions Bibliography Reason for correspondence

Interesting correspondance

Going back - maximizing posterior probability P(Jin|Jout) - using baye’s theorem relates it to P(Jout|Jin). After some algebra, we find: P(Jij|Jout) = exp

• u

n Mn k1...knJin k1 . . . Jin kn − i hiJin i

• →

exponent strikingly similar to the hamiltonian of a spin glass - minimize this ⇒ find a ground state of the spin glass we have a spin glass where

• Jin

i

• play the role of spins and Mk1...kn contain link

variables for n-spin interactions. Instead of finding ground states, demontrating encoding & decoding, we look at more similarities between SG physics and ECC. All the above are only for ising spins.

Pavithran S Iyer Guide: Prof. V.V Sreedhar Spin Glasses and Information Processing

Outline Overview Information Theory Disordered spin systems Implications of the correspondence Questions Bibliography Reason for correspondence

Interesting correspondance

Going back - maximizing posterior probability P(Jin|Jout) - using baye’s theorem relates it to P(Jout|Jin). After some algebra, we find: P(Jij|Jout) = exp

• u

n Mn k1...knJin k1 . . . Jin kn − i hiJin i

• →

exponent strikingly similar to the hamiltonian of a spin glass - minimize this ⇒ find a ground state of the spin glass we have a spin glass where

• Jin

i

• play the role of spins and Mk1...kn contain link

variables for n-spin interactions. Instead of finding ground states, demontrating encoding & decoding, we look at more similarities between SG physics and ECC. All the above are only for ising spins.

Pavithran S Iyer Guide: Prof. V.V Sreedhar Spin Glasses and Information Processing

Outline Overview Information Theory Disordered spin systems Implications of the correspondence Questions Bibliography Spin glass physics

Physics of spin glass

Spins on a lattice - total energy invariant under local transformations Simple example: SK model

Pavithran S Iyer Guide: Prof. V.V Sreedhar Spin Glasses and Information Processing

Outline Overview Information Theory Disordered spin systems Implications of the correspondence Questions Bibliography Spin glass physics

Physics of spin glass

Spins on a lattice - total energy invariant under local transformations Phases: Paramagnetic, Ferromagnetic, Spin glass Order Parameters: Phase m q Ferro > 0 > 0 SG > 0 Para In SG phase, m = 0 - random alignment but q = 0 - key difference from paramagnetic phase. Over large time, spins at two lattice sites will be correlated[1]. Simple example: SK model

Pavithran S Iyer Guide: Prof. V.V Sreedhar Spin Glasses and Information Processing

Outline Overview Information Theory Disordered spin systems Implications of the correspondence Questions Bibliography Spin glass physics

Physics of spin glass

Phases: Paramagnetic, Ferromagnetic, Spin glass Order Parameters: Phase m q Ferro > 0 > 0 SG > 0 Para In SG phase, m = 0 - random alignment but q = 0 - key difference from paramagnetic phase. Over large time, spins at two lattice sites will be correlated[1]. Simple example: SK model

Pavithran S Iyer Guide: Prof. V.V Sreedhar Spin Glasses and Information Processing

Outline Overview Information Theory Disordered spin systems Implications of the correspondence Questions Bibliography Spin glass physics

Physics of spin glass

Phases: Paramagnetic, Ferromagnetic, Spin glass Order Parameters: Phase m q Ferro > 0 > 0 SG > 0 Para In SG phase, m = 0 - random alignment but q = 0 - key difference from paramagnetic phase. Over large time, spins at two lattice sites will be correlated[1]. Information cannot be encoded in paramagnetic phase - high random fluctuations destroy spin-spin correlation. Simple example: SK model

Pavithran S Iyer Guide: Prof. V.V Sreedhar Spin Glasses and Information Processing

Outline Overview Information Theory Disordered spin systems Implications of the correspondence Questions Bibliography Spin glass physics

Physics of spin glass

Aim: Look at SG phase and overlap. Phase transitions in a disordered spin system - using Ginsburg Landau theory - expanding the free energy about critical points (small order parameter). Simple example: SK model

Pavithran S Iyer Guide: Prof. V.V Sreedhar Spin Glasses and Information Processing

Outline Overview Information Theory Disordered spin systems Implications of the correspondence Questions Bibliography Spin glass physics

Physics of spin glass

Aim: Look at SG phase and overlap. Simple example: SK model H =

i<j JijSiSj. Distribution of links, Jij- given by

P(Jij) =

• N

2πJ2 exp

• − N

2J2

• Jij − J0

N 2

Pavithran S Iyer Guide: Prof. V.V Sreedhar Spin Glasses and Information Processing

Outline Overview Information Theory Disordered spin systems Implications of the correspondence Questions Bibliography Spin glass physics

Physics of spin glass

Simple example: SK model H =

i<j JijSiSj. Distribution of links, Jij- given by

P(Jij) =

• N

2πJ2 exp

• − N

2J2

• Jij − J0

N 2 Free energy: every sample - one realization of disorder - n replicas of the system - average over all values of Jij - gaussian distribution

Pavithran S Iyer Guide: Prof. V.V Sreedhar Spin Glasses and Information Processing

Outline Overview Information Theory Disordered spin systems Implications of the correspondence Questions Bibliography Spin glass physics

Physics of spin glass

Simple example: SK model After some algebra - free energy ≡ f (m, {qαβ}, T, J, J0). Equations of state - determine value for order parameters in terms of J, J0, T.

Pavithran S Iyer Guide: Prof. V.V Sreedhar Spin Glasses and Information Processing

Outline Overview Information Theory Disordered spin systems Implications of the correspondence Questions Bibliography Spin glass physics

Physics of spin glass

Simple example: SK model Infinite range model - H =

i1<i2<···<ir Si1Si2 · · · Sin.

Similar calculations as SK model yield free energy & existence of spin glass phase at full RSB. Aim is to demonstrate shannon heartely theorem - take a system for which 1RSB is sufficient - calculation convenience.

Pavithran S Iyer Guide: Prof. V.V Sreedhar Spin Glasses and Information Processing

Outline Overview Information Theory Disordered spin systems Implications of the correspondence Questions Bibliography Spin glass physics

Physics of spin glass

Simple example: SK model REM r → ∞ limit of Infinite range model - probability of a state only depends on energy & independent distribution of energy states 1 RSB is enough → exact calculations confirm with 1RSB results Magnetic phases: Phases Condition P ↔ SG Tc = J 2 √ ln 2 SG ↔ F j0 = J √ ln 2 P ↔ F j0 = J2 4T + T ln 2 Calculations for overlap - M=1 in ferromagnetic phase - j0/J > √ ln 2. Hence, error free decoding.

Pavithran S Iyer Guide: Prof. V.V Sreedhar Spin Glasses and Information Processing

Outline Overview Information Theory Disordered spin systems Implications of the correspondence Questions Bibliography

Implications of this correspondence

Transistion from binary to ±1 since spin & link variables take ±1 values. Symbol u → (−1)u. raw input ⇔ spin orientations, code symbols ⇔ link variables in ground state signal amplitude ⇔ J0 and noise amplitude ⇔ J. word MAP decoding ⇔ finding ground state. error probability per bit ⇔ ground state magnetization. sequence of most probable bits ⇔ magnetization at T = 1. Error free decoding ⇔ overlap: M = 1 Hence we have some correspondances.

Pavithran S Iyer Guide: Prof. V.V Sreedhar Spin Glasses and Information Processing

Outline Overview Information Theory Disordered spin systems Implications of the correspondence Questions Bibliography

Implications of this correspondence

Transistion from binary to ±1 since spin & link variables take ±1 values. Symbol u → (−1)u. Information processing using Spin glass → raw input ⇔ spin orientations, encoded into code symbols ⇔ link variables in ground state by introducing interactions. signal amplitude ⇔ J0 and noise amplitude ⇔ J. word MAP decoding ⇔ finding ground state. error probability per bit ⇔ ground state magnetization. sequence of most probable bits ⇔ magnetization at T = 1. Error free decoding ⇔ overlap: M = 1 Hence we have some correspondances.

Pavithran S Iyer Guide: Prof. V.V Sreedhar Spin Glasses and Information Processing

Outline Overview Information Theory Disordered spin systems Implications of the correspondence Questions Bibliography

Implications of this correspondence

raw input ⇔ spin orientations, code symbols ⇔ link variables in ground state signal amplitude ⇔ J0 and error operation

Temp

− − − →

rise ? gaussian disorder - variance of J2 -

noise amplitude ⇔ J. word MAP decoding ⇔ finding ground state. error probability per bit ⇔ ground state magnetization. sequence of most probable bits ⇔ magnetization at T = 1. Error free decoding ⇔ overlap: M = 1 Hence we have some correspondances.

Pavithran S Iyer Guide: Prof. V.V Sreedhar Spin Glasses and Information Processing

Outline Overview Information Theory Disordered spin systems Implications of the correspondence Questions Bibliography

Implications of this correspondence

raw input ⇔ spin orientations, code symbols ⇔ link variables in ground state signal amplitude ⇔ J0 and noise amplitude ⇔ J. State of spin glass corresponding to output has variance J2

0 + J2 in link variables.

word MAP decoding ⇔ finding ground state. error probability per bit ⇔ ground state magnetization. sequence of most probable bits ⇔ magnetization at T = 1. Error free decoding ⇔ overlap: M = 1 Hence we have some correspondances.

Pavithran S Iyer Guide: Prof. V.V Sreedhar Spin Glasses and Information Processing

Outline Overview Information Theory Disordered spin systems Implications of the correspondence Questions Bibliography

Implications of this correspondence

raw input ⇔ spin orientations, code symbols ⇔ link variables in ground state signal amplitude ⇔ J0 and noise amplitude ⇔ J. Decoding ⇒ finding an assignment for Jin which maximizes P

• Jin|Jout
• same

assigmnet would minimize − ln P

• Jin|Jout

≡ H → ground state word MAP decoding ⇔ finding ground state. error probability per bit ⇔ ground state magnetization. sequence of most probable bits ⇔ magnetization at T = 1. Error free decoding ⇔ overlap: M = 1 Hence we have some correspondances.

Pavithran S Iyer Guide: Prof. V.V Sreedhar Spin Glasses and Information Processing

Outline Overview Information Theory Disordered spin systems Implications of the correspondence Questions Bibliography

Implications of this correspondence

raw input ⇔ spin orientations, code symbols ⇔ link variables in ground state signal amplitude ⇔ J0 and noise amplitude ⇔ J. word MAP decoding ⇔ finding ground state. error probability per bit ⇔ ground state magnetization. sequence of most probable bits ⇔ magnetization at T = 1. Error free decoding ⇔ overlap: M = 1 Hence we have some correspondances.

Pavithran S Iyer Guide: Prof. V.V Sreedhar Spin Glasses and Information Processing

Outline Overview Information Theory Disordered spin systems Implications of the correspondence Questions Bibliography

Implications of this correspondence

raw input ⇔ spin orientations, code symbols ⇔ link variables in ground state signal amplitude ⇔ J0 and noise amplitude ⇔ J. word MAP decoding ⇔ finding ground state. error probability per bit ⇔ ground state magnetization. Average value of a bit at index i: τiP =

• i τiP (τi|Jout)
• i P (τi|Jout) −

− − − − − − − − − − − − − − →

P(τi|Jout)=e−H[Jout ,{τi }]

• i τie−H[Jout,{τi}]
• i e−H[Jout,{τi}] . Putting ln P ∼ Z,

we see τiP = magnetization with β = 1. sequence of most probable bits ⇔ magnetization at T = 1. Error free decoding ⇔ overlap: M = 1 Hence we have some correspondances.

Pavithran S Iyer Guide: Prof. V.V Sreedhar Spin Glasses and Information Processing

Outline Overview Information Theory Disordered spin systems Implications of the correspondence Questions Bibliography

Implications of this correspondence

raw input ⇔ spin orientations, code symbols ⇔ link variables in ground state signal amplitude ⇔ J0 and noise amplitude ⇔ J. word MAP decoding ⇔ finding ground state. error probability per bit ⇔ ground state magnetization. sequence of most probable bits ⇔ magnetization at T = 1. Measuring decoding performance: overlap: overlap of original & decoded message M. Suppose original bit: ξi & MAP decoded bit: ˆ ξi = signσi, then: M = Trξ

• J P(J)ξsignσi ⇔ Hamming Distance =
• 1

both are same −1 when inverted . Error free decoding ⇔ overlap: M = 1 Hence we have some correspondances.

Pavithran S Iyer Guide: Prof. V.V Sreedhar Spin Glasses and Information Processing

Outline Overview Information Theory Disordered spin systems Implications of the correspondence Questions Bibliography

Implications of this correspondence

raw input ⇔ spin orientations, code symbols ⇔ link variables in ground state signal amplitude ⇔ J0 and noise amplitude ⇔ J. word MAP decoding ⇔ finding ground state. error probability per bit ⇔ ground state magnetization. sequence of most probable bits ⇔ magnetization at T = 1. Error free decoding ⇔ overlap: M = 1 Hence we have some correspondances.

Pavithran S Iyer Guide: Prof. V.V Sreedhar Spin Glasses and Information Processing

Outline Overview Information Theory Disordered spin systems Implications of the correspondence Questions Bibliography SK Model

SK Model

For SK Model: Phase diagram - for the SK model.

Pavithran S Iyer Guide: Prof. V.V Sreedhar Spin Glasses and Information Processing

Outline Overview Information Theory Disordered spin systems Implications of the correspondence Questions Bibliography REM

Random Energy Model

Error free decoding in ferromagnetic phase for T =. Rate: Totally N sites used for encoding and N

r

• possible sites. In the

r → ∞ limit[?]: R = r! Nr−1 . Channel Capacity: for a gaussian channel[?] C − − − →

r→∞

j2

0r!

2J2Nr−1 ln 2. where we have used[?] J0 = signal amplitude, J = noise amplitude. Shannon Heartely bound is satisfied for this encoding - interesting result !

Pavithran S Iyer Guide: Prof. V.V Sreedhar Spin Glasses and Information Processing

Outline Overview Information Theory Disordered spin systems Implications of the correspondence Questions Bibliography REM

Random Energy Model

In the r → ∞ limit[?]: R = r! Nr−1 . Channel Capacity: for a gaussian channel[?] C − − − →

r→∞

j2

0r!

2J2Nr−1 ln 2. where we have used[?] J0 = signal amplitude, J = noise amplitude. Shannon Heartely bound is satisfied for this encoding - interesting result !

Pavithran S Iyer Guide: Prof. V.V Sreedhar Spin Glasses and Information Processing

Outline Overview Information Theory Disordered spin systems Implications of the correspondence Questions Bibliography REM

Random Energy Model

In the r → ∞ limit[?]: R = r! Nr−1 . Channel Capacity: for a gaussian channel[?] C − − − →

r→∞

j2

0r!

2J2Nr−1 ln 2. where we have used[?] J0 = signal amplitude, J = noise amplitude. Shannon Heartely bound is satisfied for this encoding - interesting result !

Pavithran S Iyer Guide: Prof. V.V Sreedhar Spin Glasses and Information Processing

Outline Overview Information Theory Disordered spin systems Implications of the correspondence Questions Bibliography Convolution Codes

Convolution Codes

A convolution code (CC) is an ECC encoding m bit string to n-bit string with R = m n . Consider: CC with R = 1 2, encoding: x1

i = ui + ui−1,

x2

i = ui + ui−1 + ui−2, where ui - raw input and

xi - encoded symbol1. Corresponding spin glass: → Ground state - raw input given by spins Si & encoded bits using link variables given by J(1)

i,i−2 = SiSi−2, J(2) i,i−1,i−2 = SiSi−1Si−2 and

hamiltonian[?]: H = −

i

• J(1)

i,i−2SkSk−2 + J(2) i,i−1,i−2SkSk−1Sk−2

• .

Convolution Codes correspond to 1D spin glasses

1All in binary Pavithran S Iyer Guide: Prof. V.V Sreedhar Spin Glasses and Information Processing

Outline Overview Information Theory Disordered spin systems Implications of the correspondence Questions Bibliography Convolution Codes

Convolution Codes

A convolution code (CC) is an ECC encoding m bit string to n-bit string with R = m n . Consider: CC with R = 1 2, encoding: x1

i = ui + ui−1,

x2

i = ui + ui−1 + ui−2, where ui - raw input and

xi - encoded symbol1. Corresponding spin glass: → Ground state - raw input given by spins Si & encoded bits using link variables given by J(1)

i,i−2 = SiSi−2, J(2) i,i−1,i−2 = SiSi−1Si−2 and

hamiltonian[?]: H = −

i

• J(1)

i,i−2SkSk−2 + J(2) i,i−1,i−2SkSk−1Sk−2

• .

Convolution Codes correspond to 1D spin glasses

1All in binary Pavithran S Iyer Guide: Prof. V.V Sreedhar Spin Glasses and Information Processing

Outline Overview Information Theory Disordered spin systems Implications of the correspondence Questions Bibliography Convolution Codes

Convolution Codes

A convolution code (CC) is an ECC encoding m bit string to n-bit string with R = m n . Consider: CC with R = 1 2, encoding: x1

i = ui + ui−1,

x2

i = ui + ui−1 + ui−2, where ui - raw input and

xi - encoded symbol1. Corresponding spin glass: → Ground state - raw input given by spins Si & encoded bits using link variables given by J(1)

i,i−2 = SiSi−2, J(2) i,i−1,i−2 = SiSi−1Si−2 and

hamiltonian[?]: H = −

i

• J(1)

i,i−2SkSk−2 + J(2) i,i−1,i−2SkSk−1Sk−2

• .

Convolution Codes correspond to 1D spin glasses

1All in binary Pavithran S Iyer Guide: Prof. V.V Sreedhar Spin Glasses and Information Processing

Outline Overview Information Theory Disordered spin systems Implications of the correspondence Questions Bibliography Convolution Codes

Convolution Codes

A convolution code (CC) is an ECC encoding m bit string to n-bit string with R = m n . Consider: CC with R = 1 2, encoding: x1

i = ui + ui−1,

x2

i = ui + ui−1 + ui−2, where ui - raw input and

xi - encoded symbol1. Corresponding spin glass: → Ground state - raw input given by spins Si & encoded bits using link variables given by J(1)

i,i−2 = SiSi−2, J(2) i,i−1,i−2 = SiSi−1Si−2 and

hamiltonian[?]: H = −

i

• J(1)

i,i−2SkSk−2 + J(2) i,i−1,i−2SkSk−1Sk−2

• .

Convolution Codes correspond to 1D spin glasses

1All in binary Pavithran S Iyer Guide: Prof. V.V Sreedhar Spin Glasses and Information Processing

Outline Overview Information Theory Disordered spin systems Implications of the correspondence Questions Bibliography

some (open) questions

• Measuring overlap for convolution codes - error free decoding
• r ≥ 3 models - finite range spin-spin interactions - numerical results
• finite size effects

Pavithran S Iyer Guide: Prof. V.V Sreedhar Spin Glasses and Information Processing

Outline Overview Information Theory Disordered spin systems Implications of the correspondence Questions Bibliography

References I

R.B. Ash, Information theory, Dover books on mathematics, Dover Publications, 1990. Tommaso Castellani and Andrea Cavagna, Spin-glass theory for pedestrians, Journal of Statistical Mechanics: Theory and Experiment 2005 (2005), no. 05, P05012. S F Edwards and P W Anderson, Theory of spin glasses, Journal of Physics F: Metal Physics 5 (1975), no. 5, 965. Francesco Zamponi, Mean field theory of spin glasses, arxiv cond-mat: 1008.4844v1, ( 28 Aug, 2010), no. 5, 965.

• M. Mezard, G. Parisi, and M.A. Virasoro, Spin glass theory and beyond, World

Scientific lecture notes in physics, World Scientific, 1987.

Pavithran S Iyer Guide: Prof. V.V Sreedhar Spin Glasses and Information Processing

Outline Overview Information Theory Disordered spin systems Implications of the correspondence Questions Bibliography

References II

Ralf .R. Muller, The replica method, Lecture Notes for TT8107: Random Matrix Theory for Wireless Communications.

• H. Nishimori, Statistical physics of spin glasses and information processing: an

introduction, International series of monographs on physics, Oxford University Press, 2001. Giorgio Parisi, On spin glass theory, Physica Scripta 1987 (1987), no. T19A, 27. Nicolas Sourlas, Statistical mechanics and capacity-approaching error-correcting codes, Physica A: Statistical Mechanics and its Applications 302 (2001), no. 1-4, 14 – 21.

Pavithran S Iyer Guide: Prof. V.V Sreedhar Spin Glasses and Information Processing

Outline Overview Information Theory Disordered spin systems Implications of the correspondence Questions Bibliography

My sincere thanks to

• Prof. V.V. Sreedhar for helping me through the long calculations.

Nana Siddharth for taking time to help me with mathematica. ... and my friends for thier help and support.

Pavithran S Iyer Guide: Prof. V.V Sreedhar Spin Glasses and Information Processing