Towards a Theory of Information Flow -Transducers in the Finitary - - PowerPoint PPT Presentation

Information Flow Spencer Mathews ǫ-Transducers Dynamics

Single State Soup Evolution

Channel Capacity Partitioning Metamachine Conclusion

Towards a Theory of Information Flow in the Finitary Process Soup

Spencer Mathews

Department of Computer Science and Complexity Sciences Center University of California at Davis

June 1, 2010

Information Flow Spencer Mathews ǫ-Transducers Dynamics

Single State Soup Evolution

Channel Capacity Partitioning Metamachine Conclusion

Goals

Analyze model of evolutionary self-organization in terms of information flow. Measure channel capacity of elementary ǫ-Transducers. Develop functional partitioning based on this and language properties.

Information Flow Spencer Mathews ǫ-Transducers Dynamics

Single State Soup Evolution

Channel Capacity Partitioning Metamachine Conclusion

Outline

1

ǫ-Transducers

2

Dynamics Single State Soup Evolution

3

Channel Capacity

4

Partitioning

5

Metamachine

6

Conclusion

Information Flow Spencer Mathews ǫ-Transducers Dynamics

Single State Soup Evolution

Channel Capacity Partitioning Metamachine Conclusion

ǫ-machines

ǫ-machines Defined T = {S, T }

S is a set of causal states T is the set of transitions between them: T (s)

ij , s ∈ A

ǫ-machine Properties All of their recurrent states form a single strongly connected component. Transitions are deterministic. S is minimal: an ǫ-machine is the smallest causal representation of the transformation it implements.

Information Flow Spencer Mathews ǫ-Transducers Dynamics

Single State Soup Evolution

Channel Capacity Partitioning Metamachine Conclusion

Transducers

We interpret the symbols labeling the transitions in the alphabet A as consisting of two parts: an input symbol that determines which transition to take from a state and an output symbol which is emitted on taking that transition. Transducers implement functions:

Character to character Input string to output string Map sets to sets (languages)

Information Flow Spencer Mathews ǫ-Transducers Dynamics

Single State Soup Evolution

Channel Capacity Partitioning Metamachine Conclusion

Set of Single State Machines

15 0|0 0|1 1|0 1|1 14 0|1 1|0 1|1 13 0|0 1|0 1|1 12 1|0 1|1 11 0|0 0|1 1|1 10 0|1 1|1 9 0|0 1|1 8 1|1 7 0|0 0|1 1|0 6 0|1 1|0 5 0|0 1|0 4 1|0 3 0|0 0|1 2 0|1 1 0|0

Information Flow Spencer Mathews ǫ-Transducers Dynamics

Single State Soup Evolution

Channel Capacity Partitioning Metamachine Conclusion

Transducer Composition

Create new mapping, input language of one machine becomes input language of another Not commutative Possible exponential growth in number of states.

Information Flow Spencer Mathews ǫ-Transducers Dynamics

Single State Soup Evolution

Channel Capacity Partitioning Metamachine Conclusion

Interaction Network

Interaction Matrix G(k) G(k)

ij

= 1 ifTk = Tj ◦ Ti

• therwise

Information Flow Spencer Mathews ǫ-Transducers Dynamics

Single State Soup Evolution

Channel Capacity Partitioning Metamachine Conclusion

Single State Transition Matrix

                           T1 T2 T3 T0 T1 T2 T3 T0 T1 T2 T3 T0 T1 T2 T3 T0 T0 T0 T1 T1 T1 T1 T2 T2 T2 T2 T3 T3 T3 T3 T1 T2 T3 T1 T1 T3 T3 T2 T3 T2 T3 T3 T3 T3 T3 T4 T8 T12 T0 T4 T8 T12 T0 T4 T8 T12 T0 T4 T8 T12 T5 T10 T15 T0 T5 T10 T15 T0 T5 T10 T15 T0 T5 T10 T15 T4 T8 T12 T1 T5 T9 T13 T2 T6 T10 T14 T3 T7 T11 T15 T5 T10 T15 T1 T5 T11 T15 T2 T7 T10 T15 T3 T7 T11 T15 T0 T0 T0 T4 T4 T4 T4 T8 T8 T8 T8 T12 T12 T12 T12 T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 T11 T12 T13 T14 T15 T0 T0 T0 T5 T5 T5 T5 T10 T10 T10 T10 T15 T15 T15 T15 T1 T2 T3 T5 T5 T7 T7 T10 T11 T10 T11 T15 T15 T15 T15 T4 T8 T12 T4 T4 T12 T12 T8 T12 T8 T12 T12 T12 T12 T12 T5 T10 T15 T4 T5 T14 T15 T8 T13 T10 T15 T12 T13 T14 T15 T4 T8 T12 T5 T5 T13 T13 T10 T14 T10 T14 T15 T15 T15 T15 T5 T10 T15 T5 T5 T15 T15 T10 T15 T10 T15 T15 T15 T15 T15                           

Information Flow Spencer Mathews ǫ-Transducers Dynamics

Single State Soup Evolution

Channel Capacity Partitioning Metamachine Conclusion

Interaction Network

Interaction Graph G Nodes correspond to ǫ-transducers If TC = TB ◦ TA, place directed edge connecting node TA, to TC, labeled with the transforming machine TB.

Information Flow Spencer Mathews ǫ-Transducers Dynamics

Single State Soup Evolution

Channel Capacity Partitioning Metamachine Conclusion

Single State Interaction Network G

Information Flow Spencer Mathews ǫ-Transducers Dynamics

Single State Soup Evolution

Channel Capacity Partitioning Metamachine Conclusion

Population Dynamics

Population P N individuals

Information Flow Spencer Mathews ǫ-Transducers Dynamics

Single State Soup Evolution

Channel Capacity Partitioning Metamachine Conclusion

Population Dynamics

A single replication is determined through compositions and replacements in a two-step sequence:

1

Construct ǫ-machine TC by forming the composition TC = TB ◦ TA from TA and TB randomly selected from the population and minimizing.

2

Replace a randomly selected ǫ-machine, TD, with TC. Note that there is no imposed notion of fitness nor spatial component.

Information Flow Spencer Mathews ǫ-Transducers Dynamics

Single State Soup Evolution

Channel Capacity Partitioning Metamachine Conclusion

Relaxation to Steady State (Single State Soup of Size 100,000)

0.05 0.1 0.15 0.2 0.25 2 4 6 8 10 p t/N T1, T2, T4, T8 T3, T5, T10, T12 T6, T9 T7, T11, T13, T14 T15

Information Flow Spencer Mathews ǫ-Transducers Dynamics

Single State Soup Evolution

Channel Capacity Partitioning Metamachine Conclusion

Communication Channel

Discrete Channel [Cover and Thomas, 2006] (X, p(y|x), Y):

(X and Y) are finite sets p(y|x) are probability mass functions

Information Flow Spencer Mathews ǫ-Transducers Dynamics

Single State Soup Evolution

Channel Capacity Partitioning Metamachine Conclusion

Channel Capacity

Cover and Thomas [Cover and Thomas, 2006] define the channel capacity of a discrete memoryless channel as: C = max

p(x) I(X; Y )

(1) This capacity specifies the highest rate, in bits, at which information may be reliably transmitted through the channel, and Shannon’s second theorem states that his rate is achievable in practice.

Information Flow Spencer Mathews ǫ-Transducers Dynamics

Single State Soup Evolution

Channel Capacity Partitioning Metamachine Conclusion

Mutual Information

Remember that mutual information is the reduction in uncertainty in one random variable due to knowledge of another. I(X; Y ) = H(Y ) − H(Y |X) (2)

Information Flow Spencer Mathews ǫ-Transducers Dynamics

Single State Soup Evolution

Channel Capacity Partitioning Metamachine Conclusion

Discrete Noiseless Channel

Discrete Noiseless Channel One to one correspondence between input and output symbols

Information Flow Spencer Mathews ǫ-Transducers Dynamics

Single State Soup Evolution

Channel Capacity Partitioning Metamachine Conclusion

Languages

Partitioning based on language L Lin = Σ∗ Lout = Σ∗ union of these – possibility for positive channel capacity

Information Flow Spencer Mathews ǫ-Transducers Dynamics

Single State Soup Evolution

Channel Capacity Partitioning Metamachine Conclusion

Single State Channel Capacities

0.5 1 0.5 1 bits 0.5 1 0.5 1 0.5 1 0 0.5 1 P(X) 0.5 1 0 0.5 1

Information Flow Spencer Mathews ǫ-Transducers Dynamics

Single State Soup Evolution

Channel Capacity Partitioning Metamachine Conclusion

Machines of Maximal Channel Capacity

0.5 1 P(X) 0.5 1 bits

Information Flow Spencer Mathews ǫ-Transducers Dynamics

Single State Soup Evolution

Channel Capacity Partitioning Metamachine Conclusion

Partially Noisy Channels

0.5 1 P(X) 0.5 1 bits

Information Flow Spencer Mathews ǫ-Transducers Dynamics

Single State Soup Evolution

Channel Capacity Partitioning Metamachine Conclusion

Partially Noisy Channels

0.5 1 P(X) 0.5 1 bits

Information Flow Spencer Mathews ǫ-Transducers Dynamics

Single State Soup Evolution

Channel Capacity Partitioning Metamachine Conclusion

Single State Channel Capacities

Most are zero, except for... Machine Number Channel Capacity P(X=1) 6 1.0 0.5 7 0.32 0.6 9 1.0 0.5 11 0.32 0.6 13 0.32 0.4 14 0.32 0.4

Information Flow Spencer Mathews ǫ-Transducers Dynamics

Single State Soup Evolution

Channel Capacity Partitioning Metamachine Conclusion

Meta-machines

Meta-machine A set of machines that is closed and self-maintained under composition Ω ⊆ P is a meta-machine if and only if (i) Ti ◦ Tj ∈ Ω, for all Ti, Tj ∈ Ω and (ii) For all Tk ∈ Ω, there exists Ti, Tj ∈ Ω, such that Tk = Ti ◦ Tj. Captures notion of invariant set: Ω = G ◦ Ω

Information Flow Spencer Mathews ǫ-Transducers Dynamics

Single State Soup Evolution

Channel Capacity Partitioning Metamachine Conclusion

Single State Metamachine

Information Flow Spencer Mathews ǫ-Transducers Dynamics

Single State Soup Evolution

Channel Capacity Partitioning Metamachine Conclusion

Single State Metamachine

!" !# !$ !% !& !" !& !' !( !) !# !* !$ !* !) !# !% !) !( !' !& !)

!"#$%&'

!&

!"#%%&'

!#

!"#$%&'

!& !* !( !) !& !' !& !" !$ !# !* !% !) !# !$ !) !( !% !# !' !" !$

!"#"()'

!%

!"#$%&'

!(

!"#"()'

!& !* !( !) !& !' !& !" !$ !# !) !% !* !# !$ !(!) !) !( !' !" !)

!"#"()'

!"

!"#$%&'

!*

!"#"()'

[Crutchfield, 2006]

Information Flow Spencer Mathews ǫ-Transducers Dynamics

Single State Soup Evolution

Channel Capacity Partitioning Metamachine Conclusion

Summary

Channel capacity is a function of machine structure. High channel capacity does not necessarily lead to persistence. All of the transducers in the single state meta-machine have zero channel capacity. Composition never increases channel capacity.

Information Flow Spencer Mathews ǫ-Transducers Dynamics

Single State Soup Evolution

Channel Capacity Partitioning Metamachine Conclusion

For Further Reading

• T. M. Cover and J. A. Thomas

Elements of Information Theory Wiley-Interscience, 2006

• J. P. Crutchfield and O. G¨
• rnerup

Objects That Make Objects: The Population Dynamics of Structural Complexity Journal of the Royal Society Interface, 3 (2006) 345-349.