Evolving Neural Networks Keith L. Downing The Norwegian University - - PowerPoint PPT Presentation

Evolving Neural Networks

Keith L. Downing

The Norwegian University of Science and Technology (NTNU) Trondheim, Norway keithd@idi.ntnu.no

March 11, 2014

Keith L. Downing Evolving Neural Networks

A Brief History of Artificial Intelligence

Early (1955 - 1980) focus (and success) on tasks that humans find difficult: chess, geometry, physics... Later (1985- 2010) focus on easy human tasks, which are hard for computers. In the 1980’s, it became clear that computers lack common sense, and it’s not easy to give it to them in the same way that we give them high-level, expert knowledge of a specific domain. In the 1990’s, Situated and Embodied AI (SEAI) recognized as a promising low road to intelligence. Computers will only acquire common sense about the world by experiencing it and having to survive in it.

Keith L. Downing Evolving Neural Networks

Cognitive Incrementalism

Llinas (pg. 35) ...that which we call thinking is the evolutionary internalization of movement.. Mindware (pg. 135), Andy Clark, 2001 This is the idea that you do indeed get full-blown human cognition by gradually adding bells and whistles to basic (embodied and embedded) strategies of relating to the present at hand.

Could sensorimotor control be the basis of common sense? Is it the key to Artificial General Intelligence (AGI)?

Keith L. Downing Evolving Neural Networks

Robot (Hans Moravec)

Comparative Evolution Living Organisms Computers Sense & Act 10,000,000 years 25 years Reason 100,000 years 40 years Calculate 1,000 years 60 years Neural circuitry for cognition reuses, extends and is constrained by sensorimotor circuitry. Should AGI progress and be restricted in similar ways?

Keith L. Downing Evolving Neural Networks

Artificial Neural Networks (ANNs)

Utility for AGI via Cognitive Incrementalism Simple, homogeneous substrate Same, basic, neural signals carry information of perceptual, cognitive and motor nature - - no need for special representations for each aspect of intelligence. Relatively unbiased. Adapt to represent the salient aspects

• f a situation.

Built for learning.

Keith L. Downing Evolving Neural Networks

Training Artificial Neural Networks

Encoder Decoder E = r3 - r* r* r3 d3 Training/Test Cases: {(d1, r1) (d2, r2) (d3, r3)....} dE/dW Training Test Cases

Neural Net

N times, with learning 1 time, without learning

Keith L. Downing Evolving Neural Networks

Backpropagation

Advantages

Powerful tool for learning complex input-output mappings in diverse problem domains. Relatively simple algorithm with solid mathematical foundation.

Drawbacks

Requires a known, correct output for each input → impractical for training autonomous systems. Requires many training rounds, often hundreds or thousands. Can easily get stuck in local error minima during gradient descent. Recurrent networks are a problem. Biologically unrealistic

Keith L. Downing Evolving Neural Networks

Evolving Artificial Neural Networks (EANNs)

Encoder Decoder Test Cases

Neural Net

1 time, without learning Total Error Fitness

0 0 1 0 0 1 0 0 0 1 0 0 0 1 1 1 1 0 0 0 1 1 0 0 0 Genome (Direct Encoding)

Keith L. Downing Evolving Neural Networks

Three Levels of Adaptation

POE Systems

1

Phylogenetic or Evolutionary - Characterized by the use of an EA and thus having clearly definable genotypic and phenotypic levels, genetic operators, fitness functions, etc.

2

Ontogenetic or Developmental - Involving a non-trivial genotype-to-phenotype translation. In most cases, the genotype is a recipe that, through some recursive growth process, produces the phenotype.

3

Epigenetic or Learning - During actual performance testing, the system is able to modify itself in some manner that effects future behavior. A.k.a. TRIDAP (3-way adaptive)

Keith L. Downing Evolving Neural Networks

Genotypic Encodings

Direct - Position (in chromosome) and bits determine a phenotypic trait, independent of all other genes. Indirect Bijective - Genes may interact in determining traits. Chromosomal position and/or bits may only be relative indicators. Indirect Generative - Genes encode parameters for development.

1110001101011011.....

Direct Encoding Schedule the 3rd exam for the 10th time slot.

....1110001101011011.....

Indirect (Uncompressed) Encoding Schedule the next unscheduled exam for the 10th of the unfilled time slots.

111010100011

Generative (Developmental) Encoding 7 5 12 Exam 1 => Slot 4 Exam 2 => Slot 8

.........

Keith L. Downing Evolving Neural Networks

Two Early Direct Encodings

0 0 1 0 0 0 0 0 0 0 0 1 0 0 1 1 1 0 0 0 0 0 1 1 0 To From 1 2 3 4 5 1 2 3 4 5 0 0 1 0 0 0 0 0 0 0 0 1 0 0 1 1 1 0 0 0 0 0 1 1 0 Genotype Phenotype 1 2 3 4 5 ? ? Weights learned by back-propagation Connection Table Miller et. al. (1989) ? ? ? Recurrent connections ignored ? Montana & Davis (1989) +.45 -.32 +.89 +.55 -.07 +.61 +.89

• .32

+.45 +.55 Genotype Phenotype Net assumed to be fully connected

• .07

1 2 3 4 5 +.61

Keith L. Downing Evolving Neural Networks

The Competing Conventions Problem

A B X Y C A B X Y C

• 111 1-11

1-11 -111 Crossover A B X Y C A B X Y C XOR(X,Y) XOR(X,Y) Y ∧¬X X ∧¬Y

Genotypes

1-11 1-11

• 111 -111

Many-to-1 genotype → phenotype mapping can hinder evolution.

Keith L. Downing Evolving Neural Networks

Evolving Individual Neurons: the SANE Approach

3 2 1 A B C 1 0.2

• 0.6

0.5 1.2

• 1.0

0.4 8 0.2 311 -1.0 150 0.4 Index Weight 2 0.5 33 -0.6 130 1.2 Genome for Node A Genome for Node C Index < 127 => Input Else => Output

Genotype => Phenotype Mapping

Moriarty & Mikkulainen (1997). Still a direct representation.

Keith L. Downing Evolving Neural Networks

Cooperative Coevolution of Neurons in SANE

Generation K Neurons Create Networks Assign fitness to neurons Evaluate Networks Selection, mutation & recombination

• f neurons

Generation K+1 Neurons

Neuron fitness is based on the ability to combine well (i.e. cooperate with) other neurons in forming a good neural network Circumvents competing conventions by never linking neurons together

• n a chromosome; it just allows good combinations to form dynamically

during fitness testing.

Keith L. Downing Evolving Neural Networks

Complexification

Complexity

Viruses Bacteria Insects Reptiles Amphibians Birds Fishes Mammals Primates Humans ???

Evolution does not necessarily favor increased complexity. Evolution searches all over the complexity spectrum, but there seem to be clear LOWER limits of complexity. Evolution found those early but continues to stretch the upper limits. Full House, Stephen Jay Gould (1996). In EANNs, it’s hard to begin with large, complex genomes; all are unfit. Can we allow genomes to gradually complexify? This entails dynamic and variable chromosome sizes.

Keith L. Downing Evolving Neural Networks

Duplication and Differentiation

A B C F G H Genotype Phenotype A B C F G H A F

Duplication

A B C F G H A' U

Differentiation Useless

A B C F G H A* X

Useful NEW Function Further Differentiation

A low-risk route to complexification, since key functionalities (e.g. F) are still present during the exploratory period when variants of A arise and their phenotypic consequences are tested.

Keith L. Downing Evolving Neural Networks

Neural Complexification via Modularity, Duplication & Differentiation

!"#$%& '&()*+,+-

Hox Genes: a conserved modular component Evolving Brains (J. Allman, 1999) Evolution by Gene Duplication (S. Ohno, 1970)

Keith L. Downing Evolving Neural Networks

Vertebrate Brain Archetype

!"#$%%& '"(")"%%$* +,-.&%/'0(# 1"23$* 4-".2",5&%0. 6"3-.& 7&%%-$* 8%9&230(: ;$%)

Principles of Brain Evolution (G. Streidter, 2005)

Keith L. Downing Evolving Neural Networks

Problems with Complexification and Dynamic, Variable Genome Size

A A* B B C C* A

X

A A* B B C C* A A A* B B C C* A

X

A A* B B C C* A A A* B B C C* A

X

A A* B B C C* A Parents Children

Missing genes and copies of same or similar (i.e. from same ancestor) genes. Partially remedied by history-based alignment.

Keith L. Downing Evolving Neural Networks

Neurevolution of Augmenting Topologies (NEAT)

2 Input Nodes Connections 1 => 5 W: 0.3 1 Input 5 Hidden 3 Output 4 Hidden Genotype 2 => 4 W: 0.7 1 => 3 W: 0.5 5 => 3 W: -0.6 4 => 3 W: -0.1 2 => 5 W: 0.9 1 3 2 5 4 4 => 5 W: 0.2 0.3 0.7 0.9 0.2

• 0.6
• 0.1

0.5 Phenotype Input Hidden Output

Stanley and Miikkulainen (2002) Historical tags + speciation allow gradual complexification. Classic version restricted to one hidden layer but many connection schemes Extremely popular (direct encoding) approach to EANNs. Basis for NERO war game (Stanley et. al., 2005).

Keith L. Downing Evolving Neural Networks

Cartesian Genetic Programming (Miller, 2000, 2011)

i w s i w s

F1 C1 F2 C2 Fn Cn

O1 Ok

2 0.3 1 4 0.7 1 1 1 -0.4 1 3 0.9 1 2 1

6

5 0.6 1 4 1.0 1

1 3 2 4 6 5 0.3

0.7

• 0.4

0.9 1.0 0.6 Output Input

Beats SANE, ESP and NEAT on several benchmarks. (Khan et. al., 2013)

Keith L. Downing Evolving Neural Networks

Standard EANNs: Pros & Cons

Advantages

No training (learning) needed. Works with or without explicit test cases and explicit target outputs → useful in supervised and unsupervised learning scenarios. For fitness assessment, total error is easily replaced with other performance measures. Recurrent networks are no additional work. Better at avoiding local error minima due to parallel nature of evolutionary search.

Drawbacks

Requires a whole population of weight vectors. Scales poorly: large networks → large genotype weight vectors → large search space. General problem with direct-encoded EAs. No more biologically realistic than backpropagation, since animal genomes do not encode all synaptic strengths.

Keith L. Downing Evolving Neural Networks

Developmental Representations

Advantages Scale well: Large phenotypes generated from compact genotypes. More biologically realistic Facilitate evolution of repetitive structure. Can support gradual evolution of complexity. Disadvantages Low heritability - easy to disrupt via genetic operators. May overconstrain search Difficulty finding needle in a haystack optimal solutions.

Keith L. Downing Evolving Neural Networks

Evolving Tables

Direct Encoding Developmental Encoding

• G. Hornby & J. Pollack (2001)

Keith L. Downing Evolving Neural Networks

A Classic Early Developmental Encoding

S AB CD A aa bp B pb ba Genotype Fixed Auxiliary Rules a 00 00 b 00 01 p 11 11 Development S AB CD aapb bpba ppbb abpb Phenotype 00001100 00001101 00110000 01110100 11110000 11110101 00001100 00011101 Connection Table 1 2 3 5 ? ? ? ? Weights learned by back-propagation 6 7 4 8

Kitano’s (1990) encoding of ANNs as context-free grammars. The first complete POE system

Keith L. Downing Evolving Neural Networks

Cellular Encoding (Gruau, 1993)

Ins Outs Embryo ANN

S P S P P E A E E E

Read Head GP Tree

E

Ins Outs

S P S P P E A E E E

1

E

Ins Outs

S P S P P E A E E E

1 2

E Keith L. Downing Evolving Neural Networks

Ins Outs

S P S P P E A E E E

1 2

E

3 Ins Outs

S P S P P E A E E E

1 2

E

3 4

Keith L. Downing Evolving Neural Networks

Ins Outs

S P S P P E A E E E

1 2

E

3 4 Ins Outs

S P S P P E A E E E

1 2

E

3 4 5

Keith L. Downing Evolving Neural Networks

Ins Outs

S P S P P E A E E E

1 2

E

3 4 5 Ins Outs

S P S P P E A E E E

1 2

E

3 4 5

Keith L. Downing Evolving Neural Networks

POE in an Alife Setting (Yeager, 1994)

Red, Green, Blue Sensors Run Eat Attack Actions Size, Strength, Max Speed, Tag, Mutation Rate, Lifespan Genotype Body # Internal Neuron Groups, Neuron Group Sizes, In each group: # Excitatory/Inhibitory Neurons, Connection Density, Learning Rate Connection Density in each Group Brain Phenotype

Keith L. Downing Evolving Neural Networks

Polyworld in 1994

Hebbian Learning Agent Resource The PolyWorld Environment

Keith L. Downing Evolving Neural Networks

Polyworld in 2011

Keith L. Downing Evolving Neural Networks

G2L (Boers, 2011)

S(n,h) → A(n,h)B(h)C S(8,2) A(n,h): n > 0 → G(h,n)A(n-h,h) ⇒ A(8,2)B(2)C A(n,h): n ≤ 0 → / ⇒ G(2,8)A(6,2)b2B(1)c G(c,n) → anG(c-1,n) ⇒a8G(1,8)G(2,6)A(4,2)b2b1B(0)c G(0,n) → / ⇒ a8a8G(0,8)a6G(1,6)G(2,4)A(2,2)b2b1c B(c) → bcB(c-1) ⇒ a8a8a6a6G(0,6)a4G(1,4)G(2,2)A(0,2)b2b1c B(0) → / ⇒ a8a8a6a6a4a4G(0,4)a2G(1,2)b2b1c C → c ⇒ a8a8a6a6a4a4a2a2G(0,2)b2b1c ⇒ a8a8a6a6a4a4a2a2b2b1c

The CFG produces a direct encoding, which produces the ANN topology.

Keith L. Downing Evolving Neural Networks

The Phenotype ANN

a a a a a a a a b b c

Keith L. Downing Evolving Neural Networks

Weighted Function Graphs

X Y K 0.7

• 1.3

0.9 2.3

• 1.2
• 0.5

0.2 3.1

• 1.7

Keith L. Downing Evolving Neural Networks

Compositional Pattern-Producing Network (CPPN)

• K. Stanley (2006, 2007) - CPPNs
• J. Secretan, K. Stanley, et. al. - PicBreeder (picbreeder.org)

Keith L. Downing Evolving Neural Networks

Developmental Encoding of ANN Weights

0 0 1 0 0 1 0 0 0 1 0 0 0 1 1 Genome 3 2 1 2 1 1 2 3 sine 3 2 1 3 abs gauss 0.25 0.6 1.3 0.67 CPPN 1.2

HYPERNEAT (J. Gauci & K. Stanley, 2007, 2010)

Keith L. Downing Evolving Neural Networks

Evolving Artificial Genetic Regulatory Networks

0 0 1 0 0 1 0 0 0 1 0 0 0 1 1 1 1 0 0 0 1 1 0 0 0 Structural Unit Regulatory Region Regulatory Unit Regulatory Region Cell GRN Receptors Signals Keith L. Downing Evolving Neural Networks

Neural Networks from GRNs

N2 N3 N4 N1

?? ??

Affinity Match? N5

P . Eggenberger (1997, 2003, 2004)

Keith L. Downing Evolving Neural Networks

A Diversity of GRN Genotypes

S Ra Pa S Rb Pb S Rc Pc Gene a Gene b Gene c S Ra,1 Pa,1 S E Pa,2 Rb,1 Rb,2 E Pb,1 Gene a Gene b Rc,1

Mattiussi and Floreano (2007) Eggenberger (1997)

S Pa S T-Pa Pb S Pc T-Pa ? ?

Regulate

T-Pc ? Gene b Gene a Gene c +Ra,1 Pa,1

• Ra,2

Pa,2

• Rb,1

+ Rb,2 Pb,1 Pc,1 Pc,2 Pc,3 +Rc,1 Gene a Gene b Gene c

Reil (1999) Reisinger and Mikkulainen (2007)

Keith L. Downing Evolving Neural Networks

Josh Bongard (2001)

Coevolution

Morphology - GRNs Neural Network - Cellular Encoding (F . Gruau, 1994)

(Loading Bongard’s Pusher)

Keith L. Downing Evolving Neural Networks

Neuromeres & Vertebrate Brain Archetypes

Midbrain Hindbrain Forebrain Eye stalk Spinal Cord !"#$%& '#(" )"*+$,-#&%$ .$/&,0"$%+,/* 1/*"23).14 !"#$%&'(%)*+,-./0& 5%+"$26,7$%+/$8 +$%9"&2+/2/#+"$ &%:"$8

On the development of neuromeres to migration areas in the vertebrate cerebral tube (H. Bergquist & B. Kallen, 1953)

Keith L. Downing Evolving Neural Networks

Neural Darwinism: Exploratory Growth in Neural Devp

RNA Survival of the Best Networkers Finger 1 Finger 2 Finger 1

Gerald Edelman (1987, 1994).

Keith L. Downing Evolving Neural Networks

Displacement Theory

D E A C B Neuron Groups S1 S2 M1 M2 Sensory Inputs Motor Outputs Displacement D E A C B S1 S2 M1 M2

The Symbolic Species (Terrence Deacon, 1997)

Keith L. Downing Evolving Neural Networks

DEACANN (K. Downing, 2007)

Evolving, Developing Artificial Neural Networks based on The Neuromeric Model and Displacement Theory. Evolution: Genetic Algorithm with non-positional genes. Development:

1

Translation

2

Displacement

3

Instantiation

Keith L. Downing Evolving Neural Networks

Development in DEACANN

!"""!"!!!!"!"""""!!"!!"!!!"!""!!"!" #$%&'(%)*+& ,*'-(%./0/&) 1&')%&)*%)*+&

Keith L. Downing Evolving Neural Networks

Neural Layer Properties

!"#$ %&'( )*$+,-.* %&'( !"#$ %&'( /#0& %&'( !"#$ /1&,-$2 )*$+,-.* /1&,-$2 3*4,#0#+45&.#, /*$. 3*4,#0#+45&.#, !66*7.*+

8*&,$-$2 9:45*9;9:&.*

<"6-.&.#,=9#, >$1-?-.#,= @-0*9A#$'.&$. )*B*5#70*$.&5 9C,#D.198>0-.

Keith L. Downing Evolving Neural Networks

Evolving Starfish Neural Controllers

20 40 60 80 100 2 4 6 8 10 12 14 16 18 20 22 Generation Fitness Max Avg StDev ! " # $ %

Poor inheritance of functionality (even with cloning) due to stochastic nature

• f neural development and synaptic tuning.

Keith L. Downing Evolving Neural Networks

More Evolved Starfish Controllers

Keith L. Downing Evolving Neural Networks

Scripted Evolving ANNs (SEVANN)

ANN Generator Proxy ANN Script

Visualizations

• f ANN

Behavior

Evolutionary Algorithm Core

Time Activation EVANN Script Generation Fitness

Fitness Testing

? ? ? Local Vars

Keith L. Downing Evolving Neural Networks

Core Neural Network Structure

Link ANN Link Layer Arc Node Module

Module

Layer

Keith L. Downing Evolving Neural Networks

Simple ANN Script

Layers input transfer 20 :input? t *nodes* simple klay kohonen 15 :max-num-nodes 20 :feeder-thresh 0.01 :win-thresh 0.8 *nodes* simple :act-func simple-linear Modules koho kohonen input klay :feeder-link koholink Links koholink input klay devp 0.0 1.0 random :seed 0.25 :rate 0.2 :iter 0.7 :devp-steps 5 *learn* kohonen 0.65 :normalize t :drad -0.25 :start-rad 3.0 Execution (1 klay)

Keith L. Downing Evolving Neural Networks

ANN Script with Evolvable Parameters

Variables max-init-wgt g0-1 dsteps g1-20 kl-rate g0-1 drad g0-1-neg Layers input transfer 20 :input? t *nodes* simple klay kohonen ?g5-20 :max-num-nodes 20 :feeder-thresh 0.01 :win-thresh 0.8 *nodes* simple :act-func simple-linear Modules koho kohonen input klay :feeder-link koholink Links koholink input klay devp 0 ?max-init-wgt random :seed ?g0-1 :rate ?g0-1 :iter ?g0-1 :devp-steps ?dsteps *learn* kohonen ?kl-rate :normalize t :drad ?drad :start-rad 3.0 Execution (1 klay)

Keith L. Downing Evolving Neural Networks

Script Completion via Evolution

EVANN Script Variables X Y

g1 g2 g3 X Y g4

Layers

X

Links

g1 g2 g3 g4 g4 4 6 12 3 7

Gene Map

1 1 6 2 8

Genotypes

12 3

Layers

7

Links

4 4 6

Proxy ANN Script

g4

Keith L. Downing Evolving Neural Networks

Developmental Waves using WFGs

X Y Time

t1 t2 t3

Keith L. Downing Evolving Neural Networks

Weight Tuning via Developmental Waves

t1 t2 t3 t1 t2 t3 dW = 0.2 dW = 0.1 dW = 0.005 Layer J Layer K dW = 0.4

Keith L. Downing Evolving Neural Networks

Modularity, Duplication & Differentiation in SEVANN

Layer A Layer B Layer C

Link A => B Link B => C Link C => A Module B + C

Layer A Layer B Layer C

Link A => B Link B => C Link C => A Module B + C

Layer B2

Link A => B2 Link B2 => C Module B2 + C

Duplication Layer A Layer B Layer C

Link A => B Link B => C Link C => A Module B + C

Layer D

Link A => D Link D => C Module D + C

Differentiation

Duplication → a macro-mutation to the script of one individual, whose accompanying bit-string chromosome must be expanded. It becomes a new reproductively-isolated species.

Keith L. Downing Evolving Neural Networks

The Spectrum of Evolutionary ANNs

Problem-Solving Success Direct Indirect Developmental Genotype Encoding Phenotypic Adaptivity SANE (P) NEAT (P) P - Phylogenetic O - Ontogenetic E - Epigenetic Kitano (POE) DEACANN (POE) SEVANN (POE) HyperNEAT (PO) Montana & Davis (P) Miller (PE) Bongard (PO) PolyWorld (PE) Adaptive HyperNeat (POE) Static Learning

Keith L. Downing Evolving Neural Networks