1 GA The Schema Theorem (Holland, 1975) How do genetic algorithms - - PDF document

1

Umeå University Department of Computing Science

Emergent systems

Spring-13 Evolutionary methods and game theory

http://www.cs.umu.se/kurser/5DV017

Emergent Systems, Jonny Pettersson, UmU 18/2 - 13

Last time

❒ Genetics and evolution ❒ Genetic algorithms

Emergent Systems, Jonny Pettersson, UmU 18/2 - 13

Outline for today

❒ How do GA’s work? ❒ Evolutionary computation

❍ Overview

❒ Genetic programming ❒ Aspects of evolution ❒ Classifier systems ❒ General on cooperation ❒ Game theory

2

Emergent Systems, Jonny Pettersson, UmU 18/2 - 13

GA – The Schema Theorem (Holland, 1975)

❒ How do genetic algorithm’s work? ❒ Schema

❍ “Building blocks” ❍ Bit strings of 0, 1, * ❍ E.g. H = 1 ****1

• H = “Hyper plane” (“planes” of various dimensions)
• Represents all strings that start and end with 1
• 111111 is an instance of H

Emergent Systems, Jonny Pettersson, UmU 18/2 - 13

GA – The Schema Theorem (2)

❒ There are 2n possible bit strings of length n,

and thus 2^2n possible subsets of strings, but there are only 3n possible schemas

❍ Every possible subset of n –bit strings can not be

describe as a schema ❒ Any given bit string of length n is an instance

• f 2n different schemas

❍ Each population of n strings contains instances of

between 2n and n x 2n different schemas

❍ That is, in each generation, while the GA is

explicitly evaluating the fitness of the n strings, it is actually implicitly estimating the average fitness

• f a much larger number of schemas

Emergent Systems, Jonny Pettersson, UmU 18/2 - 13

GA – The Schema Theorem (3)

❒ The formulas…

❍ See the excerpt

❒ The schema theorem describes the growth

• f a schema from one generation to the

next

❒ Short, low-order schemas is favored ❒ Implicit parallelism

❍ Many schemas is simultaneously implicitly

evaluated ❒ Mutation prevents the loss of diversity at

a given bit position

3

Emergent Systems, Jonny Pettersson, UmU 18/2 - 13

Evolutionary Computation - History

❒ AI and ALife

❍ Alan Turing, John von Neumann, Norbert Wiener ❍ Self-replicate and adaptivity

❒ Evolutionary programming

❍ Fogel, Owens, and Walsh (1966) ❍ Differs from genetic algorithms in three ways:

• Representation: not constrained to be a string
• No crossover
• Different form of mutation, and typically reduced rate of

mutation during a run

❒ Evolution strategies

❍ Rechenberg (1965,1973), Schwefel (1975,1977) ❍ Independently developed ❍ Slightly different way of selection and mutation

compared to EP

❍ Recombination is possible Emergent Systems, Jonny Pettersson, UmU 18/2 - 13

Evolutionary Computation - History

❒ Genetic algorithms

❍ John Holland (1960s)

❒ Classifier Systems

❍ John Holland (1976 ?) ❍ A cross between a Post production system, a genetic

algorithm, and a market economy

❍ A hybrid nature: Both evolution and learning

❒ Genetic programming

❍ John Koza (1992) ❍ Evolving of whole programs ❍ Resembles GA, but program fragments are used instead

• f strings

❍ LISP Emergent Systems, Jonny Pettersson, UmU 18/2 - 13

The No Free Lunch Theorem

❒ ”The NFL theorem states that over all possible

search spaces, all methods perform equally well, including the simple technique of randomly guessing.” – Flake

❒ No single method of optimization is best for all

applications

❒ Evolutionary algorithms performs relatively well

when:

❍ there is a large number of parameters to be determined ❍ the surface of solutions is complex, having many

intermediate optima

4

Emergent Systems, Jonny Pettersson, UmU 18/2 - 13

Genetic Programming

❒ An attempt to deal with one of the central

questions in computer science (posed by Arthur Samuel in 1959), namely

❍ How can computers learn to solve problems without being

explicitly programmed? In other words, how can computers be made to do what needs to be done, without being told exactly how to do it? ❒ Any computer program can be graphically depicted

as an rooted point-labeled tree with ordered branches

❒ The search space in genetic programming is the

space of all possible computer programs composed

• f functions and terminals appropriate to the

problem domain

Emergent Systems, Jonny Pettersson, UmU 18/2 - 13

Genetic Programming - Steps

❒ In applying genetic programming to a

problem, there are five major preparing steps:

❍ The set of terminals ❍ The set of primitive functions ❍ The fitness measure ❍ The parameters for controlling the run ❍ The method for designating a result and the

criterion for terminating a run ❒ Start with an initial population of randomly

generated computer programs

Emergent Systems, Jonny Pettersson, UmU 18/2 - 13

Genetic Programming - Example

❒ Koza, Rice, and Roughgarden (1992) ❒ Foraging strategies of Anolis lizards ❒ Questions:

❍ ”What makes for an optimal foraging strategy?” ❍ ”How can an evolutionary process assemble

strategies that require complex calculations from simple components?”

5

Emergent Systems, Jonny Pettersson, UmU 18/2 - 13

Genetic Programming - Example

❒ Four variables:

❍ The abundance a of insects ❍ The sprint velocity v of the lizard ❍ The coordinate x, y of the insect in the lizard’s

view ❒ A strategy is a function of these variables

that returns 1 or -1

❒ The goal: A function that maximizes food

capture per unit time

Emergent Systems, Jonny Pettersson, UmU 18/2 - 13

Genetic Programming - Example

❒ 10 x 20 meter viewing area (fig 1a)

❍ Region 1: Insects always escape ❍ Region 2: Insects never escape ❍ Region 3: Insects escape with probability zero

• n the x axis and linearly increasing with the

angle to a maximum of 0.5 on the y axis ❒ Result, the best individual at generation

❍ 0 (fig 1b) ❍ 12 (fig 1c) ❍ 46 (fig 1d)

Emergent Systems, Jonny Pettersson, UmU 18/2 - 13

Genetic Algoritms – Example: Coevolution

❒ Hillis (1990) ❒ Host-parasite coevolution ❒ Adaptation in a static environment results

in

❍ loss of diversity ❍ overfit solutions

❒ Problem:

❍ Evolving minimal sorting networks for sorting

lists with a fixed number n of elements

❍ Ex: (3,8), (14,8), (4,9), ... ❍ With n = 16, best known solution is 60

comparisons

6

Emergent Systems, Jonny Pettersson, UmU 18/2 - 13

GA – Example: Coevolution

❒ Hillis used a GA ❒ Two criteria for networks in the population

❍ Small size, implicitly favored through the

encoding

❍ Correctness, explicitly through the fitness

function ❒ The fitness of a network, equal to the

percentage of correctly sorted cases

❒ Spatial implementation, each individual

were placed on a two-dimensional lattice

❒ Result (with static environment):

❍ The GA got stuck on local optima ❍ 65 comparisons

Emergent Systems, Jonny Pettersson, UmU 18/2 - 13

GA – Example: Coevolution

❒ Reason:

❍ After a while the test cases were not

challenging enough ❒ Solution:

❍ Let the test cases evolve ❍ The network’s fitness was the percentage of

test cases in the parasite that it sorted correctly

❍ The fitness of the parasite was the percentage

• f its test cases that the network sorted

incorrectly ❒ New result:

❍ 61 comparisons

Emergent Systems, Jonny Pettersson, UmU 18/2 - 13

The Blind Watchmaker

❒ 40% of all Americans (25% of college-

educated Americans) do not believe in Darwinian evolution (M. Mitchell, 1999)

❒ Only 40% of all Americans accept the

evolution theory (Aftonbladet, 090201)

❒ Richard Dawkins (1996) ❒ ”Biomorphs”

❍ A way to teach how evolution works

❒ Variants

❍ SimLife ❍ Creatures

7

Emergent Systems, Jonny Pettersson, UmU 18/2 - 13

Lamarckian Evolution

❒ ”... the evolution of traits that are

modified through experience and passed

• n, in their modified form, to the genotype
• f the next generation” – M. Mitchell

❒ Not possible in natural systems ❒ But artificial systems can use it

❍ Needs a mean for adapting within a generation ❍ and a way of passing new gains to the genotype

• f the next generation

Emergent Systems, Jonny Pettersson, UmU 18/2 - 13

Lamarckian Evolution

❒ Often more effective than Darwin

evolution in static environments

❍ Each individual can try out many possibilities in

each generation ❒ But, not so effective when the environment

is dynamic

Emergent Systems, Jonny Pettersson, UmU 18/2 - 13

Classifier Systems

❒ Adaptation

❍ Learning – in the lifetime of the agent ❍ Evolution – across generations

❒ What about adaptation in systems between

learning and evolution

❍ Culture ❍ Social ❍ Economic

❒ Classifier systems combine

❍ Genetic algorithms ❍ Environmental feedback ❍ Simple reinforcement learning

8

Emergent Systems, Jonny Pettersson, UmU 18/2 - 13

Feedback and Control

❒ Visible features usually correspond to a subset of

environment

❒ Reinforcement

❍ What differs adaptive systems from non-adaptive

❒ Delayed rewards and punishments ❒ How does one find the optimal controller?

Emergent Systems, Jonny Pettersson, UmU 18/2 - 13

Classifier Systems

❒ Rules

❍ if condition then action

❒ Classifier systems

❍ Are mostly used to control-like problems ❍ Almost never ”programmed”

Emergent Systems, Jonny Pettersson, UmU 18/2 - 13

Classifier Systems

❒ A classifier system consist of

❍ List of classifiers

• condition : message : strength
• Ex: 1#0#:1001:37

❍ List of messages

• Messages describe the ”current” environment
• Temporary storage space
• Actions to take

❍ Detectors

• Sensory organs, post on the message list

❍ Effectors

• Can be used to modify the environment

9

Emergent Systems, Jonny Pettersson, UmU 18/2 - 13

Classifier Systems

Emergent Systems, Jonny Pettersson, UmU 18/2 - 13

Classifier Systems

1.

The detectors place messages on the message list

2.

A match set is formed from all suitable classifiers

3.

The classifiers bid against each other. A function

• f strength and maybe specificity. An action set

is formed from the highest bidders

4.

The classifiers in the action set pay a portion of their bids to the other classifiers (if any) that were responsible for posting the message that matched their condition. The paid classifiers have their strengths increased as a result

Emergent Systems, Jonny Pettersson, UmU 18/2 - 13

Classifier Systems

• 5. The message list is erased and a new message list

is formed from the message portions of all the classifiers in the action set

• 6. If any of the new messages in the message list

correspond to a real action, then the effectors process the action appropriately

• 7. If the environment rewards the classifier

system, then the reward is divided among the classifiers in the action set, which increases the strengths of the winning classifiers

10

Emergent Systems, Jonny Pettersson, UmU 18/2 - 13

Classifier Systems

❒ The ”bucket brigade” algorithm

❍ Payments are passed down a line of classifiers,

reinforcing all in the chain

❍ Represent the basis of a long-term memory

❒ Genetic algorithms

❍ Initially randomly selected classifiers ❍ After a while some classifiers will be strong ❍ A GA weed out the weak classifiers and form new ones

from the stronger classifiers ❒ Thus,

❍ GA remove bad classifiers and introduces new,

potentially good classifiers

❍ The ”bucket brigade” algorithm strengthening the good

• nes

Emergent Systems, Jonny Pettersson, UmU 18/2 - 13

General on cooperation

❒ Example

❍ Slime molds ❍ Birds and fish that polish ❍ Symbiosis - Lichens: symbiotic relationship

between algae and bacteria

❍ Bacteria that assist in digestion in humans ❍ Society

❒ Why does it work? ❒ What happens if someone is cheating? ❒ Game theory is an attempt to provide

answers

Emergent Systems, Jonny Pettersson, UmU 18/2 - 13

Game theory

❒ It is about finding the best possible

strategy for a particular game

❒ History

❍ 1928: John von Neumann

• Optimal strategy for a 2-person zero-sum game

❍ 1944: von Neumann and Oskar Morganstern

• Theory of Games and Economic Behavior
• Morganstern – a famous mathematical economist

❍ 1950: John Nash

• Non-cooperative games, PhD thesis, 27 pages
• Genius, Nobel Prize 1994, schizophrenic

11

Emergent Systems, Jonny Pettersson, UmU 18/2 - 13

Classification of games

❒ Chance games

❍ The outcome depends not on the players'

actions

❍ ”Uninteresting", probability theory

❒ Strategy games

❍ The players' actions are important ❍ Example: Poker

Emergent Systems, Jonny Pettersson, UmU 18/2 - 13

Classification of strategy games

❒ Number of players

❍ 1, 2, 3, ..., n

❒ Zero-sum or not

❍ Zero-sum – what one wins the other lose ❍ Nonzero-sum – positive, negative, constant sum, non-

constant sum ❒ Essential or not

❍ Essential – an advantage of forming coalitions

❒ Perfect or imperfect information

❍ Perfect - all with full info on all previous actions ❍ Imperfect - some or all have partial information Emergent Systems, Jonny Pettersson, UmU 18/2 - 13

Strategies

❒ A complete sequence of actions for a

player

❒ Pure strategy

❍ The sequence is completely deterministic

❒ Mixed strategy

❍ Actions can be done with some probability

12

Emergent Systems, Jonny Pettersson, UmU 18/2 - 13

Zero-sum games

❒ Maximin

❍ Choose the strategy that maximizes the

minimum payoff ❒ Example

❍ Two competing mineral water manufacturers

• On the board

❍ Matching Pennies

• On the board

Emergent Systems, Jonny Pettersson, UmU 18/2 - 13 Emergent Systems, Jonny Pettersson, UmU 18/2 - 13

13

Emergent Systems, Jonny Pettersson, UmU 18/2 - 13

Zero-sum games

❒ Example

❍ Matching Pennies

• von Neumann: Each 2-person zero-sum game has a

maximin solution if we allow mixed strategies

Emergent Systems, Jonny Pettersson, UmU 18/2 - 13

Non-constant sum games

❒ There is no general definition of rationality

in non-constant sum game (nonzero-sum games)

❒ Two common criteria

❍ Dominant strategy equilibrium ❍ Nash equilibrium

Emergent Systems, Jonny Pettersson, UmU 18/2 - 13

Dominant strategy equilibrium

❒ Study each of your opponents' strategies,

and determine your best strategy for each case

❒ If the same strategy is the best in all

situations, it is a dominant strategy

❒ Equilibrium exists when both players have a

dominant strategy they use

❒ Example

❍ On the board

14

Emergent Systems, Jonny Pettersson, UmU 18/2 - 13

Nash equilibrium

❒ A set of strategies with the property: No

player can do better with a different strategy when the other retains its strategy

❒ For mixed strategies one look at the

expected payoff

❒ Example

❍ On the board

Emergent Systems, Jonny Pettersson, UmU 18/2 - 13

Summary

❒ How do GA’s work? ❒ Evolutionary computation

❍ Overview

❒ Genetic programming ❒ Aspects of evolution ❒ Classifier systems ❒ General on cooperation ❒ Game theory

Emergent Systems, Jonny Pettersson, UmU 18/2 - 13

Next time

❒ Prisoners’ Dilemma and other dilemmas

❍ Iterated Prisoners’ Dilemma ❍ Ecological models ❍ Spatial models

❒ Short conclusion of the course