decision-making strategies Peter Kvam Center for Adaptive - - PowerPoint PPT Presentation

decision making strategies
SMART_READER_LITE
LIVE PREVIEW

decision-making strategies Peter Kvam Center for Adaptive - - PowerPoint PPT Presentation

Computational evolution of decision-making strategies Peter Kvam Center for Adaptive Rationality, Max Planck Institute for Human Development Department of Psychology, Michigan State University Joseph Cesario Department of Psychology, Michigan


slide-1
SLIDE 1

Computational evolution of decision-making strategies

Peter Kvam

Center for Adaptive Rationality, Max Planck Institute for Human Development Department of Psychology, Michigan State University

Joseph Cesario

Department of Psychology, Michigan State University

Jory Schossau

Department of Computer Science and Engineering, Michigan State University

Heather Eisthen

Department of Integrative Biology, Michigan State University

Arend Hintze

Department of Integrative Biology, Michigan State University

slide-2
SLIDE 2

Introduction

  • We often take a strategy-first approach when

constructing decision-making models

– Assume optimal, approximately optimal, specific heuristics, utility-based, or other principled strategy – Then see what practical limitations are and in what environments it succeeds – Success = strategy is adaptive

  • This is a bit backwards

– In reality, strategies are adaptive because they arise from the environment, not vice versa – We instead take the opposite approach, examining strategies that evolve in response to task environment

slide-3
SLIDE 3

Introduction

  • Computational evolution approach

– Evolve artificial agents using mechanisms that are similar to natural selection – Guarantees adaptive strategies – Biologically based agents easy to implement on neural hardware

  • Allows us to evolve decision-making

strategies and compare against existing theories

– More complex / more information used

  • Optimal sampling models (e.g. high-

threshold diffusion)

– Less complex / less information used

  • Heuristic strategies (e.g. run rules)

𝜺 𝜾

slide-4
SLIDE 4

Goals

  • Examine a common task that animals in almost

every ecological niche have to solve

– Binary perceptual decision

  • Incoming information is from source A or source B -- agent

must determine which one

  • e.g. predator / prey, edible / inedible, track A / B

– Optional stopping (agent terminates search)

  • Examine strategies that they develop

– Examine structure of their brains – Behavioral accuracy, brain complexity, amount of information use

slide-5
SLIDE 5

Methods

  • Task

– Discriminate between sources of incoming information – Comes from either majority [01] (right) or majority [10] (left)

  • Agents

– Markov brains (16-bit)

  • Have 16 nodes, used for input, processing, output of information
  • Nodes are causally connected via logic gates
  • Genetic code gives rise to pattern of connections
slide-6
SLIDE 6

Markov Brain

slide-7
SLIDE 7

Markov Brain

Inputs come from: A) N% 1s on left, N% 0s on right B) N% 0s on left, N% 1s on right Difficulty manipulated by changing N (high [e.g. 90] = easy, low [e.g. 60] = difficult)

slide-8
SLIDE 8

Markov Brain

N = 90%, LEFT on this trial

Difficulty manipulated by changing N (high [e.g. 90] = easy, low [e.g. 60] = difficult)

81% 9% 9% 1%

slide-9
SLIDE 9

Markov Brain

Organism has to indicate which one using its output nodes

slide-10
SLIDE 10

Markov Brain

slide-11
SLIDE 11

Markov Brain

slide-12
SLIDE 12

Markov Brain

slide-13
SLIDE 13

Markov Brain

slide-14
SLIDE 14

Markov Brain

slide-15
SLIDE 15

Markov Brain

slide-16
SLIDE 16

Setup

  • 100 organisms per generation

– Each one has its own genome and node structure

  • Each organism makes 100 left [10] / right [01] decisions

during its lifetime (actual direction is random)

– Gains points for every correct answer, loses points for incorrect

  • Reproduction based on performance (roulette wheel)

– Offspring accumulate mutations in genome to set node structure – Poor performers simply die off without reproducing

slide-17
SLIDE 17

Data

  • Focus on manipulations of difficulty
  • Fitness – asymptotic performance of population
  • Brain size - # of connections between nodes in the

artificial brains

  • Strategy use – how much information do the agents

use, and how long do they gather it?

slide-18
SLIDE 18

Fitness

90% 85% 80% 75% 70% 65% 60%

  • Agents achieved near-

perfect accuracy in most conditions except most difficult

  • Even then, worst

performance was ~80% accuracy

slide-19
SLIDE 19

Strategies

  • Simple heuristic strategies

– Should use very little information – Terminate search early – Small brains

  • Complex evidence accumulation strategies

– Should use a large amount of information – Gather information over a protracted period – Large brains

slide-20
SLIDE 20

Results – brain size

  • Brain size

– Generally larger in more difficult conditions – Response to more demanding task environment

  • Brains decreased in size

when the task was easy

– Despite no explicit cost for more connections – Seems to result from the mutation load imposed by larger brain

slide-21
SLIDE 21

Results – strategy use

  • More difficult conditions led to more information use

– Agents also gathered information over a more protracted period

slide-22
SLIDE 22

Conclusions

  • Difficulty of task environment plays a huge role in

evolutionary trajectory of artificial agents

– Behavior scales based on environmental demands, – Not strictly based on optimal decision strategies

  • Agents in difficult conditions evolved large brains,

integrated lots of information over a protracted period

– Strategies resembled complex sequential sampling strategies

  • Agents in easier conditions evolved smaller brains, used

relatively little information

– Brain size seems to be limited by mutation load – Strategies resembled 2-run / 3-run heuristic rules

slide-23
SLIDE 23

Bonus slides

  • Surface plots

– Brain size – Memory (self loops) – Memory (back-and-forth loops) – Mean decision time

  • Cost-benefit payoff manipulations
  • 1000 tick results

– % correct – Fitness – Decision time

slide-24
SLIDE 24

Brain size

slide-25
SLIDE 25

Memory – self loops

slide-26
SLIDE 26

Memory – back-and-forth loops

slide-27
SLIDE 27

Waiting times

slide-28
SLIDE 28

Cost-Benefit manipulations

slide-29
SLIDE 29

1,000-tick Results - # Correct

slide-30
SLIDE 30

1,000-tick Results - Fitness

slide-31
SLIDE 31

1,000-tick Results – Decision time