Evaluating Cognitive Models and Architectures Marc Halbrgge, - PowerPoint PPT Presentation

Evaluating Cognitive Models and Architectures Marc Halbrügge, Dipl.-Psych. marc.halbruegge@unibw.de Human Factors Institute Universität der Bundeswehr München July 2007

Cognitive architectures in general Evaluation Methods for Cognitive Models Can a Cognitive Architecture be Falsified? Topics of this talk Main points Evaluation of ... Models: Current approaches to quantify model complexity are not sufficient Architectures: Cognitive architectures cannot be falsified and are therefore too weak to be considered as theories Marc Halbrügge Evaluating Cognitive Models and Architectures

Cognitive architectures in general Evaluation Methods for Cognitive Models Can a Cognitive Architecture be Falsified? Topics of this talk Main points Evaluation of ... Models: Current approaches to quantify model complexity are not sufficient Architectures: Cognitive architectures cannot be falsified and are therefore too weak to be considered as theories Marc Halbrügge Evaluating Cognitive Models and Architectures

Cognitive architectures in general Evaluation Methods for Cognitive Models Can a Cognitive Architecture be Falsified? Contents Cognitive architectures in general 1 Evaluation Methods for Cognitive Models 2 Model Complexity and Overfitting The Markov Chain Example Can a Cognitive Architecture be Falsified? 3 The Quicksort Example Conclusions Marc Halbrügge Evaluating Cognitive Models and Architectures

Cognitive architectures in general Evaluation Methods for Cognitive Models Can a Cognitive Architecture be Falsified? Cognitive Modeling The Goal The main goal of cognitive modeling is to simulate human cognition Marc Halbrügge Evaluating Cognitive Models and Architectures

Cognitive architectures in general Evaluation Methods for Cognitive Models Can a Cognitive Architecture be Falsified? Marc Halbrügge Evaluating Cognitive Models and Architectures

Cognitive architectures in general Evaluation Methods for Cognitive Models Can a Cognitive Architecture be Falsified? Marc Halbrügge Evaluating Cognitive Models and Architectures

Cognitive architectures in general Evaluation Methods for Cognitive Models Can a Cognitive Architecture be Falsified? Marc Halbrügge Evaluating Cognitive Models and Architectures

Cognitive architectures in general Evaluation Methods for Cognitive Models Can a Cognitive Architecture be Falsified? Marc Halbrügge Evaluating Cognitive Models and Architectures

Cognitive architectures in general Model Complexity and Overfitting Evaluation Methods for Cognitive Models The Markov Chain Example Can a Cognitive Architecture be Falsified? Contents Cognitive architectures in general 1 Evaluation Methods for Cognitive Models 2 Model Complexity and Overfitting The Markov Chain Example Can a Cognitive Architecture be Falsified? 3 The Quicksort Example Conclusions Marc Halbrügge Evaluating Cognitive Models and Architectures

Cognitive architectures in general Model Complexity and Overfitting Evaluation Methods for Cognitive Models The Markov Chain Example Can a Cognitive Architecture be Falsified? Evaluating Cognitive Models Goodness of Fit Concordance with human data – a high fit – is generally thought to be necessary, but not sufficient for model validation [Roberts and Pashler, 2000]. Complexity and Overfitting In order to assess the fit of a model or to compare fits between models, it is important to be aware of the complexity of the models. Models with many degrees of freedom usually achieve better fits, but bear the risk of overfitting Marc Halbrügge Evaluating Cognitive Models and Architectures

Cognitive architectures in general Model Complexity and Overfitting Evaluation Methods for Cognitive Models The Markov Chain Example Can a Cognitive Architecture be Falsified? Evaluating Cognitive Models Goodness of Fit Concordance with human data – a high fit – is generally thought to be necessary, but not sufficient for model validation [Roberts and Pashler, 2000]. Complexity and Overfitting In order to assess the fit of a model or to compare fits between models, it is important to be aware of the complexity of the models. Models with many degrees of freedom usually achieve better fits, but bear the risk of overfitting Marc Halbrügge Evaluating Cognitive Models and Architectures

Cognitive architectures in general Model Complexity and Overfitting Evaluation Methods for Cognitive Models The Markov Chain Example Can a Cognitive Architecture be Falsified? Evaluating Cognitive Models Goodness of Fit Concordance with human data – a high fit – is generally thought to be necessary, but not sufficient for model validation [Roberts and Pashler, 2000]. Complexity and Overfitting In order to assess the fit of a model or to compare fits between models, it is important to be aware of the complexity of the models. Models with many degrees of freedom usually achieve better fits, but bear the risk of overfitting Marc Halbrügge Evaluating Cognitive Models and Architectures

Cognitive architectures in general Model Complexity and Overfitting Evaluation Methods for Cognitive Models The Markov Chain Example Can a Cognitive Architecture be Falsified? Marc Halbrügge Evaluating Cognitive Models and Architectures

Cognitive architectures in general Model Complexity and Overfitting Evaluation Methods for Cognitive Models The Markov Chain Example Can a Cognitive Architecture be Falsified? Marc Halbrügge Evaluating Cognitive Models and Architectures

Cognitive architectures in general Model Complexity and Overfitting Evaluation Methods for Cognitive Models The Markov Chain Example Can a Cognitive Architecture be Falsified? Marc Halbrügge Evaluating Cognitive Models and Architectures

Cognitive architectures in general Model Complexity and Overfitting Evaluation Methods for Cognitive Models The Markov Chain Example Can a Cognitive Architecture be Falsified? Marc Halbrügge Evaluating Cognitive Models and Architectures

Cognitive architectures in general Model Complexity and Overfitting Evaluation Methods for Cognitive Models The Markov Chain Example Can a Cognitive Architecture be Falsified? Marc Halbrügge Evaluating Cognitive Models and Architectures

Cognitive architectures in general Model Complexity and Overfitting Evaluation Methods for Cognitive Models The Markov Chain Example Can a Cognitive Architecture be Falsified? Overfitting Results for the three classifiers (N=269) df misclassifications obs. data true data too low 9 10 low 6 5 high 0 11 Marc Halbrügge Evaluating Cognitive Models and Architectures

Cognitive architectures in general Model Complexity and Overfitting Evaluation Methods for Cognitive Models The Markov Chain Example Can a Cognitive Architecture be Falsified? Model Complexity and Overfitting Complexity in Cognitive Modeling Baker et al. (2003): Count the number of free weights of the subsymbolic part of an ACT-R model, use this as degrees of freedom when computing the Bayesian information criterion (BIC). BUT: Measurements of complexity like the BIC aim at closed form mathematical models. Cognitive models are sequences of actions! Marc Halbrügge Evaluating Cognitive Models and Architectures

Cognitive architectures in general Model Complexity and Overfitting Evaluation Methods for Cognitive Models The Markov Chain Example Can a Cognitive Architecture be Falsified? Model Complexity and Overfitting Complexity in Cognitive Modeling Baker et al. (2003): Count the number of free weights of the subsymbolic part of an ACT-R model, use this as degrees of freedom when computing the Bayesian information criterion (BIC). BUT: Measurements of complexity like the BIC aim at closed form mathematical models. Cognitive models are sequences of actions! Marc Halbrügge Evaluating Cognitive Models and Architectures

Cognitive architectures in general Model Complexity and Overfitting Evaluation Methods for Cognitive Models The Markov Chain Example Can a Cognitive Architecture be Falsified? Memory Retrieval in ACT-R Two cognitive models that retrieve chunks from declarative memory. Which chunk is to be retrieved next depends on the production that currently fires. The models stop when the memory retrieval fails. Model 1 Model 2 Marc Halbrügge Evaluating Cognitive Models and Architectures

Cognitive architectures in general Model Complexity and Overfitting Evaluation Methods for Cognitive Models The Markov Chain Example Can a Cognitive Architecture be Falsified? Addressing Complexity From the Baker et al. approach, model 2 (four states) should be considered the more complex one, because it has more numerical weights. But: Model 2 can only create sequences like abcdabcda Model 1 can output any sequence of a ’s and b ’s. The difference is present even if you use model run time as dependent variable. Imagine two of the four productions take a much longer time than the others. Marc Halbrügge Evaluating Cognitive Models and Architectures