Diagrams, Interfaces, and Klingons Based on Kieras & Bovair's - - PDF document

21/3/04 1

Modelling How and When Learning Happens in a Diagrammatic Reasoning Task

Problem Solving Time Decision Cycles

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 10 20 30 40 50 20 40 60 80 100 120 Subject Soar Model

Frank E. Ritter

School of Information Sciences and Technology, Psychology, and Computer Science Penn State

Peter A. Bibby

School of Psychology

• U. of Nottingh a m

Supported by Joint Council Initiative in HCI and Cognitive Science

Slides presented at the ONR, NSF, and DARPA Symposium on Reasoning and Learning in Cognitive Systems, CSLI, Stanford, March 20-21, 2004.

21/3/04 2

Diagrams, Interfaces, and Klingons

ENERGY BOOSTER ONE MAIN ACCUMULATOR POWER SOURCE SECONDARY ACCUMULATOR TWO SECONDARY ACCUMULATOR ONE ENERGY BOOSTER TWO LASER BANK

• ff
• n

eb1 eb2 sa1 sa2 ma sa

Device Schematic ■ Based on Kieras & Bovair's Starship+ (='Klingon’)

interface

21/3/04 3

Task Details

■ Subjects are given:

(i) a general intro. to the problem (ii) basic information on the interface (iii) a diagram of the underlying circuit

■ Subjects are told that ONE component in the

circuit is faulty, and are asked to indicate which

• n e

21/3/04 4

Some Example Faults

PS EB1 EB2 MA SA1 SA2 LB

• ff
• n eb1

eb2 sa1 sa2 sa ma

• ff
• ff
• ff

PS EB1 EB2 MA SA1 SA2 LB

• ff
• n eb1

eb2 sa1 sa2 sa ma

• ff
• ff
• ff

PS EB1 EB2 MA SA1 SA2 LB

• ff
• n eb1

eb2 sa1 sa2 sa ma

• ff
• ff
• ff

ENERGY BOOSTER ONE MAIN ACCUMULATOR POWER SOURCE SECONDARY ACCUMULATOR TWO SECONDARY ACCUMULATOR ONE ENERGY BOOSTER TWO LASER BANK

• ff
• n

eb1 eb2 sa1 sa2 ma sa

Device Schematic

21/3/04 5

Why pay attention?

■ How to learn with interaction ■ Illustrates human-level learning

➤Forbus gold standard, ➤Langley notes “just on this task”

■ Shows that Soar chunker models

learning and transfer of learning

■ Shows learning and/with reasoning ■ There is no wireless network

21/3/04 6

Diag-Soar (v16 )

• (R

itter & Bibby, 2 0 01 )

■ 173 rules + 220 chunks (new rules) ■ Schematic knowledge represented a s

linked lists, organized as 'routes' through the circuit

■ Visual interface information represented

as declarative structures for lights & switches

■ Status of interface diagram represented in

top goal, accessed by attend

interf.- select

INTERF. CH OICE

find- fault

CH OOSE-COMPONENT TE ST-COMPONENT

select comp

DIAG- S U GGE STION INTERFACE- S U GE STION

(SOLVE-PROBLEM)

(TE ST-COMPONENT)

(DIAG-S

U GG.)

(CHECK-LIT) (CHECK-SWITCH) (CHECK-PRE

VIOUS) S OLVE-PROBLEM COMPREHEND R EPORT

(INTER

FACE

• SU

GG.) CH OOS E-SWITCH DIAG-CH OICE

diag.-select

CHECK-SWITCH-DIAG CHECK-PRE VIOUS-DIAG

test- comp

CHECK-LIT DECIDE-STATUS ATTEND

(COMPR

EHEND)

(CH

OOSE- COMPONENT)

diagnose (DECIDE-

STATUS)

check-world

CHECK-SWITCH-DECIDE CHECK-PRE VIOUS-DE CIDE R ES ET CH OOS E-PRE VIOUS RE ALITY-CHECK

Diag-Soar’s Problem Spaces

• Rit ter, 2 0 01. Soar

ICCM tutorials

21/3/04 8

Diag-Soar (cont.)

■ Organization of components on the

interface diagram 'sequences' the checking

• f each component

■ If a subgoal requires perceptual

information, goal stack must be re-built

■ attend and comprehend operators used to

represent the perceptual components of the task

• SUG

• SU

• STATUS)

21/3/04 9

Representation?

■ Experience and knowledge stored in

Soar production rules

■ Rules organized as implementing

problem spaces, operators, and choices between them

■ All learned knowledge is also

represented in rules with these structures

• PRO

• SU

• SWITCH

• STATUS

• PRE

21/3/04 10

Op No Change

Count problem space

Soar, Learning in Action

Add 1+2 Result=3 New rule: If op is add 1+2 then result = 3

Tor et al. 2004 CaDaDis, BRIMS 21/3/04 12

What is Learned?

■ (All learning implemented as new rules)

■ Where to look - implementation of

Choose-component; and creation of Attend and additions to Attend

■ What stimuli mean - implementation of

Comprehend (and lower operations)

■ Augmentations to the state from

previous problem solving

■ Huge amounts of transfer

21/3/04 13

General Results

■ Does the task ■ General Strategy — sequential

components checking — emerges from the interface representation , consistent with Ss protocols

■ Final ('fully chunked') behaviour

reflects this strategy, behaving as if it were simply a recognition (immediate response) task

21/3/04 14

Average RTs by Subjects

10 20 30 40 50 60 70 80 2 4 6 8 10 12 14 16

Subject

Model Cycles Problem Solving Time (s)

Problem Solving Time (s) Model Cycles 1 2 3 5 6 7 8 9 10 4

21/3/04 15

The data

■ 10 subjects solved ■ 20 problems ■ Each subject saw a different series of

problems

■ Problems sampled with replacement

(nominally)

■ RTs and answers recorded ■ Incorrect trials discarded ■ Some comments taken at end of trials

21/3/04 16

RTs by Fault Type

■ Accounts for 90% of Ss' RT

variability

20 40 60 80 100 120 140 2 4 6 8 10 12 14 16 18 20 22 Fault Model Cycles Problem Solving Time (s)

Decision Cycles Problem Solving Time (s) PS EB1 EB2 MA SA1 SA2 LB

21/3/04 17

RTs by Trial Number

20 40 60 80 100 120 140 160 180 200 220 240 2 6 10 14 18 22 26 30 34

Trial

Model Cycles Problem Solving Time (s)

Problem solving time (s) Model cycles

2 4 6 8 10 12 14 16 18 20

21/3/04 18

Matches RT by Individual’s Trials (s9)

20 40 60 80 100 120 140 160 180 200 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30

Fault

Model Cycles Problem Solving Time (s)

Problem Solving Time (s) Model Cycles PS MA EB2 LB EB1 LB MA MA EB2 LB EB1 PS EB1 SA1 SA2 SA1 SA2 SA1 SA2

21/3/04 19

Learns like No Other Model Tested with Data (!?)

∴No model1 with automatic learning,

tested in detail across tasks while learning

■ Able does not do transition on the fly ■ Anderson's tutors just add rules ■ VanLehn looked at new rule

acquisition but added rules by hand

1 Altmann (‘99) programming model

does this with recognition of objects

21/3/04 20

Data and Regularities Left Behind

■ Modelling the perceptual improve-

ments in motor skills omitte d

■ No account of initial learning of

the task

■ Model accounts for < 10% of the

variability for some subjects (2 out of 10)

■ 2nd trials on a problem are too fast in

the model; Verbal protocols not reported her e

21/3/04 21

Evidence for Reflection in Participants

Trial 3 Trial 5

21/3/04 22

What Can Diag-Soar Tell Us About Reasoning with Diagrams?

■ Our subjects learn and only learn the diagram

information that is relevant to the context of each stage of the problem-solving

■ The model supports that subjects still use the

diagram information in a cyclical, iterative fashion as an external resource to support the problem-solving sequence

■ Result is recognition-driven problem solving,

rather than model-driven behaviour arising

• ut of problem solving

■ Soar’s chunking models transfer in this task

21/3/04 23

Where Next ?

■ Model transfer on a fine level of

detail

■ Explaining where learning may have

mismatched

■ Apply to a more dynamic task such

as dTank (a simple tank game)

■ Modeling the type of reflection

suggested in the protocols

21/3/04 24

Thank you

Ritter, F. E., & Bibby, P. A. (accepted pending revisions).

Modeling how and when learning happens in a diagrammatic reasoning task. To Cognitive Science.

Ritter, F. E., & Bibby, P. (2001). Modeling how and when

learning happens in a simple fault-finding task. In Proceedings

• f ICCM - 2001 - Fourth International Conference on Cognitive
• Modeling. 187-192. Mahwah, NJ: Lawrence Erlbaum.

http:/ / a c s.ist.p s u.e d u/ p a p e r s / ritterB01.pdf

Ritter, F. E. (2003). Soar. In L. Nadel (Ed.), Encyclopedia of

cognitive science. vol. 4, 60-65. London: Nature Publishing

• Group. http:/ / a c s.ist.p s u.e d u/soar-faq

http:/ / a c s.ist.p s u.e d u/ nottingh a m/ pst

Tor, K., Ritte r, F. E., Haynes, S. R., & Cohen, M. A. (in press).

CaDaDis: A tool for displaying the behavior of cognitive models and agents. In Proceedings of the 13th Conference on Behavior Representation in Modeling and Simulation. http:/ / a c s.ist.p s u.e d u/ p a p e r s / torRHC04.pdf