virtual humans for training and tutoring applications Robert - - PowerPoint PPT Presentation

U.S. Army Research, Development and Engineering Command

Unclassified – Distribution A – Unlimited

Cognitive and affective modeling in intelligent virtual humans for training and tutoring applications

Robert Sottilare, Ph.D., Associate Director for Science & Technology John Hart, Chief, Creative Technologies Branch and Program Manager, Institute for Creative Technologies Army Research Laboratory - Human Research & Engineering Directorate (HRED) 31 July 2012

Unclassified – Distribution A – Unlimited 2 of 33 Unclassified – Distribution A – Unlimited

Presentation Outline

• Artificial life forms & virtual humans
• Virtual humans in training and tutoring
• Human Cognition and Affect
• Cognitive and Affective Models
• Future directions
• Questions

Unclassified – Distribution A – Unlimited 3 of 33 Unclassified – Distribution A – Unlimited

Artificial life forms

• We will be talking about virtual

humans… can you identify these artificial life forms?

Unclassified – Distribution A – Unlimited 4 of 33 Unclassified – Distribution A – Unlimited

Composites

• Human-like entities composed of living

tissue created outside of normal reproductive processes

• driven by its own goals, cognitive and

affective processes

• includes clones, replicants, and graveyard

compilations of reanimated tissue

Unclassified – Distribution A – Unlimited 5 of 33 Unclassified – Distribution A – Unlimited

Artificial life forms

Unclassified – Distribution A – Unlimited 6 of 33 Unclassified – Distribution A – Unlimited

Robots

• electromechanical machines

driven by goals and cognitive processes defined by its creator

• constrained by the three laws
• f robotics?
• A robot may not injure a human being or,

through inaction, allow a human being to come to harm.

• A robot must obey the orders given to it by

human beings, except where such orders would conflict with the First Law.

• A robot must protect its own existence as long

as such protection does not conflict with the First or Second Laws.

yes no

Unclassified – Distribution A – Unlimited 7 of 33 Unclassified – Distribution A – Unlimited

Artificial life forms

Unclassified – Distribution A – Unlimited 8 of 33 Unclassified – Distribution A – Unlimited

Avatars

• Graphical

representation

• f a user or a

user’s alter-ego

• Usually driven

by the goals and behaviors /actions of the user (aka role player) users

Unclassified – Distribution A – Unlimited 9 of 33 Unclassified – Distribution A – Unlimited

Artificial life forms

Unclassified – Distribution A – Unlimited 10 of 33 Unclassified – Distribution A – Unlimited

Virtual Humans

• VH are graphical

representations of human behaviors driven by cognitive and affective processes

• VH are intelligent

agents that facilitate meaningful interpersonal interactions with human users in virtual reality

Unclassified – Distribution A – Unlimited 11 of 33 Unclassified – Distribution A – Unlimited

Virtual Humans

Autonomous virtual characters that can have meaningful interactions with human users

• Reason about environment
• Understand and express emotion
• Communicate through speech & gesture
• Play the role of teachers, peers, adversaries

Unclassified – Distribution A – Unlimited 12 of 33 Unclassified – Distribution A – Unlimited

Human Cognition and Affect

• Cogn

Cognit itive ive pro proce cesse sses

– beh behaviors aviors indic indicating ating increasingly complex and abstract mental capabilities – Remember Remembering ing (low) (low) – Unders Understanding tanding – Apply Applying ing – Analyz Analyzing ing – Evaluating Evaluating – Cr Creating eating (high (high)

Source: Anderson and Krathwohl's Taxonomy (2000) aka Bloom’s Revised Taxonomy

• Aff

Affec ecti tive ve pro proce cesse sses

– beh behaviors aviors indic indicating ating emotional emotional grow growth th – Receiv Receiving ing (awarenes (awareness) s) – Res Responding ponding (interes (interest) t) – Valuing Valuing (ap (appreciat preciation) ion) – Organiz Organizing ing (respon (responsibility sibility) – Char Character acterizing izing (co (commit mmitment) ment)

Source: Krathwohl’s Taxonomy

Unclassified – Distribution A – Unlimited 13 of 33

Physical World Virtual World

Virtual Physiology

Ergonomics Surgical manikins Physio models

Artificial Cognition

Planning Decision-theory Cognitive models

Human Physiology

Mechanics Anatomy Physiology

Human Cognition

Psychology Education Neuroscience

Human-Computer Interaction How do people look, sense, act, think? How can computers simulate this? What are minimum requirements to have meaningful outcomes

Virtual Humans are interdisciplinary science

Courtesy of the Institute of Creative Technologies

Unclassified – Distribution A – Unlimited 14 of 33 Unclassified – Distribution A – Unlimited

Cognitive Architecture ACT-R

• Adaptive Control of

Thought—Rational

• a theory for simulating

and understanding human cognition

• ACT-R's main

components are: – modules, buffers, and a pattern matcher

• Modules

Modules

Anderson, J. R., Bothell, D., Byrne, M. D., Douglass, S., Lebiere, C., & Qin, Y . (2004). An integrated theory of the mind. Psychological Review 111, (4). 1036-1060.

– percept perceptual ual-motor motor modules modules – memory memory modules modules – declarat declarative ive memory memory (f (facts acts) – proced procedural ural mem memory

• ry
• product

productions ions (how we do things)

Unclassified – Distribution A – Unlimited 15 of 33 Unclassified – Distribution A – Unlimited

A cognitive-affective VH framework

Courtesy of the Institute of Creative Technologies

Unclassified – Distribution A – Unlimited 16 of 33

Virtual Humans in computer-based tutoring

world learner

learner

• bserves world

learner acts

• n world

tutoring agent(s)

agent

• bserves

world agent

• bserves

learner agent acts to change world (scenario adaptation) agent acts to provide feedback or instruction via VH agent

• bserves

effect

• n learner

agent

• bserves

effect

• f

changes

• n

learning

• bjectives

Unclassified – Distribution A – Unlimited 17 of 33 Unclassified – Distribution A – Unlimited

Desired Affective Capabilities for VH

• Recognize the human’s emotional state (e.g.

mo motivate tivated, d, engaged engaged, , frus frustrate trated); d);

• Mak

Make e the the hu human man awa aware of his re of his affec affecti tive ve state state (e.g. (e.g. emo emoti tion

• nal s

al state, tate, moo mood) d) so so he ca he can n pa parti rticipa cipate te in in ma mana nagin ging h g his affec is affective sta tive state; te;

• Prov

Provide op ide opti tion

• ns (e.g. strateg

s (e.g. strategies) f ies) for

• r the

the huma human n to to man manag age/co e/control his a ntrol his affective ffective state state;

• Use

Use em emotio

• tion to

n to mo motivate tivate the the hu huma man n to a to ach chieve ieve esta establishe blished d ob

• bjectives

jectives.

Salovey, P. & Mayer, J. D. (1990). Emotional intelligence. Imagination, Cognition, & Personality, 9, 185-211. Goleman, D., (1995). Emotional Intelligence. Bantam Books: New York.

“No matter how intelligent a [virtual human] is, it will eventually become annoying if it does not have emotional intelligence.” (Picard, 2006)

Unclassified – Distribution A – Unlimited 18 of 33

Affective Modeling - EMA

1990

Courtesy of the Institute of Creative Technologies

Unclassified – Distribution A – Unlimited 19 of 33

Appraisal Theory

Unclassified – Distribution A – Unlimited 20 of 33

Emotion and Adaptation (EMA) Model Appraisal Theory

• Based on appraisal

theory

• Attitude
• Affect
• Judgement

Marsella, S., and J. Gratch. 2009. EMA: A process model of appraisal dynamics. Cognitive Systems Research 10, no. 1: 70–90.

Unclassified – Distribution A – Unlimited 21 of 33

Virtual Humans as Affect-Sensitive Tutors

• What does the VH need to know about the

learner during tutoring?

• what does the tutor

need to know about the learner to classify their affect?

• how does the tutor get

that information?

• which affective states

are important to recognize?

• how does

classification of state influence instructional decisions?

Grae aess sser er, D’Mello, Cr Craig, aig, Per erson, son, Bak Baker er, Rod

• drigo

go (20 (2012 12, in pr n press) ess)

Unclassified – Distribution A – Unlimited 22 of 33 Unclassified – Distribution A – Unlimited

Future Directions for VH

• Values – modeling the influence of values on:

– virtual human decision making (e.g., moral judgments) – preferences (e.g., personality)

• Sensory capabilities

– better perception of humans, VHs and the world

• Persistent models of:

– memory to support long-term rapport (e.g., personal learning assistants) – previous actions to support reinforcement learning (e.g., expert modeling)

Unclassified – Distribution A – Unlimited 23 of 33 Unclassified – Distribution A – Unlimited

Future Directions for VH

• Cognitive Architecture for VH (Rosenbloom, 2011)

– Broadly and incrementally functional – Theoretically elegant and simple for simple things – Mixed and hybrid – Support truly robust systems – Maintainable and extendible – Enhance VH:

• learning, perception, problem solving

Rosenbloom, P. S. (2011). From memory to problem solving: Mechanism reuse in a graphical cognitive

• architecture. Proceedings of the Fourth Conference on Artificial General Intelligence

Unclassified – Distribution A – Unlimited 24 of 33 Unclassified – Distribution A – Unlimited

Virtual humans in training & tutoring

Unclassified – Distribution A – Unlimited 25 of 33 Unclassified – Distribution A – Unlimited

Stuff we talked about…

• Artificial life forms & virtual humans
• Virtual humans in training and tutoring
• Human Cognition and Affect
• Cognitive and Affective Models
• Future directions
• Questions

Unclassified – Distribution A – Unlimited 26 of 33 Unclassified – Distribution A – Unlimited

Thank y hank you f

• u for y
• r your a
• ur attention

ttention Que Questions stions?