Making Systems Sensitive to the User’s Time and Working Memory Constraints Anthony Jameson, Ralph Schäfer, Thomas Weis, André Berthold, Thomas Weyrath Project "Resource-Adaptive Dialog" DFG Collaborative Research Center on Resource-Adaptive Cognitive Processes SFB 378 University of Saarbrücken, Germany 1. Why is this an important problem? 2. Can we start with a simple, 80/20 solution? 3. How can a more complex, theoretically oriented solution look? Avoiding speech recognition and generation Empirical basis Illustrative example dialog The core: A Bayesian causal model 4. What can we conclude from all this?
What’s in � README? Late Second Millenium Late Second Millenium .cshrc Frankfurt-Mannheim emacs-19 README
➧ ✩ ✮ ✦ ✫ ✴ ★ ✾ ✼ ✽ ✼ ✧ ✧ ✻ ★ ✰ ✦ ✗ ✚ ✙ ✛✺ ✗ ✗ ✘ ✙ ✗ ✦ ✫ ✯ ✫ ✫ ✷ ✈ ✦ ✫ ✴ ★ ✾ ✼ ✽ ✼ ✧ ✧ ✻ ✶ ✬ ✦ ✵ ✴ ✦ ✳ ✤ ✤ ✘ ✣ ✗ ✥ ✱ ✬✹ ✧ ✷ ✧ ✯ ✮ ✬✭ ✫ ✩ ✦ ✦ ✧ ✪ ✩ ★ ✦ ✫ ✥ ✱ ✣ ✢ ✜ ✛ ✚ ✙ ✘ ✗ ✗ ✰ ✤ ✬ ✶ ✮ ✭ ✬ ✫ ✩ ✦ ✦ ✧ ✪ ✷ ✱ ✭ ✦ ✵ ✴ ✦ ✳ ✲ ✤ ✘ ✗ ✗ ✙ ✸ Early Third Millenium ❸❹❸❺❸❙❻✕❼❾❽ ❿➁➀❹➂☛➃➅➄➇➆❺➈❹➉ ➂➋➊➌➄❈➍➁➎➐➏➒➑➔➓→❿ ➏↔➣➙↕■➈➜➛q➈❺➝ ➍➞➂➠➟➡➍➒➂➋➢➤➆➥↕➦❸❺❸❹❸ ✿✆❀✏❁ ❀❃❂❅❄ ❆❈❇❃❉❃❉✕❊●❋■❍✏❏✠❑▲❋■▼ ❊●◆P❖✆◗❙❘ ❚✏▼ ❯❱❉ ✿✆❀✏❁ ❉❃❀❳❲ ❊●❨❬❩●❊●❭❫❪❴❑▲❋■▼ ❊●◆P❖❃❵❜❛✌❯❝❪✓❚❞▼ ❡❱❚✏❊ ✿✆❀✏❁ ❂❃❢❅❣✐❤❴❂❜❥✏✿✆❉✕❊✐❋❦❍✄❏♠❧❬❋■❋■♥❫♦■♥q♣●❍✏❨r❚✏❊s❖✆◗❙❘ ❚✏▼ ❯❞❇ ✿✆❉✏❁ ❢❃t❳❲ ❊●❨❬❩●❊●❭❫❪✉❧❬❋■❋■♥❫♦■♥❫♣✐❍✏❨r❚✏❊ ♥q❚❞▼ ❯✇❁①t❃t✏❖ ②④③❴❢✏✿❃❖ ②✐▼ ❊⑤✿❃⑥ ③❴❉✏⑥❃⑦r❘q⑥✆⑧❃⑨④⑩❶❩●❯❷❍✏❏●❘ ❋■❡ ✁✄✂✆☎✞✝✠✟☛✡✌☞✌✍✏✎✒✑✓✎✕✔ ✖✕✖✕✖ ✁✄✂✆☎✞✝✠✟☛✡✌☞✌✍✏✎✒✑✓✎✕✔ ✖✕✖✕✖ ✁✄✂✆☎✞✝✠✟☛✡✌☞✌✍✏✎✒✑✓✎✕✔ ✖✕✖✕✖
➨ Possible Simple Approaches 1. When designing, just assume minimal user resources Problems: Resources are variable When resources are there, they should be exploited 2. Let users influence system behaviors Problems: Requires familiarity or obstrusiveness Consumes user resources 3. Allow users to report on the state of their resources Problems: Same as for 2 4. Specify simple input-output relationships Examples: 1. ”If talks fast then should synthesize fast speech” 2. ”If asks for clarification of ’s output then should simplify subsequent outputs” 3. ”. . . ” Problems: Is Rule N really a good idea? Why? When?
➩ Example Scenario of READY Prototype
➫ Salient Features of the Present Approach 1. Many problems involving input and output techni- ques are avoided through use of simple simulati- ons. 2. The core of the system is a probabilistic causal model of: CAUSES OF RESOURCE CONSTRAINTS AVAILABLE WORKING TIME PRESSURE MEMORY CAPACITY CONSEQUENCES OF RESOURCE CONSTRAINTS 3. The causal model is used for: Assessing resource constraints on the basis of observations Evaluating possible system actions
➭ Empirical Basis 1. Previous laboratory research Result: Partial quantification of relationships among selected variables in artificial situations 2. New laboratory research Result: Similar but more quantitative results, concerning the variables you don’t yet know about 3. Quantitative analysis of data from real interactions Result: Distribution of observable phenomena in realistic si- tuations 4. Knowledge elicitation from experienced persons (e.g., firemen) Result: More or less reliable qualitative opinions about causal relationships
➯ Menu for Simulation of Speech Input
➲ READY’s Main Interaction Screen
Coarse View of READY’s Architecture INSTANTIATED DYNAMIC BAYESIAN NETWORK USER ➳❜➵ GENERIC NETWORK RULES FOR NETWORK MODELING SCHEMATA INSTANTIATION DIALOG TRANSITION NETWORKS SCRIPTS ENCODING MANAGEMENT ENCODING DISCOURSE TROUBLESHOOTING AND KNOWLEDGE REPAIR KNOWLEDGE USER INTERFACE
Time WM Pressure Load S Good morning to you. You are Good morning, automatic connected to the automatic repair repair advice service advice service READY of the READY of the SFB378. SFB378. U Good day to you, too. This is Joseph Smith here. Joseph Smith from Eckerhausen. S Mr. Smith, how can I help you? Are What kind of problem do ➳✄➳ you having problems with your you have? engine? U Uh ... the lamp ... under, um next to Cooling water warning the speedometer for the cooling lamp is on. water is lit up red. S Have you noticed anything else Anything else unusual unusual about the cooling system, about the cooling system? maybe steam around the grill at the front of the hood? U At the front of the car - uh, engine, The cooler is ... steaming there’s, there’s something and the fan isn’t working. steaming there ... and the fan is off.
U At the front of the car - uh, engine, The cooler is ... steaming there’s, there’s something and the fan isn’t working. steaming there ... and the fan is off. S Under the hood, on the right, in the Check the plug on the front, near the yellow oil measuring cooling water filter. stick, there’s a cable. Please check its plug. U Cooling water filter ... OK. U What was that again? Got it. S Under the hood, on the Check the plug on the cooling water filter. right, in the front, near the ➳➸� yellow oil measuring stick, there’s a cable. Please check its plug. U Plug next to the oil stick. OK. U What was that again? Got it. S From the plug, there’s a Is the connecting cable damaged? cable that goes to the fan. Is the cable damaged somehow - a bit loose or bent?
Two Time Slices of a Dynamic Bayesian Net SYNTACTIC USE OF PROVISION OF INTRINSIC DIFFICULTY DOMAIN TECHNICAL STRUCTURE DIFFICULTY KNOWLEDGE TERMS ACTION ELLIPSIS / ANAPHORA DIFFICULTY OF FORMULATION AMOUNT OF INFORMATION TO CONVEY ACTION NOISES COMPLEXITY BACKGROUND SITUATIONAL SITUATIONAL BACKGROUND NOISES DISTRACTIONS DISTRACTIONS NOISES UTTERANCE LENGTH LENGTH OF ANSWER BREATHING EMOTIONAL EMOTIONAL BREATHING NOISES STRAIN STRAIN NOISES PRIORITY FOR WM DEMANDS WM DEMANDS WM DEMANDS OF AVAILABLE WM AVAILABLE WM WM DEMANDS OF PRIORITY FOR LANGUAGE OF GENERATION OF COMPREHEN- ACTION CAPACITY CAPACITY ACTION LANGUAGE ➳➸➧ PROCESSING SION PROCESSING DURATION OF GENERATION USE OF WM FOR USE OF WM FOR USE OF WM FOR TIME PRESSURE TIME PRESSURE USE OF WM FOR FLUENCY REPETITIONS GENERATION COMPREHEN- ACTION ACTION SION NUMBER OF COGNITIVE INCREASE IN BASELINE PAUSES UNNECESSARY ARTICULATION ARTICULATION RATE INFORMATION RATE DURATION OF SUCCESS OF SUCCESS IN COMPREHEN- COMPREHEN- CONCEPTUALIZ- OBSERVED SUCCESS IN TOTAL LENGTH SION SION ING ARTICULATION FORMULATION OF COGNITIVE RATE PAUSES ELLIPSIS / ANAPHORA IN ANSWER FEEDBACK ON CONTENT CLARITY OF COMPREHEN- ERRORS PRONUNCIATION SION APPROPRIATE- SENTENCE SPEECH NESS OF FRAGMENTS ERRORS CONTENT Time Slice t i Time Slice t i+1
➨ ➳ SYNTACTIC USE OF PROVISION OF DIFFICULTY TECHNICAL STRUCTURE TERMS ELLIPSIS / ANAPHORA DIFFICULTY OF FORMULATION COMPLEXITY BACKGROUND SITUATIONAL NOISES DISTRACTIONS UTTERANCE LENGTH BREATHING EMOTIONAL NOISES STRAIN WM DEMANDS WM DEMANDS OF AVAILABLE WM PRIORITY FOR OF COMPREHEN- ACTION CAPACITY LANGUAGE SION PROCESSING
USE OF WM FOR USE OF WM FOR TIME PRESSURE COMPREHEN- ACTION SION ➳➸➩ DURATION OF SUCCESS OF COMPREHEN- COMPREHEN- SION SION ELLIPSIS ANAPHORA ANSWER FEEDBACK ON COMPREHEN- SION
INTRINSIC DOMAIN DIFFICULTY KNOWLEDGE ACTION AMOUNT OF INFORMATION TO CONVEY ACTION NOISES ➳➸➫ SITUATIONAL BACKGROUND DISTRACTIONS NOISES LENGTH OF ANSWER EMOTIONAL BREATHING STRAIN NOISES PRIORITY FOR WM DEMANDS AVAILABLE WM WM DEMANDS OF LANGUAGE OF GENERATION CAPACITY ACTION PROCESSING DURATION GENERATION
➭ ➳ USE OF WM FOR TIME PRESSURE USE OF WM FOR FLUENCY REPETITIONS GENERATION ACTION NUMBER OF COGNITIVE INCREASE IN BASELINE PAUSES ARTICULATION UNNECESSARY ARTICULATION RATE INFORMATION RATE SUCCESS IN CONCEPTUALIZ- OBSERVED SUCCESS IN TOTAL LENGTH ING ARTICULATION FORMULATION OF COGNITIVE RATE PAUSES ELLIPSIS / ANAPHORA IN ANSWER CONTENT CLARITY OF ERRORS PRONUNCIATION APPROPRIATE- SENTENCE SPEECH NESS OF FRAGMENTS ERRORS CONTENT
Recommend
More recommend
