Creating an Empirical Basis for Adaptation Decisions These slides - - PDF document

▶
creating an empirical basis for adaptation decisions
SMART_READER_LITE
LIVE PREVIEW

Creating an Empirical Basis for Adaptation Decisions These slides - - PDF document

Sep 25, 2022 363 likes •494 views

1 Introduction 2 Creating an Empirical Basis for Adaptation Decisions These slides and the full paper are available from: Anthony Jameson Department of Barbara Gromann-Hutter Computer Science Leonie March Department of Ralf Rummer

slide-1
SLIDE 1

1 Introduction 2

Creating an Empirical Basis for Adaptation Decisions

Anthony Jameson Barbara Großmann-Hutter Department of Computer Science Leonie March Ralf Rummer Department of Psychology University of Saarbrücken, Germany http://w5.cs.uni-sb.de/~ready/ (slides, etc.)

These slides and the full paper are available from: http://w5.cs.uni-sb.de/~ready/

Overview

2

General issue

IUIs often adapt their behavior

  • to material being presented
  • to properties of the situation
  • to properties of the user

How can we help them make sound adaptation decisions?

Overview

  • 1. Rule-based vs. decision-theoretic adaptation
  • 2. Method for empirically based decision-theoretic adaptation
  • 3. What to do when this method is infeasible?
slide-2
SLIDE 2

3 Introduction 4

Contents

3

Introduction 1

Overview 2 Contents 3

Basic Concepts 5

Good Adaptation? 5 Rule-Based Adaptation 6 Decision-Theoretic Adaptation 7

Experiment 8

Everyday Example 8 Experimental Setup 9 Stepwise vs. Bundled Instructions 10 Variables in Experiment 11 Main Results 12

The Decision Mechanism 14

Learned Bayesian Network 14 Influence Diagram 17 Properties of the Learned Decision Mechanism 18

Fallbacks 19

Fallback 1: Modify the Model by Hand 19 Fallback 2: Collect Real Usage Data 21 Fallback 3: Do Analysis Without Data 22 Interactors Revisited 23

Conclusions 24

Conclusions 24

slide-3
SLIDE 3

5 Basic Concepts 6

Basic Concepts

Good Adaptation?

5

In the original talk, this slide is accompanied by a playback of the answering machine recording.

Good day. You’ve reached my mobile communication center. I don’t wanna waste your time, so I’m gonna make this really quick.

To leave a voice message, wait for the tag. To page me, press 5. You can also leave a voice message after you page me. Or, email me, at 318-367-3135@airtouch.nick

Well now, mate, that wasn’t so bad, was it?

Rule-Based Adaptation

6

(Cf. the decision trees of Eisenstein & Puerta, IUI2000)

Rules for choosing interactors for an interface

If the type of data is boolean and the type of form is a control panel then use a check box If the type of data is boolean and the type of form is a questionnaire then use radio buttons

slide-4
SLIDE 4

7 Basic Concepts 8

Decision-Theoretic Adaptation

7

TYPE OF FORM SPEED VALIDITY INTER- ACTOR UTILITY RADIO BUTTONS CHECK BOX SPEED QUESTION- NAIRE CONTROL PANEL RADIO BUTTONS CHECK BOX VALIDITY QUESTION- NAIRE CONTROL PANEL

When is decision-theoretic adaptation useful?

  • There are multiple criterion variables
  • There are quantitative tradeoffs
  • The (exact) nature of the relationships is an empirical question

Experiment

Everyday Example

8

Possible output of spoken help system:

Choose "PostScript Level 2 only" Set "Fit to Page" to "off". Set "Print to File" to "on". Set "Use Printer Halftone Screens" to "on". Set "Download Asian Fonts" to "off".

slide-5
SLIDE 5

9 Experiment 10

Experimental Setup

9

Stepwise vs. Bundled Instructions

10

Stepwise: S: Set X to 3 U: ... OK S: Set M to 1 U: ... OK S: Set V to 4 U: ... Done Bundled: S: Set X to 3, set M to 1, set V to 4 U: ... ... ... Done

slide-6
SLIDE 6

11 Experiment 12

Variables in Experiment

11

Independent variables: Presentation Mode

  • Stepwise vs.

bundled

Distraction?

  • No secondary task
  • vs. "monitor the

flashing lights"

Number of Steps

  • 2, 3 or 4 steps

Dependent variables (selection): Execution Time

  • Total time to execute an

instruction sequence (including "OK"s, etc.)

Error

  • All instructed buttons

pressed (and no others)?

Main Results (1)

12

Distraction?

sequences:

Distraction?

No Yes

Errors (%)

10 20 30 40

Distraction?

No Yes

Execution time (msec)

1000 2000 3000 4000 5000 6000

Three-step sequences:

Execution time (msec)

slide-7
SLIDE 7

13 Experiment 14

Main Results (2)

13 Distraction?

No Yes

Errors (%)

10 20

Distraction?

No Yes

Execution time (msec)

1000 2000 3000

Two-step sequences:

Distraction?

No Yes

Errors (%)

10 20 30 40

Distraction?

No Yes

Execution time (msec)

1000 2000 3000 4000 5000 6000

Three-step sequences:

Distraction?

No Yes

Errors (%)

10 20 30 40 50

Distraction?

No Yes

Execution time (msec)

1000 2000 3000 4000 5000 6000 7000

Four-step sequences:

The Decision Mechanism

Learned Bayesian Network (1)

14

Bayesian network learned on the basis of the experimental data, showing a prediction for a specific combination of values of the independent variables

Number of Steps Four Three Two 100 Presentation Mode Bundled Stepwise 100 Distraction? Yes No 100 Error Yes No 18.3 81.7 0.18 ± 0.39 Execution Time s9 s8 s7 s6 s5 s4 s3 s2 s1 0.65 0.65 1.96 1.31 8.50 14.4 39.2 30.7 2.61 3.1 ± 1.3

In the original talk, demonstrations of the example network and influence diagram are given instead of this and the following three slides. The tool depicted is Netica, which is available from http://www.norsys.com.

slide-8
SLIDE 8

15 The Decision Mechanism 16

Learned Bayesian Network (2)

15

The same network as in the previous slide, showing a prediction made under uncertainty about the independent variable "Distraction?"

Number of Steps Four Three Two 100 Presentation Mode Bundled Stepwise 100 Distraction? Yes No 40.0 60.0 Execution Time s9 s8 s7 s6 s5 s4 s3 s2 s1 0.65 0.65 1.57 0.92 3.79 7.71 31.8 48.8 4.18 2.7 ± 1.2 Error Yes No 10.0 90.0 0.1 ± 0.3

Learned Bayesian Network (3)

16

The same network as in the previous two slides, showing the interpretation of an observation of U’s performance

Number of Steps Four Three Two 100 Presentation Mode Bundled Stepwise 100 Distraction? Yes No 92.3 7.72 Execution Time s9 s8 s7 s6 s5 s4 s3 s2 s1 100 4 Error Yes No 100 1

slide-9
SLIDE 9

17 The Decision Mechanism 18

Influence Diagram

17

An influence diagram defined as an extension to the BN of the previous slides

Distraction? Yes No 100 Execution Time s9 s8 s7 s6 s5 s4 s3 s2 s1 0.65 0.65 1.96 1.31 8.50 14.4 39.2 30.7 2.61 3.1 ± 1.3 Error Yes No 18.3 81.7 0.18 ± 0.39 Weight of Error w28 w25 w22 w19 w16 w13 w10 w7 w4 w1 100 16 Number of Steps Four Three Two 100 Utility Presentation Mode Bundled Stepwise

  • 6.0679

Properties of the Learned Decision Mechanism

18

Learned Bayes net

  • Embodies experimental results
  • Also allows probabilistic prediction and interpretation

Influence diagram

  • Generates a decision for each situation
  • Computes general policies

Summary of Internal Policy Table

Steps Distraction = "No" Distraction = "Yes" Four Stepwise iff w > 9 Stepwise iff w > 3 Three Stepwise iff w > 21 Stepwise iff w > 6 Two Always bundled Stepwise iff w > 9

slide-10
SLIDE 10

19 Fallbacks 20

Fallbacks

Fallback 1: Modify the Model by Hand (1)

19

Motivation

  • Real application situation is different from data-collection situation

Procedure

  • Replace learned relationships by theoretically based formulas

Prospects

− New aspects may be highly speculative + Decision-theoretic tools permit sensitivity analyses

Fallback 1: Modify the Model by Hand (2)

20

Original situation

Distraction?

No Yes

Errors (%)

10 20 30 40 50 60

Distraction?

No Yes

Execution time (msec)

1000 2000 3000 4000 5000 6000 7000 8000 9000

Three-step sequences:

Sliders instead of buttons

Distraction?

No Yes

Errors (%)

10 20 30 40 50 60

Distraction?

No Yes

Execution time (msec)

1000 2000 3000 4000 5000 6000 7000 8000 9000

Three-step sequences:

??

slide-11
SLIDE 11

21 Fallbacks 22

Fallback 2: Collect Real Usage Data

21

Motivation

  • Experiment is not feasible or could not be realistic

Procedure

  • Learn influence diagrams while system is in real use

Prospects

− Massively missing data may make useful learning impossible

Fallback 3: Do Analysis Without Data

22

Motivation

  • [Same problems as above]
  • Unwillingness to deal with decision-theoretic tools

Procedure

  • 1. Draw influence diagrams on paper
  • 2. Graph hypotheses about causal relationships
  • 3. What’s the reasoning behind adaptation?

Prospects

+ You can check the assumptions on which your adaptation policy is based + You may decide to change or reject the policy

slide-12
SLIDE 12

23 Fallbacks 24

Interactors Revisited

23

Check boxes or radio buttons for Boolean data?

TYPE OF FORM SPEED VALIDITY INTER- ACTOR UTILITY RADIO BUTTONS CHECK BOX SPEED QUESTION- NAIRE CONTROL PANEL RADIO BUTTONS CHECK BOX VALIDITY QUESTION- NAIRE CONTROL PANEL

Conclusions

Conclusions

24

Content

[Empirical results and theoretical analysis concerning ways of presenting instructions]

Methodology

  • 1. Rule-based adaptation is often too simple

⇒ Consider decision-theoretic adaptation

  • 2. An optimal adaptation mechanism can in principle be learned fully

automatically from empirical data And it has useful additional functions

  • 3. Theory-based tweaking is often necessary

It can be more or less reliable Decision-theoretic tools can help explore possible strategies

  • 4. Even purely conceptual decision-theoretic analysis can be useful