Automatically Generating User Interfaces Adapted To Users Motor And - - PowerPoint PPT Presentation

▶
automatically generating user interfaces adapted to users
SMART_READER_LITE
LIVE PREVIEW

Automatically Generating User Interfaces Adapted To Users Motor And - - PowerPoint PPT Presentation

Sep 19, 2023 322 likes •1.58k views

Automatically Generating User Interfaces Adapted To Users Motor And Vision Capabilities Krzysztof Z. Gajos, Jacob O. Wobbrock and Daniel S. Weld Road Map Introduction Interface generation as optimization Modeling motor capabilities

slide-1
SLIDE 1

Automatically Generating User Interfaces Adapted To Users’ Motor And Vision Capabilities

Krzysztof Z. Gajos, Jacob O. Wobbrock and Daniel S. Weld

slide-2
SLIDE 2
slide-3
SLIDE 3
slide-4
SLIDE 4
slide-5
SLIDE 5
slide-6
SLIDE 6
slide-7
SLIDE 7
slide-8
SLIDE 8
slide-9
SLIDE 9
slide-10
SLIDE 10
slide-11
SLIDE 11
slide-12
SLIDE 12
slide-13
SLIDE 13
slide-14
SLIDE 14
slide-15
SLIDE 15
slide-16
SLIDE 16
slide-17
SLIDE 17
slide-18
SLIDE 18
slide-19
SLIDE 19

Road Map

Introduction Interface generation as optimization Modeling motor capabilities Adapting to motor capabilities Adapting to vision capabilities Adapting to motor and vision capabilities together Pilot User Study Lessons learned

slide-20
SLIDE 20

Automatic Interface Generation

  • Manually capture design knowledge as rules
  • Automatically apply rules to generate new

interfaces

slide-21
SLIDE 21

Automatic Interface Generation

  • Manually capture design knowledge as rules
  • Automatically apply rules to generate new

interfaces

  • Problems:
slide-22
SLIDE 22

Automatic Interface Generation

  • Manually capture design knowledge as rules
  • Automatically apply rules to generate new

interfaces

  • Problems:
  • New rule sets necessary for every device

type, interaction style or even screen size

slide-23
SLIDE 23

Automatic Interface Generation

  • Manually capture design knowledge as rules
  • Automatically apply rules to generate new

interfaces

  • Problems:
  • New rule sets necessary for every device

type, interaction style or even screen size

  • No good way to adapt to individual

preferences or needs

slide-24
SLIDE 24

Our Approach: User Interface Generation as Optimization

$( )=...

Driven by a “cost function”: and constraints

[Gajos and Weld, IUI’05]

slide-25
SLIDE 25

Find the interface that minimizes

$

without violating constraints

[Gajos and Weld, IUI’05]

slide-26
SLIDE 26

Single Algorithm -- Many Devices

slide-27
SLIDE 27

Single Algorithm -- Many Devices

slide-28
SLIDE 28

Robustly Adapting to Different Screen Sizes

slide-29
SLIDE 29

Robustly Adapting to Different Screen Sizes

slide-30
SLIDE 30
slide-31
SLIDE 31
slide-32
SLIDE 32

Highly Personalizable!

$( )=??

slide-33
SLIDE 33

Adapting to Preferences

$( )=

Preference estimate

[Gajos and Weld, UIST’05]

slide-34
SLIDE 34

Adapting to Motor Capabilities

$( )=

Estimated task completion time

slide-35
SLIDE 35

Road Map

Introduction Interface generation as optimization Modeling motor capabilities Adapting to motor capabilities Adapting to vision capabilities Adapting to motor and vision capabilities together

slide-36
SLIDE 36

Participants

Device used Impairment

Mouse Muscular Dystrophy Trackball Spinal cord injury Head Mouse Spinal cord injury Eye Tracker none Vocal Joystick none Vocal Joystick none Mouse none Track Pad none

Motor- impaired Unusual device

slide-37
SLIDE 37

Participants

Device used Impairment

Mouse Muscular Dystrophy Trackball Spinal cord injury Head Mouse Spinal cord injury Eye Tracker none Vocal Joystick none Vocal Joystick none Mouse none Track Pad none

Motor- impaired Unusual device

slide-38
SLIDE 38

Participants

Device used Impairment

Mouse Muscular Dystrophy Trackball Spinal cord injury Head Mouse Spinal cord injury Eye Tracker none Vocal Joystick none Vocal Joystick none Mouse none Track Pad none

Motor- impaired Unusual device

slide-39
SLIDE 39

Participants

Device used Impairment

Mouse Muscular Dystrophy Trackball Spinal cord injury Head Mouse Spinal cord injury Eye Tracker none Vocal Joystick none Vocal Joystick none Mouse none Track Pad none

Motor- impaired Unusual device

slide-40
SLIDE 40

Elicit a Person’s Motor Abilities Model

Pointing Dragging

slide-41
SLIDE 41

Predicting Movement Time

Actual Data

2000 4000 6000 8000 10000 12000 100 200 300 400 500 600 Distance to target size 15 size 25
slide-42
SLIDE 42 2000 4000 6000 8000 10000 12000 100 200 300 400 500 600 Distance to target Movement time (ms) size-10 size-15 size-25 size-40 size-60

Fitts’ Law Prediction

Predicting Movement Time

Actual Data

2000 4000 6000 8000 10000 12000 100 200 300 400 500 600 Distance to target size 15 size 25
slide-43
SLIDE 43 2000 4000 6000 8000 10000 12000 100 200 300 400 500 600 Distance to target Movement time (ms) size-10 size-15 size-25 size-40 size-60

Fitts’ Law Prediction

Predicting Movement Time

Actual Data

X

2000 4000 6000 8000 10000 12000 100 200 300 400 500 600 Distance to target size 15 size 25
slide-44
SLIDE 44

Custom Motor Capability Models

  • Supple++
  • Automatically selects the best set of

features for a custom regression model for each participant from: ID, 1, log(D), log(W), D, 1/W, W

  • Trains the models
slide-45
SLIDE 45 2000 4000 6000 8000 10000 12000 100 200 300 400 500 600 Distance to target Movement time (ms) size-10 size-15 size-25 size-40 size-60

Fitts’ Law Prediction

Predicting Movement Time

X

2000 4000 6000 8000 10000 12000 100 200 300 400 500 600 Distance to target size 15 size 25

Actual Data

slide-46
SLIDE 46 2000 4000 6000 8000 10000 12000 100 200 300 400 500 600 Distance to target

Custom Model

2000 4000 6000 8000 10000 12000 100 200 300 400 500 600 Distance to target Movement time (ms) size-10 size-15 size-25 size-40 size-60

Fitts’ Law Prediction

Predicting Movement Time

X

2000 4000 6000 8000 10000 12000 100 200 300 400 500 600 Distance to target size 15 size 25

Actual Data

slide-47
SLIDE 47

Road Map

Introduction Interface generation as optimization Modeling motor capabilities Adapting to motor capabilities Adapting to vision capabilities Adapting to motor and vision capabilities together Pilot User Study Lessons learned

slide-48
SLIDE 48

UI Building Blocks

  • Widgets
slide-49
SLIDE 49

UI Building Blocks

  • Widgets
  • Layout
slide-50
SLIDE 50

UI Building Blocks

  • Widgets
  • Layout
  • Structure (divide into windows, pop-ups, tabs)
slide-51
SLIDE 51

UI Building Blocks

  • Widgets
  • Layout
  • Structure (divide into windows, pop-ups, tabs)
  • Size of interactors
slide-52
SLIDE 52

UI Building Blocks

  • Widgets
  • Layout
  • Structure (divide into windows, pop-ups, tabs)
  • Size of interactors
slide-53
SLIDE 53 Light Level: : <int, [0,10]> Power: : bool Light: : { , } Light Bank: : { , , } Light ... Light ... A/V: : { , } Projector: : { , } Classroom: : { , , } Input: : <string, {data1,data2, video}> Vent: : <int, [0,3]> Power: : bool Screen: : bool

Optimization as Search

slide-54
SLIDE 54 Light Level: : <int, [0,10]> Power: : bool Light: : { , } Light Bank: : { , , } Light ... Light ... A/V: : { , } Projector: : { , } Classroom: : { , , } Input: : <string, {data1,data2, video}> Vent: : <int, [0,3]> Power: : bool Screen: : bool

Optimization as Search

slide-55
SLIDE 55 Light Level: : <int, [0,10]> Power: : bool Light: : { , } Light Bank: : { , , } Light ... Light ... A/V: : { , } Projector: : { , } Classroom: : { , , } Input: : <string, {data1,data2, video}> Vent: : <int, [0,3]> Power: : bool Screen: : bool

Optimization as Search

slide-56
SLIDE 56 Light Level: : <int, [0,10]> Power: : bool Light: : { , } Light Bank: : { , , } Light ... Light ... A/V: : { , } Projector: : { , } Classroom: : { , , } Input: : <string, {data1,data2, video}> Vent: : <int, [0,3]> Power: : bool Screen: : bool

Optimization as Search

slide-57
SLIDE 57 Light Level: : <int, [0,10]> Power: : bool Light: : { , } Light Bank: : { , , } Light ... Light ... A/V: : { , } Projector: : { , } Classroom: : { , , } Input: : <string, {data1,data2, video}> Vent: : <int, [0,3]> Power: : bool Screen: : bool

Optimization as Search

Right Center Left
slide-58
SLIDE 58

Optimization as Search

Light Level: : <int, [0,10]> Power: : bool Light: : { , } Light Bank: : { , , } Light ... Light ... A/V: : { , } Projector: : { , } Classroom: : { , , } Input: : <string, {data1,data2, video}> Vent: : <int, [0,3]> Power: : bool Screen: : bool Right Center Left
slide-59
SLIDE 59

Optimization as Search

Light Level: : <int, [0,10]> Power: : bool Light: : { , } Light Bank: : { , , } Light ... Light ... A/V: : { , } Projector: : { , } Classroom: : { , , } Input: : <string, {data1,data2, video}> Vent: : <int, [0,3]> Power: : bool Screen: : bool Right Center Left Light Level: : <int, [0,10]> Power: : bool Light: : { , } Light Bank: : { , , } Light ... Light ... A/V: : { , } Projector: : { , } Classroom: : { , , } Input: : <string, {data1,data2, video}> Vent: : <int, [0,3]> Power: : bool Screen: : bool
slide-60
SLIDE 60

Optimization as Search

Light Level: : <int, [0,10]> Power: : bool Light: : { , } Light Bank: : { , , } Light ... Light ... A/V: : { , } Projector: : { , } Classroom: : { , , } Input: : <string, {data1,data2, video}> Vent: : <int, [0,3]> Power: : bool Screen: : bool
slide-61
SLIDE 61

Optimization as Search

Light Level: : <int, [0,10]> Power: : bool Light: : { , } Light Bank: : { , , } Light ... Light ... A/V: : { , } Projector: : { , } Classroom: : { , , } Input: : <string, {data1,data2, video}> Vent: : <int, [0,3]> Power: : bool Screen: : bool
slide-62
SLIDE 62

Optimization as Search

Light Level: : <int, [0,10]> Power: : bool Light: : { , } Light Bank: : { , , } Light ... Light ... A/V: : { , } Projector: : { , } Classroom: : { , , } Input: : <string, {data1,data2, video}> Vent: : <int, [0,3]> Power: : bool Screen: : bool
slide-63
SLIDE 63

Optimization as Search

Light Level: : <int, [0,10]> Power: : bool Light: : { , } Light Bank: : { , , } Light ... Light ... A/V: : { , } Projector: : { , } Classroom: : { , , } Input: : <string, {data1,data2, video}> Vent: : <int, [0,3]> Power: : bool Screen: : bool

Enumerate all the options: ~370 years

slide-64
SLIDE 64

Optimization as Search

Light Level: : <int, [0,10]> Power: : bool Light: : { , } Light Bank: : { , , } Light ... Light ... A/V: : { , } Projector: : { , } Classroom: : { , , } Input: : <string, {data1,data2, video}> Vent: : <int, [0,3]> Power: : bool Screen: : bool

Enumerate all the options: ~370 years Supple: ~2 seconds

slide-65
SLIDE 65

Optimization as Search

Light Level: : <int, [0,10]> Power: : bool Light: : { , } Light Bank: : { , , } Light ... Light ... A/V: : { , } Projector: : { , } Classroom: : { , , } Input: : <string, {data1,data2, video}> Vent: : <int, [0,3]> Power: : bool Screen: : bool Right Center Left
slide-66
SLIDE 66

Optimization as Search

Light Level: : <int, [0,10]> Power: : bool Light: : { , } Light Bank: : { , , } Light ... Light ... A/V: : { , } Projector: : { , } Classroom: : { , , } Input: : <string, {data1,data2, video}> Vent: : <int, [0,3]> Power: : bool Screen: : bool Right Center Left
slide-67
SLIDE 67

Optimization as Search

Light Level: : <int, [0,10]> Power: : bool Light: : { , } Light Bank: : { , , } Light ... Light ... A/V: : { , } Projector: : { , } Classroom: : { , , } Input: : <string, {data1,data2, video}> Vent: : <int, [0,3]> Power: : bool Screen: : bool Right Center Left

$( ) $( ) $( )

$( )

slide-68
SLIDE 68

Optimization as Search

Light Level: : <int, [0,10]> Power: : bool Light: : { , } Light Bank: : { , , } Light ... Light ... A/V: : { , } Projector: : { , } Classroom: : { , , } Input: : <string, {data1,data2, video}> Vent: : <int, [0,3]> Power: : bool Screen: : bool Right Center Left

$( ) $( )

slide-69
SLIDE 69

Optimization as Search

Light Level: : <int, [0,10]> Power: : bool Light: : { , } Light Bank: : { , , } Light ... Light ... A/V: : { , } Projector: : { , } Classroom: : { , , } Input: : <string, {data1,data2, video}> Vent: : <int, [0,3]> Power: : bool Screen: : bool Right Center Left

MT( ) MT( )

slide-70
SLIDE 70

Optimization as Search

Light Level: : <int, [0,10]> Power: : bool Light: : { , } Light Bank: : { , , } Light ... Light ... A/V: : { , } Projector: : { , } Classroom: : { , , } Input: : <string, {data1,data2, video}> Vent: : <int, [0,3]> Power: : bool Screen: : bool Right Center Left

MT( )

?? ??

MT( )=? =?

slide-71
SLIDE 71

Optimization as Search

Light Level: : <int, [0,10]> Power: : bool Light: : { , } Light Bank: : { , , } Light ... Light ... A/V: : { , } Projector: : { , } Classroom: : { , , } Input: : <string, {data1,data2, video}> Vent: : <int, [0,3]> Power: : bool Screen: : bool Right Center Left

MT( )

?? ??

MT( )=? =?

Use Supple to optimize for motor abilities: hours or days

slide-72
SLIDE 72

Supple++

  • Novel optimization-based algorithm

computes a lower bound on movement time through a layout even before all the widgets are chosen

slide-73
SLIDE 73

Supple++

  • Novel optimization-based algorithm

computes a lower bound on movement time through a layout even before all the widgets are chosen Use Supple++ to optimize for motor abilities: seconds or minutes

slide-74
SLIDE 74

Results

slide-75
SLIDE 75
slide-76
SLIDE 76
slide-77
SLIDE 77
slide-78
SLIDE 78
slide-79
SLIDE 79
slide-80
SLIDE 80
slide-81
SLIDE 81

Able-bodied Impaired dexterity Low strength

slide-82
SLIDE 82

Road Map

Introduction Interface generation as optimization Modeling motor capabilities Adapting to motor capabilities Adapting to vision capabilities Adapting to motor and vision capabilities together Pilot User Study Lessons learned

slide-83
SLIDE 83
slide-84
SLIDE 84
slide-85
SLIDE 85
slide-86
SLIDE 86
slide-87
SLIDE 87

Popular OS Solution: “Large Fonts”

slide-88
SLIDE 88

Popular OS Solution: “Large Fonts”

slide-89
SLIDE 89

Supple++ Solution

slide-90
SLIDE 90

Supple++ Solution

slide-91
SLIDE 91

Supple++ Solution

slide-92
SLIDE 92
  • User selects desired

minimum size for all visual cues

  • User interface generation

constrained to meet desired visual cue size

  • Fast “resize and reflow”

Our Approach

slide-93
SLIDE 93
  • User selects desired

minimum size for all visual cues

  • User interface generation

constrained to meet desired visual cue size

  • Fast “resize and reflow”

Our Approach

slide-94
SLIDE 94
  • User selects desired

minimum size for all visual cues

  • User interface generation

constrained to meet desired visual cue size

  • Fast “resize and reflow”

Our Approach

slide-95
SLIDE 95
  • User selects desired

minimum size for all visual cues

  • User interface generation

constrained to meet desired visual cue size

  • Fast “resize and reflow”

Our Approach

Content arranged in tabs

slide-96
SLIDE 96

Road Map

Introduction Interface generation as optimization Modeling motor capabilities Adapting to motor capabilities Adapting to vision capabilities Adapting to motor and vision capabilities together Pilot User Study Lessons learned

slide-97
SLIDE 97

for “typical” users

slide-98
SLIDE 98

for “typical” users impaired dexterity

modified cost function

slide-99
SLIDE 99

modified constraints

for “typical” users impaired dexterity low vision

modified cost function

slide-100
SLIDE 100

modified constraints

for “typical” users impaired dexterity low vision low vision & impaired dexterity

modified cost function

slide-101
SLIDE 101

Results of a Pilot Study

slide-102
SLIDE 102

Results of a Pilot Study

  • Enabled interaction for one participant
slide-103
SLIDE 103

Results of a Pilot Study

  • Enabled interaction for one participant
  • For others, personalized interfaces improved

performance by 20%

slide-104
SLIDE 104

Results of a Pilot Study

  • Enabled interaction for one participant
  • For others, personalized interfaces improved

performance by 20% But

slide-105
SLIDE 105

Results of a Pilot Study

  • Enabled interaction for one participant
  • For others, personalized interfaces improved

performance by 20% But

  • Large variance
slide-106
SLIDE 106

Results of a Pilot Study

  • Enabled interaction for one participant
  • For others, personalized interfaces improved

performance by 20% But

  • Large variance
  • Better but not best
slide-107
SLIDE 107

Problems uncovered

  • List selection ≠ drag + point
slide-108
SLIDE 108

Problems uncovered

  • List selection ≠ drag + point

R2 = .09

slide-109
SLIDE 109

Modeling List Selection Times

direct model: R2 = .64

slide-110
SLIDE 110

Since this paper...

slide-111
SLIDE 111

Since this paper...

  • In the new study, we found that participants

with motor impairments

  • were consistently faster (by 26%) using

interfaces generated by Supple++

  • made 73% fewer errors
  • strongly preferred Supple++ interfaces

to manufacturers’ baselines

slide-112
SLIDE 112

Summary of Contributions

slide-113
SLIDE 113

Summary of Contributions

  • Supple++ for automatically generating user

interfaces adapted to user’s motor and vision capabilities

slide-114
SLIDE 114

Summary of Contributions

  • Supple++ for automatically generating user

interfaces adapted to user’s motor and vision capabilities

  • No expert necessary
slide-115
SLIDE 115

Summary of Contributions

  • Supple++ for automatically generating user

interfaces adapted to user’s motor and vision capabilities

  • No expert necessary
  • Custom regression models for individual motor

capabilities

slide-116
SLIDE 116

Summary of Contributions

  • Supple++ for automatically generating user

interfaces adapted to user’s motor and vision capabilities

  • No expert necessary
  • Custom regression models for individual motor

capabilities

  • Novel optimization-based algorithm for efficient

ability-based GUI generation

slide-117
SLIDE 117

Limitations & Future Work

slide-118
SLIDE 118

Limitations & Future Work

  • Limitations of modeling
slide-119
SLIDE 119

Limitations & Future Work

  • Limitations of modeling
  • Pointer-only interactions
slide-120
SLIDE 120

Limitations & Future Work

  • Limitations of modeling
  • Pointer-only interactions
  • Movement time as proxy for performance
slide-121
SLIDE 121

Limitations & Future Work

  • Limitations of modeling
  • Pointer-only interactions
  • Movement time as proxy for performance
  • How to address other concerns?
slide-122
SLIDE 122

Limitations & Future Work

  • Limitations of modeling
  • Pointer-only interactions
  • Movement time as proxy for performance
  • How to address other concerns?
  • Design tools
slide-123
SLIDE 123
  • SUPPLE:

http://supple.cs.washington.edu/

  • r Google : supple interfaces
  • Krzysztof Gajos:

kgajos@cs.washington.edu

  • Jacob Wobbrock:

wobbrock@u.washington.edu

  • Daniel Weld:

weld@cs.washington.edu

More Information