Fast And Robust Interface Generation for Ubiquitous Applications - - PowerPoint PPT Presentation

▶
fast and robust interface generation for ubiquitous
SMART_READER_LITE
LIVE PREVIEW

Fast And Robust Interface Generation for Ubiquitous Applications - - PowerPoint PPT Presentation

Apr 25, 2023 42 likes •1.15k views

Fast And Robust Interface Generation for Ubiquitous Applications The S UPPLE Project University of Washington, Seattle Krzysztof Gajos, David Christianson, Raphael Hoffmann, Tal Shaked, Kiera Henning, Jing Jing Long, and Daniel S. Weld

slide-1
SLIDE 1

Fast And Robust Interface Generation for Ubiquitous Applications

The SUPPLE Project University of Washington, Seattle Krzysztof Gajos, David Christianson, Raphael Hoffmann, Tal Shaked, Kiera Henning, Jing Jing Long, and Daniel S. Weld

slide-2
SLIDE 2

Fast And Robust Interface Generation for Ubiquitous Applications

Automatic, On The Fly

The SUPPLE Project University of Washington, Seattle Krzysztof Gajos, David Christianson, Raphael Hoffmann, Tal Shaked, Kiera Henning, Jing Jing Long, and Daniel S. Weld

slide-3
SLIDE 3

Motivation

slide-4
SLIDE 4

Motivation

slide-5
SLIDE 5

Motivation

slide-6
SLIDE 6

Motivation

slide-7
SLIDE 7

Motivation

slide-8
SLIDE 8

Motivation

  • Variety of display devices & interaction contexts

makes hand-designed interfaces expensive ⇒ Adapt to device characteristics

slide-9
SLIDE 9

Motivation

  • Variety of display devices & interaction contexts

makes hand-designed interfaces expensive ⇒ Adapt to device characteristics

  • Current interfaces: complex & “One size fits all”

⇒ Adapt to users and tasks

slide-10
SLIDE 10

Motivation

  • Variety of display devices & interaction contexts

makes hand-designed interfaces expensive ⇒ Adapt to device characteristics

  • Current interfaces: complex & “One size fits all”

⇒ Adapt to users and tasks ⇒ Automatic interface generation is a scalable solution

slide-11
SLIDE 11

Approach

  • Develop abstract representation for:
  • Interfaces
  • Display devices
  • Users
  • Automatically generate interfaces from the

abstractions

slide-12
SLIDE 12

SUPPLE Architecture

Display Device Model Target Device SUPPLE Interface Model Application

  • r

Appliance User Model User's Info Space

slide-13
SLIDE 13

SUPPLE Architecture

Display Device Model Target Device SUPPLE Interface Model Application

  • r

Appliance User Model User's Info Space

slide-14
SLIDE 14

Road Map

Motivation Modeling user interfaces in SUPPLE User interface generation as optimization Automatically adapting user interfaces A preliminary user study Adaptation in SUPPLE Customization support in SUPPLE Conclusions

slide-15
SLIDE 15
  • simple types:
  • media types:
  • containers:
  • derivative types:
  • vectors:
  • actions: τ1 → τ2

τ, Cτ

Modeling User Interfaces

vector(τ) int|f loat|string|bool {τi

i∈1...n}

image|interactiveMap

slide-16
SLIDE 16

Modeling User Interfaces

Light Level: τ: <int, [0,10]> Power: τ: bool Light: τ: { , } Light Bank: τ: { , , } Light ... Light ... A/V: τ: { , } Projector: τ: { , } Classroom: τ: { , , } Input: τ: <string, {data1,data2, video}> Vent: τ : <int, [0,3]> Power: τ : bool Screen: τ: bool

slide-17
SLIDE 17

Modeling User Interfaces

Light Level: τ: <int, [0,10]> Power: τ: bool Light: τ: { , } Light Bank: τ: { , , } Light ... Light ... A/V: τ: { , } Projector: τ: { , } Classroom: τ: { , , } Input: τ: <string, {data1,data2, video}> Vent: τ : <int, [0,3]> Power: τ : bool Screen: τ: bool

slide-18
SLIDE 18

Light Level: : <int, [0,10]> Power: : bool Light: : { , } Light Bank: : { , , } Light ... Light ... A/V: : { , } Projector: : { , } Classroom: : { , , } Input: : <string, {data1,data2, video}> Vent: : <int, [0,3]> Power: : bool Screen: : bool

Modeling User Interfaces

slide-19
SLIDE 19

Model (UI, Device, User) - Algorithm - Results - Personalization (Adaptation, Customization) - Arnauld - Consistency - Task Models

slide-20
SLIDE 20

Model (UI, Device, User) - Algorithm - Results - Personalization (Adaptation, Customization) - Arnauld - Consistency - Task Models

Click!

slide-21
SLIDE 21

Model (UI, Device, User) - Algorithm - Results - Personalization (Adaptation, Customization) - Arnauld - Consistency - Task Models

slide-22
SLIDE 22

Model (UI, Device, User) - Algorithm - Results - Personalization (Adaptation, Customization) - Arnauld - Consistency - Task Models

slide-23
SLIDE 23

Model (UI, Device, User) - Algorithm - Results - Personalization (Adaptation, Customization) - Arnauld - Consistency - Task Models

slide-24
SLIDE 24

Model (UI, Device, User) - Algorithm - Results - Personalization (Adaptation, Customization) - Arnauld - Consistency - Task Models

Click!

slide-25
SLIDE 25

Model (UI, Device, User) - Algorithm - Results - Personalization (Adaptation, Customization) - Arnauld - Consistency - Task Models

slide-26
SLIDE 26

Model (UI, Device, User) - Algorithm - Results - Personalization (Adaptation, Customization) - Arnauld - Consistency - Task Models

slide-27
SLIDE 27

Media Types

slide-28
SLIDE 28

Subtyping

slide-29
SLIDE 29

Subtyping

slide-30
SLIDE 30

Subtyping

Click!

slide-31
SLIDE 31
slide-32
SLIDE 32
slide-33
SLIDE 33
slide-34
SLIDE 34
slide-35
SLIDE 35

Road Map

Motivation Modeling user interfaces in SUPPLE User interface generation as optimization Automatically adapting user interfaces A preliminary user study Adaptation in SUPPLE Customization support in SUPPLE Conclusions

slide-36
SLIDE 36

Model (UI, Device, User) - Algorithm - Results - Personalization (Adaptation, Customization) - Arnauld - Consistency - Task Models

User Interface Generation as Optimization

slide-37
SLIDE 37

Model (UI, Device, User) - Algorithm - Results - Personalization (Adaptation, Customization) - Arnauld - Consistency - Task Models

  • Driven by a Cost function ($) -- estimated

user effort to manipulate a rendering of the interface

User Interface Generation as Optimization

slide-38
SLIDE 38

Model (UI, Device, User) - Algorithm - Results - Personalization (Adaptation, Customization) - Arnauld - Consistency - Task Models

  • Driven by a Cost function ($) -- estimated

user effort to manipulate a rendering of the interface

  • Cost function derived from device model

User Interface Generation as Optimization

slide-39
SLIDE 39

Model (UI, Device, User) - Algorithm - Results - Personalization (Adaptation, Customization) - Arnauld - Consistency - Task Models

  • Driven by a Cost function ($) -- estimated

user effort to manipulate a rendering of the interface

  • Cost function derived from device model
  • Algorithm: branch-and-bound search with

full constraint propagation at each step

User Interface Generation as Optimization

slide-40
SLIDE 40

Model (UI, Device, User) - Algorithm - Results - Personalization (Adaptation, Customization) - Arnauld - Consistency - Task Models

  • Driven by a Cost function ($) -- estimated

user effort to manipulate a rendering of the interface

  • Cost function derived from device model
  • Algorithm: branch-and-bound search with

full constraint propagation at each step

  • Performance: 0.2 - 2.0 seconds on a

desktop computer

User Interface Generation as Optimization

slide-41
SLIDE 41

Model (UI, Device, User) - Algorithm - Results - Personalization (Adaptation, Customization) - Arnauld - Consistency - Task Models

  • Driven by a Cost function ($) -- estimated

user effort to manipulate a rendering of the interface

  • Cost function derived from device model
  • Algorithm: branch-and-bound search with

full constraint propagation at each step

  • Performance: 0.2 - 2.0 seconds on a

desktop computer

User Interface Generation as Optimization

slide-42
SLIDE 42

User Interface Generation as Optimization

  • Flexible with respect to screen size
  • Versatile: Same algorithm for different

devices

  • Allows new concerns to be included in the

rendering process, e.g.:

  • Cross-device consistency
  • Adaptation to usage patterns
slide-43
SLIDE 43

Robustly Adapting to Different Screen Sizes

slide-44
SLIDE 44

Robustly Adapting to Different Screen Sizes

slide-45
SLIDE 45

Robustly Adapting to Different Screen Sizes

slide-46
SLIDE 46

Single Algorithm -- Many Devices

slide-47
SLIDE 47

Single Algorithm -- Many Devices

slide-48
SLIDE 48

Single Algorithm -- Many Devices

slide-49
SLIDE 49

Single Algorithm -- Many Devices

slide-50
SLIDE 50

Single Algorithm -- Many Devices

slide-51
SLIDE 51

Adapting To Usage Patterns

SUPPLE with an empty trace

slide-52
SLIDE 52

Adapting To Usage Patterns

SUPPLE with an empty trace SUPPLE with a

“lights-heavy” trace

slide-53
SLIDE 53

Road Map

Motivation Modeling user interfaces in SUPPLE User interface generation as optimization Automatically adapting user interfaces A preliminary user study Adaptation in SUPPLE Customization support in SUPPLE Conclusions

slide-54
SLIDE 54

Adapting To Usage Patterns By Complete Makeover

SUPPLE with an empty trace SUPPLE with a

“lights-heavy” trace

slide-55
SLIDE 55

Adapting To Usage Patterns By Complete Makeover

SUPPLE with an empty trace SUPPLE with a

“lights-heavy” trace

Is this the only way to adapt?

slide-56
SLIDE 56

Content Eliding

slide-57
SLIDE 57

Content Eliding

slide-58
SLIDE 58

Content Eliding

slide-59
SLIDE 59

Content Eliding

slide-60
SLIDE 60

Visual Popout

slide-61
SLIDE 61

Visual Popout

slide-62
SLIDE 62

Split Interfaces

slide-63
SLIDE 63

Split Interfaces

slide-64
SLIDE 64

Split Interfaces

slide-65
SLIDE 65

Adaptation Strategies

Efficiency Confusion

slide-66
SLIDE 66

Adaptation Strategies

Efficiency Confusion Efficiency Confusion

Complete Makeover

slide-67
SLIDE 67

Adaptation Strategies

Efficiency Confusion

Complete Makeover Content eliding

slide-68
SLIDE 68

Adaptation Strategies

Efficiency Confusion

Visual Popout Complete Makeover Content eliding

slide-69
SLIDE 69

Adaptation Strategies

Efficiency Confusion

Split Visual Popout Complete Makeover Content eliding

slide-70
SLIDE 70

Adaptation Strategies

Efficiency Confusion

Split Visual Popout Complete Makeover Content eliding

slide-71
SLIDE 71

Preliminary User Study

Split Interface

slide-72
SLIDE 72

Preliminary User Study

Split Interface

slide-73
SLIDE 73

Preliminary User Study

Split Interface

slide-74
SLIDE 74

Preliminary User Study

Split Interface

slide-75
SLIDE 75

Preliminary User Study

Split Interface Visual Popout Interface

slide-76
SLIDE 76

Preliminary User Study Results

slide-77
SLIDE 77

Preliminary User Study Results

  • Split Interface strongly preferred over

non-adaptive

slide-78
SLIDE 78

Preliminary User Study Results

  • Split Interface strongly preferred over

non-adaptive

  • Both adaptive interfaces slightly faster than

non-adaptive

slide-79
SLIDE 79

Preliminary User Study Results

  • Split Interface strongly preferred over

non-adaptive

  • Both adaptive interfaces slightly faster than

non-adaptive

  • But: Visual Popout interface often found

distracting

slide-80
SLIDE 80

Sneak Peak: Followup Study

with Mary Czerwinski and Desney Tan

slide-81
SLIDE 81

Sneak Peak: Followup Study

  • In a different setting, we tested three

adaptation strategies including Split Interface and Visual Popout Interface with Mary Czerwinski and Desney Tan

slide-82
SLIDE 82

Sneak Peak: Followup Study

  • In a different setting, we tested three

adaptation strategies including Split Interface and Visual Popout Interface

  • Users strongly preferred and were

significantly faster using Split Interface with Mary Czerwinski and Desney Tan

slide-83
SLIDE 83

Sneak Peak: Followup Study

  • In a different setting, we tested three

adaptation strategies including Split Interface and Visual Popout Interface

  • Users strongly preferred and were

significantly faster using Split Interface

  • Users strongly disliked

Visual Popout Interface with Mary Czerwinski and Desney Tan

slide-84
SLIDE 84

Split Interfaces in SUPPLE

  • Adapting by promoting hard-to-reach but

frequently used functionality

slide-85
SLIDE 85

Split Interfaces in SUPPLE

slide-86
SLIDE 86

Click!

Split Interfaces in SUPPLE

slide-87
SLIDE 87

Split Interfaces in SUPPLE

slide-88
SLIDE 88

Click!

Split Interfaces in SUPPLE

slide-89
SLIDE 89

Click!

Split Interfaces in SUPPLE

slide-90
SLIDE 90

Click! Click!

Split Interfaces in SUPPLE

slide-91
SLIDE 91

Split Interfaces in SUPPLE

slide-92
SLIDE 92

Click!

Split Interfaces in SUPPLE

slide-93
SLIDE 93

Split Interfaces in SUPPLE

slide-94
SLIDE 94

Split Interfaces in SUPPLE

Four extra clicks required just to print in landscape mode!

slide-95
SLIDE 95

Split Interfaces in SUPPLE

slide-96
SLIDE 96

Split Interfaces in SUPPLE

slide-97
SLIDE 97

Split Interfaces in SUPPLE

slide-98
SLIDE 98

System-

  • Vs. User-Initiated

Adaptation: Customization

  • Customizing any part of the interface with

drag and drop

  • Out of order undo
  • Generalization
slide-99
SLIDE 99

Customization

slide-100
SLIDE 100

Customization

slide-101
SLIDE 101

Customization

slide-102
SLIDE 102

Customization

slide-103
SLIDE 103

Customization

slide-104
SLIDE 104

Customization

slide-105
SLIDE 105

Road Map

Motivation Modeling user interfaces in SUPPLE User interface generation as optimization Automatically adapting user interfaces A preliminary user study Adaptation in SUPPLE Customization support in SUPPLE Conclusions

slide-106
SLIDE 106

Conclusions

  • A powerful, flexible and practical tool for

automatically generating user interfaces for multiple devices

  • Adaptation and customization offer the possibility
  • f creating custom-made UIs for each user
  • Support for distributed operation and caching

makes it practical even on small devices

slide-107
SLIDE 107

Can I Have It?

slide-108
SLIDE 108

Can I Have It?

  • We are awaiting permission to release

SUPPLE as an open source toolkit

  • Visit SUPPLE web site to sign up for the

user’s mailing list

Yes!

slide-109
SLIDE 109

Contributors

Daniel Weld Raphael Hoffmann Kiera Henning Jing Jing Long me Anthony Wu Dave Christianson Tal Shaked

slide-110
SLIDE 110

More Information

  • SUPPLE:

http://www.cs.washington.edu/ai/supple/ : supple interfaces

  • Krzysztof Gajos:

kgajos@cs.washington.edu

  • Daniel Weld:

weld@cs.washington.edu

slide-111
SLIDE 111
  • SUPPLE:

http://www.cs.washington.edu/ai/supple/ : supple interfaces

  • Krzysztof Gajos:

kgajos@cs.washington.edu

  • Daniel Weld:

weld@cs.washington.edu

More Information