Predictive Interaction Jeffrey Heer @jeffrey_heer U. Washington / - - PowerPoint PPT Presentation

▶
predictive interaction
SMART_READER_LITE
LIVE PREVIEW

Predictive Interaction Jeffrey Heer @jeffrey_heer U. Washington / - - PowerPoint PPT Presentation

May 29, 2023 409 likes •1.61k views

Predictive Interaction Jeffrey Heer @jeffrey_heer U. Washington / Trifacta My software doesnt know what Im trying to do. What if it did? Spell / Grammar Checker Hints of Intelligent Interaction Type-ahead uses context and data to

slide-1
SLIDE 1

Jeffrey Heer @jeffrey_heer

  • U. Washington / Trifacta

Predictive Interaction

slide-2
SLIDE 2
slide-3
SLIDE 3

My software doesn’t know what I’m trying to do.

slide-4
SLIDE 4

What if it did?

slide-5
SLIDE 5
slide-6
SLIDE 6

Spell / Grammar Checker

slide-7
SLIDE 7

Type-ahead uses context and data to predict search terms and preview results.

Hints of Intelligent Interaction

slide-8
SLIDE 8
slide-9
SLIDE 9
slide-10
SLIDE 10
slide-11
SLIDE 11

Search Engine Query

Search Query Auto-Complete

slide-12
SLIDE 12

Search Engine Query

Search Query Auto-Complete

Textbox

slide-13
SLIDE 13

Search Engine Query

Search Query Auto-Complete

Textbox Response

type

slide-14
SLIDE 14

Search Engine Query

Search Query Auto-Complete

Textbox Response Suggestions

type predict

slide-15
SLIDE 15

Search Engine Query

Search Query Auto-Complete

Textbox Query Response Suggestions

pick type predict

slide-16
SLIDE 16

Search Engine Query

Search Query Auto-Complete

Textbox Query Response Suggestions

pick type predict

slide-17
SLIDE 17

Search Engine Query

Search Query Auto-Complete

Textbox Query Response Suggestions

pick type

GUIDE

predict

slide-18
SLIDE 18

Search Engine Query

Search Query Auto-Complete

Textbox Query Response Suggestions

pick type

GUIDE DECIDE

predict

slide-19
SLIDE 19

Search Engine Query

Search Query Auto-Complete

Textbox Query Response Suggestions

pick type

GUIDE DECIDE

predict

The input and output domains are the same: text.

slide-20
SLIDE 20

Search Engine Query

Search Query Auto-Complete

Textbox Query Response Suggestions

pick type

GUIDE DECIDE

predict

What about more complex input/output relations?

slide-21
SLIDE 21

Predictive Interaction Goals

Accelerate successful task completion. Scale to large data or batch repetition. Support discovery and ambiguous intent. Long-term learning and improvement.

slide-22
SLIDE 22

Predictive Interaction Goals

Accelerate successful task completion. Scale to large data or batch repetition. Support discovery and ambiguous intent. Long-term learning and improvement.

Strategy

Model user interface actions in a domain-specific language (DSL). Leverage the language to (1) predict potential actions, and (2) decouple UI from underlying runtime.

slide-23
SLIDE 23

Predictive Interaction Goals

Accelerate successful task completion. Scale to large data or batch repetition. Support discovery and ambiguous intent. Long-term learning and improvement.

Strategy

Model user interface actions in a domain-specific language (DSL). Leverage the language to (1) predict potential actions, and (2) decouple UI from underlying runtime.

slide-24
SLIDE 24

Predictive Interaction Goals

Accelerate successful task completion. Scale to large data or batch repetition. Support discovery and ambiguous intent. Long-term learning and improvement.

Strategy

Model user interface actions in a domain-specific language (DSL). Leverage the language to (1) predict potential actions, and (2) decouple UI from underlying runtime.

slide-25
SLIDE 25

E X A M P L E :

Data Visualization

slide-26
SLIDE 26
slide-27
SLIDE 27
slide-28
SLIDE 28
slide-29
SLIDE 29
slide-30
SLIDE 30
slide-31
SLIDE 31
slide-32
SLIDE 32
slide-33
SLIDE 33
slide-34
SLIDE 34

x \ +

slide-35
SLIDE 35

x \ +

—> GROUP BY Category, Region, Segment

slide-36
SLIDE 36

VizQL: A DSL for Tabular Visualization

slide-37
SLIDE 37

VizQL: A DSL for Tabular Visualization

Operators: concatenation (+) cross product (x) nest (\) Operands: Ordinal fields Quantitative fields

slide-38
SLIDE 38

VizQL: A DSL for Tabular Visualization

Operators: concatenation (+) cross product (x) nest (\) Operands: Ordinal fields Quantitative fields The operators (+, x, \) and operands (O, Q) provide an algebra for tabular visualization. Algebraic statements are then compiled to: Queries: selection, projection, group-by… Visualizations: partitions, visual encodings Users make statements via drag-and-drop This specifies parameters, not operators! Operators inferred by data type (O, Q)

slide-39
SLIDE 39

Domain Specific Language (DSL)

Mapping from Textual to Visual Languages

Data Vis

VizQL

slide-40
SLIDE 40

Domain Specific Language (DSL)

Mapping from Textual to Visual Languages

Data Vis

VizQL

Visualization and Interaction

Schema Shelves

drag-and-drop

slide-41
SLIDE 41

Domain Specific Language (DSL)

Mapping from Textual to Visual Languages

Data Vis

VizQL

Lift

Visualization and Interaction

Schema Shelves

drag-and-drop

slide-42
SLIDE 42

Domain Specific Language (DSL)

Mapping from Textual to Visual Languages

Data Vis

VizQL

Lift

Visualization and Interaction

Schema Shelves

drag-and-drop

Ground

slide-43
SLIDE 43

Domain Specific Language (DSL)

Mapping from Textual to Visual Languages

Data Vis

VizQL

Lift

Visualization and Interaction

Schema Shelves

drag-and-drop

Ground DSL = Lift ∘ Interact ∘ Ground

slide-44
SLIDE 44

Domain Specific Language (DSL)

Are the Languages Isomorphic?

Data Vis

VizQL

Lift

Visualization and Interaction

Schema Shelves

drag-and-drop

Ground

slide-45
SLIDE 45

Domain Specific Language (DSL)

Are the Languages Isomorphic?

Data Vis

VizQL

Lift

Visualization and Interaction

Schema Shelves

drag-and-drop

Ground

O + O? O x O?

slide-46
SLIDE 46

Domain Specific Language (DSL)

Are the Languages Isomorphic?

Data Vis

VizQL

Lift

Visualization and Interaction

Schema Shelves

drag-and-drop

Ground

O + O? O x O?

DSL more expressive than UI

slide-47
SLIDE 47

E X A M P L E :

Data Wrangling

slide-48
SLIDE 48

I spend more than half of my time integrating, cleansing and transforming data without doing any actual analysis. Most of the time I’m lucky if I get to do any “analysis” at all.

Anonymous Data Scientist from our 2012 interview study

slide-49
SLIDE 49
slide-50
SLIDE 50
slide-51
SLIDE 51
slide-52
SLIDE 52

DataWrangler

Wrangler: Interactive Visual Specification of Data Transformation Scripts Sean Kandel, Joseph M. Hellerstein, et al. CHI’11

slide-53
SLIDE 53
slide-54
SLIDE 54
slide-55
SLIDE 55
slide-56
SLIDE 56
slide-57
SLIDE 57
slide-58
SLIDE 58
slide-59
SLIDE 59
slide-60
SLIDE 60
slide-61
SLIDE 61
slide-62
SLIDE 62

Visualization and Interaction Data Transformation Code

Traditional Specification

slide-63
SLIDE 63

Visualization and Interaction Data Transformation Code

User authors a draft transformation script

1.

Traditional Specification

slide-64
SLIDE 64

Visualization and Interaction Data Transformation Code

User authors a draft transformation script User tests the script on a small amount of data

1. 2.

Traditional Specification

slide-65
SLIDE 65

Visualization and Interaction Data Transformation Code

User authors a draft transformation script User tests the script on a small amount of data User inspects output data to assess effects

1. 2. 3.

Traditional Specification

slide-66
SLIDE 66

Visualization and Interaction Data Transformation Code

Predictive Interaction

slide-67
SLIDE 67

Visualization and Interaction Data Transformation Code

Predictive Interaction

slide-68
SLIDE 68

Visualization and Interaction Data Transformation Code

User highlights features of a data visualization

1.

Predictive Interaction

slide-69
SLIDE 69

Visualization and Interaction Data Transformation Code

User highlights features of a data visualization Algorithms predict distribution over DSL statements

1. 2.

Predictive Interaction

slide-70
SLIDE 70

Visualization and Interaction Data Transformation Code

User highlights features of a data visualization Data previews allow user to choose, adjust and confirm Algorithms predict distribution over DSL statements

1. 3. 2.

Predictive Interaction

slide-71
SLIDE 71

Wrangle Language Building Blocks

Transforms Split Extract Filter Derive Header Pivot Aggregate Join Union …

slide-72
SLIDE 72

Wrangle Language Building Blocks

Transforms Split Extract Filter Derive Header Pivot Aggregate Join Union … Parameter Types Text Selection Column Selection Row Selection Formula Enumeration Number String Boolean

slide-73
SLIDE 73

Text Selection: A Language within a Language

Transforms Split Extract Filter Derive Header Pivot Aggregate Join Union … Parameter Types Text Selection Column Selection Row Selection Formula Enumeration Number String Boolean

slide-74
SLIDE 74
slide-75
SLIDE 75
slide-76
SLIDE 76

/(?<=adtam_source\=)[^\&]*(?=\&)/

slide-77
SLIDE 77

/(?<=adtam_source\=)[^\&]*(?=\&)/

Look-behind Look-ahead What (not) to match

slide-78
SLIDE 78

/(?<=adtam_source\=)[^\&]*(?=\&)/

Escaped Literal Characters Control Characters

slide-79
SLIDE 79

Write once, read never.

slide-80
SLIDE 80

after: ‘adtam_source=’ before: ‘&’

slide-81
SLIDE 81
slide-82
SLIDE 82

Text Selection: A Language within a Language

Transforms Split Extract Filter Derive Header Pivot Aggregate Join Union … Parameter Types Text Selection Column Selection Row Selection Formula Enumeration Number String Boolean

slide-83
SLIDE 83

Text Selection: A Language within a Language

Transforms Split Extract Filter Derive Header Pivot Aggregate Join Union … Parameter Types Text Selection Column Selection Row Selection Formula Enumeration Number String Boolean Text Selection Prepositions

  • n

from / to after / before

slide-84
SLIDE 84

Text Selection: A Language within a Language

Transforms Split Extract Filter Derive Header Pivot Aggregate Join Union … Parameter Types Text Selection Column Selection Row Selection Formula Enumeration Number String Boolean Text Selection Prepositions

  • n

from / to after / before Inference Procedure

slide-85
SLIDE 85

Text Selection: A Language within a Language

Transforms Split Extract Filter Derive Header Pivot Aggregate Join Union … Parameter Types Text Selection Column Selection Row Selection Formula Enumeration Number String Boolean Text Selection Prepositions

  • n

from / to after / before Inference Procedure

  • 1. User Selects Text(s)
slide-86
SLIDE 86

Text Selection: A Language within a Language

Transforms Split Extract Filter Derive Header Pivot Aggregate Join Union … Parameter Types Text Selection Column Selection Row Selection Formula Enumeration Number String Boolean Text Selection Prepositions

  • n

from / to after / before Inference Procedure

  • 1. User Selects Text(s)
  • 2. Tokenize / Generalize
slide-87
SLIDE 87

Text Selection: A Language within a Language

Transforms Split Extract Filter Derive Header Pivot Aggregate Join Union … Parameter Types Text Selection Column Selection Row Selection Formula Enumeration Number String Boolean Text Selection Prepositions

  • n

from / to after / before Inference Procedure

  • 1. User Selects Text(s)
  • 2. Tokenize / Generalize
  • 3. Generate Clauses
slide-88
SLIDE 88

Text Selection: A Language within a Language

Transforms Split Extract Filter Derive Header Pivot Aggregate Join Union … Parameter Types Text Selection Column Selection Row Selection Formula Enumeration Number String Boolean Text Selection Prepositions

  • n

from / to after / before Inference Procedure

  • 1. User Selects Text(s)
  • 2. Tokenize / Generalize
  • 3. Generate Clauses
  • 4. Combine Clauses
slide-89
SLIDE 89

Text Selection: A Language within a Language

Transforms Split Extract Filter Derive Header Pivot Aggregate Join Union … Parameter Types Text Selection Column Selection Row Selection Formula Enumeration Number String Boolean Text Selection Prepositions

  • n

from / to after / before Inference Procedure

  • 1. User Selects Text(s)
  • 2. Tokenize / Generalize
  • 3. Generate Clauses
  • 4. Combine Clauses
  • 5. Filter & Rank Patterns
slide-90
SLIDE 90

vs.

slide-91
SLIDE 91

Wrangle Language Building Blocks

Transforms Split Extract Filter Derive Header Pivot Aggregate Join Union … Parameter Types Text Selection Column Selection Row Selection Formula Enumeration Number String Boolean

slide-92
SLIDE 92

Wrangle Language Building Blocks

Transforms Split Extract Filter Derive Header Pivot Aggregate Join Union … Parameter Types Text Selection Column Selection Row Selection Formula Enumeration Number String Boolean

slide-93
SLIDE 93

Wrangle Language Building Blocks

Transforms Split Extract Filter Derive Header Pivot Aggregate Join Union … Parameter Types Text Selection Column Selection Row Selection Formula Enumeration Number String Boolean Inference Procedure

slide-94
SLIDE 94

Wrangle Language Building Blocks

Transforms Split Extract Filter Derive Header Pivot Aggregate Join Union … Parameter Types Text Selection Column Selection Row Selection Formula Enumeration Number String Boolean Inference Procedure

  • 1. User Makes Selection(s)
slide-95
SLIDE 95

Wrangle Language Building Blocks

Transforms Split Extract Filter Derive Header Pivot Aggregate Join Union … Parameter Types Text Selection Column Selection Row Selection Formula Enumeration Number String Boolean Inference Procedure

  • 1. User Makes Selection(s)
  • 2. Infer Parameter Sets
slide-96
SLIDE 96

Wrangle Language Building Blocks

Transforms Split Extract Filter Derive Header Pivot Aggregate Join Union … Parameter Types Text Selection Column Selection Row Selection Formula Enumeration Number String Boolean Inference Procedure

  • 1. User Makes Selection(s)
  • 2. Infer Parameter Sets
  • 3. Generate Compatible Transforms
slide-97
SLIDE 97

Wrangle Language Building Blocks

Transforms Split Extract Filter Derive Header Pivot Aggregate Join Union … Parameter Types Text Selection Column Selection Row Selection Formula Enumeration Number String Boolean Inference Procedure

  • 1. User Makes Selection(s)
  • 2. Infer Parameter Sets
  • 3. Generate Compatible Transforms
  • 4. Rank & Cluster Transforms
slide-98
SLIDE 98

Wrangle Language Building Blocks

Transforms Split Extract Filter Derive Header Pivot Aggregate Join Union … Parameter Types Text Selection Column Selection Row Selection Formula Enumeration Number String Boolean Inference Procedure

  • 1. User Makes Selection(s)
  • 2. Infer Parameter Sets
  • 3. Generate Compatible Transforms
  • 4. Rank & Cluster Transforms
  • 5. Present Top Results
slide-99
SLIDE 99

Predictive Interaction

slide-100
SLIDE 100

Search Engine Query

Auto-Complete

Textbox Query Response Suggestions

pick type

GUIDE DECIDE

predict

slide-101
SLIDE 101

Domain Specific Language (DSL) Visualization and Interaction

Lifting from DSL to Visual Language

Data Output

write code, compile, run

View Result

visualize interact

Lift Ground

compile

slide-102
SLIDE 102

Domain Specific Language (DSL)

Predictive Interaction

Data Output

write code, compile, run

slide-103
SLIDE 103

Domain Specific Language (DSL)

Predictive Interaction

Data Output

write code, compile, run visualize

Lift

slide-104
SLIDE 104

Domain Specific Language (DSL) Visualization and Interaction

Predictive Interaction

Data Output

write code, compile, run

View

visualize

Lift

slide-105
SLIDE 105

Domain Specific Language (DSL) Visualization and Interaction

Predictive Interaction

Data Output

write code, compile, run

View

visualize

Response

interact

Lift

slide-106
SLIDE 106

Domain Specific Language (DSL) Visualization and Interaction

Predictive Interaction

Data Output

write code, compile, run

View

visualize

Response

interact predict

Lift

slide-107
SLIDE 107

Domain Specific Language (DSL) Visualization and Interaction

Predictive Interaction

Data Output

write code, compile, run

View

visualize

Response

interact predict generate

Lift

slide-108
SLIDE 108

Domain Specific Language (DSL) Visualization and Interaction

Predictive Interaction

Data Output

write code, compile, run

View

visualize

Response Preview

interact predict generate present

Lift

slide-109
SLIDE 109

Domain Specific Language (DSL) Visualization and Interaction

Predictive Interaction

Data Output

write code, compile, run

View Result

visualize

Response Preview

pick interact predict generate present

Lift

slide-110
SLIDE 110

Domain Specific Language (DSL) Visualization and Interaction

Predictive Interaction

Data Output

write code, compile, run

View Result

visualize compile

Response Preview

pick interact predict generate present

Lift Ground

slide-111
SLIDE 111

Domain Specific Language (DSL) Visualization and Interaction

Predictive Interaction

Data Output

write code, compile, run

View Result

visualize compile

Response Preview

pick interact predict

GUIDE

generate present

Lift Ground

slide-112
SLIDE 112

Domain Specific Language (DSL) Visualization and Interaction

Predictive Interaction

Data Output

write code, compile, run

View Result

visualize compile

Response Preview

pick interact predict

GUIDE DECIDE

generate present

Lift Ground

slide-113
SLIDE 113

Why Domain-Specific Languages?

Model the task (often as a sequence). Formalism for reasoning about actions. Provides means of learning from usage. Can be re-applied to new inputs. Cross-compile to different runtimes.

Necessary Components:

  • 1. Content Representations
  • 2. Language Model
  • 3. Preview Mechanisms
slide-114
SLIDE 114

Language Design Considerations

  • Expressivity. Supports the tasks.

Problem domain fit. Nouns and verbs match domain understanding. Small surface area. Permits tractable inference, less for users to learn. Bootstrap ranking. Can the language model provide useful suggestions without extensive training data? Interfaces that learn. Improving prediction by observing user behavior.

slide-115
SLIDE 115

Research Directions

slide-116
SLIDE 116

Opportunities & Challenges

Applicable Problem Domains & Language Designs

slide-117
SLIDE 117
  • Voyager. Kanit Wongsuphasawat, Dominik Moritz et al. InfoVis’15, CHI’17
slide-118
SLIDE 118

Opportunities & Challenges

Applicable Problem Domains & Language Designs Mixed-Initiative Interaction, Proactive Suggestions User Performance Cliffs, Error Handling Learning from Usage; How to Interpret Observed Input? Reusable Abstractions for UI, Language & Inference?

slide-119
SLIDE 119

Jeffrey Heer @jeffrey_heer

  • U. Washington / Trifacta

Predictive Interaction