Discovering Natural Language Commands in Multimodal Interfaces - - PowerPoint PPT Presentation

▶
discovering natural language commands in multimodal
SMART_READER_LITE
LIVE PREVIEW

Discovering Natural Language Commands in Multimodal Interfaces - - PowerPoint PPT Presentation

Nov 18, 2022 219 likes •880 views

Discovering Natural Language Commands in Multimodal Interfaces Arjun Srinivasan Mira Dontcheva Eytan Adar Seth Walker Speech-enabled multimodal interfaces are becoming popular (1) What operations can I perform? (2) How do I ask the

slide-1
SLIDE 1

Discovering Natural Language Commands in Multimodal Interfaces

Arjun Srinivasan Mira Dontcheva Eytan Adar Seth Walker

slide-2
SLIDE 2

Speech-enabled multimodal interfaces are becoming popular…

slide-3
SLIDE 3
slide-4
SLIDE 4
slide-5
SLIDE 5
slide-6
SLIDE 6
slide-7
SLIDE 7

(1) What operations can I perform? (2) How do I ask the system to perform them?

slide-8
SLIDE 8

Discoverability

(1) What operations can I perform? (2) How do I ask the system to perform them?

slide-9
SLIDE 9

Discoverability

(1) What operations can I perform? (2) How do I ask the system to perform them?

2nd most common challenge with Voice User Interfaces

Patterns for How Users Overcome Obstacles in Voice User Interfaces, Myers et al. CHI 2018

slide-10
SLIDE 10
slide-11
SLIDE 11

Can we leverage multimodal input to enhance discoverability by suggesting contextually-relevant natural language commands?

slide-12
SLIDE 12

Tooltips

slide-13
SLIDE 13
slide-14
SLIDE 14
slide-15
SLIDE 15
slide-16
SLIDE 16
slide-17
SLIDE 17
slide-18
SLIDE 18
slide-19
SLIDE 19
slide-20
SLIDE 20
slide-21
SLIDE 21
slide-22
SLIDE 22

When?

  • Onboarding
  • During a session
  • On failure
slide-23
SLIDE 23

When? What?

  • Onboarding
  • During a session
  • On failure
  • Number of commands
  • Coverage vs. Relevance
  • Complexity
  • Phrasing
  • Parameters
slide-24
SLIDE 24

When? What? Where?

  • Onboarding
  • During a session
  • On failure
  • Number of commands
  • Coverage vs. Relevance
  • Complexity
  • Phrasing
  • Parameters
  • Pop-up window
  • Tooltips
  • Embedded in GUI
  • Panels
slide-25
SLIDE 25

Three interface variants to present command suggestions

Exhaustive Embedded Adaptive

slide-26
SLIDE 26

Exhaustive Embedded Adaptive

slide-27
SLIDE 27

Exhaustive Embedded Adaptive

slide-28
SLIDE 28
slide-29
SLIDE 29
slide-30
SLIDE 30
slide-31
SLIDE 31
slide-32
SLIDE 32
slide-33
SLIDE 33

Exhaustive Embedded Adaptive

slide-34
SLIDE 34
slide-35
SLIDE 35
slide-36
SLIDE 36
slide-37
SLIDE 37

Exhaustive Embedded Adaptive

Command Suggestions

slide-38
SLIDE 38

Command Templates Filter, Rank, Parameterize Examples

slide-39
SLIDE 39

Add a name filter on target Make the name filter strength Set fill color to color Change border to color Set stroke size to size Make count copies Remove target Highlight entities in the image ...

Command Templates

slide-40
SLIDE 40

Add a name filter on target Make the name filter strength Set fill color to color Change border to color Set stroke size to size Make count copies Remove target Highlight entities in the image ... Change border to color Set fill color to color Set stroke size to size Make count copies ...

Command Templates Filter, Rank, Parameterize

color = [red, blue, …] size = [1-10] ...

slide-41
SLIDE 41

Add a name filter on target Make the name filter strength Set fill color to color Change border to color Set stroke size to size Make count copies Remove target Highlight entities in the image ... Change border to color Set fill color to color Set stroke size to size Make count copies ... Change border to blue Set fill color to red Set stroke size to 10 Make 5 copies ...

Command Templates Filter, Rank, Parameterize Examples

color = [red, blue, …] size = [1-10] ...

slide-42
SLIDE 42

Add a name filter on target Make the name filter strength Set fill color to color Change border to color Set stroke size to size Make count copies Remove target Highlight entities in the image ... Change border to color Set fill color to color Set stroke size to size Make count copies ... Change border to blue Set fill color to red Set stroke size to 10 Make 5 copies ...

Command Templates Filter, Rank, Parameterize Examples

color = [red, blue, …] size = [1-10] ...

slide-43
SLIDE 43

Color this green Add a red stroke Make 2 copies Delete

slide-44
SLIDE 44

Copy Available Operations Fill Border Delete Add Effect

slide-45
SLIDE 45

Available Operations

Operation Selection Target Type Usage Freq. Display Freq.

Fill Border Copy Delete Add Effect

slide-46
SLIDE 46

Available Operations

Operation Selection

Phrasing Templates

Target Type Usage Freq. Display Freq.

Delete Border Copy Fill

slide-47
SLIDE 47

Available Operations

Operation Selection

Phrasing Templates

Target Type Usage Freq. Display Freq. Color this ___ Change color to ___ Set the fill to ___

Delete Border Copy Fill

Change fill of ___ to ___ …

slide-48
SLIDE 48

Available Operations

Operation Selection

Phrasing Templates

Target Type Usage Freq. Display Freq. Color this ___ Change color to ___ Set the fill to ___

Delete Border Copy Fill

Change fill of ___ to ___ … Template Selection & Parameterization Input Type Usage Freq. Display Freq. Target State

slide-49
SLIDE 49

Available Operations

Operation Selection

Phrasing Templates

Target Type Usage Freq. Display Freq. Color this ___ Change color to ___ Set the fill to ___

Delete Border Copy Fill

Change fill of ___ to ___ … Template Selection & Parameterization Input Type Usage Freq. Display Freq. Target State ___ = [blue, green, red, …]

slide-50
SLIDE 50

Available Operations

Operation Selection

Phrasing Templates

Target Type Usage Freq. Display Freq. Color this ___ Change color to ___ Set the fill to ___

Delete Border Copy Fill

Change fill of ___ to ___ … Template Selection & Parameterization Input Type Usage Freq. Display Freq. Target State

Examples

Color this green

___ = [blue, green, red, …]

slide-51
SLIDE 51

Available Operations

Operation Selection

Phrasing Templates

Target Type Usage Freq. Display Freq.

Delete Border Copy Fill

Template Selection & Parameterization Input Type Usage Freq. Display Freq. Target State

Examples

Color this green Add a red stroke Make 2 copies Delete

slide-52
SLIDE 52

Evaluation

slide-53
SLIDE 53

Evaluation

  • Between-subjects online study with 24 participants on UserTesting.com
  • Platform: Chrome running on a touch-enabled Microsoft Surface Pro
  • Minimal Training: Short videos about the basic interface and how to

invoke suggestions (no details about available operations and speech commands)

slide-54
SLIDE 54

Evaluation

  • Task: Three before-after image editing tasks
  • Duration: 32 min (avg.)
  • Compensation: $10
slide-55
SLIDE 55

(source) (target)

Edit the image on the left to make it look like the image on the right. Note that it is okay if your

  • utput does not look exactly the same as the target image below but try to make it look as similar

as possible.

slide-56
SLIDE 56

Speech Usage Summary

  • Total of 834 spoken commands issued (avg. 49) during 17/24

sessions (6 exhaustive, 5 adaptive, 6 embedded)

slide-57
SLIDE 57

Speech Command Failures

  • 369/834 (44%) spoken commands failed:

Error % Error Type 65% Speech recognition & recording errors 18% Phrasing errors 7% Operation-object mapping errors 5% Unsupported operations 5% Parameter errors

slide-58
SLIDE 58

Speech Command Failures

  • 369/834 (44%) spoken commands failed:

Error % Error Type 65% Speech recognition & recording errors 18% Phrasing errors 7% Operation-object mapping errors 5% Unsupported operations 5% Parameter errors

slide-59
SLIDE 59

Suggestions encourage and aid natural language interaction

Exhaustive (avg.) Embedded (avg.) Adaptive (avg.) Overall (avg.) Suggestions helped me learn how to talk to the system 4 3.67 4.4 4.02 Suggestions encouraged me to talk to the system 3.83 3.67 4.2 3.88

*scores between 1-5 5 is “strongly agree”

slide-60
SLIDE 60

Explanations for domain specific commands

  • Suggestions do not overcome lack of domain knowledge
slide-61
SLIDE 61

Explanations for domain specific commands

  • Command suggestions as interactive widgets
slide-62
SLIDE 62

Future work

  • Validating framework in other application

domains (e.g. data visualization)

PixelTone: A Multimodal Interface for Image Editing Laput et al., CHI 2013

  • Supporting additional command

types (e.g. gesture + speech)

slide-63
SLIDE 63

Conclusions

  • Contextual command suggestions aid discoverability and encourage

natural language interaction

  • Direct manipulation can be used to teach natural language interaction
slide-64
SLIDE 64

Thank you

  • Contextual command suggestions aid discoverability

and encourage natural language interaction

  • Direct manipulation can be used to teach natural

language interaction

Arjun Srinivasan ( @10_arjun) Mira Dontcheva Eytan Adar Seth Walker

bit.ly/

voice-hints

slide-65
SLIDE 65