Eric Lecolinet LTCI Tlcom ParisTech - - PowerPoint PPT Presentation

Eric Lecolinet– LTCI Télécom ParisTech 1

HABILITATION A DIRIGER DES RECHERCHES

Eric Lecolinet LTCI – Télécom-ParisTech 10 December 2018

Eric Lecolinet– LTCI Télécom ParisTech 2

Research domains

§ Cursive script recognition § Graphical toolkits and software architecture § Information visualization § Tactile feedback § Target acquisition on small devices § Augmented reality § Wall-sized displays

§ Gestural interaction

Preamble

Eric Lecolinet– LTCI Télécom ParisTech 3

Why using gestures?

Eric Lecolinet– LTCI Télécom ParisTech 4

Analogy

§ A thing which is comparable to something else in significant respects ‘works of art were seen as an analogy for works of nature’

(oxforddictionaries.com)

Analogical user interfaces

Eric Lecolinet– LTCI Télécom ParisTech 5

Idea

§ Metaphors make UIs 'intuitive' § "Intuitive = familiar" [Raskin94] § Tremendously successful!

Analogical user interfaces

Eric Lecolinet– LTCI Télécom ParisTech 6

Analogical user interfaces

Limitation

§ Efficient for novice users § Not so for expert users § Everyone is an expert of something!

Eric Lecolinet– LTCI Télécom ParisTech 7

Analogical user interfaces

Reasons

§ Don't leverage user knowledge § Based on recognition: (search) / point / recognize / click § Transient objects § Limited syntactic capabilities

Eric Lecolinet– LTCI Télécom ParisTech 8

Small devices

§ Occlusion, accuracy § Mobility conditions § Eyes-free interaction

Large devices

§ Large gestures § Users must constantly walk

Solution?

Analogical user interfaces

Eric Lecolinet– LTCI Télécom ParisTech 9

Limitations

§ Only on the PC § Explicit learning (different modality) § Limited expressivity (name collisions) § Hard to customize

Consequence

§ Largely underused [Lane05]

Hotkeys

Eric Lecolinet– LTCI Télécom ParisTech 10

Paradox of the active user [Carroll87]

§ Users tend to favor short-term solutions

• Even if they know more efficient methods exist [Krisler08]
• True in economics [Goodwin18] ... and HCI!

Changing modalities

§ Performance drop [Scarr11]

Solution?

Users are not rational!

Eric Lecolinet– LTCI Télécom ParisTech 11

Speech interaction

§ Rely on recall of prior knowledge § Direct access § Rich syntax

Recall vs. recognition

Eric Lecolinet– LTCI Télécom ParisTech 12

Limitations

§ Slow § Privacy concerns § So 'natural'?

Solution?

Recall vs. recognition

Eric Lecolinet– LTCI Télécom ParisTech 13

Advantages

§ Recall => direct access § Fast § Possibly eyes-free § Limited conflicts

Problem

§ "Most gestures are neither natural nor easy to learn or remember" [Norman10],

Solution?

Gestural interaction!

Eric Lecolinet– LTCI Télécom ParisTech 14

User-defined gestures

§ Elicitation studies § "Ask users to perform the gestures causing given effects" [Wobbrock09]

Limitations

§ Few commands elicit agreement § Especially abstract commands

'Natural' gestures

Eric Lecolinet– LTCI Télécom ParisTech 15

Nature dimension [Wobbrock09]

§ Physical gestures § Metaphorical gestures § Symbolic gestures § Abstract gestures

'Natural' gestures

Eric Lecolinet– LTCI Télécom ParisTech 16

Nature dimension [Wobbrock09]

§ Physical gestures § Metaphorical gestures § Symbolic gestures § Abstract gestures

'Natural' gestures

physical or metaphorical? symbolic or abstract? abstract, symbolic, metaphorical?

Eric Lecolinet– LTCI Télécom ParisTech 17

Analogue-abstract spectrum [Zhai12]

§ "The more analogous gestures are to the user’s prior experience, the easier they are to learn"

'Natural' gestures

Eric Lecolinet– LTCI Télécom ParisTech 18

Analogue-abstract spectrum [Zhai12]

§ "The more analogous gestures are to the user’s prior experience, the easier they are to learn"

It's all about familiarity!

§ "Intuitive = familiar" [Raskin94] § 'Natural' have nothing to do with 'nature'! § Relative/evolving notion: users have a brain, they do learn!

'Natural' gestures

Eric Lecolinet– LTCI Télécom ParisTech 19

Useless

§ Whatever can be done easily § Avoid solving problems that do not exist!

Useful

§ Small and large devices § User or application specific tasks:

• Find and Select Previous
• Open the NYT webpage
• Open XCode on this file
• Show CNN on my TV, etc.

Useful gestures

Eric Lecolinet– LTCI Télécom ParisTech 20

Research questions

§ How to to increase the expressivity of gestures? § How to improve the learning and retention of gestures? (and of gesture/command associations) § How to design efficient + user-friendly interaction techniques?

Useful gestures must be learnt!

Eric Lecolinet– LTCI Télécom ParisTech 21

§ Expressivity of gestures § Memorization of gestures & gesture/command associations § Conclusion § Perspectives

Outline

Eric Lecolinet– LTCI Télécom ParisTech 22

First part: Expressivity of gestures

Eric Lecolinet– LTCI Télécom ParisTech 23

§ Dimensionality (2D, 3D) § Shape § Kinematics § With or without friction § Pression § Multitouch § etc.

A few examples...

Gestures = a large resource deposit

§ Frame of reference:

• Movements on the device
• Movements around the device
• Movements of the device

Eric Lecolinet– LTCI Télécom ParisTech 24

Directional radial gestures

§ Simple and fast

Learning by repeating

§ Implicit learning

Marking menus

Eric Lecolinet– LTCI Télécom ParisTech 25

Limitations

§ Limited breadth § No inner groupings

Marking menus

Eric Lecolinet– LTCI Télécom ParisTech 26

Flower menus

§ Gilles Bailly, Laurence Nigay

Three dimensions

§ Direction § Curvature § Curvature direction

Advantages

§ Time efficient § Large cardinality (56 commands) § Semantic groupings

Shape > curvature

Eric Lecolinet– LTCI Télécom ParisTech 27

Leaf menus

§ Gilles Bailly, Anne Roudaut § Adaptation for small devices § Mirrored representation

Shape > curvature

Eric Lecolinet– LTCI Télécom ParisTech 28

MicroRolls

§ Anne Roudaut, Yves Guiard § Touchscreen: no hotkeys, no menu button § Rolling gestures (no friction) § Specific signature: don't conflict with usual gestures

Shape > no-friction gestures

MicroRolls Drag Flick/Swipe Rubbing

Eric Lecolinet– LTCI Télécom ParisTech 29

Shape > no-friction gestures

Eric Lecolinet– LTCI Télécom ParisTech 30

CycloStar

§ Sylvain Malacria, Yves Guiard § Elliptic oscillatory gestures § Easy to perform [Kugler87] § Up to 7 dimensions! § Commands & parameters control

Shape & Kinematics

Eric Lecolinet– LTCI Télécom ParisTech 31

Shape & Kinematics

Eric Lecolinet– LTCI Télécom ParisTech 32

JerkTilts

§ Mathias Baglioni, Yves Guiard § 3D gestures indistinguishable from everyday motions § Self-delimited tilting gesture

Shape & Kinematics > 3D gestures

Eric Lecolinet– LTCI Télécom ParisTech 33

FingerCount

§ Gilles Bailly § Postures § # of fingers § Two-handed

Multitouch

Eric Lecolinet– LTCI Télécom ParisTech 34

InAir FingerCount

§ Gilles Bailly et al. § Not the same fingers, more tiring § N fingers increase perceived difficulty, decreases with familiarity [Rekik14]

Multitouch > 3D gestures

Eric Lecolinet– LTCI Télécom ParisTech 35

BezelTap

§ Marcos Serrano, Yves Guiard § Micro-interactions on mobile devices § Two modalities: Bumps + Slides § Reactivates the device when sleeping, avoids involuntary activations

Combining modalities

Eric Lecolinet– LTCI Télécom ParisTech 36

Combining modalities

Eric Lecolinet– LTCI Télécom ParisTech 37

Bi-positional gestures + visuotactile feedback

§ Bruno Fruchard, Olivier Chapuis § Trackpad, Bezel gestures § Starting + ending position (= position + orientation + distance) § Visual or tactile marks

Combining modalities > MarkPad

Eric Lecolinet– LTCI Télécom ParisTech 38

MarkPad

Eric Lecolinet– LTCI Télécom ParisTech 39

§ 680 possible gestures (42 tested) with different configurations § Longitudinal experiment with 6 users § Used daily for more than 2 years § Commands & gestures are very user-specific (sometimes unexpected) § Customization is the key!

MarkPad

Eric Lecolinet– LTCI Télécom ParisTech 40

WatchIt

§ Simon Perrault, Yves Guiard § Wristband of a smartwatch § Analog gestures without occlusion § Eyes-free interaction (15 gestures)

Small devices

!

Eric Lecolinet– LTCI Télécom ParisTech 41

Small devices

Eric Lecolinet– LTCI Télécom ParisTech 42

CoReach

§ Can Liu, Olivier Chapuis, Michel Beaudouin-Lafon § Collaborative gestures: Help multiple users to manipulate content

Large devices

Eric Lecolinet– LTCI Télécom ParisTech 43

Large devices

Eric Lecolinet– LTCI Télécom ParisTech 44

Using additional dimensions is the key!

§ Alternate shapes (Flower & Leaf menus) § Shapes and Kinematics (CycloStar, BezelTap) § Specific signature (MicroRolls, JerkTilts) § Shapes and visuotactile feedback (MarkPad) § Multi-touch (FingerCount, etc.) § And various other techniques...

What we learned

Eric Lecolinet– LTCI Télécom ParisTech 45

Using additional dimensions is the key!

§ Miller law on multidimensional judgments [Miller56]

Large expressivity Complementary to standard interactions

What we learned

Eric Lecolinet– LTCI Télécom ParisTech 46

Second part: Gesture memorability

Eric Lecolinet– LTCI Télécom ParisTech 47

Research questions

§ How to improve the learning and retention of gesture/command associations § Are certain gestures easier to learn/remember? § Which factors improve memorization? § Is the learning by repeating paradigm sufficient? § How to design efficient + user-friendly interaction techniques?

Gesture memorability

Eric Lecolinet– LTCI Télécom ParisTech 48

Memorization performance

§ Gilles Bailly, Laurence Nigay § Extension of MM § 16 items, 18 participants, within-subjects

Flower vs. Polygon vs. Linear menus

Eric Lecolinet– LTCI Télécom ParisTech 49

Flower menus more efficient

§ Flower menus: 81% recalled items § Polygon menus: 40% § Linear menus: 35%

Conclusions

§ Learning by repeating not sufficient! § Radial gestures favor memorization

Flower vs. Polygon vs. Linear menus

Eric Lecolinet– LTCI Télécom ParisTech 50

Radial gestures & memorization

§ Human directional abilities innate? [Wills10] § Treated as egocentric gestures? [Klatzky98] § Intensive prior learning (familiarity)

Curved radial gestures

§ Semantic relationships § Multidimensional judgments [Miller56]

Flower vs. Polygon vs. Linear menus

Eric Lecolinet– LTCI Télécom ParisTech 51

Does it make a difference?

§ Bruno Fruchard, Olivier Chapuis § Spatial memory efficient [Scarr12, Gutwin14, etc.] § Same memorization process? § 16 items, 16 participants, within-subjects, 3 sessions

Positions vs. directions

Eric Lecolinet– LTCI Télécom ParisTech 52

Puzzling differences in favor of positions

§ Recall: systematic trend § Novice mode use (significative differences):

• Less often
• Shorter amount of time

Positions vs. directions

Eric Lecolinet– LTCI Télécom ParisTech 53

Open questions

§ MM harder to use? § Less spatial and graphical cues?

• MM representation not optimal?

§ More fundamental differences?

• Navigation vs. spatial memory

Positions vs. directions

Eric Lecolinet– LTCI Télécom ParisTech 54

Advantages

§ Structure / categories improve memorability [Mandler67, Bower69, etc.] § Intuitively used by participants

Multi-Finger Chords

§ Julie Wagner, Ted Selker § 3 families of 3 gestures § Categorical vs. random mapping

Structure & categories

Eric Lecolinet– LTCI Télécom ParisTech 55

Results

§ Less error-prone long-term memorization (0.56 vs. 3.4 errors) § No completely wrong mappings

Structure & categories

Categorical

• vs. random

2 sessions (1 week) 18 participants, between-subjects

Eric Lecolinet– LTCI Télécom ParisTech 56

Letters and digits

§ Straightforward semantics § Overlearned thus 'natural' § Name collisions § Slower than straight gestures

Overlearning

Gesture Avatar [Lu & Li 11]

Eric Lecolinet– LTCI Télécom ParisTech 57

Augmented Letters

§ Quentin Roy, Yves Guiard § 1 symbol + 1 Marking menu § Balances memory and time performance § Syntactic approach § 202 commands

Overlearning

SAVE SMILE SELECT SEARCH SAVE SMILE SELECT SEARCH

Eric Lecolinet– LTCI Télécom ParisTech 58

Comparison

§ With 2-level Marking menus § 15 commands, 12 participants

Results

§ Recall: 85.4% vs. 63.2% § Reaction time compensates execution time

Overlearning

Eric Lecolinet– LTCI Télécom ParisTech 59

Several subsystems components [Baddeley13]

§ Verbal / visuospatial information processed differently § Different properties:

• e.g. explicit vs. implicit memory

Using multiple components improves memorization

§ Dual Coding Theory [Paivio71] § Elaborative encoding [Anderson79]

Memory components

Eric Lecolinet– LTCI Télécom ParisTech 60

Important memory components (for our concern)

§ Spatial memory

= directional memory?

§ Object/image memory § Verbal/Semantic encoding § Procedural memory

("muscle" memory)

Memory components

Eric Lecolinet– LTCI Télécom ParisTech 61

Mnemonic technique [Yates92, Higbee01]

§ Used since Antiquity (Cicero) § Many experiments, used by champions! § Relies on Spatial, Object and Semantic memory

Method of loci

Eric Lecolinet– LTCI Télécom ParisTech 62

Method of loci

Eric Lecolinet– LTCI Télécom ParisTech 63

Method of Loci

§ Very efficient § But requires much training!

Physical loci

§ For interacting with smart home § "Palace" in front of user!

Physical loci

Sri Vyshnavi Yarlagadda

Eric Lecolinet– LTCI Télécom ParisTech 64

Eric Lecolinet– LTCI Télécom ParisTech 65

• 1. Object memory

Eric Lecolinet– LTCI Télécom ParisTech 66

• 2. Spatial memory

Eric Lecolinet– LTCI Télécom ParisTech 67

• 3. Semantic memory (stories)

The frame contains a picture of me skiing Nature reminds me of holidays

Eric Lecolinet– LTCI Télécom ParisTech 68

Experiment

48 items Day 1 +2 months +1 day

• Active group: Creates mapping (8 participants)
• Passive group: Uses active group mappings (8 participants)

+1 week 16 participants

Eric Lecolinet– LTCI Télécom ParisTech 69

Results

§ Amazingly efficient! § User-defined mappings: No effect considering total time

Eric Lecolinet– LTCI Télécom ParisTech 70

§ 25 items (5 categories / clusters) § 2 sessions § Clusters changed in 5 different ways

Room reconfiguration

• Baseline
• Move cluster
• Move inside cluster
• Both
• Scattered

Eric Lecolinet– LTCI Télécom ParisTech 71

71

Results

§ Recall: 99.3% no significant effect § Time: 26.3% slower § Spatial memory: Less important than expected?

Baseline Move cluster Move inside cluster Both Scattered

Eric Lecolinet– LTCI Télécom ParisTech 72

Users make 'stories'

§ “The dog barks at the cat from the floor” § “The peach goes with the red door”

Passive group

§ Invented other stories!

Importance of semantic encoding

Eric Lecolinet– LTCI Télécom ParisTech 73

Humans very skilled at inventing stories!

§ It's fun! (kind of a game) § They remember them a long time

Importance of semantic encoding

Shrereshveskskii's way of remembering equations (from [Baddeley13])

Eric Lecolinet– LTCI Télécom ParisTech 74

Body = interaction surface

§ Bruno Fruchard, Olivier Chapuis § Eyes-free interaction § Helps memorization?

BodyLoci

Eric Lecolinet– LTCI Télécom ParisTech 75

Experiments

§ 1) BodyLoci vs. in air MultiStroke menus

• 24 participants, within-groups

§ 2) With stories and:

• With/without background images
• 24 participants, within-groups

BodyLoci

Eric Lecolinet– LTCI Télécom ParisTech 76

Results

§ Memorization? No difference § Impact of images? No difference § Impact of semantic encoding? Strong difference (up to 28% for MM)

BodyLoci

Eric Lecolinet– LTCI Télécom ParisTech 77

Results

§ Memorization? No difference

• Unusal
• People don't see their own body

§ Impact of images? No difference

• Redundancy / selective attention
• Should be more "bizarre" [Briggs70]

§ Impact of semantic encoding? Strong difference

• Confirms previous hypothesis
• How to leverage this capability in user interfaces?

BodyLoci

Eric Lecolinet– LTCI Télécom ParisTech 78

Expressivity of gestures

§ Using additional dimensions is the key! § Increased cardinality => improves structure / semantic mappings

Conclusions

Eric Lecolinet– LTCI Télécom ParisTech 79

Memorization

§ Using multiple dimensions is the key! § Learning != retention § Memory components:

• Spatial memory: Less important than expected?
• Image memory: Mixed results
• Semantic memory: Very efficient!
• Procedural memory: Overclaimed?

Conclusions

Eric Lecolinet– LTCI Télécom ParisTech 80

Memorization

§ Individual and combined effects § Where is the limit? [PhysicalLoci, Zhai03] § Images and graphical cues § How to incitate users to use semantic encoding § Syntax for user-defined tasks

Perspectives

SAVE SMILE SELECT SEARCH SAVE SMILE SELECT SEARCH

IconHK [Giannisakis17] CommandBoard [Alvina17]

Eric Lecolinet– LTCI Télécom ParisTech 81

Application domains

§ Small and very small devices § On-body /wearable interfaces § Wall-sized displays § AR/VR devices § Tactile feedback § etc.

Perspectives

Eric Lecolinet– LTCI Télécom ParisTech 83

Social Touch

§ Leverage communicative and emotional channels

• Between humans and machines
• Between humans via machines

§ Techniques and devices for simulating human touch

Perspectives

MobilLimb [Teyssier18]