SLIDE 1
Margarita Grinvald
Gesture recognition for Smartphones/Wearables
2
Gestures
▪ hands, face, body movements ▪ non-verbal communication ▪ human interaction
Gesture recognition for Smartphones/Wearables Gestures hands, face, - - PowerPoint PPT Presentation
Gesture recognition for Smartphones/Wearables Gestures hands, face, - - PowerPoint PPT Presentation
Margarita Grinvald Gesture recognition for Smartphones/Wearables Gestures hands, face, body movements non-verbal communication human interaction 2 Gesture recognition interface with computers increase usability intuitive
SLIDE 2
SLIDE 3
3
Gesture recognition
▪ interface with computers ▪ increase usability ▪ intuitive interaction
SLIDE 4
4
▪ Contact type: ▪ Touch based ▪ Non-contact type: ▪ Device gesture ▪ Vision based ▪ Electrical Field Sensing (EFS)
Gesture sensing
SLIDE 5
▪ miniaturisation ▪ lack tactile clues ▪ no link between physical and digital interactions ▪ computational power
5
Issues on mobile devices
SLIDE 6
▪ augment environment with digital information
6
Approaches
Sixthsense [Mistry et al. SIGGRAPH 2009] Skinput [Harrison et al. CHI 2010] OmniTouch [Harrison et al. UIST 2011]
SLIDE 7
7
Approaches
▪ augment hardware
In-air typing interface for mobile devices with vibration feedback [Niikura et al. SIGGRAPH 2010] A low-cost transparent electric field sensor for 3D interaction [Le Goc et al. CHI 2014] MagGetz [Hwang et al. UIST 2013]
SLIDE 8
8
▪ combine devices
Approaches
▪ efficient algorithms
In-air gestures around unmodified mobile devices [Song et al. UIST 2014] Duet: Exploring Joint interactions on a smart phone and a smart watch [Chen et al. CHI 2014]
SLIDE 9
▪ augment environment with visual information ▪ interact through natural hand gestures ▪ wearable to be truly mobile
9
Sixthsense [Mistry et al. SIGGRAPH 2009]
SLIDE 10
10
Color markers Camera Projector Mirror Smartphone
SLIDE 11
11
Support for arbitrary surfaces
SLIDE 12
12
Support for multitouch
SLIDE 13
13
Limitations
▪ inability track surfaces ▪ differentiate hover and click ▪ accuracy limitations
SLIDE 14
▪ skin as input canvas ▪ wearable bio-acoustic sensor ▪ localisation of finger tap
14
Skinput [Harrison et al. CHI 2010]
SLIDE 15
15
Projector Armband
SLIDE 16
16
Mechanical phenomena
▪ finger tap on skin generates acoustic energy ▪ some energy becomes sound waves ▪ some energy transmitted through the arm
SLIDE 17
17
SLIDE 18
18
Transverse waves
SLIDE 19
19
Longitudinal waves
SLIDE 20
▪ array of tuned vibrations sensors ▪ sensitive only to motion perpendicular to skin ▪ two sensing arrays to disambiguate different armband positions.
20
Sensing
SLIDE 21
21
Sensor packages Weights
SLIDE 22
▪ sensor data segmented into taps ▪ ML classification of location ▪ initial training stage
22
Tap localisation
SLIDE 23
23
SLIDE 24
24
▪ lack of support of other surfaces than skin ▪ no multitouch support ▪ no touch drag movement
Limitations
SLIDE 25
25
▪ appropriate on demand ad hoc surfaces ▪ depth sensing and projection wearable ▪ depth driven template matching
OmniTouch [Harrison et al. UIST 2011]
SLIDE 26
26
Depth Camera Projector
SLIDE 27
▪ multitouch finger tracking on arbitrary surfaces ▪ no calibration or training ▪ resolve position and distinguish hover from click
27
Finger tracking
SLIDE 28
28
Finger segmentation
Depth map Depth map gradient
SLIDE 29
29
Finger segmentation
Candidates Tip estimation
SLIDE 30
30
Click detection
Finger hovering Finger clicking
SLIDE 31
▪ expand application space with graphical feedback ▪ track surface on which rendered ▪ update interface as surface moves
31
On demand interfaces
SLIDE 32
32
Interface ‘glued’ to surface
SLIDE 33
33
SLIDE 34
▪ vision based 3D input interface ▪ detect keystroke action in the air ▪ provide vibration feedback
34
In-air typing interface for mobile devices with vibration feedback
[Niikura et al. SIGGRAPH 2010]
SLIDE 35
35
Camera white LEDs vibration motor
SLIDE 36
▪ high frame rate camera ▪ wide angle lens needs distortion correction ▪ skin colour extraction to detect fingertip ▪ estimate fingertip translation, rotation and scale
36
Tracking
SLIDE 37
▪ difference of the dominant frequency of the fingertips scale to detect keystroke ▪ tactile feedback is important ▪ vibration feedback is conveyed after a keystroke
37
Keystroke feedback
SLIDE 38
38
▪ camera is rich and flexible but with limitations ▪ minimal distance between sensor and scene ▪ sensitivity to lighting changes ▪ computational overheads ▪ high power requirements
Vision limitations
SLIDE 39
39
▪ smartphone augmented with EFS ▪ resilient to illumination changes ▪ mapping measurements to 3D finger positions.
A low-cost transparent electric field sensor for 3D interaction
[Le Goc et al. CHI 2014]
SLIDE 40
40
Drive electronics Electrode array
SLIDE 41
▪ microchip built-in 3D positioning has low accuracy ▪ Random Decision Forests for regression on raw signal data ▪ speed and accuracy
41
Recognition
SLIDE 42
42
SLIDE 43
▪ tangible control widgets for richer tactile clues ▪ wider interaction area ▪ low cost and user configurable unpowered magnets
43
MagGetz [Hwang et al. UIST 2013]
SLIDE 44
44
Magnetic fields Tangibles
SLIDE 45
▪ traditional physical input controls with magnets ▪ magnetic traces change on widget state change ▪ track physical movement of control widgets
45
Tangibles
SLIDE 46
46
Tangibles magnetism
Toggle switch Slider
SLIDE 47
47
SLIDE 48
▪ object damage by magnets ▪ magnetometer limitations
48
Limitations
SLIDE 49
▪ extend interaction space with gesturing ▪ mobile devices RGB camera ▪ robust ML based algorithm
49
In-air gestures around unmodified mobile devices
[Song et al. UIST 2014]
SLIDE 50
▪ detection of salient hand parts (fingertips) ▪ works without relying on highly discriminative depth data and rich computational resources ▪ no strong assumption about users environment ▪ reasonably robust to rotation and depth variation
50
Gesture recognition
SLIDE 51
▪ real time algorithm ▪ pixel labelling with random forests ▪ techniques to reduce memory footprint of classifier
51
Recognition algorithm
SLIDE 52
52
Recognition steps
RGB input Segmentation Labeling
SLIDE 53
▪ division of labor ▪ works on many devices ▪ new apps enabled just by collecting new data
53
Applications
SLIDE 54
54
SLIDE 55
55
SLIDE 56
▪ beyond usage of single device ▪ allow individual input and output ▪ joint interactions smart phone and smart watch
56
Duet: Exploring joint interactions on a smart phone and a smart watch
[Chen et al. CHI 2014]
SLIDE 57
▪ conversational duet ▪ foreground interaction ▪ background interaction
57
Design space theory
SLIDE 58
58
Design space
SLIDE 59
SLIDE 60
60
Design space
SLIDE 61
SLIDE 62
▪ ML techniques on accelerometer data ▪ handedness recognition ▪ promising accuracy
62
Gesture recognition
SLIDE 63
▪ wearables extend interaction space to everyday surfaces ▪ augmented hardware in general provides an intuitive interface ▪ no additional hardware is preferable but there are still computational limitations ▪ combination of devices may be redundant
63
Summary
SLIDE 64
▪ SixthSense: a wearable gestural interface [Mistry et al. SIGGRAPH 2009] ▪ Skinput: Appropriating the Body As an Input Surface [Harrison et al. CHI 2010] ▪ OmniTouch: Wearable Multitouch Interaction Everywhere [Harrison et al. UIST 2011] ▪ In-air typing interface for mobile devices with vibration feedback [Niikura et al. SIGGRAPH 2010] ▪ A Low-cost Transparent EF Sensor for 3D Interaction on Mobile Devices [Le Goc et al. CHI 2014] ▪ MagGetz: customizable passive tangible controllers on and around [Hwang et al. UIST 2013] ▪ In-air gestures around unmodified mobile devices mobile devices [Song et al. UIST 2014] ▪ Duet: Exploring Joint Interactions on a Smart Phone and a Smart Watch [Chen et al. CHI 2014]
64