SLIDE 1
1
Authoring Sensor-Based Interactions by Demonstration
with Direct Manipulation and Pattern Recognition
Björn Hartmann, Leith Abdulla Manas Mittal, Scott R. Klemmer
2
How would you prototype…
[Apple, Nike]
Authoring Sensor-Based Interactions by Demonstration with Direct - - PowerPoint PPT Presentation
Authoring Sensor-Based Interactions by Demonstration with Direct - - PowerPoint PPT Presentation
Authoring Sensor-Based Interactions by Demonstration with Direct Manipulation and Pattern Recognition Bjrn Hartmann, Leith Abdulla Manas Mittal, Scott R. Klemmer 1 How would you prototype a workout monitoring system? [Apple, Nike] 2
SLIDE 2
SLIDE 3
3
How would you explore…
[Nintendo]
…motion-based game controllers?
SLIDE 4
4
How would you build…
[Ju, Lee, Akers]
…a proximity-aware whiteboard?
SLIDE 5
5
Sensor-based Interaction Design
SLIDE 6
6
[Hartmann et al., UIST ’06]
d.tools
SLIDE 7
7
d.Tools Student Project
d.tools
SLIDE 8
8
Designing Sensor-based Interactions
PROTOTYPE APPLICATION LOGIC SPECIFY RELATIONSHIP BETWEEN SENSOR DATA AND APPLICATION LOGIC PROVIDE SOFTWARE ABSTRACTION FOR HARDWARE SPECIFY RELATIONSHIP BETWEEN SENSOR DATA AND APPLICATION LOGIC
SLIDE 9
9
Types of Sensors
Continuous Categorical Discrete
IR/Ultrasonic Rangers Force Sensitive Resistors RFID Accelerometers Phototransistors
SLIDE 10
10
Current Tools: Oscilloscopes
[CC image by Timo Arnall]
SLIDE 11
11
LabView
Current Tools: LabView
SLIDE 12
12
Representation Matters
//detect accelerometer peaks //read data sample xVal[t++]=readA2DValue(xPin); //look for changes in derivative if(((xVal[t]-xVal[t-1]) >= 0 && (xVal[t-1]-xVal[t-2]) < 0) (((xVal[t]-xVal[t-1]) < 0 && (xVal[t-1]-xVal[t-2]) >= 0 //peak detected //send message
- scSendMessageInt("/x/peak",1);
} else { //no peak }
Accelerometer X axis Accelerometer Y axis
SLIDE 13
13
Idea: Programming by Demonstration
SLIDE 14
14
Idea: Programming by Demonstration
SLIDE 15
15
Tacit Knowledge
SLIDE 16
16
Crux: Generalization
SLIDE 17
17
Crux: Generalization
SLIDE 18
18
Crux: Generalization
SLIDE 19
19
Exemplar
SLIDE 20
20
Exemplar
SLIDE 21
21
Exemplar
SLIDE 22
22
Exemplar
SLIDE 23
23
SLIDE 24
24
Exemplar UI
SLIDE 25
25
Exemplar UI
SLIDE 26
26
Exemplar UI
SLIDE 27
27
Exemplar UI
SLIDE 28
28
Exemplar UI
SLIDE 29
29
Exemplar UI
SLIDE 30
30
Exemplar UI
SLIDE 31
31
[Flash animations from the talk are not included in this presentation]
SLIDE 32
32
[Sakoe, H. Chiba, S. ‘78]
Dynamic Time Warping
Demonstration Signal Matching Input Signal
SLIDE 33
33
Offset Y-Axis Scaling Smoothing Rate of Change
Filters
SLIDE 34
34
Extensible Filter Set
SLIDE 35
35
Understanding the Exemplar Approach
Cognitive Dimensions of Notation (CDN) Inspection First-Use Laboratory Study Class Deployment Used as Design Tools for “Building Upon Everyday Play” CHI Interactivity Exhibit Analysis of Exemplar as a visual authoring environment Threshold and usability Real-world stress test Complexity ceiling for knowledgeable users
What? Why?
SLIDE 36
36
Lab Study
[sketches by Wendy Ju]
SLIDE 37
37
Lab Study
[sketches by Wendy Ju]
47 min 22 min 26 min 31 min 27 min 18 min
SLIDE 38
38
SLIDE 39
39
Mini Joystick Bend Sensor taped to table Navigation with Accelerometer Navigation with XY Joystick Test Play
SLIDE 40
40
Mini Joystick Bend Sensor taped to table Test Play Test Play Trigger with Bend sensor Refine trigger detection
SLIDE 41
41
Post-Test Questionnaire
SLIDE 42
42
Post-Test Questionnaire
SLIDE 43
43
CHI Interactivity
[Control Freaks by Haiyan Zhang]
SLIDE 44
44
CHI Interactivity
Compulsory latency of pattern matching precludes use for fast-paced actions
Threshold crossed here Pattern matched here time
[Control Freaks by Haiyan Zhang]
SLIDE 45
45
Related Work
a Capella [Dey et al., CHI 04] Crayons [Fails & Olsen, CHI 03] Monet [Li, Landay, UIST 05] Papier-Mâché [Klemmer et al., CHI 04] HID Toolkit [Steiner, NIME 05] MnM [Bevilacqua et al., NIME 05] FlexiGesture [Merrill & Paradiso, CHI 05] LabView/ Lego Mindstorms NXT
PBD Tools for Ubicomp Musical controller Design tools Commercial Dsp software
SLIDE 46
46
A CAPella (Dey et al., CHI 04)
Related Work
SLIDE 47
47
FlexiGesture (Merrill CHI05)
Related Work
SLIDE 48
48
LabView
Related Work
SLIDE 49
49
Lego Mindstorms NXT
Related Work
SLIDE 50
50
Future Directions
Export: Generate code + specifications Programming sensor networks by demonstration d.mix: Programming by A Sample for Web Service APIs Integration of direct manipulation environments with textual programming
SLIDE 51
51
Acknowledgments
We thank MediaX/DNP for funding, Intel for equipment donation, Wendy Ju for illustrations, David Merrill & Timo Arnall for photos.
SLIDE 52
52