Authoring Sensor-Based Interactions by Demonstration with Direct - PowerPoint PPT Presentation

Authoring Sensor-Based Interactions by Demonstration with Direct Manipulation and Pattern Recognition Björn Hartmann, Leith Abdulla Manas Mittal, Scott R. Klemmer 1

How would you prototype… …a workout monitoring system? [Apple, Nike] 2

How would you explore… …motion-based game controllers? [Nintendo] 3

How would you build… …a proximity-aware whiteboard? [Ju, Lee, Akers] 4

Sensor-based Interaction Design 5

d.tools [Hartmann et al., UIST ’06] 6

d.Tools d.tools Student Project 7

Designing Sensor-based Interactions PROTOTYPE APPLICATION LOGIC SPECIFY RELATIONSHIP SPECIFY RELATIONSHIP BETWEEN SENSOR DATA BETWEEN SENSOR DATA AND APPLICATION LOGIC AND APPLICATION LOGIC PROVIDE SOFTWARE ABSTRACTION FOR HARDWARE 8

Types of Sensors Force Sensitive Resistors Accelerometers RFID IR/Ultrasonic Rangers Phototransistors Discrete Continuous Categorical 9

Current Tools: Oscilloscopes [CC image by Timo Arnall] 10

Current Tools: LabView LabView 11

Representation Matters Accelerometer X axis Accelerometer Y axis //detect accelerometer peaks //read data sample xVal[t++]=readA2DValue(xPin); //look for changes in derivative if(((xVal[t]-xVal[t-1]) >= 0 && (xVal[t-1]-xVal[t-2]) < 0) (((xVal[t]-xVal[t-1]) < 0 && (xVal[t-1]-xVal[t-2]) >= 0 //peak detected //send message oscSendMessageInt("/x/peak",1); } else { //no peak } 12

Idea: Programming by Demonstration 13

Idea: Programming by Demonstration 14

Tacit Knowledge 15

Crux: Generalization 16

Crux: Generalization 17

Crux: Generalization 18

Exemplar 19

Exemplar 20

Exemplar 21

Exemplar 22

23

Exemplar UI 24

Exemplar UI 25

Exemplar UI 26

Exemplar UI 27

Exemplar UI 28

Exemplar UI 29

Exemplar UI 30

[Flash animations from the talk are not included in this presentation] 31

Dynamic Time Warping Demonstration Signal Matching Input Signal [Sakoe, H. Chiba, S. ‘78] 32

Filters Offset Y-Axis Scaling Smoothing Rate of Change 33

Extensible Filter Set 34

Understanding the Exemplar Approach What? Why? Cognitive Dimensions of Analysis of Exemplar as a Notation (CDN) Inspection visual authoring environment First-Use Laboratory Study Threshold and usability Class Deployment Real-world stress test Used as Design Tools for Complexity ceiling for “ Building Upon Everyday Play ” knowledgeable users CHI Interactivity Exhibit 35

Lab Study [sketches by Wendy Ju] 36

Lab Study 26 min 18 min 27 min 47 min 22 min 31 min [sketches by Wendy Ju] 37

38

Mini Joystick Bend Sensor taped to table Navigation Navigation Test Play with Accelerometer with XY Joystick 39

Mini Joystick Bend Sensor taped to table Trigger with Refine trigger Test Play Test Play Bend sensor detection 40

Post-Test Questionnaire 41

Post-Test Questionnaire 42

CHI Interactivity [Control Freaks by Haiyan Zhang] 43

CHI Interactivity Compulsory latency of pattern matching precludes use for fast-paced actions Threshold Pattern crossed here matched here time [Control Freaks by Haiyan Zhang] 44

Related Work PBD Tools Musical controller Commercial for Ubicomp Design tools Dsp software a Capella HID Toolkit LabView/ [Dey et al., CHI 04] [Steiner, NIME 05] Lego Mindstorms NXT Crayons MnM [Fails & Olsen, CHI 03] [Bevilacqua et al., NIME 05] Monet FlexiGesture [Li, Landay, UIST 05] [Merrill & Paradiso, CHI 05] Papier-Mâché [Klemmer et al., CHI 04] 45

Related Work A CAPella (Dey et al., CHI 04) 46

Related Work FlexiGesture (Merrill CHI05) 47

Related Work LabView 48

Related Work Lego Mindstorms NXT 49

Future Directions Export: Generate code + specifications Programming sensor networks by demonstration d.mix: Programming by A Sample for Web Service APIs Integration of direct manipulation environments with textual programming 50

Acknowledgments We thank MediaX/DNP for funding, Intel for equipment donation, Wendy Ju for illustrations, David Merrill & Timo Arnall for photos. 51

Download Exemplar: http:// hci.stanford.edu /exemplar CHI Interactivity “ Building Upon Everyday Play ” Talk tomorrow, 11:30am, Room C2 52