M ULTIMODAL A UGMENTED R EALITY – A UGMENTING A UDITORY-TACTILE F EEDBACK TO C HANGE THE P ERCEPTION OF T HICKNESS Geert Lugtenberg 1,2 Wolfgang Hürst 1 Nina Rosa 1 Christian Sandor 2 Alexander Plopski 2 1 Utrecht University (Netherlands) Takafumi Taketomi 2 2 Nara Institute of Science and Hirokazu Kato 2 Technology (Japan)
Augmented Reality (AR) Different implementations exist All combine real with virtual elements Spatial / AR with head-worn Handheld AR projected AR displays
AR Turing test We are not there yet, but … Which object is real, which one is virtual? … progress in vision (light field displays) … very good in audio augmentation But : reality is multimodal, incorporating other senses E.g., haptics, smell, taste, …
Multimodal AR Modalities other than vision and sound are hard to create (E.g., haptics via skin versus visuals and sound via eyes and ears, respectively) But we can “trick” human perception (E.g., temperature via BurnAR (ISMAR 2012, Weir et al.) visuals; red, fire, …) People feel heat, although demo is just adding visuals
AR Turing test, here: sound (and haptics) Context / usage example: Assume seeing a souvenir. What is it made of? Look at it (visuals) Does it look real, old, expensive, …? Lift it up (haptics) How does it feel like? What does it weight? Tap on it (sound (and haptics)) What material does it sound like? Is it hollow or solid? How thick is it?
1 st research question : Can we achieve a different perception of thickness (solid or hollow) of an object when tapping it Psychophysical experiment by solely modifying auditory feedback & Two cubes (solid & hollow), otherwise fixed physical properties? identity unknown to subjects Auditory feedback: • No sound augmentation (control condition) • White noise (no sound at all) • Resynthesized sound (solid or hollow) 8 participants, 16 trials, testing all combinations of conditions Experiment 1: augmented sound
Real sound: high performance 1 Major results Thickness can be classified by in plain English tapping White noise: solid can be classify, 2 but hollow not (assumed as solid) Audio essential for classification Artificial hollow sound: high when 3 matching physical property, low when not We can trick people with audio to perceive solid cubes as hollow 3 2 4 1 Results on chance level (guessing) 4 Audio has impact on perception, but not enough to consistently Experiment 1: augmented sound create target scenarios
2 nd research question : Can we achieve a Psychophysical experiment different perception of thickness (solid or Change of setup from cubes hollow) of an object when tapping it to plates (and thicker/thinner) by modifying auditory and tactile feedback & Two-alternative forced-choice otherwise fixed physical properties? psychophysical experiment (2AFC, “which one is thinner?”) Conditions: • Tactile stimuli (thick / thin) from real object • Auditory-tactile stimuli (congruent or discrepant with each other) vibration-cues are added to vibration in real material 10 participants, 60 trials Experiment 2: augmented sound & tactile feedback
Haptics only condition: misclassi- 1 Major results fication for non-matching case in plain English Perception can be tricked by changing haptics (if sole modality) Congruent stimuli (both stimuli 2 match or not): misclassification for non-matching case Perception can be tricked by congruently changing both stimuli Incongruent stimuli (either sound 3 or haptics don’t match): results at chance level (guessing) One modality is enough to “destroy 1 2 3 real perception” but not enough to consistently create a new one Experiment 2: augmented sound & haptics
Audio has an impact 1 st research question : Can we achieve a different perception of thickness (solid or In some situation this is enough to reliably change perception (hollow hollow) of an object when tapping it perceived as solid) by solely modifying auditory feedback & otherwise fixed physical properties? In others not (solid not perceived (Experiment 1) as hollow but results on chance level) 2 nd research question : Can we achieve a Haptics have an impact different perception of thickness (solid or If auditory and tactile stimuli are hollow) of an object when tapping it presented congruently, we can by modifying auditory and tactile feedback reliably change perception & otherwise fixed physical properties? If only one of them is changed, (Experiment 2) perception is manipulated but not reliably changed
Summary & conclusion Motivation True AR requires multimodality Multimodal AR is VERY hard But we can “trick” human perception Pilot study if this is true for auditory feedback Results show that there is potential (proof of concept) Yet, audio alone is only sufficient in some cases (change perception from solid to hollow) For others, additional modalities (haptics) need to be considered
Recommend
More recommend
