SLIDE 1 MULTIMODAL AUGMENTED REALITY – AUGMENTING AUDITORY-TACTILE FEEDBACK
TO CHANGE THE PERCEPTION OF THICKNESS
Geert Lugtenberg 1,2 Wolfgang Hürst 1 Nina Rosa 1 Christian Sandor 2 Alexander Plopski 2 Takafumi Taketomi 2
2 Nara Institute of Science and
Technology (Japan)
1 Utrecht University (Netherlands)
Hirokazu Kato 2
SLIDE 2
Augmented Reality (AR)
AR with head-worn displays Spatial / projected AR Handheld AR
All combine real with virtual elements Different implementations exist
SLIDE 3
AR Turing test Which object is real, which one is virtual? But: reality is multimodal, incorporating other senses … progress in vision (light field displays) E.g., haptics, smell, taste, … We are not there yet, but … … very good in audio augmentation
SLIDE 4 Multimodal AR Modalities other than vision and sound are hard to create But we can “trick” human perception
BurnAR (ISMAR 2012, Weir et al.) People feel heat, although demo is just adding visuals
(E.g., haptics via skin versus visuals and sound via eyes and ears, respectively) (E.g., temperature via visuals; red, fire, …)
SLIDE 5
AR Turing test, here: sound (and haptics) Context / usage example: Assume seeing a souvenir. What is it made of? Look at it (visuals) Lift it up (haptics) Tap on it (sound (and haptics))
Does it look real, old, expensive, …? How does it feel like? What does it weight? What material does it sound like? Is it hollow or solid? How thick is it?
SLIDE 6 1st research question: Can we achieve a different perception of thickness (solid or hollow) of an object when tapping it by solely modifying auditory feedback &
- therwise fixed physical properties?
Psychophysical experiment Two cubes (solid & hollow), identity unknown to subjects Experiment 1: augmented sound Auditory feedback:
(control condition)
- White noise (no sound at all)
- Resynthesized sound
(solid or hollow) 8 participants, 16 trials, testing all combinations of conditions
SLIDE 7 Major results Real sound: high performance Experiment 1: augmented sound Results on chance level (guessing) Audio essential for classification Artificial hollow sound: high when matching physical property, low when not Thickness can be classified by tapping We can trick people with audio to perceive solid cubes as hollow in plain English Audio has impact on perception, but not enough to consistently create target scenarios White noise: solid can be classify, but hollow not (assumed as solid)
1 2 3 4 1 2 3 4
SLIDE 8 2nd research question: Can we achieve a different perception of thickness (solid or hollow) of an object when tapping it by modifying auditory and tactile feedback &
- therwise fixed physical properties?
Psychophysical experiment
Experiment 2: augmented sound & tactile feedback
Change of setup from cubes to plates (and thicker/thinner) Two-alternative forced-choice psychophysical experiment (2AFC, “which one is thinner?”) Conditions:
- Tactile stimuli (thick / thin)
from real object
(congruent or discrepant with each other) vibration-cues are added to vibration in real material 10 participants, 60 trials
SLIDE 9 Major results Haptics only condition: misclassi- fication for non-matching case Experiment 2: augmented sound & haptics Congruent stimuli (both stimuli match or not): misclassification for non-matching case Perception can be tricked by congruently changing both stimuli Incongruent stimuli (either sound
- r haptics don’t match): results
at chance level (guessing) One modality is enough to “destroy real perception” but not enough to consistently create a new one in plain English 1 2 3 1 2 3 Perception can be tricked by changing haptics (if sole modality)
SLIDE 10
Audio has an impact In some situation this is enough to reliably change perception (hollow perceived as solid) In others not (solid not perceived as hollow but results on chance level) Haptics have an impact If auditory and tactile stimuli are presented congruently, we can reliably change perception If only one of them is changed, perception is manipulated but not reliably changed 1st research question: Can we achieve a different perception of thickness (solid or hollow) of an object when tapping it by solely modifying auditory feedback & otherwise fixed physical properties? (Experiment 1) 2nd research question: Can we achieve a different perception of thickness (solid or hollow) of an object when tapping it by modifying auditory and tactile feedback & otherwise fixed physical properties? (Experiment 2)
SLIDE 11
Summary & conclusion Motivation True AR requires multimodality Multimodal AR is VERY hard But we can “trick” human perception Pilot study if this is true for auditory feedback Results show that there is potential (proof of concept) Yet, audio alone is only sufficient in some cases (change perception from solid to hollow) For others, additional modalities (haptics) need to be considered
