How Do We Measure Depth Perception in Near-Field Augmented Reality - - PowerPoint PPT Presentation

How Do We Measure Depth Perception in Near-Field Augmented Reality Inspired by Medical Applications?

• Dr. J. Edward Swan II

25 June 2014 Professor Dept of Computer Science & Engineering Adjunct Professor Dept of Psychology

Augmented Reality (AR)

• AR demo from my lab
• Virtual objects
• At real-world locations
• Spatially related to

real-world objects

AR Medical Applications and Perceived Depth

• AR-based training system for

planning brain tumor resection

– Robarts Research Institute, Western University, Ontario, Canada – Kamyar Abhari, Terry Peters, and many collaborators

• What is the perceived depth
• f the tumor?

How does depth perception operate?

• We see 3D world from

ambiguous 2D retinal images

• Important scientific question

since mid-1800’s

• Current theories

emphasize importance of depth cues

Near-Field Depth Cues

n Occlusion n Relative Size n Relative Density n Binocular Disparity n Accommodation n Convergence n Motion Perspective

Near-Field Depth Cues

n Occlusion n Relative Size n Relative Density n Binocular Disparity n Accommodation n Convergence n Motion Perspective

Near-Field Depth Cues

n Occlusion n Relative Size n Relative Density n Binocular Disparity n Accommodation n Convergence n Motion Perspective

Near-Field Depth Cues

n Occlusion n Relative Size n Relative Density n Binocular Disparity n Accommodation n Convergence n Motion Perspective

Near-Field Depth Cues

n Occlusion n Relative Size n Relative Density n Binocular Disparity n Accommodation n Convergence n Motion Perspective

Near-Field Depth Cues

n Occlusion n Relative Size n Relative Density n Binocular Disparity n Accommodation n Convergence n Motion Perspective

Near-Field Depth Cues

n Occlusion n Relative Size n Relative Density n Binocular Disparity n Accommodation n Convergence n Motion Perspective

Near-Field Depth Cues

n Occlusion n Relative Size n Relative Density n Binocular Disparity n Accommodation n Convergence n Motion Perspective

How Do We Measure Depth Perception?

• Problem:

perception is invisible

• Solution:

depth judgment tasks

• Based on cognition

– Verbal report

• Based on perception-action

– Perceptual matching – Blind reaching

That’s about a meter… Who cares? Let me hit the ball!

Previous work in near-field AR depth perception

• Rolland et al. [1995]
• Rolland et al. [2002]
• Ellis & Menges [1998]
• McCandless, Ellis,

Adelstein [2000]

• Edwards et al. [2004]
• Singh, Ellis,

Swan [2010, 2013]

• Hua, Ellis, Swan [2014]

Exp I: Matching vs. Reaching

• Used perceptual matching and blind reaching tasks
• Extend Ellis & Menges [1998] apparatus and task
• Effect of occluding surface (x-ray vision)
• Within-subjects design

Exp I: Tasks

• ccluder = absent, present

judgment = blind reach judgment = match hydraulic jack calibration cross display

Exp I: Apparatus

Judgment Occluder Distance (cm) reach match present absent present absent 50 46 42 38 34 50 46 42 38 34 50 46 42 38 34 50 46 42 38 34 2 1

• 1
• 2
• 3
• 4
• 5
• 6
• 7

Error (cm), +/– 1 SEM

• ccluder
• ccluder

Experiment I

Exp I: Results

Exp I Matching/Reaching

X-ray Vision Within-Subjects

Exp II Real/AR

Matching , Reaching Similar Biomechanics Between-Subjects Matching > Reaching Calibration

Task ¡= ¡reach Task ¡= ¡match Environment ¡= ¡Real, ¡AR

Exp II: Real vs. AR

• Compare real and AR targets
• Compare perceptual matching

and blind reaching judgments

• Same biomechanical movement

for matching and reaching

• Between-subjects design

background curtain closed-loop slider foam ridge (restricts HMD movement) real-world target cameras

Judgment Environment Distance (cm) reach match AR real AR real 50 46 42 38 34 50 46 42 38 34 50 46 42 38 34 50 46 42 38 34

3 2 1

• 1
• 2
• 3
• 4
• 5
• 6

Error (cm), +/– 1 SEM

Experiment II

Exp II: Results

The Accommodative / Vergence Problem for VR and AR

Normal viewing Viewing with vergence farther than accommodation

vergence ¡distance ¡ focal ¡distance ¡ vergence ¡distance ¡ focal ¡distance ¡ vergence ¡distance ¡ focal ¡distance ¡

Viewing with accommodation farther than vergence

Judgment Environment Distance (cm) reach match AR real AR real 50 46 42 38 34 50 46 42 38 34 50 46 42 38 34 50 46 42 38 34

3 2 1

• 1
• 2
• 3
• 4
• 5
• 6

Error (cm), +/– 1 SEM

Experiment II

Exp II: Disparity Changes for AR Matching

α β Δv = α – β Δv = α – β α β

34 38 42 46 50

• 0.4
• 0.2

0.0 0.2 0.4

Experiment II AR Matching

Distance (cm) Δvergence°

Judgment Environment Distance (cm) reach match AR real AR real 50 46 42 38 34 50 46 42 38 34 50 46 42 38 34 50 46 42 38 34

3 2 1

• 1
• 2
• 3
• 4
• 5
• 6

Error (cm), +/– 1 SEM

Experiment II

Exp II: Results

Exp I Matching/Reaching

X-ray Vision Within-Subjects

Exp II Real/AR

Matching , Reaching Similar Biomechanics Between-Subjects

Exp III Feedback Use Finger

Matching , Reaching Real, AR Pretest, Intervention, Posttest Matching > Reaching Calibration Matching > Reaching No Calibration Matching > Reaching Calibration

¡

Mon-Williams & Tresilian [1999], fig 2

Exp III: Feedback, Proprioception

• Feedback:

– Pretest, Expose, Posttest design – Reach-1, Match-2, Reach-3 – Match-1, Reach-2, Match-3

• Point with Finger:

– Reaching and matching both use finger

• Compare real and AR

targets

Exp III: Tasks

perceptual matching blind reaching cardboard ridge

Judgment Environment Distance (cm) reach match AR real AR real 50 46 42 38 34 50 46 42 38 34 50 46 42 38 34 50 46 42 38 34

5 4 3 2 1

• 1
• 2
• 3
• 4
• 5
• 6

Error (cm), +/– 1 SEM

Experiment II

Comparing Exp II and Exp III Pretest

Judgment Environment Distance (%) reach match AR real AR real 87 79 71 63 55 87 79 71 63 55 87 79 71 63 55 87 79 71 63 55

5 4 3 2 1

• 1
• 2
• 3
• 4
• 5
• 6

Error (cm), +/– 1 SEM

Experiment III

Exp III: Feedback (Pretest, Expose, Posttest)

Judgment Block Distance (%) match-3 reach-2 match-1 reach-3 match-2 reach-1 87 79 71 63 55 87 79 71 63 55 87 79 71 63 55 87 79 71 63 55 87 79 71 63 55 87 79 71 63 55

5 4 3 2 1

• 1
• 2
• 3
• 4

Error (cm), +/– 1 SEM

Experiment III

Environment = real

Judgment Block Distance (%) m a t c h

• 3

r e a c h

• 2

m a t c h

• 1

r e a c h

• 3

m a t c h

• 2

r e a c h

• 1

8 7 7 9 7 1 6 3 5 5 8 7 7 9 7 1 6 3 5 5 8 7 7 9 7 1 6 3 5 5 8 7 7 9 7 1 6 3 5 5 8 7 7 9 7 1 6 3 5 5 8 7 7 9 7 1 6 3 5 5

5 4 3 2 1

• 1
• 2
• 3
• 4

Error (cm), +/– 1 SEM

Experiment III

Environment = AR

Judgment Environment Distance (%) reach match AR real AR real 87 79 71 63 55 87 79 71 63 55 87 79 71 63 55 87 79 71 63 55

5 4 3 2 1

• 1
• 2
• 3
• 4
• 5
• 6

Error (cm), +/– 1 SEM

Experiment III (Posttest)

Exp III: Posttest Results (Reach-3, Match-3)

Exp I Matching/Reaching

X-ray Vision Within-Subjects

Exp II Real/AR

Matching , Reaching Similar Biomechanics Between-Subjects

Exp III Feedback Use Finger

Matching , Reaching Real, AR Pretest, Intervention, Posttest

Exp IV Accommodation

Collimated, Consistent, Midpoint, Real Younger Between-Subjects Matching > Reaching Calibration Overestimation AR-Matching Overestimation AR-Matching Overestimation AR-Matching Matching > Reaching No Calibration Matching > Reaching Calibration

32

Haploscopes

• Commercial HMDs don’t allow

enough control

• Vision scientists have long built

haploscopes: Presents separately- controlled image to each eye

Berkeley Haploscope Clemson Haploscope HASA Haploscope (HMD)

Haploscope

Image generated by minimization lens 10 cm 10 cm 5 cm Stimulus Object Monitor +10 D

• 10 D

Minimization Lens (Concave) Collimation Lens (Convex) Accommodation Lens (Concave) Image is formed at 33.3 - 50 cm based on the power of the accommodation lens

Image generator Optics Eye positions as pivot points

n m f

§ Adjustable focus display § Can present objects with varying accommodation and vergence demands

Exp IV: Accommodation

• Environment /

Accommodation:

– Real / Consistent – AR / Collimated – AR / Consistent – AR / Midpoint

• Perceptual Matching
• 33 to 50 cm

Haploscope

Environment ¡= ¡Real, ¡AR Task ¡= ¡match

Condition Distance (cm) M i d p

• i

n t C

• n

s i s t e n t C

• l

l i m a t e d R e a l 5 . 4 4 . 4 4 . 3 6 . 4 3 3 . 3 5 . 4 4 . 4 4 . 3 6 . 4 3 3 . 3 5 . 4 4 . 4 4 . 3 6 . 4 3 3 . 3 5 . 4 4 . 4 4 . 3 6 . 4 3 3 . 3

2.5 2.0 1.5 1.0 0.5 0.0

• 0.5
• 1.0

Error (cm), +/– 1 SEM

Experiment IV

Exp IV: Results

Exp I Matching/Reaching

X-ray Vision Within-Subjects

Exp II Real/AR

Matching , Reaching Similar Biomechanics Between-Subjects

Exp III Feedback Use Finger

Matching , Reaching Real, AR Pretest, Intervention, Posttest

Exp IV Accommodation

Collimated, Consistent, Midpoint, Real Younger Between-Subjects

Exp V Brightness

Bright, Dim Collimated, Consistent, Midpoint Younger Between-Subjects Matching > Reaching Calibration Overestimation AR-Matching Overestimation AR-Matching Overestimation AR-Matching Matching > Reaching No Calibration Matching > Reaching Calibration

Real-world AR Exp IV AR Exp V

Exp V: Brightness

• Exp IV luminance:

– Real-world = 15.5 cd/m2 – AR = 33.2 cd/m2

• Exp V:

– Replicated Exp 4 AR conditions – dimmer object = 6.8 cd/m2

Exp V: Results

Condition Brightness Distance (cm) M i d p

• i

n t C

• n

s i s t e n t C

• l

l i m a t e d R e a l D i m B r i g h t D i m B r i g h t D i m B r i g h t N

• r

m a l 5 . 4 4 . 4 4 . 3 6 . 4 3 3 . 3 5 . 4 4 . 4 4 . 3 6 . 4 3 3 . 3 5 . 4 4 . 4 4 . 3 6 . 4 3 3 . 3 5 . 4 4 . 4 4 . 3 6 . 4 3 3 . 3 5 . 4 4 . 4 4 . 3 6 . 4 3 3 . 3 5 . 4 4 . 4 4 . 3 6 . 4 3 3 . 3 5 . 4 4 . 4 4 . 3 6 . 4 3 3 . 3

2.5 2.0 1.5 1.0 0.5 0.0

• 0.5
• 1.0

Error (cm), +/– 1 SEM

Experiment V

Exp I Matching/Reaching

X-ray Vision Within-Subjects

Exp II Real/AR

Matching , Reaching Similar Biomechanics Between-Subjects

Exp III Feedback Use Finger

Matching , Reaching Real, AR Pretest, Intervention, Posttest

Exp IV Accommodation

Collimated, Consistent, Midpoint, Real Younger Between-Subjects

Exp V Brightness

Bright, Dim Collimated, Consistent, Midpoint Younger Between-Subjects

Exp VI Age

Younger, Older Collimated, Consistent, Midpoint Bright Between-Subjects Matching > Reaching Calibration Overestimation AR-Matching Overestimation AR-Matching Overestimation AR-Matching Matching > Reaching No Calibration Matching > Reaching Calibration

Exp VI: Age

Duane [1912], 1500 subjects

• Observers 40+
• Same as Exp IV:

– Real / Consistent – AR / Collimated – AR / Consistent – AR / Midpoint

• Used bright AR
• bjects

Experiment VI: Results

Condition Age Distance (cm) Midpoint Consistent Collimated Real Older Younger Older Younger Older Younger Older Younger 50.0 44.4 40.0 36.4 33.3 50.0 44.4 40.0 36.4 33.3 50.0 44.4 40.0 36.4 33.3 50.0 44.4 40.0 36.4 33.3 50.0 44.4 40.0 36.4 33.3 50.0 44.4 40.0 36.4 33.3 50.0 44.4 40.0 36.4 33.3 50.0 44.4 40.0 36.4 33.3

2.5 2.0 1.5 1.0 0.5 0.0

• 0.5
• 1.0

Error (cm), +/– 1 SEM

Experiment VI

Exp VII: Replicate Edwards et al. (2004)

Exp VII: Effect of Occluding Surface

• Introduce highly salient occluding surface
• Densely sample around occluding surface
• Use matching judgment

Exp VII: Results

Surface Relative Depth (mm) Error (mm), ± 1 SEM

• 6
• 4
• 2

2 4 6

• ccluder = absent
• 80
• 60
• 40
• 20

20

• 6
• 4
• 2

2 4 6

• ccluder = present

Surface Relative Depth (mm) Error (mm), ± 1 SEM

• 6
• 4
• 2

2 4 6

y = –0.06x – 2.35 r2 = 0.98

• ccluder = absent
• 80
• 60
• 40
• 20

20

• 6
• 4
• 2

2 4 6

y = –0.07x – 3.33 r2 = 0.70

• ccluder = present

Exp VII: Results

Exp VII: Linear Underestimation and IPD Change

Surface Relative Depth (mm) Error (mm), ± 1 SEM

• 6
• 4
• 2

2 4 6

y = –0.06x – 2.35 r2 = 0.98

• ccluder = absent
• 80
• 60
• 40
• 20

20

• 6
• 4
• 2

2 4 6

y = –0.07x – 3.33 r2 = 0.70

• ccluder = present

Surface Relative Depth (mm) ΔError (mm), ± 1 SEM

• 6
• 4
• 2

2 4 6

• 80
• 60
• 40
• 20

20

(occluder = absent) – (occluder = present)

Exp VII: Effect of Introducing Occluder

Surface Relative Depth (mm) Error (mm), ± 1 SEM

• 6
• 4
• 2

2 4 6

• ccluder = absent
• 80
• 60
• 40
• 20

20

• 6
• 4
• 2

2 4 6

• ccluder = present

Summary of Findings

• Accommodative / vergence mismatch matters;

but sufficient to focus at midpoint of workspace

• Brightness of virtual object matters; needs to be

similar to surrounding real objects

• Older observers as accurate as younger observers
• Proper modeling of IPD (likely) matters
• Salience of occluding surface (likely) matters
• Vergence angle explains many depth errors

The AR Depth Perception Team

• Students:

– Kenny Moser (Computer Science PhD) – Nate Phillips (Computer Science PhD)

• Alumni:

– Chunya Hua (Computer Science MS) – Gurjot Singh (Computer Science PhD) – Adam Jones (University of Southern California)

• Collaborators:

– Stephen R. Ellis (NASA Ames) – Christain Sandor (University of South Australia)

• Funding:

– NSF, NASA, Office of Naval Research

How Do We Measure Depth Perception in Near-Field Augmented Reality Inspired by Medical Applications?

• Dr. J. Edward Swan II

25 June 2014 Professor Dept of Computer Science & Engineering Adjunct Professor Dept of Psychology