Computational Methods for Immersive Perception Qi Sun Stony Brook - - PowerPoint PPT Presentation

▶
computational methods for immersive perception
SMART_READER_LITE
LIVE PREVIEW

Computational Methods for Immersive Perception Qi Sun Stony Brook - - PowerPoint PPT Presentation

Feb 21, 2024 14 likes •645 views

Computational Methods for Immersive Perception Qi Sun Stony Brook University Warehouse/Arts District, New Orleans, USA Traditional Visual Media Immersive Visual Media Perception for Better Immersive Experience Where we are? What we see?

slide-1
SLIDE 1

Computational Methods for Immersive Perception

Qi Sun

Stony Brook University

slide-2
SLIDE 2

Warehouse/Arts District, New Orleans, USA

slide-3
SLIDE 3

Traditional Visual Media

slide-4
SLIDE 4

Immersive Visual Media

slide-5
SLIDE 5

Perception for Better Immersive Experience

Where we are? What we see?

Miquel Perello Nieto

slide-6
SLIDE 6

Where we are?

slide-7
SLIDE 7

Nobel Brain Study

slide-8
SLIDE 8

Locomotion System in the Brain

slide-9
SLIDE 9

Locomotion System in the Brain

slide-10
SLIDE 10

VR Locomotion: Gadgets

 Un‐natural  Portable

slide-11
SLIDE 11

VR Locomotion: Walk‐in‐Place

 Natural?  Bulky

slide-12
SLIDE 12

Free Locomotion  Natural  Limited space

 Portable

slide-13
SLIDE 13

Constraint: Spatial Mismatching

Virtual World Real Space

https://vid.me/gTuI

slide-14
SLIDE 14

Redirected Walking

slide-15
SLIDE 15

Inception

slide-16
SLIDE 16

Challenges

slide-17
SLIDE 17
  • Match large virtual and small real spaces
  • Users move with comfort
  • Accurate locomotion and visual fidelity

and safety

Mapping Virtual and Physical Reality Qi Sun, Li‐Yi Wei, Arie Kaufman, SIGGRAPH 2016

slide-18
SLIDE 18

Automatic Content Re‐Creation

slide-19
SLIDE 19

virtual floor plan HMD view physical floor plan

Mapping Virtual and Physical Reality Qi Sun, Li‐Yi Wei, Arie Kaufman, SIGGRAPH 2016

slide-20
SLIDE 20

Angle Preserving

β α min (| ‐ |)

slide-21
SLIDE 21

Local Distance Preserving

slide-22
SLIDE 22

Geometric Preserving Visualization

slide-23
SLIDE 23

Mapping Virtual and Physical Reality Qi Sun, Li‐Yi Wei, Arie Kaufman, SIGGRAPH 2016

Boundary

slide-24
SLIDE 24

Mapping Virtual and Physical Reality Qi Sun, Li‐Yi Wei, Arie Kaufman, SIGGRAPH 2016

Interior Obstacles

slide-25
SLIDE 25

λ1 λ2

+ =

Warped

Original

Blended Rendering

High distortion! Motion sickness!

slide-26
SLIDE 26

Different Weights

slide-27
SLIDE 27

Different Weights

slide-28
SLIDE 28

Different Weights

slide-29
SLIDE 29

Different Weights

slide-30
SLIDE 30

Where we are

  • Manipulate visual stimuli for spatial matching
  • Extend image‐based modulations to immersive worlds

Content‐aware media re‐creation

slide-31
SLIDE 31

What we see?

slide-32
SLIDE 32

Depth Cue from Accommodation

Photo credit: Fu‐Chung Huang

Accommodation

slide-33
SLIDE 33

Traditional Displays

slide-34
SLIDE 34

Critical to Immersive Media

  • Vergence–Accommodation Conflict ‐> Sickness
slide-35
SLIDE 35

Critical to Immersive Media

  • Vergence–Accommodation Conflict ‐> Sickness
  • Depth Understanding, esp. AR/MR
slide-36
SLIDE 36

Wide Field Of View Varifocal Near‐eye Display Using See‐through Deformable Membrane Mirrors, Dunn et al. 2017

slide-37
SLIDE 37

Critical to Immersive Media

  • Vergence–Accommodation Conflict ‐> Sickness
  • Depth Understanding, esp. AR/MR
  • Interaction Accuracy
slide-38
SLIDE 38

Traditional Displays

slide-39
SLIDE 39

Light Field Displays

slide-40
SLIDE 40

Perceptually‐Guided Foveation for Light Field Displays Qi Sun et al., SIGGRAPH Asia 2017

slide-41
SLIDE 41

Light Field Displays

Perceptually‐Guided Foveation for Light Field Displays Qi Sun et al., SIGGRAPH Asia 2017

Light Field Displays

slide-42
SLIDE 42

Perceptually‐Guided Foveation for Light Field Displays Qi Sun et al., SIGGRAPH Asia 2017

Light Field Displays

slide-43
SLIDE 43

Perceptually‐Guided Foveation for Light Field Displays Qi Sun et al., SIGGRAPH Asia 2017

Light Field Displays

slide-44
SLIDE 44

Perceptually‐Guided Foveation for Light Field Displays Qi Sun et al., SIGGRAPH Asia 2017

Light Field Displays

slide-45
SLIDE 45

The light field stereoscope, Huang et al. SIGGRAPH 2015

slide-46
SLIDE 46

high latency → AR/VR sickness

slide-47
SLIDE 47

Cone Cells

Fovea

Foveated Human Vision

slide-48
SLIDE 48

Perceptually‐Based Foveated Virtual Reality, Patney et al. 2016

slide-49
SLIDE 49

Perceptually‐Guided Foveation for Light Field Displays Qi Sun et al., SIGGRAPH Asia 2017

Foveated Light Fields

slide-50
SLIDE 50

Display DoF Receptor Field

(a)

Focusing Plane

A B

LF Display

  • eccentricity

spatial bandwidth

? ?

slide-51
SLIDE 51

Psychophysical Studies – Blur Stimuli

Blur Diff Detection Depth Diff Detection Depth Discrimination are they different? are they different? which is closer?

slide-52
SLIDE 52

Perceptual Studies for Foveated Light Field Displays Joohwan Kim et al., arXiv 2017

slide-53
SLIDE 53

Psychophysical Studies – Light Field Stimuli

Blur Diff Detection Depth Diff Detection Depth Discrimination are they different? are they different? which is closer?

slide-54
SLIDE 54

Perceptual Studies for Foveated Light Field Displays Joohwan Kim, et al. arXiv 2017

slide-55
SLIDE 55

Spatial Angular

slide-56
SLIDE 56

Perceptually‐Guided Foveation for Light Field Displays Qi Sun et al., SIGGRAPH Asia 2017

slide-57
SLIDE 57

foveated light field full light field

slide-58
SLIDE 58

49.2 48 44.4 46.5 32 33.5 29.8 29.9 15 15.4 15.7 15.1 10 20 30 40 50 60 Scene 1 Scene 2 Scene 3 Scene 4

Ours Full resolution [Huang et al. 2015]

FPS

slide-59
SLIDE 59

What we see

  • Track, understand, analyzing the anatomical eyes

Faster, more comfortable, natural Content representation/authorization/viewing

slide-60
SLIDE 60

What’s Next?

slide-61
SLIDE 61

Eye‐Tracked Immersive Perception

  • Foveal VS peripheral vision
  • Perception of dynamics
  • (Large) Immersive media interface
  • Cinematography
  • Other actions
  • Vergence
  • Blink

Panning and Zooming High‐Resolution Panoramas in Virtual Reality, Chang and Cohen, UIST 2017

Why Important?

http://bair.berkeley.edu/blog/2017/11/09/learn‐to‐attend‐fovea/

slide-62
SLIDE 62

Perception‐assisted Immersive Creation

  • Immersive media editing
  • Painting/Sculpting
  • Light Field Capturing
  • Collaborative authorization

Vremiere: In‐headset Virtual Reality Video Editing, Nguyen et al., CHI 2017

CollaVR: Collaborative In‐Headset Review for VR Video, Nguyen et al., UIST 2017

slide-63
SLIDE 63

Thank You! & Discussion

Major References:

  • Mapping Virtual and Physical Reality. Qi Sun, Li‐Yi Wei, and Arie Kaufman, SIGGRAPH 2016
  • Perceptually‐Guided Foveation for Light Field Displays. Qi Sun, Fu‐Chung Huang, Joohwan Kim,

Li‐Yi Wei, David Luebke, and Arie Kaufman, SIGGRAPH Asia 2017

More Materials: http://qisun.me/