PERCEPTUAL INSIGHTS INTO FOVEATED VIRTUAL REALITY Anj ul Patney S - - PowerPoint PPT Presentation

▶
perceptual insights into foveated virtual reality
SMART_READER_LITE
LIVE PREVIEW

PERCEPTUAL INSIGHTS INTO FOVEATED VIRTUAL REALITY Anj ul Patney S - - PowerPoint PPT Presentation

Apr 04, 2024 10 likes •405 views

PERCEPTUAL INSIGHTS INTO FOVEATED VIRTUAL REALITY Anj ul Patney S enior Research S cientist INTRODUCTION Virtual reality is an exciting challenging workload for computer graphics Most VR pixels are peripheral Efficient

slide-1
SLIDE 1

Anj ul Patney S enior Research S cientist

PERCEPTUAL INSIGHTS INTO FOVEATED VIRTUAL REALITY

slide-2
SLIDE 2

2

INTRODUCTION

  • Virtual reality is an exciting challenging

workload for computer graphics

  • Most VR pixels are peripheral
  • Efficient peripheral rendering is poorly

understood

  • Quick perceptual experiments can

expose opportunities for significant speedups

slide-3
SLIDE 3

3

Foveal Pixels Peripheral Pixels

FOVEATED RENDERING

slide-4
SLIDE 4

4

THIS TALK

Why foveated rendering? S tate of the art in efficient peripheral rendering How else could we reduce peripheral details?

slide-5
SLIDE 5

5

WHY FOVEATED RENDERING?

slide-6
SLIDE 6

6 *VR render resolut ion f or HTC Vive

PC Gaming : 60 MP/s

(1920 x 1080 @

30 Hz)

1920 1080

slide-7
SLIDE 7

7 *VR render resolut ion f or HTC Vive

PC Gaming : 60 MP/s

(1920 x 1080 @

30 Hz)

1920 1080

Virtual Reality : 450 MP/s

(3024 x 1680* @

90 Hz)

1512 1680 1512

slide-8
SLIDE 8

8

VR IS UP TO 7X MORE DEMANDING TODAY

*VR render resolut ion f or HTC Vive

PC Gaming : 60 MP/s

(1920 x 1080 @

30 Hz)

1920 1080

Virtual Reality : 450 MP/s

(3024 x 1680* @

90 Hz)

1512 1680 1512

slide-9
SLIDE 9

9

Column1

MOST VR PIXELS ARE PERIPHERAL

4% 73% 96%

iPhone 7 Plus 27" Deskt op Monitor 2016 VR HMD 20° Peripheral Peripheral Foveal

slide-10
SLIDE 10

10 10

Column1

MOST VR PIXELS ARE PERIPHERAL

4% 73% 96%

iPhone 7 Plus 27" Deskt op Monitor 2016 VR HMD 20° Peripheral Peripheral Foveal

Efficient peripheral rendering can unlock significant performance gains

slide-11
SLIDE 11

11

EFFICIENT PERIPHERAL RENDERING

What is the ideal peripheral image? How to efficiently render the ideal peripheral image?

slide-12
SLIDE 12

12

STATE OF THE ART IN EFFICIENT PERIPHERAL RENDERING

slide-13
SLIDE 13

13 13

MULTI-PASS FOVEATED RENDERING

[Guenter et al. 2012]

slide-14
SLIDE 14

14

GPU-ACCELERATED FOVEATED RENDERING

NVIDIA Multi-Res S hading

slide-15
SLIDE 15

15

GPU-ACCELERATED FOVEATED RENDERING

NVIDIA Lens-Matched S hading

slide-16
SLIDE 16

16

BEYOND FOVEATED IMAGE RESOLUTION

  • Peripheral vision shows non-uniform
  • ptical, retinal, and neural degradation
  • Over an order of magnitude potential

improvement

10 100 1K 10K 100K 20 40 60 80 Density (per mm2) Eccentricity (degrees)

Rods Cones Ganglion Cells

slide-17
SLIDE 17

17 17

BEYOND FOVEATED IMAGE RESOLUTION

LOW SENSITIVITY TOWARD HIGH SENSITIVITY TOWARD

http://pics.psych.stir.ac.uk/

Flicker Motion Faces High S patial Frequencies Color Crowded S timuli

slide-18
SLIDE 18

18 18

BEYOND FOVEATED IMAGE RESOLUTION

LOW SENSITIVITY TOWARD HIGH SENSITIVITY TOWARD

http://pics.psych.stir.ac.uk/

Flicker Motion Faces High S patial Frequencies Color Crowded S timuli

slide-19
SLIDE 19

19 19

BEYOND FOVEATED IMAGE RESOLUTION

LOW SENSITIVITY TOWARD HIGH SENSITIVITY TOWARD

http://pics.psych.stir.ac.uk/

Flicker Motion Faces High S patial Frequencies Color Crowded S timuli

slide-20
SLIDE 20

20 20

PERCEPTUAL FOVEATED RENDERING

[S tengel et al. 2016]

Acuity Fall-off, Eye/ Obj ect Motion, Contrast, Brightness Adaptation 2-3x lower fragment shading workload

slide-21
SLIDE 21

21 21

PERCEPTUAL FOVEATED RENDERING [Patney et al. 2016]

PERIPHERAL BLUR

slide-22
SLIDE 22

22 22

PERIPHERAL BLUR + CONTRAST PRESERVATION

PERCEPTUAL FOVEATED RENDERING [Patney et al. 2016]

slide-23
SLIDE 23

23 23

PERCEPTUAL FOVEATED RENDERING [Patney et al. 2016]

PRACTICAL RENDERING SYSTEM WITH 2X-3X FEWER PIXEL SHADES

slide-24
SLIDE 24

24

HOW ELSE COULD WE REDUCE PERIPHERAL DETAILS?

slide-25
SLIDE 25

25

PERCEPTUAL HYPOTHESES

“ Can we reduce peripheral color saturation? ” “ Can we foveate alternate frames? ” “ S hould we foveate symmetrically around the gaze center? ”

slide-26
SLIDE 26

26

EARLY EXPERIMENTS

  • S

imulated foveation following each hypothesis

  • Performed ad-hoc Yes/ No tests with two subj ects
  • Estimated threshold using MATLAB psychtoolbox
  • Total time per trained subj ect: < 1 hour
slide-27
SLIDE 27

27

BASELINE: FOVEATED BLUR

slide-28
SLIDE 28

28

CAN WE REDUCE PERIPHERAL COLOR SATURATION?

slide-29
SLIDE 29

29

Even Frame Odd Frame

CAN WE FOVEATE ALTERNATE FRAMES?

slide-30
SLIDE 30

30

Even Frame

CAN WE FOVEATE ALTERNATE FRAMES?

slide-31
SLIDE 31

31 31

Temporal Foveation Nasal Foveation

SHOULD WE FOVEATE SYMMETRICALLY AROUND THE GAZE CENTER?

slide-32
SLIDE 32

32 32

Temporal Foveation

SHOULD WE FOVEATE SYMMETRICALLY AROUND THE GAZE CENTER?

slide-33
SLIDE 33

33 33

RESULTS – MAGNITUDE OF FOVEATION

10 20 30 40 50 Baseline Color Flicker Nasal Temporal

Foveation threshold

S ubj ect 1 S ubj ect 2

slide-34
SLIDE 34

34 34

0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 Color Flicker Nasal + Temporal

Relative performance

RESULTS – ESTIMATED PERFORMANCE

slide-35
SLIDE 35

35 35

0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 Color Flicker Nasal + Temporal

Relative performance

RESULTS – ESTIMATED PERFORMANCE

Potential Win!

slide-36
SLIDE 36

36

SUMMARY

  • Efficient peripheral rendering is fundamental for VR rendering
  • But we have a lot to learn about efficient peripheral rendering
  • What is the ideal peripheral image?
  • How to efficiently render the ideal peripheral image?
  • Perceptual evaluations can expose opportunities for significant speedups
slide-37
SLIDE 37

37

ACKNOWLEDGMENTS

Joohwan Kim Rachel Albert Peter S hirley Chris Wyman Marco S alvi Jason Paul David Luebke Aaron Lefohn

slide-38
SLIDE 38

38

THANK YOU

apatney@ nvidia.com

slide-39
SLIDE 39