THE VIRTUAL FRONTIER: Computer graphics challenges in virtual - - PowerPoint PPT Presentation

▶
the virtual frontier
SMART_READER_LITE
LIVE PREVIEW

THE VIRTUAL FRONTIER: Computer graphics challenges in virtual - - PowerPoint PPT Presentation

Sep 27, 2022 152 likes •668 views

THE VIRTUAL FRONTIER: Computer graphics challenges in virtual reality Dr. Morgan McGuire, NVIDIA Research May 9, 2017 GPU Technology Conference VISION 1977 HPC 2017 HPC 1997 HPC Today, everyone is a high-performance computer user, with GPUs

slide-1
SLIDE 1
  • Dr. Morgan McGuire, NVIDIA Research

May 9, 2017 GPU Technology Conference

THE VIRTUAL FRONTIER:

Computer graphics challenges in virtual reality

slide-2
SLIDE 2

2

VISION

Today, everyone is a high-performance computer user, with GPUs in phones, tablets, desktops, game consoles, and cars 1997 HPC 1977 HPC 2017 HPC

slide-3
SLIDE 3

3

VISION

Pervasive Power User Technology

FUTURE VR

slide-4
SLIDE 4

4

  • 2. Virtual reality requires a new graphics system

sensors, algorithms, data structures, processors, and displays

  • 1. Virtual reality will be the new interface to computing for everyone
slide-5
SLIDE 5

5

MODERN GRAPHICS SYSTEMS

slide-6
SLIDE 6

6

Deadpool (Marvel)

VISUAL FIDELITY OF FILM CGI

slide-7
SLIDE 7

7

Deadpool (Marvel)

slide-8
SLIDE 8

8

FILM CGI: CONCEPT TO PHOTONS

Performance Modeling Rigging Animation Texturing Lighting Simulation Path Tracer Particles Triangles Sub-D Screen 24 Hz 9 Mpix Composite & Color Grade Direction Primitives Preproduction Renderer Production Display Script Characters Costumes Sets (Hours)

slide-9
SLIDE 9

9

FILM CGI: CONCEPT TO PHOTONS

Performance Modeling Rigging Animation Texturing Lighting Simulation Path Tracer Particles Triangles Sub-D Screen 24 Hz 9 Mpix Composite & Color Grade Direction Primitives Preproduction Renderer Production Script Characters Costumes Sets Display (Hours)

slide-10
SLIDE 10

10

3D GAME SYSTEM

AI Network Simulation HDMI, Sync User Input Monitor Renderer Primitives Rasterization Particles Triangles Shadow Maps Display 30Hz 2 MPix AO Shade Post FX* Direction Preproduction Production

Performance Modeling Rigging Animation Texturing Lighting Simulation Script Characters Costumes Sets

slide-11
SLIDE 11

11

3D GAME SYSTEM

AI Network Simulation HDMI, Sync User Input Monitor Renderer Primitives Rasterization Particles Triangles Shadow Maps Display 30Hz 2 MPix AO Shade Post FX* Direction Preproduction Production

Performance Modeling Rigging Animation Texturing Lighting Simulation Script Characters Costumes Sets

slide-12
SLIDE 12

12

3D GAME SYSTEM

AI Network Simulation HDMI, Sync User Input Monitor Renderer Primitives Rasterization Particles Triangles Shadow Maps Display 30Hz 2 MPix AO Shade Post FX*

* Includes depth of field, reflections, fog, color grading, motion blur, antialiasing

slide-13
SLIDE 13

13

3D GAME SYSTEM

AI Network Simulation HDMI, Sync User Input Monitor Renderer Primitives Rasterization Particles Triangles Shadow Maps Display 30Hz 2 MPix AO Shade Post FX*

* Includes depth of field, reflections, fog, color grading, motion blur, antialiasing

slide-14
SLIDE 14

14

Star Wars: Battlefront II (DICE)

slide-15
SLIDE 15

15

15

Forza Motorsport 6 (Turn 10 Studios)

slide-16
SLIDE 16

16

7X THROUGHPUT INCREASE

3D GAME = 60 MPIX/S

(1920 X 1080 @ MIN 30 FPS)

MODERN VR = 450 MPIX/S

(3024 X 1680* @ MIN 90 FPS) 1920 1080 1512 1680

*VR render resolution

1512

slide-17
SLIDE 17

17

AI Network Simulation HDMI, Sync User Input Monitor Renderer Primitives Rasterization Particles Triangles Shadow Maps Display 90Hz 5 MPix AO Shade Post FX*

* Includes depth of field, reflections, fog, color grading, motion blur, antialiasing

slide-18
SLIDE 18

18

MODERN VR SYSTEM

AI Network Simulation HDMI, Sync User Input and Tracking Head Tracking HMD Renderer Primitives Rasterization Particles Triangles Shadow Maps Time Warp Display 90Hz 5 MPix

slide-19
SLIDE 19

19

LENS DISTORTION

User’s View Rendered Image Optics

slide-20
SLIDE 20

20

MODERN VR SYSTEM

AI Network Simulation HDMI, Sync User Input and Tracking Head Tracking HMD Renderer Primitives Rasterization Particles Triangles Shadow Maps Time Warp + Lens Distortion Display 90Hz

slide-21
SLIDE 21

21

  • 1. Virtual reality will be the new interface to computing for everyone
  • 2. Virtual reality requires a new graphics system

sensors, algorithms, representations, processors, displays

  • 3. Pascal architecture upgrades the gaming system to modern VR

warping, lens matched shading, multiprojection, stereo projection, variable resolution

slide-22
SLIDE 22

22

22

FUTURE GRAPHICS SYSTEMS The remainder of the talk describes active research, including new results not previously presented in public. These are not products.

slide-23
SLIDE 23

23

LIMITS OF HUMAN PERCEPTION

Head image from http://jeffsearle.blogspot.com/2015/09/drawing-head-from-different-angles.html

x (120 pixels/degree)2 ≈ 400,000,000 pixels = 200 x 1080p TVs 220o Horizontal x 135o Vertical

100,000x to 1Mx Beyond Modern VR

+ High dynamic range (x2), photorealistic dynamic lighting (x10,000), …

x 240 fps Modern VR = 450 Mpix/s Future VR = 100,000 Mpix/s

slide-24
SLIDE 24

24

FOVEATED RENDERING

AI Network Simulation HDMI, Sync User Input and Tracking Eye Tracking Head Tracking HMD Renderer Primitives Foveated Rasterization Particles Triangles Shadow Maps Time Warp + Lens Distortion Display 90Hz

slide-25
SLIDE 25

25

FOVEATED RENDERING

Conventional Approach: Aliasing

Patney et al., Towards Foveated Rendering for Gaze-Tracked Virtual Reality, SIGGRAPH Asia 2016

slide-26
SLIDE 26

26

FOVEATED RENDERING

Our Approach: Perceptually Optimized

Patney et al., Towards Foveated Rendering for Gaze-Tracked Virtual Reality, SIGGRAPH Asia 2016

slide-27
SLIDE 27

27

BEYOND TRIANGLES

AI Network Simulation HDMI, Sync User Input and Tracking Eye Tracking Head Tracking HMD Renderer Primitives Foveated Rasterization Particles Triangles Points Text Voxels Light Fields Shadow Maps Time Warp + Lens Distortion Display 90Hz

slide-28
SLIDE 28

28

BEYOND TRIANGLES

Light Fields

McGuire et al., Real-time global illumination with light field probes, I3D 2017

slide-29
SLIDE 29

29

COMPUTATIONAL DISPLAYS

AI Network Simulation HDMI, Sync User Input and Tracking

Light Field

Eye Tracking Head Tracking HMD Renderer Primitives Foveated Rasterization Particles Triangles Points Text Voxels Light Fields Shadow Maps Time Warp Display 90Hz Varifocal Lens Distortion

slide-30
SLIDE 30

30

COMPUTATIONAL DISPLAYS

Light Field Display

Lanman and Luebke, Near-Eye Light Field Displays, SIGGRAPH Asia 2013

Display Prototype GPU Output Observed Image

slide-31
SLIDE 31

31

COMPUTATIONAL DISPLAYS

Varifocal Optics

Akşit et al., Varifocal Virtuality: A Novel Optical Layout for Near-Eye Display, SIGGRAPH 2017 Emerging Technologies

Hologram

slide-32
SLIDE 32

32

COMPUTATIONAL DISPLAYS

Varifocal Optics

Dunn et al, Wide field of view varifocal near-eye display using see-through deformable membrane mirrors, Proc. of IEEE VR 2017

slide-33
SLIDE 33

33

PNEUMATIC HAPTICS

AI Network Simulation HDMI, Sync User Input and Tracking Light Field Eye Tracking Haptics Head Tracking HMD Renderer Primitives Foveated Rasterization Particles Triangles Points Text 2½D Video Voxels Light Fields Shadow Maps Time Warp Display 90Hz Varifocal Lens Distortion

Shepherd et al., Stretchable Transducers for Haptic Interactions in Virtual Reality, GTC VR Village 2017 & SIGGRAPH Emerging Technologies, 2017

slide-34
SLIDE 34

34

LOW LATENCY

AI Network Simulation HDMI, Sync User Input and Tracking Light Field Eye Tracking Haptics Remote GPU Head Tracking HMD Renderer Primitives Foveated Rasterization Particles Triangles Points Text 2½D Video Voxels Light Fields Shadow Maps

Hierarchical Rendering

Time Warp Display 90Hz Varifocal Lens Distortion

slide-35
SLIDE 35

35

LOW LATENCY

Hierarchical Rendering

Crassin et al., CloudLight: A system for amortizing indirect lighting in real-time, JCGT 2015

Near-HMD Tegra Local GeForce

slide-36
SLIDE 36

36

LOW LATENCY

Hierarchical Rendering

Crassin et al., CloudLight: A system for amortizing indirect lighting in real-time, JCGT 2015

Near-HMD Tegra Cloud GRID Platform Tesla GPU Local GeForce

slide-37
SLIDE 37

37

LOW LATENCY

Hierarchical Rendering

Crassin et al., CloudLight: A system for amortizing indirect lighting in real-time, JCGT 2015

Near-HMD Tegra Compressed lighting data High speed network Cloud GRID Platform Tesla GPU

slide-38
SLIDE 38

38

LOW LATENCY

AI Network Simulation HDMI, Sync User Input and Tracking Light Field Eye Tracking Haptics Remote GPU Head Tracking HMD Renderer Primitives Foveated Rasterization Particles Triangles Points Text 2½D Video Voxels Light Fields Deep Focus Display 16000Hz Time Warp + Varifocal Lens Distortion Shadow Maps

Binary Frames

slide-39
SLIDE 39

39

ULTRA LOW LATENCY & HIGH THROUGHPUT

Binary Frames

Lincoln et al., From Motion to Photons in 80 Microseconds: Towards Minimal Latency for Virtual and Augmented Reality, IEEE VR 2016

slide-40
SLIDE 40

40

Binary Frames

Lincoln et al., From Motion to Photons in 80 Microseconds: Towards Minimal Latency for Virtual and Augmented Reality, IEEE VR 2016

0.08 ms LOW LATENCY

Lincoln et al., From Motion to Photons in 80 Microseconds: Towards Minimal Latency for Virtual and Augmented Reality, IEEE VR 2016

Binary Frames

slide-41
SLIDE 41

41

LOW LATENCY

On-HMD Warping

Hardware Warp Prototype Head-Mounted Display Warped Static Point Set

slide-42
SLIDE 42

42

RAY & PATH TRACING

AI Network Simulation HDMI, Sync User Input and Tracking Light Field Eye Tracking Haptics Remote GPU Head Tracking HMD Renderer Primitives Foveated Path Tracing Particles Triangles Points Text Voxels Light Fields Denoising Deep Focus Display 16000Hz Time Warp + Varifocal Lens Distortion Shadow Maps

slide-43
SLIDE 43

43

Deadpool (Marvel)

PATH TRACING

10 rays/path

slide-44
SLIDE 44

44

Deadpool (Marvel)

PATH TRACING

10 rays/path

Visualization of path tracing noise

2 paths/pixel

slide-45
SLIDE 45

45

10 rays/path 1000 paths/pixel

slide-46
SLIDE 46

46

PATH TRACING

Mara et al., Towards real-time path tracing: An efficient denoising algorithm for global illumination, in review 2017

Naïve result at 2 paths/pixel

slide-47
SLIDE 47

47

PATH TRACING

Mara et al., Towards real-time path tracing: An efficient denoising algorithm for global illumination, in review 2017

Real-Time Result with 2 paths/pixel

slide-48
SLIDE 48

48

MODERN VR SYSTEM

AI Network Simulation HDMI, Sync User Input and Tracking Head Tracking HMD Renderer Primitives Rasterization Particles Triangles Shadow Maps Time Warp + Lens Distortion Display 90Hz

slide-49
SLIDE 49

49

FUTURE VR SYSTEM

Particles Triangles Deep Focus Display 16000Hz AI Network Simulation HDMI, Sync User Input and Tracking Time Warp + Varifocal Lens Distortion Points Text Voxels Light Fields Light Field Eye Tracking Foveated Path Tracing Haptics Remote GPU Head Tracking Denoising HMD Renderer Primitives Shadow Maps

slide-50
SLIDE 50

50

  • 1. Virtual reality will be the new interface to computing for everyone
  • 2. Virtual reality requires a new graphics system

sensors, algorithms, representations, processors, displays

  • 3. Pascal architecture upgrades the gaming system to modern VR

warping, lens matched shading, multiprojection, stereo projection, simulation computational displays, varifocal optics, foveated & hierarchical rendering, voxels, light fields, binary frames, near-display warp, beam racing, haptics, path tracing

  • 4. NVIDIA research is innovating on the entire VR system