SLIDE 1
- Dr. Morgan McGuire | NVIDIA Research
3
NVIDIA RESEARCH
120 World-Class Ph.D. Researchers
THE VIRTUAL FRONTIER: COMPUTER GRAPHICS CHALLENGES IN VIRTUAL - - PowerPoint PPT Presentation
THE VIRTUAL FRONTIER: COMPUTER GRAPHICS CHALLENGES IN VIRTUAL - - PowerPoint PPT Presentation
THE VIRTUAL FRONTIER: COMPUTER GRAPHICS CHALLENGES IN VIRTUAL REALITY Dr. Morgan McGuire | NVIDIA Research NVIDIA RESEARCH 120 World-Class Ph.D. Researchers 3 VISION 1977 HPC 1997 HPC 2017 HPC Today, everyone is a high-performance computer
SLIDE 2
SLIDE 3
4
VISION
Today, everyone is a high-performance computer user, with GPUs in phones, tablets, desktops, game consoles, and cars
1977 HPC 2017 HPC 1997 HPC
SLIDE 4
5
Power User Technology Pervasive
FUTURE VR VISION
SLIDE 5
6
- 1. Virtual reality will be the new interface to computing for everyone
- 2. Virtual reality requires a new graphics system
sensors, algorithms, physics, rendering, AI, data structures, processors, optics, and displays
SLIDE 6
7
MODERN GRAPHICS SYSTEMS
SLIDE 7
8
VISUAL FIDELITY OF FILM CGI
Deadpool (Marvel)
SLIDE 8
9
VISUAL FIDELITY OF FILM CGI
Deadpool (Marvel)
SLIDE 9
10
FILM CGI: CONCEPT TO PHOTONS
Path Tracer Screen 24 Hz 9 Mpix Composite & Color Grade (Hours) Script Characters Costumes Sets Direction
Renderer Display Preproduction
Performance Modeling Rigging Animation Texturing Lighting Simulation
Production
Particles Triangles Sub-D
Primitives
SLIDE 10
12
Production Preproduction Renderer Display
3D GAME SYSTEM
Display 30Hz 2 MPix Performance Modeling Rigging Animation Texturing Lighting Simulation Script Characters Costumes Sets
Rasterization Shadow Maps AO Shade Post FX*
User Input Direction
* Includes depth of field, reflections, fog, color grading, motion blur, antialiasing
Primitives
Particles Triangles HDMI, Sync AI Network Simulation
SLIDE 11
13
3D GAME SYSTEM
Display 30Hz 2 MPix Direction Performance Modeling Rigging Animation Texturing Lighting Simulation Script Characters Costumes Sets User Input
* Includes depth of field, reflections, fog, color grading, motion blur, antialiasing
Production Preproduction Renderer Display
Particles Triangles
Primitives
Rasterization Shadow Maps AO Shade PostFX*
HDMI, Sync AI Network Simulation
SLIDE 12
14
Primitives Renderer Display
3D GAME SYSTEM
Display 30Hz 2 MPix Particles Triangles
Rasterization Shadow Maps AO Shade Post FX*
AI Network Simulation User Input HDMI, Sync
* Includes depth of field, reflections, fog, color grading, motion blur, antialiasing
SLIDE 13
15
Primitives Renderer Display
3D GAME SYSTEM
Display 30Hz 2 MPix Particles Triangles
Rasterization Shadow Maps AO Shade Post FX*
AI Network Simulation User Input HDMI, Sync
* Includes depth of field, reflections, fog, color grading, motion blur, antialiasing
SLIDE 14
16
Star Wars: Battlefront II (DICE)
SLIDE 15
17
17 Forza Motorsport 6 (Turn 10 Studios)
SLIDE 16
18
7X THROUGHPUT INCREASE
3D GAME = 60 MPIX/S
(1920 X 1080 @ MIN 30 FPS)
MODERN VR = 450 MPIX/S
(3024 X 1680* @ MIN 90 FPS) 1920 1080 1512 1680 1512
* VR render resolution
SLIDE 17
19
Primitives Renderer Display
3D GAME SYSTEM
Display 30Hz 2 MPix Particles Triangles
Rasterization Shadow Maps AO Shade Post FX*
AI Network Simulation User Input HDMI, Sync
* Includes depth of field, reflections, fog, color grading, motion blur, antialiasing
SLIDE 18
20
Display 90 Hz 5 MPix Particles Triangles
Rasterization Shadow Maps AO Shade Post FX*
AI Network Simulation User Input HDMI, Sync
* Includes depth of field, reflections, fog, color grading, motion blur, antialiasing
3D GAME SYSTEM
Primitives Renderer Display
SLIDE 19
21
Primitives Renderer
Head Tracking
HMD
Particles Triangles
Rasterization Shadow Maps Time Warp
AI Network Simulation User Input and Tracking HDMI, Sync
MODERN VR SYSTEM
Display 90 Hz 5 MPix
SLIDE 20
22
LENS DISTORTION
User’s View Predistorted Image Optics
SLIDE 21
23
Renderer
Head Tracking
HMD
Particles Triangles
Rasterization Shadow Maps Time Warp + Lens Distortion
AI Network Simulation User Input and Tracking HDMI, Sync
MODERN VR SYSTEM
Display 90Hz
Primitives
SLIDE 22
24
- 1. Virtual reality will be the new interface to computing for everyone
- 2. Virtual reality requires a new graphics system
sensors, algorithms, data structures, processors, and displays
- 3. Pascal architecture upgrades the gaming system to modern VR
warping, lens matched shading, multiprojection, stereo projection, variable resolution
SLIDE 23
25
FUTURE GRAPHICS SYSTEMS
The remainder of the talk describes active research, including new results not previously presented in public. These are not products.
SLIDE 24
26
LIMITS OF HUMAN PERCEPTION
x (120 pixels/degree)2 ≈ 400,000,000 pixels = 200 x 1080p TVs 220o Horizontal x 135o Vertical
100,000x to 1Mx beyond modern VR
+ High dynamic range (x2), photorealistic dynamic lighting (x10,000), … x 240 Hz Modern VR = 450 Mpix/s Future VR = 100,000 Mpix/s
Head image from http://jeffsearle.blogspot.com/2015/09/drawing-head-from-different-angles.html
SLIDE 25
27
Display 90Hz Particles Triangles
Foveated Rasterization Shadow Maps Time Warp + Lens Distortion
AI Network Simulation User Input and Tracking HDMI, Sync
FOVEATED RENDERING
Eye Tracking
Primitives Renderer
Head Tracking
HMD
SLIDE 26
28
FOVEATED RENDERING
Conventional Approach: Aliasing
Patney et al., Towards Foveated Rendering for Gaze-Tracked Virtual Reality, SIGGRAPH Asia 2016
Patney et al., Towards Foveated Rendering for Gaze-Tracked Virtual Reality, SIGGRAPH Asia 2016
SLIDE 27
29
FOVEATED RENDERING
Our Approach: Perceptually Optimized
Patney et al., Towards Foveated Rendering for Gaze-Tracked Virtual Reality, SIGGRAPH Asia 2016
SLIDE 28
31
Primitives
Display 90Hz Particles Triangles
Foveated Rasterization Shadow Maps Time Warp + Lens Distortion
AI Network Simulation User Input and Tracking HDMI, Sync
BEYOND TRIANGLES
Points Text Voxels Light Fields Eye Tracking Head Tracking
Renderer HMD
SLIDE 29
32
McGuire et al., Real-time global illumination with light field probes, I3D 2017
SLIDE 30
33
Renderer HMD Primitives
Display 90Hz Particles Triangles AI Network Simulation User Input and Tracking HDMI, Sync
COMPUTATIONAL DISPLAYS
Foveated Rasterization Shadow Maps Time Warp
Points Text Voxels Light Fields Varifocal Lens Distortion Light Field Eye Tracking Head Tracking
SLIDE 31
34
COMPUTATIONAL DISPLAYS
Light Field Display
GPU Output Display Prototype Observed Image
Lanman and Luebke, Near-Eye Light Field Displays, SIGGRAPH Asia 2013
SLIDE 32
35
COMPUTATIONAL DISPLAYS
Varifocal Optics
Hologram
Akşit et al., Varifocal Virtuality: A Novel Optical Layout for Near-Eye Display, SIGGRAPH 2017 Emerging Technologies
Curved sunglasses Laser
SLIDE 33
36
COMPUTATIONAL DISPLAYS
Varifocal Optics
Dunn et al, Wide field of view varifocal near-eye display using see-through deformable membrane mirrors, Proc. of IEEE VR 2017
SLIDE 34
37
Head Tracking Particles Triangles Points Text 2½D Video Voxels Light Fields
Renderer Primitives HMD
Display 90Hz Renderer AI Network Simulation User Input and Tracking HDMI, Sync
PNEUMATIC HAPTICS
Foveated Rasterization Shadow Maps Time Warp
Varifocal Lens Distortion Haptics Particles Triangles Points Text 2½D Video Voxels Light Fields
Primitives
Light Field
SLIDE 35
38
Display 90Hz AI Network Simulation User Input and Tracking HDMI, Sync
LOW LATENCY
Hierarchical Rendering
Foveated Rasterization Shadow Maps Time Warp
Varifocal Lens Distortion Haptics Remote GPU Head Tracking Light Field Eye Tracking Particles Triangles Points Text 2½D Video Voxels Light Fields
Renderer Primitives HMD
SLIDE 36
39
LOW LATENCY
Hierarchical Rendering
Crassin et al., CloudLight: A system for amortizing indirect lighting in real-time, JCGT 2015
Wearable Tegra Local GeForce
SLIDE 37
40
LOW LATENCY
Hierarchical Rendering
Wearable Tegra Cloud GRID Platform Tesla GPU Local GeForce
Crassin et al., CloudLight: A system for amortizing indirect lighting in real-time, JCGT 2015
SLIDE 38
41
LOW LATENCY
Hierarchical Rendering
Wearable Tegra Compressed lighting data High speed network Cloud GRID Platform Tesla GPU
Crassin et al., CloudLight: A system for amortizing indirect lighting in real-time, JCGT 2015
SLIDE 39
42
Primitives
Head Tracking Particles Triangles Points Text 2½D Video Voxels Light Fields
Renderer HMD
AI Network Simulation User Input and Tracking HDMI, Sync
LOW LATENCY
Binary Frames
Foveated Rasterization Shadow Maps
Eye Tracking Time Warp + Varifocal Lens Distortion Haptics Remote GPU Deep Focus Display 16000 Hz Light Field
SLIDE 40
43
ULTRA LOW LATENCY & HIGH THROUGHPUT
Binary Frames
Lincoln et al., From Motion to Photons in 80 Microseconds: Towards Minimal Latency for Virtual and Augmented Reality , IEEE VR 2016
SLIDE 41
44
LOW LATENCY
Binary Frames
Lincoln et al., From Motion to Photons in 80 Microseconds: Towards Minimal Latency for Virtual and Augmented Reality, IEEE VR 2016
0.08 ms
Motion Initiated Data Received Pixel Transmitted Light Emitted
1 2 3 4
C1 C2 Dly 27.560µs C1 C3 Dly 74.360µs C1 C4 Dly 97.312µs C2 C3 Dly 46.802µs
Ch1 Ch3 2.00 V 2.00 V
BWCh1 Ch3 2.00 V Ch2 Ch4 1.00 V Ώ
BWM 20.0µs Ch1 √ 1.20 V
SLIDE 42
45
LOW LATENCY
On-HMD Warping
Hardware Warping Prototype Photographed in HMD Warped Static Point Set
SLIDE 43
46
Particles Triangles AI Network Simulation User Input and Tracking HDMI, Sync
RAY & PATH TRACING
Foveated Rasterization Shadow Maps
Eye Tracking Points Text Voxels Light Fields Time Warp + Varifocal Lens Distortion Haptics Remote GPU Deep Focus Display 16000Hz Denoising Light Field
Renderer HMD Primitives
Head Tracking
SLIDE 44
47
PATH TRACING
10 rays/path
Deadpool (Marvel)
SLIDE 45
48
Deadpool (Marvel)
PATH TRACING
10 rays/path
Visualization of path tracing noise
1 path/pixel
SLIDE 46
49
10 rays/path 1000 paths/pixel
SLIDE 47
50
DENOISING PATH TRACING
Naïve Real-time Result
Chaitanya et al., Interactive reconstruction of Monte Carlo image sequences using a recurrent denoising autoencoder, SIGGRAPH 2017 Schied et al., Spatiotemporal variance guided filtering: real-time reconstruction for path tracing, High Performance Graphics 2017 Mara et al., An efficient denoising algorithm for global illumination, High Performance Graphics 2017
SLIDE 48
51
DENOISING PATH TRACING
Denoised Real-Time Result
Chaitanya et al., Interactive reconstruction of Monte Carlo image sequences using a recurrent denoising autoencoder, SIGGRAPH 2017 Schied et al., Spatiotemporal variance guided filtering: real-time reconstruction for path tracing, High Performance Graphics 2017 Mara et al., An efficient denoising algorithm for global illumination, High Performance Graphics 2017
SLIDE 49
52
Chaitanya et al., Interactive reconstruction of Monte Carlo image sequences using a recurrent denoising autoencoder, SIGGRAPH 2017 Schied et al., Spatiotemporal variance guided filtering: real-time reconstruction for path tracing, High Performance Graphics 2017 Mara et al., An efficient denoising algorithm for global illumination, High Performance Graphics 2017
DENOISING PATH TRACING
SLIDE 50
53
NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.
AI GRAPHICS NVIDIA RESEARCH
AI Facial Animation
SIGGRAPH 2017
AI Denoising AI Anti-Aliasing AI Light Transport
SLIDE 51
54
Display 90Hz Particles Triangles
Rasterization Shadow Maps Time Warp + Lens Distortion
AI Network Simulation User Input and Tracking HDMI, Sync
MODERN VR SYSTEM
Primitives Renderer HMD
Head Tracking
SLIDE 52
55
Primitives
Particles Triangles AI Network Simulation User Input and Tracking HDMI, Sync
FUTURE VR SYSTEM
Foveated Rasterization Shadow Maps
Eye Tracking Points Text Voxels Light Fields Time Warp + Varifocal Lens Distortion Haptics Light Field Remote GPU Deep Focus Display 16000Hz Denoising
Renderer HMD
Head Tracking
SLIDE 53
56
- 1. Virtual reality will be the new interface to computing for everyone
- 2. Pascal architecture upgrades the gaming system to modern VR
GPU warping, lens matched shading, multiprojection, stereo projection, variable resolution
- 3. NVIDIA is innovating for a revolutionary new future VR system
computational displays, varifocal optics, foveated & cloud rendering, light fields, binary frames,
- n-display warping, beam racing, haptics, path tracing, denoising
SLIDE 54
57
http://research.nvidia.com
SLIDE 55
58
SLIDE 56
59
MODERN VR EXPERIENCES
Mechanics & Design The Climb (Crytek) SUPERHOT (Superhot Team) Narrative & Characters The Labs (Valve) Simulation & Performance NVIDIA VR Funhouse Content Google Earth VR
+Batman replacing Aperture
SLIDE 57
60
PROJECTOR-BASED DISPLAY
SLIDE 58
61
The Vanishing of Ethan Carter (The Astronauts)
SLIDE 59
62
CHALLENGE: FOCUS CUES
Video from Narain et al., Optimal Presentation of Imagery with Focus Cueson Multi-Plane Displays, SIGGRAPH 2015
SLIDE 60
63
BEYOND TRIANGLES
Light Fields
McGuire et al., Real-time global illumination with light field probes, I3D 2017
SLIDE 61
64
NVIDIA AR/VR RESEARCH
Computational Displays
Light field displays and varifocal optics
Foveated Rendering
Perceptually-guided rendering for massive throughput
Ultra-Low Latency
Hierarchical & binary rendering, beam racing, near-display warp
Beyond Triangles
Points, voxels, light fields, and text
Path Tracing
Extending ray tracing leadership to cinematic quality rendering
SLIDE 62
65
NVIDIA VRWORKS SDK
SIGHT SOUND PHYSICS & TOUCH CAPTURE
BRINGING REALITY TO VR
SLIDE 63
66
NVIDIA HOLODECK HANDS-ON DEMO AT SIGGRAPH
SLIDE 64
67
NVIDIA PROJECT HOLODECK
PHOTOREALISTIC MODELS COLLABORATION INTERACTIVE PHYSICS
SLIDE 65
68
NVIDIA VRWORKS & PASCAL
Accelerating Modern VR
GRAPHICS HEADSET AUDIO TOUCH & PHYSICS PROFESSIONAL VIDEO
SLIDE 66
69
VRWORKS & PASCAL
Hardware Acceleration for Modern VR
Single-Pass Stereo Lens Matched Shading & Multiprojection Preemption for Timewarp
SLIDE 67
70
VRWORKS & PASCAL
PhysX Flex Hairworks Flow VRWorks Audio Cloth
Hardware Acceleration for Modern VR
SLIDE 68
71
FOVEATED RENDERING
High-fidelity foveal pixels Low-fidelity peripheral pixels