THE VIRTUAL FRONTIER: COMPUTER GRAPHICS CHALLENGES IN VIRTUAL REALITY Dr. Morgan McGuire | NVIDIA Research
NVIDIA RESEARCH 120 World-Class Ph.D. Researchers 3
VISION 1977 HPC 1997 HPC 2017 HPC Today, everyone is a high-performance computer user, with GPUs in phones, tablets, desktops, game consoles, and cars 4
VISION Power User Technology Pervasive FUTURE VR 5
1. Virtual reality will be the new interface to computing for everyone 2. Virtual reality requires a new graphics system sensors, algorithms, physics, rendering, AI, data structures, processors, optics, and displays 6
MODERN GRAPHICS SYSTEMS 7
VISUAL FIDELITY OF FILM CGI Deadpool (Marvel) 8
VISUAL FIDELITY OF FILM CGI Deadpool (Marvel) 9
FILM CGI: CONCEPT TO PHOTONS Direction Preproduction Production Primitives Display Renderer Performance Modeling Script Screen Rigging Particles Characters Composite & Path Tracer 24 Hz Animation Triangles Costumes Color Grade Texturing 9 Mpix Sub-D Sets Lighting (Hours) Simulation 10
3D GAME SYSTEM Direction Preproduction Production Primitives Display Renderer Performance Modeling Script AI Display Rigging Characters HDMI, Particles Network Shadow Post 30Hz Animation Costumes Rasterization AO Shade Sync Triangles Maps FX* Simulation Texturing 2 MPix Sets Lighting Simulation User Input * Includes depth of field, reflections, fog, color grading, motion blur, antialiasing 12
3D GAME SYSTEM Direction Preproduction Production Primitives Display Renderer Performance Modeling Script AI Display Rigging Characters HDMI, Particles Network Shadow 30Hz Animation Costumes Rasterization AO Shade PostFX* Sync Triangles Maps Simulation Texturing 2 MPix Sets Lighting Simulation User Input * Includes depth of field, reflections, fog, color grading, motion blur, antialiasing 13
3D GAME SYSTEM Primitives Renderer Display AI Display Particles HDMI, Network Shadow 30Hz Triangles Rasterization AO Shade Post FX* Sync Maps Simulation 2 MPix User Input * Includes depth of field, reflections, fog, color grading, motion blur, antialiasing 14
3D GAME SYSTEM Primitives Renderer Display AI Display Particles HDMI, Network Shadow 30Hz Triangles Rasterization AO Shade Post FX* Sync Maps Simulation 2 MPix User Input * Includes depth of field, reflections, fog, color grading, motion blur, antialiasing 15
Star Wars: Battlefront II (DICE) 16
17 Forza Motorsport 6 (Turn 10 Studios) 17
7X THROUGHPUT INCREASE MODERN VR = 450 MPIX/S 3D GAME = 60 MPIX/S ( 3024 X 1680* @ MIN 90 FPS) ( 1920 X 1080 @ MIN 30 FPS) 1680 1080 1920 1512 1512 * VR render resolution 18
3D GAME SYSTEM Primitives Renderer Display AI Display Particles HDMI, Network Shadow 30Hz Triangles Rasterization AO Shade Post FX* Sync Maps Simulation 2 MPix User Input * Includes depth of field, reflections, fog, color grading, motion blur, antialiasing 19
3D GAME SYSTEM Primitives Renderer Display AI Display Particles HDMI, Network Shadow 90 Hz Triangles Rasterization AO Shade Post FX* Sync Maps Simulation 5 MPix User Input * Includes depth of field, reflections, fog, color grading, motion blur, antialiasing 20
MODERN VR SYSTEM Head Tracking Primitives Renderer HMD AI Display Particles HDMI, Network Shadow 90 Hz Triangles Rasterization Time Warp Sync Maps Simulation 5 MPix User Input and Tracking 21
LENS DISTORTION Predistorted Image Optics User’s View 22
MODERN VR SYSTEM Head Tracking Primitives Renderer HMD AI Particles Display HDMI, Network Shadow Time Warp + Triangles Rasterization Sync 90Hz Maps Lens Distortion Simulation User Input and Tracking 23
1. Virtual reality will be the new interface to computing for everyone 2. Virtual reality requires a new graphics system sensors, algorithms, data structures, processors, and displays 3. Pascal architecture upgrades the gaming system to modern VR warping, lens matched shading, multiprojection, stereo projection, variable resolution 24
FUTURE GRAPHICS SYSTEMS The remainder of the talk describes active research, including new results not previously presented in public. These are not products. 25
LIMITS OF HUMAN PERCEPTION 100,000x to 1Mx beyond modern VR 220 o Horizontal x 135 o Vertical x (120 pixels/degree) 2 ≈ 400,000,000 pixels = 200 x 1080p TVs x 240 Hz Future VR = 100,000 Mpix/s Modern VR = 450 Mpix/s + High dynamic range (x2), photorealistic dynamic lighting (x10,000), … Head image from http://jeffsearle.blogspot.com/2015/09/drawing-head-from-different-angles.html 26
FOVEATED RENDERING Head Tracking Primitives Renderer HMD AI Particles Display HDMI, Network Shadow Foveated Time Warp + Triangles Sync 90Hz Maps Rasterization Lens Distortion Simulation Eye Tracking User Input and Tracking 27
FOVEATED RENDERING Conventional Approach: Aliasing Patney et al., Towards Foveated Rendering for Gaze-Tracked Virtual Reality, SIGGRAPH Asia 2016 Patney et al., Towards Foveated Rendering for Gaze-Tracked Virtual Reality, SIGGRAPH Asia 2016 28
FOVEATED RENDERING Our Approach: Perceptually Optimized Patney et al., Towards Foveated Rendering for Gaze-Tracked Virtual Reality, SIGGRAPH Asia 2016 29
BEYOND TRIANGLES Head Tracking Primitives Renderer HMD Particles Triangles AI Display HDMI, Points Network Shadow Foveated Time Warp + Sync 90Hz Maps Rasterization Lens Distortion Text Simulation Voxels Light Fields Eye Tracking User Input and Tracking 31
McGuire et al., Real-time global illumination with light field probes, I3D 2017 32
COMPUTATIONAL DISPLAYS Head Tracking Primitives Renderer HMD Particles Triangles AI Varifocal Display HDMI, Points Network Shadow Foveated Time Warp Sync Lens Distortion 90Hz Maps Rasterization Text Simulation Voxels Light Fields Light Field Eye Tracking User Input and Tracking 33
COMPUTATIONAL DISPLAYS Light Field Display Display Prototype GPU Output Observed Image Lanman and Luebke, Near-Eye Light Field Displays, SIGGRAPH Asia 2013 34
COMPUTATIONAL DISPLAYS Varifocal Optics Laser Curved sunglasses Hologram Akşit et al., Varifocal Virtuality: A Novel Optical Layout for Near-Eye Display, SIGGRAPH 2017 Emerging Technologies 35
COMPUTATIONAL DISPLAYS Varifocal Optics Dunn et al, Wide field of view varifocal near-eye display using see-through deformable membrane mirrors, Proc. of IEEE VR 2017 36
PNEUMATIC HAPTICS Head Tracking Light Field Primitives Primitives Renderer HMD Particles Particles Triangles Triangles AI Points Points Varifocal Display HDMI, Network Shadow Foveated Text Text Time Warp Sync Lens Distortion 90Hz Maps Rasterization Simulation 2½D Video 2½D Video Voxels Voxels Light Fields Light Fields Renderer User Input and Tracking Haptics 37
LOW LATENCY Hierarchical Rendering Head Tracking Light Field Primitives Renderer HMD Remote Particles GPU Triangles AI Points Varifocal Display HDMI, Network Shadow Foveated Text Time Warp Sync Lens Distortion 90Hz Maps Rasterization Simulation 2½D Video Voxels Light Fields Eye Tracking User Input and Tracking Haptics 38
LOW LATENCY Hierarchical Rendering Local GeForce Wearable Tegra Crassin et al., CloudLight: A system for amortizing indirect lighting in real-time, JCGT 2015 39
LOW LATENCY Hierarchical Rendering Cloud GRID Platform Local GeForce Wearable Tegra Tesla GPU Crassin et al., CloudLight: A system for amortizing indirect lighting in real-time, JCGT 2015 40
LOW LATENCY Hierarchical Rendering High speed network Compressed lighting data Cloud GRID Platform Wearable Tegra Tesla GPU Crassin et al., CloudLight: A system for amortizing indirect lighting in real-time, JCGT 2015 41
LOW LATENCY Binary Frames Head Tracking Primitives Renderer HMD Light Remote Particles Field GPU Triangles AI Time Warp + Deep Focus Points HDMI, Network Shadow Foveated Varifocal Display Text Sync Maps Rasterization Simulation Lens Distortion 16000 Hz 2½D Video Voxels Light Fields Eye Tracking User Input and Tracking Haptics 42
ULTRA LOW LATENCY & HIGH THROUGHPUT Binary Frames Lincoln et al., From Motion to Photons in 80 Microseconds: Towards Minimal Latency for Virtual and Augmented Reality , IEEE VR 2016 43
LOW LATENCY Binary Frames C1 C2 Dly 27.560 µs Motion Initiated 1 C1 C3 Dly 74.360 µs 0.08 ms Data Received 2 C1 C4 Dly 97.312 µs Pixel Transmitted 3 C2 C3 Dly 46.802 µs Light Emitted 4 Ch1 2.00 V 2.00 V M 20.0µs Ch1 √ 1.20 V Ch1 Ch2 BW Ch3 2.00 V 1.00 V Ώ Ch3 Ch4 BW Lincoln et al., From Motion to Photons in 80 Microseconds: Towards Minimal Latency for Virtual and Augmented Reality, IEEE VR 2016 44
LOW LATENCY On-HMD Warping Hardware Warping Prototype Photographed in HMD Warped Static Point Set 45
Recommend
More recommend
