Virtual Reality Features of NVIDIA GPUs Mark Kilgard, July 27, 2016 - PowerPoint PPT Presentation

Virtual Reality Features of NVIDIA GPUs Mark Kilgard, July 27, 2016

Mark Kilgard My Background Principal System Software Engineer OpenGL driver implementation & API evolution Other interests Programmable shading languages, original Cg GPU shading language Web, vector graphics & path rendering, GPU rendering for 2D graphics Also known for creating OpenGL Utility Toolkit (GLUT) 2

GPU Challenges for Virtual Reality Rendering 1. Must render unique view for each eye Two scene renders per frame 2. Wide field-of-view rendered More pixels, distributed towards periphery 3. HMD lens magnifies rendered scene Non-linear inverse lens warp Allows the HMD lens magnification to provide convincing sense of presence 3

Magnification Effect of HMD Lens Graph paper viewed & magnified through HMD lens 4

Head Mounted Display Lens Effect Barrel Distorted Images on HMD Screen Magnified Overlapping Wide Field-of-view Binocular View 5

Focus on Virtual Reality Features of Pascal GPUs Virtual Reality rendering features in Pascal GPUs GeForce 1080/1070/1060 NEW Quadro P6000 & P5000 As exposed in standard 3D APIs 6

Background: NVIDA GPU Architecture Road Map What is Pascal mentioned on last slide? Our interest NVIDIA GPU architectures of interest: Pascal 7

Conventional 3D Single-eye Scene 8

Conventional 3D Single-eye Scene 9

Wide Stereo Views of Same Scene Left Right 10

Left Right 11

Right Left 12

Left & Right Views Are Similar but Different Left and Right eye views have different occlusion While left & right eye views look similar, must be draw independently Really only different in a small translation in X NVIDIA Single Pass Stereo supports generating left & right eye positions GPU automatically makes left & right eye versions of each primitive Then renders each view to a different framebuffer slice Exposed in OpenGL by NV_stereo_view_rendering extension Exposed for Direct3D 11 by NVAPI SinglePassStereoMode feature 13

Image Difference of Two Views − + 0.5 = Left eye view Right eye view Clamped difference image 14

Left & Right Views Rendered Efficiently in One Pass 15

Lens Matched Shading Left Right 16

Visualizing Lens Matched Shading 17

Lens Matched Shading Left Right 18

Visualization of Lens Matched Shading Rendering 19

Still Needs Non-linear Warp Non-linear image Lens Matched Shading Warped image suitable rendering for HMD display 20

Still Needs Non-linear Warp Non-linear image BUT substantially less pixel motion and distortion than conventional VR re-warping Lens Matched Shading Warped image suitable rendering for HMD display 21

Example HMD Post-rendering Warp 22

Lens Matched Shading with Window Rectangle Testing 23

Lens Matched Shading with Window Rectangle Testing Nothing in black corners is shaded or even rasterized Yellow lines show overlaid 8 inclusive window rectangles 24 Same 8 window rectangles “shared” by each view’s texture array layer

Lens Matched Shading with Window Rectangle Testing 25

What Didn’t Have to be Rendered 17% of left view pixels 17% of right view pixels 26

Window Rectangle Testing During Rasterization New multi-vendor OpenGL extension EXT_window_rectangles Window Rectangles Test Provides a set of window-space rectangles, 8 for NVIDIA GPUs NEW Rasterizer either includes or excludes stage rasterization from rectangles Discards at speed-of-light As if free to rasterize to irregular rectangular regions OpenGL’s fragment processing pipeline 27

Warped Lens Matched Scene Warped version of Lens Matched Shading to match HMD lens 28

Warped Lens Matched Shading 29

Visualizing Warp Window Rectangles 30

What Didn’t Have to be Warped 17% of left view pixels 17% of right view pixels 31

VR Rendering Pipeline in Images LMS Left Eye View Warped Left Eye View Scene LMS Right Eye View Warped Right Eye View LMS = Lens Matched Shading 32

Pascal does all this VR Rendering Pipeline in Images efficiently in a single LMS Left Eye View Warped Left Eye View rendering pass! 8 viewports, 1 pass Displayed within HMD Scene LMS Right Eye View Warped Right Eye View Single Rendering Pass Drawn with Single Triangle Perception to user is linear rendering Single Pass Stereo + Fragment Shader Warping HMD lens “undoes” warping to provide Lens Matched Shading + Window Rectangle Testing 33 a perceived wide field-of-view Window Rectangle Testing

Traditional 3D Rendering Pipeline uses pinhole camera projection, rasterizing to uniform rectilinear pixel grid App Transform Rasterize Display Downsample Processing 3D View 3D View Frame monitor directly displays uniform rectilinear pixel grid if multisampled 34

Virtual Reality Rendering Pipeline Rendering Left & Right Eye Views warps uniform rectilinear grid based on HMD’s device profile Transform Rasterize HMD Downsample Left View Left View Warp Display App Frame to HMD Processing Transform Transform Transform Transform Transform Rasterize Rasterize Rasterize Rasterize HMD HMD Downsample Downsample Downsample Right View Right View Right View Right View Right View Right View Right View Right View Right View Warp Warp second view rendering 35

Virtual Reality Rendering Pipeline With Lens Matched Shading less pixel resampling effort & error operating on LMS rasterized images Lens Transform Rasterize HMD Downsample Matched Left View Left View Warp Shading Display App Frame to HMD Processing Lens Transform Rasterize HMD Downsample Matched Right View Right View Warp Shading replicates primitive to 4 projected quadrants 36

Virtual Reality Rendering Pipeline With Single Pass Stereo Rasterize HMD Downsample Left View Warp Transform Display App Left & Right Frame to HMD Processing View Rasterize HMD Downsample Right View Warp one transform instance, outputs left & right position 37

Virtual Reality Rendering Pipeline Single Pass Stereo + Lens Matched Shading use window rectangle testing to bound lens region, during both rasterization & warp Lens Rasterize HMD Downsample Matched Left View Warp Shading Transform Display App Left & Right Frame to HMD Processing View Lens Rasterize HMD Downsample Matched Right View Warp Shading Advantages + 1. Single transform for both views 2. Closer match between rendering & lens images 3. Fewer overall pixels for similar quality 38

Pascal GPU Features for VR Unique to NVIDIA Pascal renders left & right eye Better matches rendering to HMD lens In single rendering pass Single pass Halves CPU load for rendering Four separately projected quadrants Better concurrent texture locality Less rendering, less resampling error OpenGL: NV_stereo_view_rendering OpenGL: NV_clip_space_w_scaling Direct3D: NVAPI SinglePassStereoMode Direct3D: NVAPI ModifiedWSupport 39

Other NVIDIA Virtual Reality Initiatives 40

More Information VRWORKS 2.0 Software Development Kit Pascal OpenGL extensions for Virtual Reality Available NOW Specification now public In official OpenGL.org extension registry OpenGL Extension Wrangler 2.0 (GLEW) support Example code in VRWORKS 2.0 SDK VRWORKS also documents NVAPI usage For Direct3D 11 Includes Examples code 41