ACCELERATING YOUR VR APPLICATIONS WITH VRWORKS Cem Cebenoyan - PowerPoint PPT Presentation

ACCELERATING YOUR VR APPLICATIONS WITH VRWORKS Cem Cebenoyan Edward Liu 1

ACCELERATING YOUR VR APPLICATIONS WITH VRWORKS Talk Overview • VRWorks Overview Graphics, audio, video stitching, physics Lens Matched Shading Deep Dive • Fundamental technology Engine integration details 2

NVIDIA VRWORKS Bringing Reality to VR SIGHT SOUND PHYSICS & TOUCH CAPTURE SIGHT SOUND TOUCH BEHAVIOR PHYSX VRWorks 360 Video Simultaneous Ray Traced PhysX & CUDA Multi-projection Audio 3

COMPUTING CHALLENGES IN REPRODUCING REALITY PHYSICS & GRAPHICS AUDIO CAPTURE TOUCH 4

VR RENDERING Ultra-High Resolution and Frame Rate TRADITIONAL = 60 MP/S VIRTUAL REALITY = 450 MP/S ( 1920 X 1080 @ 30 FPS) ( 3024 X 1680* @ 90 FPS) 1080 1680 1920 1512 1512 *VR render resolution 5

VR PERFORMANCE DEMANDS Ultra-Low Latency Motion to Photon: ≤ 20 ms 6

VR OPTICS Optics LCD display User’s view 7

VR RENDERING Rendered Image Warped Image 8

VR RENDERING GPU renders many pixels that never make it to the screen Warped Image Rendered Image 9

VRWORKS MULTI-RES SHADING 10

VRWORKS LENS MATCHED SHADING Renders to a lens corrected surface 11 11

VRWORKS MRS AND LMS Fast viewport broadcast hardware on NVIDIA Maxwell and beyond GPUs Viewport 1 Viewport 2 Geometry Pipeline ... Viewport N 12

TRADITIONAL STEREO RENDERING Requires 2 geometry passes 13 13

VRWORKS SINGLE PASS STEREO Renders left & right eye in one geometry pass Left Eye Right Eye 14 14

VRWORKS VR SLI Scales performance across multiple GPUs Frame 1 (Left eye) Warped Frame Frame 1 (Right eye) Head Tracking (t) Head Tracking (t+1) 15

“ Normal ” SLI GPUs render alternate frames CPU N N+1 GPU 0 N GPU 1 N+1 Display N N+1 Latency

VR SLI Each GPU renders one eye — lower latency N N+1 CPU N L N+1 L GPU 0 N R N+1 R GPU 1 Display N N+1 Latency

VR SLI GPU affinity masking: full control Left eye rendering Shadow maps, GPU physics, etc. Right eye rendering UINT SetGPUMask( void RenderGPUMaskNV( [in] UINT GPUMask [in] bitfield mask ); );

VR SLI Per-GPU constant buffers, viewports, scissors L L R Multi-GPU Engine API R void MulticastBufferSubDataNV( NvAPI_Status VSSetConstantBuffers( bitfield gpuMask, [in] ID3D11DeviceContext *pContext, uint buffer, [in] UINT GPUMask, intptr offset, [in] UINT StartSlot, sizeiptr size, [in] UINT NumBuffers, const void *data ); );

VRWORKS BRINGS MAJOR SPEEDUPS TO TOP VR APPS Without VRWorks With VRWorks 2.0 1.6 Relative Performance 1.2 0.8 0.4 0.0 Funhouse Everest Raw Data SportsBar VR Trials of Tatooine *Performance measured on GeForce GTX 1080 using VRWorks MRS, LMS, or VR SLI

COMPUTING CHALLENGES IN REPRODUCING REALITY GRAPHICS / PHYSICS & AUDIO CAPTURE DISPLAY TOUCH 21 21

SIMULATING AUDIO IN VR SYNTHESIS DIRECTION PROPAGATION Creation of Source Sounds Location of Incoming Sound How Sound Moves in Space 22 22

AUDIO REFLECTS YOUR ENVIRONMENT Elevator Symphony Hall Meadow 23 23

24 24

25 25

Models direction and propagation using Ray tracing 26 26

VRWORKS AUDIO SDK Features: NOW AVAILABLE! • Directionality and Reverbs • Sound Propagation with Attenuation, Reflection, Diffraction, Transmission • Realistic Occlusion • Dynamic Geometry Easy To Use Presets: • Effect Strength • Materials • Compute Available as SDK and Plugin for UE 4.15+ 27 27

COMPUTING CHALLENGES IN REPRODUCING REALITY GRAPHICS / PHYSICS & AUDIO CAPTURE DISPLAY TOUCH 28 28

HAPTICS Collision detection & deformation modeling FORCE FRICTION RESTITUTION PhysX API PhysX Constraint Solver Haptics Layer 29

REALISTIC PHYSICS Simulating behavior in VR PHYSX FLEX DESTRUCTION HAIRWORKS FLOW CLOTH 30 30

31 31

COMPUTING CHALLENGES IN REPRODUCING REALITY GRAPHICS / TOUCH / AUDIO CAPTURE DISPLAY PHYSICS 32 32

SIGNIFICANT COMPUTATION REQUIRED TO DELIVER 360 VIDEO Display Capture Stitch Decode  Calibrate  Equalize  Stitch  Encode 4k cameras Single 360 video 33 33

"The fact that NVIDIA manages to stitch 4K 360 VRWORKS 360 VIDEO SDK stereoscopic video in real time making live streaming possible changes the production pipeline NOW AVAILABLE! and enables entirely new use cases in VR“ Kinson Loo, CEO of Z CAM. Features - Real-Time & Offline Stitching - Up to 32 x 4k Camera Rigs GPU Accelerated Pipeline Decode -> Calibration -> Equalization -> Stitching -> Encode Mono SDK Available in Beta Now! Stereo SDK Available Soon 34

VRWORKS FOR UE4 AVAILABLE NOW Features: Multi-res Shading Lens Matched Shading Single Pass Stereo VR SLI Available for 4.12 through 4.15 Download at: developer.nvidia.com/nvidia- vrworks-and-ue4 35

VRWORKS FOR UNITY AVAILABLE NOW Features: Multi-res Shading Lens Matched Shading Single Pass Stereo VR SLI Supported in Unity 2017.1 Beta 2 Download VRWorks plugin at www.assetstore.unity3d.com 36

LENS MATCHED SHADING DEEP DIVE 37

38

39

40

41

SHADING RATE AND LENS WARP Two effects of applying lens warp to an image: 1.Periphery squashed 2.Central region magnified This distortion makes image center undersampled and periphery supersampled 42

SHADING RATE AND LENS WARP 43

SHADING RATE AND LENS WARP Therefore VR applications usually render at a higher resolution than display resolution. • HTC Vive Display Resolution 2160x1200, Render Resolution 3024x1680 • Oculus Rift Display Resolution 2160x1200, Render Resolution 2664x1586 44

SHADING RATE AND LENS WARP 45

LENS MATCHED SHADING BREAKDOWN LMS approximate post lens warp shading rate by: 1. Enlarge the entire viewport to increase overall shading rate 2. Modify clip space w to reduce periphery shading while maintaining the center w’ = Ax + By + w 3. Apply different coefficient A, B per quadrant to always warp inward 46

LENS MATCHED SHADING BREAKDOWN LMS approximate post lens warp shading rate by: 1. Enlarge the entire viewport to increase overall shading rate 2. Modify clip space w to reduce periphery shading while maintaining the center w’ = Ax + By + w 3. Apply different coefficient A, B per quadrant to always warp inward 47

LENS MATCHED SHADING BREAKDOWN LMS approximate post lens warp shading rate by: 1. Enlarge the entire viewport to increase overall shading rate 2. Modify clip space w to reduce periphery shading while maintaining the center w’ = Ax + By + w 3. Apply different coefficient A, B per quadrant to always warp inward 48

LENS MATCHED SHADING BREAKDOWN LMS approximate post lens warp shading rate by: 1. Enlarge the entire viewport to increase overall shading rate 2. Modify clip space w to reduce periphery shading while maintaining the center w’ = Ax + By + w 3. Apply different coefficient A, B per quadrant to always warp inward 49

LENS MATCHED SHADING BREAKDOWN 50

LENS MATCHED SHADING BREAKDOWN 51

PRE-SET CONFIGURATIONS We provide 3 sets of configurations for both HTC Vive and Oculus Rift 1.Quality: No undersampling accross the image (while reducing total # of pixels) 2.Conservative: Undersampling no worse than baseline 3.Aggressive: ¾ Resolution of the Conservative config (keeping center shading rate high) 52

PRE-SET CONFIGURATIONS We also provide a scalar variable that smoothly change the shading rate It keeps the center shading rate constant Allows finer grain tuning between image quality and performance 53

LMS VS. MRS The 1/x profile of LMS can closer approximate the lens profile than the piecewise constant of MRS LMS needs fewer shading to achieve the same image quality with MRS LMS has a smoother shading rate transition across the image LMS uses 4 viewports per eye, while MRS uses 9. This makes LMS easier to work along with Instanced Stereo and Single Pass Stereo Fewer viewports also benefits performance 54

LMS VS. MRS shading Desired (inverse lens warp) rate Default (planar projection) MRS (Quality) 1.35 LMS (Quality) 1.04 1.0 (screen) x 55

Conservative Aggressive Quality LMS vs. MRS (no undersampling) (no worse than baseline) (3/4 Reso. of conservative) LMS 1.17 MPix / eye 0.87 MPix / eye 1.57 MPix / eye Baseline MRS (no warp) 2.54 MPix / eye 1.58 MPix / eye 2.03 MPix / eye 1.40 MPix / eye 56

LMS/MRS Shading Rate Comparison 3 LMS MRS Baseline 2.54 Number of Pixels (Millions) 2.5 2.03 2 1.58 1.57 1.5 1.4 1.17 1 0.87 0.5 0 Baseline Quality Conservative Aggressive 57

ENGINE INTEGRATION OVERVIEW 1. Fill Gbuffer in LMS Space 2. Shading in LMS Space 3. Post Processing in LMS Space 4. Resample to Linear Space 58

LMS COORDINATES SYSTEM (0, 0) (1, 0) (0, 0) (1, 0) (0, 0) (0.5, 0) (0.5, 0) (1, 0) (0, 1) (0.5, 1) (0.5, 1) (1, 1) (0, 1) (1, 1) (0, 1) (1, 1) Linear Space UV LMS Space UV LMS Space Stereo UV Z value changed too! 59