ACCELERATING YOUR VR APPLICATIONS WITH VRWORKS Cem Cebenoyan - - PowerPoint PPT Presentation

▶
accelerating your vr applications with vrworks
SMART_READER_LITE
LIVE PREVIEW

ACCELERATING YOUR VR APPLICATIONS WITH VRWORKS Cem Cebenoyan - - PowerPoint PPT Presentation

Sep 07, 2022 119 likes •888 views

ACCELERATING YOUR VR APPLICATIONS WITH VRWORKS Cem Cebenoyan Edward Liu 1 ACCELERATING YOUR VR APPLICATIONS WITH VRWORKS Talk Overview VRWorks Overview Graphics, audio, video stitching, physics Lens Matched Shading Deep Dive

slide-1
SLIDE 1

1

ACCELERATING YOUR VR APPLICATIONS WITH VRWORKS

Cem Cebenoyan Edward Liu

slide-2
SLIDE 2

2

ACCELERATING YOUR VR APPLICATIONS WITH VRWORKS

  • VRWorks Overview

Graphics, audio, video stitching, physics

  • Lens Matched Shading Deep Dive

Fundamental technology Engine integration details

Talk Overview

slide-3
SLIDE 3

3

PHYSX

NVIDIA VRWORKS

Bringing Reality to VR

SIGHT BEHAVIOR SOUND TOUCH

Simultaneous Multi-projection VRWorks 360 Video & CUDA PhysX Ray Traced Audio SIGHT SOUND PHYSICS & TOUCH CAPTURE

slide-4
SLIDE 4

4

GRAPHICS CAPTURE AUDIO PHYSICS & TOUCH

COMPUTING CHALLENGES IN REPRODUCING REALITY

slide-5
SLIDE 5

5

TRADITIONAL = 60 MP/S

(1920 X 1080 @ 30 FPS)

VIRTUAL REALITY = 450 MP/S

(3024 X 1680* @ 90 FPS) 1920 1080 1512 1680 1512

*VR render resolution

VR RENDERING

Ultra-High Resolution and Frame Rate

slide-6
SLIDE 6

6

Motion to Photon: ≤ 20 ms

VR PERFORMANCE DEMANDS

Ultra-Low Latency

slide-7
SLIDE 7

7

VR OPTICS

LCD display Optics User’s view

slide-8
SLIDE 8

8

Warped Image Rendered Image

VR RENDERING

slide-9
SLIDE 9

9

Rendered Image Warped Image

VR RENDERING

GPU renders many pixels that never make it to the screen

slide-10
SLIDE 10

10

VRWORKS MULTI-RES SHADING

slide-11
SLIDE 11

11 11

VRWORKS LENS MATCHED SHADING

Renders to a lens corrected surface

slide-12
SLIDE 12

12

VRWORKS MRS AND LMS

Viewport 1 Viewport 2 Viewport N

...

Geometry Pipeline Fast viewport broadcast hardware on NVIDIA Maxwell and beyond GPUs

slide-13
SLIDE 13

13 13

TRADITIONAL STEREO RENDERING

Requires 2 geometry passes

slide-14
SLIDE 14

14 14

VRWORKS SINGLE PASS STEREO

Renders left & right eye in one geometry pass

Left Eye Right Eye

slide-15
SLIDE 15

15

VRWORKS VR SLI

Scales performance across multiple GPUs

Frame 1 (Left eye) Frame 1 (Right eye) Head Tracking (t) Warped Frame Head Tracking (t+1)

slide-16
SLIDE 16

“Normal” SLI

N N+1 N N+1 N N+1 CPU GPU 0 GPU 1 Display Latency GPUs render alternate frames

slide-17
SLIDE 17

VR SLI

NL N+1R N N+1 NR N+1L N N+1 CPU GPU 0 GPU 1 Display Latency Each GPU renders one eye—lower latency

slide-18
SLIDE 18

VR SLI

Shadow maps, GPU physics, etc. Left eye rendering Right eye rendering

UINT SetGPUMask( [in] UINT GPUMask ); void RenderGPUMaskNV( [in] bitfield mask );

GPU affinity masking: full control

slide-19
SLIDE 19

VR SLI

L R

Multi-GPU API

R L

Engine

NvAPI_Status VSSetConstantBuffers( [in] ID3D11DeviceContext *pContext, [in] UINT GPUMask, [in] UINT StartSlot, [in] UINT NumBuffers, ); void MulticastBufferSubDataNV( bitfield gpuMask, uint buffer, intptr offset, sizeiptr size, const void *data );

Per-GPU constant buffers, viewports, scissors

slide-20
SLIDE 20

VRWORKS BRINGS MAJOR SPEEDUPS TO TOP VR APPS

0.0 0.4 0.8 1.2 1.6 2.0 Funhouse Everest Raw Data SportsBar VR Trials of Tatooine Without VRWorks With VRWorks

*Performance measured on GeForce GTX 1080 using VRWorks MRS, LMS, or VR SLI

Relative Performance

slide-21
SLIDE 21

21 21

COMPUTING CHALLENGES IN REPRODUCING REALITY

GRAPHICS / DISPLAY CAPTURE AUDIO PHYSICS & TOUCH

slide-22
SLIDE 22

22 22

SIMULATING AUDIO IN VR

Creation of Source Sounds Location of Incoming Sound How Sound Moves in Space

DIRECTION PROPAGATION SYNTHESIS

slide-23
SLIDE 23

23 23

Elevator Symphony Hall Meadow

AUDIO REFLECTS YOUR ENVIRONMENT

slide-24
SLIDE 24

24 24

slide-25
SLIDE 25

25 25

slide-26
SLIDE 26

26 26

Models direction and propagation using Ray tracing

slide-27
SLIDE 27

27 27

VRWORKS AUDIO SDK NOW AVAILABLE!

Features:

  • Directionality and Reverbs
  • Sound Propagation with Attenuation,

Reflection, Diffraction, Transmission

  • Realistic Occlusion
  • Dynamic Geometry

Easy To Use Presets:

  • Effect Strength
  • Materials
  • Compute

Available as SDK and Plugin for UE 4.15+

slide-28
SLIDE 28

28 28

COMPUTING CHALLENGES IN REPRODUCING REALITY

GRAPHICS / DISPLAY CAPTURE AUDIO PHYSICS & TOUCH

slide-29
SLIDE 29

29

HAPTICS

Collision detection & deformation modeling

FORCE FRICTION PhysX API PhysX Constraint Solver Haptics Layer RESTITUTION

slide-30
SLIDE 30

30 30

PHYSX FLEX HAIRWORKS FLOW DESTRUCTION CLOTH

REALISTIC PHYSICS

Simulating behavior in VR

slide-31
SLIDE 31

31 31

slide-32
SLIDE 32

32 32

COMPUTING CHALLENGES IN REPRODUCING REALITY

GRAPHICS / DISPLAY CAPTURE AUDIO TOUCH / PHYSICS

slide-33
SLIDE 33

33 33

SIGNIFICANT COMPUTATION REQUIRED TO DELIVER 360 VIDEO

Capture Stitch Display

Decode  Calibrate  Equalize  Stitch  Encode 4k cameras Single 360 video

slide-34
SLIDE 34

34

VRWORKS 360 VIDEO SDK NOW AVAILABLE!

Features

  • Real-Time & Offline Stitching
  • Up to 32 x 4k Camera Rigs

GPU Accelerated Pipeline

Decode -> Calibration -> Equalization -> Stitching -> Encode

Mono SDK Available in Beta Now! Stereo SDK Available Soon

"The fact that NVIDIA manages to stitch 4K 360 stereoscopic video in real time making live streaming possible changes the production pipeline and enables entirely new use cases in VR“ Kinson Loo, CEO of Z CAM.

slide-35
SLIDE 35

35

VRWORKS FOR UE4 AVAILABLE NOW

Features:

Multi-res Shading Lens Matched Shading Single Pass Stereo VR SLI

Available for 4.12 through 4.15 Download at: developer.nvidia.com/nvidia- vrworks-and-ue4

slide-36
SLIDE 36

36

VRWORKS FOR UNITY AVAILABLE NOW

Features:

Multi-res Shading Lens Matched Shading Single Pass Stereo VR SLI

Supported in Unity 2017.1 Beta 2 Download VRWorks plugin at www.assetstore.unity3d.com

slide-37
SLIDE 37

37

LENS MATCHED SHADING DEEP DIVE

slide-38
SLIDE 38

38

slide-39
SLIDE 39

39

slide-40
SLIDE 40

40

slide-41
SLIDE 41

41

slide-42
SLIDE 42

42

SHADING RATE AND LENS WARP

Two effects of applying lens warp to an image: 1.Periphery squashed 2.Central region magnified This distortion makes image center undersampled and periphery supersampled

slide-43
SLIDE 43

43

SHADING RATE AND LENS WARP

slide-44
SLIDE 44

44

SHADING RATE AND LENS WARP

Therefore VR applications usually render at a higher resolution than display resolution.

  • HTC Vive

Display Resolution 2160x1200, Render Resolution 3024x1680

  • Oculus Rift

Display Resolution 2160x1200, Render Resolution 2664x1586

slide-45
SLIDE 45

45

SHADING RATE AND LENS WARP

slide-46
SLIDE 46

46

LENS MATCHED SHADING BREAKDOWN

LMS approximate post lens warp shading rate by:

  • 1. Enlarge the entire viewport to increase overall shading rate
  • 2. Modify clip space w to reduce periphery shading while maintaining the center

w’ = Ax + By + w

  • 3. Apply different coefficient A, B per quadrant to always warp inward
slide-47
SLIDE 47

47

LENS MATCHED SHADING BREAKDOWN

LMS approximate post lens warp shading rate by:

  • 1. Enlarge the entire viewport to increase overall shading rate
  • 2. Modify clip space w to reduce periphery shading while maintaining the center

w’ = Ax + By + w

  • 3. Apply different coefficient A, B per quadrant to always warp inward
slide-48
SLIDE 48

48

LENS MATCHED SHADING BREAKDOWN

LMS approximate post lens warp shading rate by:

  • 1. Enlarge the entire viewport to increase overall shading rate
  • 2. Modify clip space w to reduce periphery shading while maintaining the center

w’ = Ax + By + w

  • 3. Apply different coefficient A, B per quadrant to always warp inward
slide-49
SLIDE 49

49

LENS MATCHED SHADING BREAKDOWN

LMS approximate post lens warp shading rate by:

  • 1. Enlarge the entire viewport to increase overall shading rate
  • 2. Modify clip space w to reduce periphery shading while maintaining the center

w’ = Ax + By + w

  • 3. Apply different coefficient A, B per quadrant to always warp inward
slide-50
SLIDE 50

50

LENS MATCHED SHADING BREAKDOWN

slide-51
SLIDE 51

51

LENS MATCHED SHADING BREAKDOWN

slide-52
SLIDE 52

52

PRE-SET CONFIGURATIONS

We provide 3 sets of configurations for both HTC Vive and Oculus Rift 1.Quality:

No undersampling accross the image (while reducing total # of pixels)

2.Conservative:

Undersampling no worse than baseline

3.Aggressive:

¾ Resolution of the Conservative config (keeping center shading rate high)

slide-53
SLIDE 53

53

PRE-SET CONFIGURATIONS

We also provide a scalar variable that smoothly change the shading rate It keeps the center shading rate constant Allows finer grain tuning between image quality and performance

slide-54
SLIDE 54

54

LMS VS. MRS

The 1/x profile of LMS can closer approximate the lens profile than the piecewise constant of MRS LMS needs fewer shading to achieve the same image quality with MRS LMS has a smoother shading rate transition across the image LMS uses 4 viewports per eye, while MRS uses 9. This makes LMS easier to work along with Instanced Stereo and Single Pass Stereo Fewer viewports also benefits performance

slide-55
SLIDE 55

55

LMS VS. MRS

Desired (inverse lens warp) Default (planar projection) MRS (Quality) LMS (Quality) (screen) x shading rate 1.0 1.04 1.35

slide-56
SLIDE 56

56

Baseline (no warp) 2.54 MPix / eye

MRS LMS

Conservative

(no worse than baseline)

Aggressive

(3/4 Reso. of conservative)

1.57 MPix / eye 1.17 MPix / eye 0.87 MPix / eye 2.03 MPix / eye 1.58 MPix / eye 1.40 MPix / eye Quality

(no undersampling)

LMS vs. MRS

slide-57
SLIDE 57

57

1.57 1.17 0.87 2.03 1.58 1.4 2.54

0.5 1 1.5 2 2.5 3

Baseline Quality Conservative Aggressive

Number of Pixels (Millions)

LMS/MRS Shading Rate Comparison

LMS MRS Baseline

slide-58
SLIDE 58

58

ENGINE INTEGRATION OVERVIEW

  • 1. Fill Gbuffer in LMS Space
  • 2. Shading in LMS Space
  • 3. Post Processing in LMS Space
  • 4. Resample to Linear Space
slide-59
SLIDE 59

59

LMS COORDINATES SYSTEM

Linear Space UV LMS Space UV (0, 0) (0, 1) (1, 0) (1, 1) (1, 1) (1, 0) (0, 0) (0, 1) LMS Space Stereo UV (1, 1) (1, 0) (0, 0) (0, 1) (0.5, 0) (0.5, 1) (0.5, 1) (0.5, 0)

Z value changed too!

slide-60
SLIDE 60

60

RENDER PASSES

  • Passes that renders geometries
  • Need to enable w modification, bind fast geometry shaders and set up

multiple viewports & scissors before draw call submission, for which we have provided helper functions.

View.BeginVRProjectionStates(RHICmdList); // Invoke draw calls.. View.EndVRProjectionStates(RHICmdList);

  • Screen space passes, eg. SSR, SSAO and other post processing
  • Invoke full view rendering by rendering an octagon
slide-61
SLIDE 61

61

FULL SCREEN OCTAGON

  • Normally, full screen quads are used

to invoke shading for most screen space passes

  • With LMS, full screen octagon is used

to avoid drawing to pixels outside of the valid region

  • The UV in the octagon should span

linearly across the view.

(0 ,0) (1, 0) (1, 1) (0, 1)

slide-62
SLIDE 62

62

BOUNDARY MASK

  • A boundary mask is rendered during

Z pre-pass to avoid rendering to pixels outside of the octagon region

  • Need to carefully bind depth buffer

for passes that did not have it bound

slide-63
SLIDE 63

63

BASE PASS

  • Fast GS, multiple viewport & scissors, w modification
  • Z values used in base pass PS needed to be remapped from LMS space to linear
  • Modified FHLSLMaterialTranslator::PixelDepth() so the generated

code produce Z value in linear space

  • Instanced Stereo requires more changes
slide-64
SLIDE 64

64

INSTANCED STEREO

  • Using a single viewport that encompasses both view breaks W modification
  • We set up 8 viewports and rely on multiprojection to send primitives to the

correct viewports.

1 4 3 5 8 6 7 2

Left shift 4 bits

slide-65
SLIDE 65

65

DEFERRED LIGHTING

  • Directional light
  • Use full screen octagon to invoke shading
  • Point and spot light
  • Apply w modification when render light volume geometries
  • Boundary mask does not work when camera is inside light volume, Z test disabled
  • Set boundary mark in the stencil buffer instead
  • Dynamic shadows
  • Shadow map generation not affected. Apply w modification to Shadow projection.
  • Tiled based lighting
  • Kill thread group if all threads are outside of the octagon covered region
slide-66
SLIDE 66

66

SCREEN SPACE REFLECTIONS AND AO

  • Invoke shading with full screen octagon
  • SSR and SSAO both samples the HZB
  • The Z values are remapped to linear space at HZB creation time to avoid

sampling time overhead

slide-67
SLIDE 67

67

SCREEN SPACE REFLECTIONS AND AO

  • It’s difficult to apply marching offset in LMS space, so we transform sample coordinates

to linear space before applying offset in linear space. Remap it back to LMS space for sampling

To Linear To LMS Sample Origin in LMS Space Stepped Position in Linear Space Stepped Position Remapped to LMS Space

slide-68
SLIDE 68

68

COORDINATE REMAP IN SHADERS

  • All PS input data are in LMS space (eg. SvPosition, ScreenPos, UV)
  • Fetching from GBuffer should use LMS space coordinate
  • Gbuffer indexed in LMS space, so we can directly use UV passed in from Octagon VS

to fetch Gbuffer

  • Use linear space coordinates to do any world space compupation
  • Therefore PS input needs to be remapped to linear space before doing computation

with GBuffer data

  • GBuffer content are in linear space, or world space
slide-69
SLIDE 69

69

SHADER MODIFICATION EXAMPLE

The input SVPos is in LMS space. So convert it to linear space, since CameraVector is used to calculate lighting with GBuffer data, which is also in linear space. InUV is LMS space. When fetching data from GBuffers, use LMS space coordinates directly Since GBuffer is indexed in LMS space.

slide-70
SLIDE 70

70

2 4 6 8 10 12 Everest Realistic Rendering Landscape Mountain

Frame time (Miliseconds)

LMS/MRS Performance Benchmark

Regular LMS1 LMS2 LMS3 MRS3

slide-71
SLIDE 71

71

MRS LMS LMS MRS

slide-72
SLIDE 72

72

NOTE ON PERFORMANCE

  • Pixel shading is only part of the frame
  • LMS won’t help with geometry or CPU work
  • Linear and LMS coordinate conversion isn’t free
  • Nonideal in passes like SSAO, SSR
  • Linear resampling at the end isn’t free
  • We do this at high resolution to keep the center sharpness
slide-73
SLIDE 73

73

DEMO

slide-74
SLIDE 74

74

slide-75
SLIDE 75

75

Thank You!

cem@nvidia.com edliu@nvidia.com developer.nvidia.com/vr