DLSS 2.0 IMAGE RECONSTRUCTION FOR REAL-TIME RENDERING WITH DEEP - - PowerPoint PPT Presentation

▶
dlss 2 0 image reconstruction for real time rendering
SMART_READER_LITE
LIVE PREVIEW

DLSS 2.0 IMAGE RECONSTRUCTION FOR REAL-TIME RENDERING WITH DEEP - - PowerPoint PPT Presentation

Oct 16, 2023 129 likes •899 views

DLSS 2.0 IMAGE RECONSTRUCTION FOR REAL-TIME RENDERING WITH DEEP LEARNING Shiqiu (Edward) Liu, Sr. Research Scientist, August 6 th 2020 ABOUT ME 7 years of real-time rendering research and development at NVIDIA Key contributors to several

slide-1
SLIDE 1

Shiqiu (Edward) Liu, Sr. Research Scientist, August 6th 2020

DLSS 2.0 – IMAGE RECONSTRUCTION FOR REAL-TIME RENDERING WITH DEEP LEARNING

slide-2
SLIDE 2

3

ABOUT ME

7 years of real-time rendering research and development at NVIDIA Key contributors to several next-gen rendering technologies

  • DLSS, real-time ray tracing and denoising, VR Rendering

Developed technologies/algorithms that shipped in many titles and game engines

slide-3
SLIDE 3

4

NEXT GEN GAMES NEED SUPER RESOLUTION

Ray tracing, physics, AR/VR, and higher resolution displays drive up GPU computing needs exponentially Ray tracing alone can demand many times the computing power of traditional rendering techniques Super resolution technique become necessary RTX GPUs have tensor cores to accelerate deep learning workloads

slide-4
SLIDE 4

5

DLSS 1.0

slide-5
SLIDE 5

6

INTRODUCING DLSS 2.0

1.5ms at 4K on 2080Ti

Works on all RTX GPUs at all resolution

Great Image Quality

Details rival native resolution

Generalized Model

One model to rule them all!

4x Upscaling Ratio

540p to 1080p, 1080p to 4k

1080p 4K

slide-6
SLIDE 6

8

720p

slide-7
SLIDE 7

9

DLSS 1.0 720p to 1080p

slide-8
SLIDE 8

10

720p

slide-9
SLIDE 9

11

DLSS 1.0 720p to 1080p

slide-10
SLIDE 10

12

DLSS 2.0 720p to 1080p

slide-11
SLIDE 11

13

DLSS 1.0 720p to 1080p

slide-12
SLIDE 12

14

DLSS 2.0 720p to 1080p

slide-13
SLIDE 13

15

1080p

slide-14
SLIDE 14

16

DLSS 2.0 720p to 1080p

slide-15
SLIDE 15

17

1080p

slide-16
SLIDE 16

18

DLSS 2.0 720p to 1080p

slide-17
SLIDE 17

540p

slide-18
SLIDE 18

20

DLSS 2.0 540p to 1080p

slide-19
SLIDE 19

540p

slide-20
SLIDE 20

22

DLSS 2.0 540p to 1080p

slide-21
SLIDE 21

23

23

1080p TAA 540p TAA 540p DLSS2.0 720p DLSS1.0

slide-22
SLIDE 22

24

24

1080p TAA 540p TAA 540p DLSS2.0 720p DLSS1.0

slide-23
SLIDE 23

25

540p - 89fps

slide-24
SLIDE 24

26

1080p - 48fps

slide-25
SLIDE 25

27

540p to 1080p w/ DLSS2.0 - 86fps

slide-26
SLIDE 26

28

1080p - 48fps

slide-27
SLIDE 27

29

540p to 1080p w/ DLSS2.0 - 86fps

slide-28
SLIDE 28

30

32spp Reference 1080p

slide-29
SLIDE 29

31

540p to 1080p w/ DLSS2.0 - 86fps

slide-30
SLIDE 30

32

32spp Reference 1080p

slide-31
SLIDE 31

33

32spp Reference 540p TAA 540p DLSS2.0 1080p TAA

slide-32
SLIDE 32

34

DLSS 2.0 - ACCELERATED RENDERING

16 6 1.5

DLSS OFF DLSS ON

Rendering Cost DLSS Cost

slide-33
SLIDE 33

35

DLSS 2.0 PERFORMANCE BOOSTS

Performance Mode 1080p to 4K

slide-34
SLIDE 34

36

DLSS is impressive to the point where I believe you'd be nuts not to use it.

Digital Foundry

The upscaling power of this new AI driven algorithm is extremely impressive… its basically a free performance button.

Hardware Unboxed

slide-35
SLIDE 35

37

SHIPPING IN THE FOLLOWING TITLES

slide-36
SLIDE 36

38

CHALLENGES IN IMAGE SUPER-RES FOR REAL-TIME RENDERING

slide-37
SLIDE 37

39

RECONSTRUCTION 101

Ground Truth Function Discrete Samples Reconstructed Function

slide-38
SLIDE 38

40

RECONSTRUCTION 101

Double the sampling rate

Ground Truth Function Discrete Samples Reconstructed Function

slide-39
SLIDE 39

41

RECONSTRUCTION 101

slide-40
SLIDE 40

42

RECONSTRUCTION 101

slide-41
SLIDE 41

43

DLSS PROBLEM STATEMENT

Low resolution sampling rate High resolution reconstruction

slide-42
SLIDE 42

44

DLSS PROBLEM STATEMENT

Resolution / Image Quality Cost of Rendering With DLSS Traditional Better Performance

slide-43
SLIDE 43

45

Reconstruct high resolution image by interpolating the low-resolution pixels Common choices are bilinear, bicubic, lanczos Contrast aware sharpening deep neural networks can hallucinate new pixels conditioned on existing pixels based on priors or training data

SINGLE IMAGE SUPER-RES

Previous work

[Ledig et al. 2017]

slide-44
SLIDE 44

46

Resulted images lack details compared to native high-resolution images Images may be inconsistent with native rendering because of hallucination, and temporally unstable

SINGLE IMAGE SUPER-RES

Reconstructed with linear interpolation Reconstructed with DL or other interpolation High res samples

slide-45
SLIDE 45

47

Resulted images lack details compared to native high-resolution images Images may be inconsistent with native rendering because of hallucination, and temporally unstable

SINGLE IMAGE SUPER-RES

DL Upscaled 720p to 1080p Native Rendering 1080p

slide-46
SLIDE 46

48

1080p with TAA

slide-47
SLIDE 47

49

540p to 1080p DLSS2.0

slide-48
SLIDE 48

50

540p Bicubic Upsampled to 1080p

slide-49
SLIDE 49

51

540p to 1080p with ESRGAN

slide-50
SLIDE 50

52

540p ESRGAN 540p TAA Bicubic 540p DLSS2.0 1080p TAA

slide-51
SLIDE 51

53

Less ill-posed than single image super-res, restore true optical details better Designed for videos or burst mode photography, not leveraging rendering specific information

  • Optical flow vs. geometric motion vector
  • Pixels vs samples
  • Using frames in the future

MULTI-FRAME SUPER-RES

Previous work

[Wronski et al. 2019] [Sajjadi et al. 2018]

slide-52
SLIDE 52

54

SPATIAL-TEMPORAL SUPER SAMPLING

Temporal Antilasing (TAA) Checkerboard Rendering (CBR) Temporal Upsampling

Previous work

[Yang09, Lottes11, Sousa11, Karis14, Salvi16] [ElMansouri16, Carpentier17, Wilidal17] [Yang09, Herzog10, Malan12, Valient14, Aalto16, Epic18] References can be found in <A Survey of Temporal Antialiasing Techniques>, Yang et al.

slide-53
SLIDE 53

55

Reconstruct high resolution image using samples from across multiple frames Effective sampling rate drastically increased Reconstructed image much closer to ground truth

SPATIAL-TEMPORAL SUPER SAMPLING

Ground Truth Function Discrete Samples Reconstructed Function

  • Prev. Function
  • Prev. Discrete Samples
slide-54
SLIDE 54

56

Samples from previous frames might no longer be correct due to content changes Using samples from previous frame naively might lead to artifacts like ghosting

SPATIAL-TEMPORAL SUPER SAMPLING

The deil i he deail

slide-55
SLIDE 55

57

Traditional spatial-temporal upsampling algorithms leverages heuristics to rectify invalid samples from previous frames However common rectification heuristics often trade off between different artifacts: Blurriness, temporal instability, even moire pattern vs. lagging and ghosting

SPATIAL-TEMPORAL UPSAMPLING

History Rectification

[Yang et al. 2020]

slide-56
SLIDE 56

58

NEIGHBORHOOD CLAMPING

Most commonly used sample rectification technique [Karis14], [Salvi16] Clamp previous frames samples to the min/max of the neighboring current frame samples Resulted in loss in details in the reconstructed image

Discrete Samples

  • Prev. Discrete Samples
slide-57
SLIDE 57

59

Ghosting Happens without History Rectification

slide-58
SLIDE 58

60

NEIGHBORHOOD CLAMPING

Blurriness/Losing details

Reconstruction with clamping Reconstruction without clamping 1spp Input

[Yang et al. 2020]

slide-59
SLIDE 59

61

NEIGHBORHOOD CLAMPING

When perform temporal upsampling, clamping introduces more loss in detail Since bounding boxes are calculated from a low-resolution image

Blurriness/Losing detail

slide-60
SLIDE 60

62

Reconstruction with clamping, ¼ res input Reconstruction w/o clamping, ¼ res input

slide-61
SLIDE 61

63

Reconstruction with clamping, ¼ res input Reconstruction w/o clamping, ¼ res input

slide-62
SLIDE 62

65

Temporal Instability 1080p TAA with clamping

slide-63
SLIDE 63

66

Temporal Instability 540p to 1080p TAAU with Clamping

slide-64
SLIDE 64

67

NEIGHBORHOOD CLAMPING

Temporal Instability and Moire

Frame N Frame N+1 Frame N+2 Frame N+3

[Yang et al. 2020] Before Clamping After Clamping

Discrete Samples

  • Prev. Discrete Samples
slide-65
SLIDE 65

68

540p to 1080p TAAU w/o Clamping

slide-66
SLIDE 66

69

REAL-TIME SUPER-RES CHALLENGES

Single frame approach

Blurry image quality Inconsistent with native rendering Temporally unstable

Multi-frame approach

Heuristics to detect and rectifies changes across frames Limitation in heuristics causing blurriness, temporal instability and ghosting

slide-67
SLIDE 67

70

DLSS 2.0: DL BASED MULTI-FRAME RECONSTRUCTION

DLSS uses a neural network trained from tens of thousands of high-quality images Neural networks are much more powerful than handcrafted heuristics Much higher quality reconstructions using samples from multiple frames

Multi-frame samples and GT function Ground Truth Function Discrete Samples Reconstructed Function

  • Prev. Function
  • Prev. Discrete Samples
slide-68
SLIDE 68

71

DLSS 2.0: DL BASED MULTI-FRAME RECONSTRUCTION

DLSS uses a neural network trained from tens of thousands of high-quality images Neural networks are much more powerful than handcrafted heuristics Much higher quality reconstructions using samples from multiple frames

Multi-frame samples and GT function Ground Truth Function Discrete Samples Reconstructed Function

  • Prev. Function
  • Prev. Discrete Samples
slide-69
SLIDE 69

72

DLSS 2.0: DL BASED MULTI-FRAME RECONSTRUCTION

DLSS uses a neural network trained from tens of thousands of high-quality images Neural networks are much more powerful than handcrafted heuristics Much higher quality reconstructions using samples from multiple frames

Multi-frame samples and GT function Non-DL reconstruction DLSS reconstruction Ground Truth Function Discrete Samples Reconstructed Function

  • Prev. Function
  • Prev. Discrete Samples
slide-70
SLIDE 70

73

DLSS 2.0 540p to 1080p

slide-71
SLIDE 71

77

1080p TAA 540p DLSS 2.0 540p TAAU

slide-72
SLIDE 72

81

1080p TAA 540p DLSS 2.0 540p TAAU

slide-73
SLIDE 73

Thank you!

: @edliu1105 : @文刀秋二