TACKLING 3D TOF ARTIFACTS THROUGH LEARNING AND THE FLAT DATASET - - PowerPoint PPT Presentation

▶
tackling 3d tof artifacts through learning and the flat
SMART_READER_LITE
LIVE PREVIEW

TACKLING 3D TOF ARTIFACTS THROUGH LEARNING AND THE FLAT DATASET - - PowerPoint PPT Presentation

Oct 20, 2022 160 likes •1.59k views

TACKLING 3D TOF ARTIFACTS THROUGH LEARNING AND THE FLAT DATASET Iuri Frosio, GTC 2019 (San Jose, CA) THE IMPORTANCE OF NEGATIVE RESULTS I shall require that [the] logical form [of the theory] shall be such that it can be singled out, by

slide-1
SLIDE 1

Iuri Frosio, GTC 2019 (San Jose, CA)

TACKLING 3D TOF ARTIFACTS THROUGH LEARNING AND THE FLAT DATASET

slide-2
SLIDE 2

2

THE IMPORTANCE OF NEGATIVE RESULTS

“I shall require that [the] logical form [of the theory] shall be such that it can be singled

  • ut,

by means

  • f

empirical tests, in a negative sense: it must be possible for an empirical scientific system to be refuted by experience” (Karl Popper, The Logic

  • f

Scientific Discovery, 1959). In simple words, “negative results are fundamentals for the advancement

  • f

science”.

slide-3
SLIDE 3

3

TACKLING 3D TOF ARTIFACTS THROUGH LEARNING AND THE FLAT DATASET

  • Time Of Flight (TOF) cameras & artifacts
  • Naïve Machine Learning (ML) for TOF reconstruction
  • TOF cameras: working principles
  • Camera calibration
  • The FLAT dataset
  • Spoiler: our non-Naïve ML solution works*
  • Back to physics
  • DNN architecture
  • Results
  • Conclusion

Agenda

* See Qi Guo, Iuri Frosio, Orazio Gallo, Todd Zickler, Jan Kautz, Tackling 3D ToF Artifacts Through Learning and the FLAT Dataset, ECCV 2018, Munich (Germany),

  • Sept. 2018.
slide-4
SLIDE 4

4

TIME OF FLIGHT (TOF) CAMERAS & ARTIFACTS

E.g., Kinect 2

Image from https://stackoverflow.com/questions/22921390/how-to-scale-a-kinect- depth-image-for-applying-lbp-on-it-in-matlab?rq=1

slide-5
SLIDE 5

5

TIME OF FLIGHT (TOF) CAMERAS & ARTIFACTS

Applications

Image from https://www.physio- pedia.com/The_emerging_role_of_Microsoft_Kinect_in_physiotherapy_ rehab ilitation_for_stroke_patients Image from amazon.com Image from https://www.eenewsautomotive.com/news/3d-lidar-generates-e nviro nmental-model-time-flight- measurement

slide-6
SLIDE 6

6

TIME OF FLIGHT (TOF) CAMERAS & ARTIFACTS

Artifact #1: shot noise

Image from https://ptgrey.com

slide-7
SLIDE 7

7

TIME OF FLIGHT (TOF) CAMERAS & ARTIFACTS

Artifact #2: movement

Image from https://support.xbox.com

slide-8
SLIDE 8

8

TIME OF FLIGHT (TIF) CAMERAS & ARTIFACTS

Artifact #3: multiple reflections

Image from https://www.sciencedirect.com/science/article/pii/S0262885613001650

slide-9
SLIDE 9

9

TIME OF FLIGHT (TIF) CAMERAS & ARTIFACTS

Artifact #3: multiple reflections

slide-10
SLIDE 10

10

TACKLING 3D TOF ARTIFACTS THROUGH LEARNING AND THE FLAT DATASET

  • Time Of Flight (TOF) cameras & artifacts
  • Naïve Machine Learning (ML) for TOF reconstruction
  • TOF cameras: working principles
  • Camera calibration
  • The FLAT dataset
  • Spoiler: our non-Naïve ML solution works*
  • Back to physics
  • DNN architecture
  • Results
  • Conclusion

Agenda

* See Qi Guo, Iuri Frosio, Orazio Gallo, Todd Zickler, Jan Kautz, Tackling 3D ToF Artifacts Through Learning and the FLAT Dataset, ECCV 2018, Munich (Germany),

  • Sept. 2018.
slide-11
SLIDE 11

11

NAÏVE MACHINE LEARNING (ML) FOR TOF RECONSTRUCTION

What do we need?

(1) A large dataset of scenes… (2) … corrupted by: (1.1) photon noise, (1.2) motion, (1.3) multiple reflections… (3) … with clean output data… (4) … And a DNN.

slide-12
SLIDE 12

12

TACKLING 3D TOF ARTIFACTS THROUGH LEARNING AND THE FLAT DATASET

  • Time Of Flight (TOF) cameras & artifacts
  • Naïve Machine Learning (ML) for TOF reconstruction
  • TOF cameras: working principles
  • Camera calibration
  • The FLAT dataset
  • Spoiler: our non-Naïve ML solution works*
  • Back to physics
  • DNN architecture
  • Results
  • Conclusion

Agenda

* See Qi Guo, Iuri Frosio, Orazio Gallo, Todd Zickler, Jan Kautz, Tackling 3D ToF Artifacts Through Learning and the FLAT Dataset, ECCV 2018, Munich (Germany),

  • Sept. 2018.
slide-13
SLIDE 13

13

TOF WORKING PRINCIPLES

Time of flight is not time of flight ☺

Image from https://www.semanticscholar.org/paper/Interference-mitigation-technique-for-(ToF)-camera-Islam-Hossain/

slide-14
SLIDE 14

14

TOF WORKING PRINCIPLES

Time of flight is not time of flight ☺

Images from https://www.semanticscholar.org/paper/Interference-mitigation-technique-for-(ToF)-camera-Islam-Hossain/

slide-15
SLIDE 15

15

TOF WORKING PRINCIPLES

Multiple measurements

Pulse method Continuous wave method

slide-16
SLIDE 16

16

TOF WORKING PRINCIPLES

Camera functions and scene response

slide-17
SLIDE 17

17

TOF WORKING PRINCIPLE

More on scene response

= න

−𝑈 2 𝑈 2 𝑔 𝑢 ⨂𝑕𝑗 𝑢

𝑠 𝑢 𝑒𝑢 = න

−𝑈 2 𝑈 2 𝑔 𝑢 ∗ 𝑠 𝑢

𝑕𝑗 𝑢 𝑒𝑢 Raw measurement: Emitted signal: 𝑔 𝑢 Returned signal: ℎ 𝑢 Demodulation signal: 𝑕𝑗 𝑢 Impulse response: 𝑠(𝑢) Camera Scene 𝑅𝑗(𝑢) = න

−𝑈 2 𝑈 2 ℎ 𝑢 𝑕𝑗 𝑢 𝑒𝑢

Depth: 𝑎 = 𝑈𝑑 4𝜌arctan σ𝑗 sin𝜄𝑗 𝑅𝑗 𝑢 σ𝑗 cos𝜄𝑗 𝑅𝑗 𝑢 t t t Demodulation: gi(𝑢)

slide-18
SLIDE 18

18

TOF WORKING PRINCIPLES

Multiple frequencies

  • Different max length (combine

them)

  • Different resolutions
  • Agreement between different

measurements

slide-19
SLIDE 19

19

TACKLING 3D TOF ARTIFACTS THROUGH LEARNING AND THE FLAT DATASET

  • Time Of Flight (TOF) cameras & artifacts
  • Naïve Machine Learning (ML) for TOF reconstruction
  • TOF cameras: working principles
  • Camera calibration
  • The FLAT dataset
  • Spoiler: our non-Naïve ML solution works*
  • Back to physics
  • DNN architecture
  • Results
  • Conclusion

Agenda

* See Qi Guo, Iuri Frosio, Orazio Gallo, Todd Zickler, Jan Kautz, Tackling 3D ToF Artifacts Through Learning and the FLAT Dataset, ECCV 2018, Munich (Germany),

  • Sept. 2018.
slide-20
SLIDE 20

20

CAMERA CALIBRATION

Camera response functions (flat scene)

Inside coated with black-out material

slide-21
SLIDE 21

21

CAMERA CALIBRATION

Camera response functions (flat scene)

  • Three “frequencies”
  • Three

measurements per frequency

slide-22
SLIDE 22

22

CAMERA CALIBRATION

Photon noise

Other calibration details (pixel delay, vignetting, … in Qi Guo, Iuri Frosio, Orazio Gallo, Todd Zickler, Jan Kautz, Tackling 3D ToF Artifacts Through Learning and the FLAT Dataset, ECCV 2018, Munich (Germany), Sept. 2018.

slide-23
SLIDE 23

23

TACKLING 3D TOF ARTIFACTS THROUGH LEARNING AND THE FLAT DATASET

  • Time Of Flight (TOF) cameras & artifacts
  • Naïve Machine Learning (ML) for TOF reconstruction
  • TOF cameras: working principles
  • Camera calibration
  • The FLAT dataset
  • Spoiler: our non-Naïve ML solution works*
  • Back to physics
  • DNN architecture
  • Results
  • Conclusion

Agenda

* See Qi Guo, Iuri Frosio, Orazio Gallo, Todd Zickler, Jan Kautz, Tackling 3D ToF Artifacts Through Learning and the FLAT Dataset, ECCV 2018, Munich (Germany),

  • Sept. 2018.
slide-24
SLIDE 24

24

THE FLAT DATASET

Flexible, Large, Augmentable, ToF (FLAT)

To disk (FLAT) https://github.com/NVlabs/FLAT

slide-25
SLIDE 25

25

THE FLAT DATASET

Flexible, Large, Augmentable, ToF (FLAT)

Transient rendering (scene response function) based on Jarabo, A., Marco, J., Muñoz, A., Buisan, R., Jarosz, W., Gutierrez, D.: A framework for transient rendering. In: ACM Transactions on Graphics (SIGGRAPH ASIA), (2014).

slide-26
SLIDE 26

26

THE FLAT DATASET

Flexible, Large, Augmentable, ToF (FLAT)

Transient rendering (scene response function) based on Jarabo, A., Marco, J., Muñoz, A., Buisan, R., Jarosz, W., Gutierrez, D.: A framework for transient rendering. In: ACM Transactions on Graphics (SIGGRAPH ASIA), (2014). Brightness, travel time

slide-27
SLIDE 27

27

Impulse response

slide-28
SLIDE 28

28

Impulse response

slide-29
SLIDE 29

29

Impulse response

slide-30
SLIDE 30

30

Impulse response

slide-31
SLIDE 31

31

Impulse response

slide-32
SLIDE 32

32

Impulse response

slide-33
SLIDE 33

33

Impulse response

slide-34
SLIDE 34

34

Impulse response

slide-35
SLIDE 35

35

Impulse response

slide-36
SLIDE 36

36

Impulse response

slide-37
SLIDE 37

37

Impulse response

slide-38
SLIDE 38

38

Impulse response

slide-39
SLIDE 39

39

Impulse response

slide-40
SLIDE 40

40

Impulse response

slide-41
SLIDE 41

41

Impulse response

slide-42
SLIDE 42

42

Impulse response

slide-43
SLIDE 43

43

Impulse response

slide-44
SLIDE 44

44

Impulse response

slide-45
SLIDE 45

45

Impulse response

slide-46
SLIDE 46

46

Impulse response

slide-47
SLIDE 47

47

Impulse response

slide-48
SLIDE 48

48

Impulse response

slide-49
SLIDE 49

49

Impulse response

slide-50
SLIDE 50

50

Impulse response

slide-51
SLIDE 51

51

Impulse response

slide-52
SLIDE 52

52

Impulse response

slide-53
SLIDE 53

53

Impulse response

slide-54
SLIDE 54

54

Impulse response

slide-55
SLIDE 55

55

Impulse response

slide-56
SLIDE 56

56

Impulse response

slide-57
SLIDE 57

57

Impulse response

slide-58
SLIDE 58

58

Impulse response

slide-59
SLIDE 59

59

Impulse response

slide-60
SLIDE 60

60

Impulse response

slide-61
SLIDE 61

61

Impulse response

slide-62
SLIDE 62

62

Impulse response

slide-63
SLIDE 63

63

Impulse response

slide-64
SLIDE 64

64

Impulse response

slide-65
SLIDE 65

65

Impulse response

slide-66
SLIDE 66

66

Impulse response

slide-67
SLIDE 67

67

Impulse response

slide-68
SLIDE 68

68

Impulse response

slide-69
SLIDE 69

69

Impulse response

slide-70
SLIDE 70

70

Impulse response

slide-71
SLIDE 71

71

Impulse response

slide-72
SLIDE 72

72

Impulse response

slide-73
SLIDE 73

73

Impulse response

slide-74
SLIDE 74

74

Impulse response

slide-75
SLIDE 75

75

Impulse response

slide-76
SLIDE 76

76

Impulse response

slide-77
SLIDE 77

77

Impulse response

slide-78
SLIDE 78

78

Impulse response

slide-79
SLIDE 79

79

Impulse response

slide-80
SLIDE 80

80

Impulse response

slide-81
SLIDE 81

81

Impulse response

slide-82
SLIDE 82

82

Impulse response

slide-83
SLIDE 83

83

Impulse response

slide-84
SLIDE 84

84

Impulse response

slide-85
SLIDE 85

85

Impulse response

slide-86
SLIDE 86

86

Impulse response

slide-87
SLIDE 87

87

Impulse response

slide-88
SLIDE 88

88

Impulse response

slide-89
SLIDE 89

89

Impulse response

slide-90
SLIDE 90

90

Impulse response

slide-91
SLIDE 91

91

Impulse response

slide-92
SLIDE 92

92

Impulse response

slide-93
SLIDE 93

93

Impulse response

slide-94
SLIDE 94

94

Impulse response

slide-95
SLIDE 95

95

Impulse response

slide-96
SLIDE 96

96

Impulse response

slide-97
SLIDE 97

97

Impulse response

slide-98
SLIDE 98

98

Impulse response

slide-99
SLIDE 99

99

Impulse response

slide-100
SLIDE 100

100

Impulse response

slide-101
SLIDE 101

101

Impulse response

slide-102
SLIDE 102

102

Impulse response

slide-103
SLIDE 103

103

Impulse response

slide-104
SLIDE 104

104

Impulse response

slide-105
SLIDE 105

105

Impulse response

slide-106
SLIDE 106

106

Impulse response

slide-107
SLIDE 107

107

Impulse response

slide-108
SLIDE 108

108

Impulse response

slide-109
SLIDE 109

109

Impulse response

slide-110
SLIDE 110

110

Impulse response

slide-111
SLIDE 111

111

Impulse response

slide-112
SLIDE 112

112

Impulse response

slide-113
SLIDE 113

113

THE FLAT DATASET

Flexible, Large, Augmentable, ToF (FLAT)

Different cameras (beyond Kinect 2) can be simulated, after calibration.

slide-114
SLIDE 114

114

THE FLAT DATASET

Flexible, Large, Augmentable, ToF (FLAT)

Noise can be added…

slide-115
SLIDE 115

115

THE FLAT DATASET

Flexible, Large, Augmentable, ToF (FLAT)

… As well as motion (approximate model) and texture…

slide-116
SLIDE 116

116

THE FLAT DATASET

Flexible, Large, Augmentable, ToF (FLAT)

… and multiple reflections.

slide-117
SLIDE 117

117

THE FLAT DATASET

Samples

slide-118
SLIDE 118

118

TACKLING 3D TOF ARTIFACTS THROUGH LEARNING AND THE FLAT DATASET

  • Time Of Flight (TOF) cameras & artifacts
  • Naïve Machine Learning (ML) for TOF reconstruction
  • TOF cameras: working principles
  • Camera calibration
  • The FLAT dataset
  • Spoiler: our non-Naïve ML solution works*
  • Back to physics
  • DNN architecture
  • Results
  • Conclusion

Agenda

* See Qi Guo, Iuri Frosio, Orazio Gallo, Todd Zickler, Jan Kautz, Tackling 3D ToF Artifacts Through Learning and the FLAT Dataset, ECCV 2018, Munich (Germany),

  • Sept. 2018.
slide-119
SLIDE 119

119

NAÏVE MACHINE LEARNING (ML) FOR TOF RECONSTRUCTION

Supervised learning

Take it easy: supervised learning, from raw data to 3D map. Training input/output pairs from the FLAT dataset.

slide-120
SLIDE 120

120

NAÏVE MACHINE LEARNING (ML) FOR TOF RECONSTRUCTION

Supervised learning

slide-121
SLIDE 121

121

THE LESSON WE LEARNED*…

* To advance science.

That’s a nice negative result!

slide-122
SLIDE 122

122

… AND HOW WE IMPROVED

*1 Yes, it’s a fake picture… *2 … But the message is correct. *1 *2

slide-123
SLIDE 123

123

TACKLING 3D TOF ARTIFACTS THROUGH LEARNING AND THE FLAT DATASET

  • Time Of Flight (TOF) cameras & artifacts
  • Naïve Machine Learning (ML) for TOF reconstruction
  • TOF cameras: working principles
  • Camera calibration
  • The FLAT dataset
  • Spoiler: our non-Naïve ML solution works*
  • Back to physics
  • DNN architecture
  • Results
  • Conclusion

Agenda

* See Qi Guo, Iuri Frosio, Orazio Gallo, Todd Zickler, Jan Kautz, Tackling 3D ToF Artifacts Through Learning and the FLAT Dataset, ECCV 2018, Munich (Germany),

  • Sept. 2018.
slide-124
SLIDE 124

124

BACK TO PHYSICS

Cause: Sequential measurements Effect: Misaligned moving object Solution:Warping

And, more generally speaking, any a-priori knowledge.

time

slide-125
SLIDE 125

125

BACK TO PHYSICS

Cause: DNN architecture and learningmocks physics Effect: Sub-optimal results Solution:Include physics in the DNN architecture / reconstruction pipeline.

And, more generally speaking, any a-priori knowledge.

Naïve machine learning Physics

slide-126
SLIDE 126

126

DNN ARCHITECTURE

#1: Motion Correction Module

Trained to warp images to the central one

slide-127
SLIDE 127

127

DNN ARCHITECTURE

#2: Motion Reflection Module

Trained to reduce multiple reflection after re-alignment.

slide-128
SLIDE 128

128

DNN ARCHITECTURE

#3: Differential reconstruction pipeline

Non-trainable, but differentiable,physics- based reconstruction pipeline.

slide-129
SLIDE 129

129

DNN ARCHITECTURE

#3: Differential reconstruction pipeline

Can be refined with end-to-end training.

slide-130
SLIDE 130

130

TACKLING 3D TOF ARTIFACTS THROUGH LEARNING AND THE FLAT DATASET

  • Time Of Flight (TOF) cameras & artifacts
  • Naïve Machine Learning (ML) for TOF reconstruction
  • TOF cameras: working principles
  • Camera calibration
  • The FLAT dataset
  • Spoiler: our non-Naïve ML solution works*
  • Back to physics
  • DNN architecture
  • Results
  • Conclusion

Agenda

* See Qi Guo, Iuri Frosio, Orazio Gallo, Todd Zickler, Jan Kautz, Tackling 3D ToF Artifacts Through Learning and the FLAT Dataset, ECCV 2018, Munich (Germany),

  • Sept. 2018.
slide-131
SLIDE 131

131 NVIDIA CONFIDENTIAL. DO NOT DISTRIBUTE.

RESULTS

Compare against LF2 [1] [Kinect, non DL] DToF [3] [DL, Raw to 3D, no motion] Phasor [2] [High frequencies reduce MPI]

[1] Xiang, et al. libfreenect2: Release 0.2 [2] Marco, et al. DeepToF: Off-the-shelf real -time correction of multipath interference in time-of-flight imaging. In: ACM Transactions on Graphics (SIGGRAPH ASIA). [3] Gupta, et al. Phasor imaging: A generalization of correlation-based time-of- flight imaging. ACM Transactions on Graphics.

Competitors & ablation study

Ablation study MOM [motion only] MRM [multiple reflection and noise only] MOM-MRM [motion, multiple reflection and noise ]

slide-132
SLIDE 132

132

RESULTS

Ablation study: none, MRM, MOM+MRM [simulation]

Median [Med] and Inter Quartile Range [IQR] of the error decreased by MRM / MOM-MRM, in cm.

slide-133
SLIDE 133

133

RESULTS

Ablation study: none, MRM, MOM+MRM [simulation]

slide-134
SLIDE 134

134

RESULTS

Compare against: DTOF, Phasor imaging [simulation]

Smaller error when compared to DToF or Phasor.

slide-135
SLIDE 135

135

RESULTS

Compare against: DTOF, Phasor imaging on multi-reflection and shot noise [simulation]

Field of view

Multiple reflection removed through local reflection / a-priori information, no bias.

slide-136
SLIDE 136

136

RESULTS

Compare against: LF2, on multi-reflection and shot noise [real data]

Multiple reflection removed through local reflection / coherence / a-priori information, no bias.

slide-137
SLIDE 137

137

RESULTS

Compare against: LF2, on movement [real data]

Realignment of raw data reduce motion artifacts, specular reflections (red box) generate errors.

slide-138
SLIDE 138

138

TACKLING 3D TOF ARTIFACTS THROUGH LEARNING AND THE FLAT DATASET

  • Time Of Flight (TOF) cameras & artifacts
  • Naïve Machine Learning (ML) for TOF reconstruction
  • TOF cameras: working principles
  • Camera calibration
  • The FLAT dataset
  • Spoiler: our non-Naïve ML solution works*
  • Back to physics
  • DNN architecture
  • Results
  • Conclusion

Agenda

* See Qi Guo, Iuri Frosio, Orazio Gallo, Todd Zickler, Jan Kautz, Tackling 3D ToF Artifacts Through Learning and the FLAT Dataset, ECCV 2018, Munich (Germany),

  • Sept. 2018.
slide-139
SLIDE 139

139

CONCLUSION

  • 1. Naïve ML does not always work…
slide-140
SLIDE 140

140

CONCLUSION

  • 1. Naïve ML does not always work…
  • 2. ….But going back to a priori knowledge may help.
slide-141
SLIDE 141

141

CONCLUSION

The physics of ToF cameras: acquisition, reconstruction, artifacts Photon shot noise, motion artifacts, multiple reflection A large dataset of simulated data Design the DNN architecture accordingly to a-priori knowledge Effective reduction of reconstruction artifacts

slide-142
SLIDE 142

142

RESOURCES

Tackling 3D ToF Artifacts Through Learning and the FLAT Dataset, Qi Guo, Iuri Frosio, Orazio Gallo, Todd Zickler, Jan Kautz; The European Conference on Computer Vision (ECCV), 2018,

  • pp. 368-383,

http://openaccess.thecvf.com/content_ECCV_2018/html/Qi_Guo_Tackling_3D_ToF_ECCV_20 18_paper.html The FLAT dataset (code and data): https://github.com/NVlabs/FLAT Contact: {ifrosio, ogallo}@nvidia.com, qiguo@g.harvard.edu

Online