Monocular Depth Estimation Using Atrous Convolutions Group 5 - - - PowerPoint PPT Presentation

Monocular Depth Estimation Using Atrous Convolutions

Group 5 - Faraz Saeedan Fabian Kessler, Dominik Straub, Steven Lang February 15, 2019

Technische Universität Darmstadt

Introduction Experiments Results Discussion Conclusion

Fabian Kessler, Dominik Straub, Steven Lang February 15, 2019

1

Technische Universität Darmstadt

Monocular Depth Estimation

◮ Monodepth: Estimate depth from a single image at test-time (instead of stereo pair) ◮ Existing approaches treat this as a supervised regression problem ◮ Godard et al., CVPR 2017: Depth estimation as a stereo reconstruction problem

Figure: Godard et al. 2017

Fabian Kessler, Dominik Straub, Steven Lang February 15, 2019

2

Technische Universität Darmstadt

Atrous Convolutions

Figure: Chen et al., 2017

Fabian Kessler, Dominik Straub, Steven Lang February 15, 2019

3

Technische Universität Darmstadt

Project Objective

◮ Get Monodepth baseline to run ◮ Design and implement encoder-decoder network using atrous convolutions ◮ Consider memory and runtime in architectural decisions ◮ Quantitative comparison of proposed architectures to baseline

Fabian Kessler, Dominik Straub, Steven Lang February 15, 2019

4

Technische Universität Darmstadt

Our Architecture with ASPP

1x1 Conv 3x3 Conv rate=6 3x3 Conv rate=12 3x3 Conv rate=18 Image Pooling

Atrous Spatial Pyramid Pooling

Block 1 Block 2 Block 3 Block 4 Output Stride 4 8 16 16 Conv1 + Pool1 Concat + 1x1 Conv

Prediction Input Fabian Kessler, Dominik Straub, Steven Lang February 15, 2019

5

Technische Universität Darmstadt

Remember last time? Sky Artifacts

Fabian Kessler, Dominik Straub, Steven Lang February 15, 2019

6

Technische Universität Darmstadt

How we solved it: Reimplementation

Fabian Kessler, Dominik Straub, Steven Lang February 15, 2019

7

Technische Universität Darmstadt

How we solved it: Reimplementation

Differences of their implementation compared to standard ResNet: ◮ No batch-normalization ◮ Nearest-neighbor instead of bilinear interpolation ◮ Order of convolution strides in ResNet switched + Fixed bug in author’s implementation of loss function

Fabian Kessler, Dominik Straub, Steven Lang February 15, 2019

7

Technische Universität Darmstadt

New Disparity Maps

Fabian Kessler, Dominik Straub, Steven Lang February 15, 2019

8

Technische Universität Darmstadt

General Experimental Setup

◮ General setup

◮ Batch Size 16 ◮ Learning Rate 2e-4 ◮ 50 epochs

◮ KITTI for training

◮ Rescaled to 256 x 512 px

◮ KITTI Stereo 2015 for testing

◮ Sparse ground truth Fabian Kessler, Dominik Straub, Steven Lang February 15, 2019

9

Technische Universität Darmstadt

Output Strides

Issues

Output stride 64 with Atrous rate > 1

Convolution Kernel

F e a t u r e s a f t e r E n c

• d

e r

Fabian Kessler, Dominik Straub, Steven Lang February 15, 2019

10

Technische Universität Darmstadt

Output Strides

Fabian Kessler, Dominik Straub, Steven Lang February 15, 2019

11

Technische Universität Darmstadt

Output Strides

Stride

• Abs. Rel

Runtime Memory Params (M) 64 0.1120 NaN 6115MiB 58.4 32 0.1048 NaN 8871MiB 58.4 16 0.1041 NaN 8883MiB 58.4 8 NaN NaN 10609MiB 58.4

Fabian Kessler, Dominik Straub, Steven Lang February 15, 2019

11

Technische Universität Darmstadt

ASPP vs. no ASPP (with Skip Connections)

Fabian Kessler, Dominik Straub, Steven Lang February 15, 2019

12

Technische Universität Darmstadt

ASPP vs. no ASPP (without Skip Connections)

Fabian Kessler, Dominik Straub, Steven Lang February 15, 2019

13

Technische Universität Darmstadt

Experiments: Atrous Rates

1x1 Conv 3x3 Conv rate=6 3x3 Conv rate=12 3x3 Conv rate=18 Image Pooling

Atrous Spatial Pyramid Pooling

0.095 0.100 0.105 0.110

Abs.Rel.

stride 32 stride 16 Baseline 1,1,1,1 1,2,3,4 1,3,5,7

ASPP rates

Fabian Kessler, Dominik Straub, Steven Lang February 15, 2019

14

Technische Universität Darmstadt

Experiments: Atrous Rates

1x1 Conv 3x3 Conv rate=6 3x3 Conv rate=12 3x3 Conv rate=18 Image Pooling

Atrous Spatial Pyramid Pooling

0.095 0.100 0.105 0.110

Abs.Rel.

stride 32 stride 16 Baseline 1,1,1,1 1,2,3,4 1,3,5,7

ASPP rates

Fabian Kessler, Dominik Straub, Steven Lang February 15, 2019

14

Technische Universität Darmstadt

Experiments: Different number of modules

1x1 Conv 3x3 Conv rate=1 3x3 Conv rate=1 Image Pooling

Atrous Spatial Pyramid Pooling

0.095 0.100 0.105 0.110

Abs.Rel.

Baseline 1 2 3 4 5

ASPP modules

10 20 30 40 50 60 70

Million Parameters

Fabian Kessler, Dominik Straub, Steven Lang February 15, 2019

15

Technische Universität Darmstadt

Experiments: Different number of modules

1x1 Conv 3x3 Conv rate=1 3x3 Conv rate=1 Image Pooling

Atrous Spatial Pyramid Pooling

0.095 0.100 0.105 0.110

Abs.Rel.

Baseline 1 2 3 4 5

ASPP modules

10 20 30 40 50 60 70

Million Parameters

Fabian Kessler, Dominik Straub, Steven Lang February 15, 2019

15

Technische Universität Darmstadt

Experiments: Atrous Convolutions in Encoder

Block 1 Block 2 Block 3 Block 4 Conv1 + Pool1 1x1 Conv 3x3 Conv rate=6 3x3 Conv rate=12 3x3 Conv rate=18 Image Pooling

Atrous Spatial Pyramid Pooling Input

No improvement with atrous convolutions in ResNet blocks

Fabian Kessler, Dominik Straub, Steven Lang February 15, 2019

16

Technische Universität Darmstadt

Experiments: ASPP in ResNet

Block 1 Block 2 Conv1 + Pool1

1x1 Conv 3x3 Conv rate=6 3x3 Conv rate=12 3x3 Conv rate=18 Image Pooling

Atrous Spatial Pyramid Pooling

Input

.....

No improvement with ASPP module between ResNet blocks

Fabian Kessler, Dominik Straub, Steven Lang February 15, 2019

17

Technische Universität Darmstadt

Final Results

Pretrained on CityScapes & finetuned on KITTI

Output-stride ASPP Abs.Rel. Sq.Rel. RMSE Log RMSE a1 a2 a3 Params (M) ∆ Abs. Rel. (%) 64 - Godard

• 0.0970

0.8960 5.093 0.176 0.962 0.962 0.986 58.4 – 16 - Ours 1-1-1-1 0.0927 0.8132 4.865 0.168 0.888 0.967 0.987 58.4 4.43 32 - Ours 1 0.0941 0.8196 4.910 0.173 0.882 0.963 0.986 44.1 2.99 32 - Ours 1-2-3-4 0.0936 0.8281 4.941 0.172 0.884 0.963 0.987 58.4 3.50

Fabian Kessler, Dominik Straub, Steven Lang February 15, 2019

18

Technische Universität Darmstadt

Discussion

◮ Loss hyperparameters ◮ Learning rate ◮ Output stride 8 ◮ Decoder architecture ◮ Robustness of predictions

Fabian Kessler, Dominik Straub, Steven Lang February 15, 2019

19

Technische Universität Darmstadt

Recap

◮ Reproduced baseline ◮ Experiments

◮ Output stride ◮ Skip connections ◮ ASPP rates ◮ Number of ASPP modules Fabian Kessler, Dominik Straub, Steven Lang February 15, 2019

20

Technische Universität Darmstadt

Conclusion

◮ Atrous convolutions . . .

◮ do not improve monocular depth estimation ◮ need a decreased output stride, which harms runtime and memory

◮ Channel reduction after encoder . . .

◮ decreases parameter count and improves runtime ◮ without losing predictive power Fabian Kessler, Dominik Straub, Steven Lang February 15, 2019

21

Technische Universität Darmstadt

References

◮ C. Godard, O. Aodha and G. Brostow: Unsupervised Monocular Depth Estimation with Left- Right Consistency, CVPR 2017. ◮ L. Chen, Y. Zhu, G. Papandreou, F. Schroff, H. Adam: Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation. arXiv 2018. ◮ M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, B. Schiele: The cityscapes dataset for semantic urban scene understanding. CVPR. 2016. ◮ M. Menze and A. Geiger: Object Scene Flow for Autonomous

• Vehicles. CVPR. 2015.

Fabian Kessler, Dominik Straub, Steven Lang February 15, 2019

22

Technische Universität Darmstadt

Experiments: Atrous Rates (Visual Comparison)

Fabian Kessler, Dominik Straub, Steven Lang February 15, 2019

23

Technische Universität Darmstadt

Experiments: Atrous Rates (Visual Comparison)

Fabian Kessler, Dominik Straub, Steven Lang February 15, 2019

24

Technische Universität Darmstadt

Experiments: Atrous Rates (Visual Comparison)

Fabian Kessler, Dominik Straub, Steven Lang February 15, 2019

25

Technische Universität Darmstadt

Experiments: Atrous Rates (Visual Comparison)

Fabian Kessler, Dominik Straub, Steven Lang February 15, 2019

26

Technische Universität Darmstadt

Experiments: Atrous Rates (Squared relative error)

0.90 0.95 1.00 1.05 1.10

Sq.Rel.

stride 32 stride 16 Baseline 1,1,1,1 1,2,3,4 1,3,5,7

ASPP rates

Fabian Kessler, Dominik Straub, Steven Lang February 15, 2019

27

Technische Universität Darmstadt

Experiments: Atrous Rates (RMSE)

4.8 5.0 5.2 5.4 5.6

RMSE

stride 32 stride 16 Baseline 1,1,1,1 1,2,3,4 1,3,5,7

ASPP rates

Fabian Kessler, Dominik Straub, Steven Lang February 15, 2019

28

Technische Universität Darmstadt

Experiments: Atrous Rates (RMSE of log)

log.pdf log.pdf

0.17 0.18 0.19 0.20

RMSE log

stride 32 stride 16 Baseline 1,1,1,1 1,2,3,4 1,3,5,7

ASPP rates

Fabian Kessler, Dominik Straub, Steven Lang February 15, 2019

29

Technische Universität Darmstadt

Experiments: Atrous Rates (% inliers 1)

0.78 0.80 0.82 0.84 0.86 0.88 0.90

<1.25

stride 32 stride 16 Baseline 1,1,1,1 1,2,3,4 1,3,5,7

ASPP rates

Fabian Kessler, Dominik Straub, Steven Lang February 15, 2019

30

Technische Universität Darmstadt

Experiments: Atrous Rates (% inliers 2)

0.875 0.900 0.925 0.950 0.975 1.000

<1.25^2

stride 32 stride 16 Baseline 1,1,1,1 1,2,3,4 1,3,5,7

ASPP rates

Fabian Kessler, Dominik Straub, Steven Lang February 15, 2019

31

Technische Universität Darmstadt

Experiments: Atrous Rates (% inliers 3)

0.900 0.925 0.950 0.975 1.000 1.025

<1.25^3

stride 32 stride 16 Baseline 1,1,1,1 1,2,3,4 1,3,5,7

ASPP rates

Fabian Kessler, Dominik Straub, Steven Lang February 15, 2019

32

Technische Universität Darmstadt

Experiments: Atrous Rates (Squared relative error)

0.90 0.95 1.00 1.05 1.10

Sq.Rel.

Baseline 1 2 3 4 5

ASPP modules

10 20 30 40 50 60 70

Million Parameters

Fabian Kessler, Dominik Straub, Steven Lang February 15, 2019

33

Technische Universität Darmstadt

Experiments: Atrous Rates (RMSE)

4.8 5.0 5.2 5.4 5.6

RMSE

Baseline 1 2 3 4 5

ASPP modules

10 20 30 40 50 60 70

Million Parameters

Fabian Kessler, Dominik Straub, Steven Lang February 15, 2019

34

Technische Universität Darmstadt

Experiments: Atrous Rates (RMSE of log)

log.pdf log.pdf

0.17 0.18 0.19 0.20

RMSE log

Baseline 1 2 3 4 5

ASPP modules

10 20 30 40 50 60 70

Million Parameters

Fabian Kessler, Dominik Straub, Steven Lang February 15, 2019

35

Technische Universität Darmstadt

Experiments: Atrous Rates (% inliers 1)

figures/results/experiment2_d<125.pdf

Fabian Kessler, Dominik Straub, Steven Lang February 15, 2019

36

Technische Universität Darmstadt

Experiments: Atrous Rates (% inliers 2)

figures/results/experiment2_d<125^2.pdf

Fabian Kessler, Dominik Straub, Steven Lang February 15, 2019

37

Technische Universität Darmstadt

Experiments: Atrous Rates (% inliers 3)

figures/results/experiment2_d<125^3.pdf

Fabian Kessler, Dominik Straub, Steven Lang February 15, 2019

38

Technische Universität Darmstadt

Experiments: Output Stride Comparison

Fabian Kessler, Dominik Straub, Steven Lang February 15, 2019

39

Technische Universität Darmstadt

Experiments: Output Stride Comparison

Fabian Kessler, Dominik Straub, Steven Lang February 15, 2019

40