CS381V Experiment Presentation Chun-Chen Kuo The Paper Indoor - - PowerPoint PPT Presentation

▶
cs381v experiment presentation
SMART_READER_LITE
LIVE PREVIEW

CS381V Experiment Presentation Chun-Chen Kuo The Paper Indoor - - PowerPoint PPT Presentation

Mar 17, 2024 139 likes •673 views

CS381V Experiment Presentation Chun-Chen Kuo The Paper Indoor Segmentation and Support Inference from RGBD Images. N. Silberman, D. Hoiem, P. Kohli, and R. Fergus. ECCV 2012. 50 100 150 200 250 300 350 400 50 100 150 200 250

slide-1
SLIDE 1

CS381V Experiment Presentation

Chun-Chen Kuo

slide-2
SLIDE 2

The Paper

  • Indoor Segmentation and Support Inference from

RGBD Images. N. Silberman, D. Hoiem, P. Kohli, and R. Fergus. ECCV 2012.

50 100 150 200 250 300 350 400 450 500 550 50 100 150 200 250 300 350 400

slide-3
SLIDE 3

Pipeline

segmentation support inference

slide-4
SLIDE 4

Outline

  • Run the segmentation pipeline
  • Experiment on the segmentation pipeline
  • Run the support inference pipeline
  • Address strength and weakness
slide-5
SLIDE 5

Outline

  • Run the segmentation pipeline
  • Experiment on the segmentation pipeline
  • Run the support inference pipeline
  • Address strength and weakness
slide-6
SLIDE 6

Segmentation Pipeline

Image920, RGB Depth Map

slide-7
SLIDE 7

Compute Surface Normal

y x z

slide-8
SLIDE 8

Align to room coordinates

slide-9
SLIDE 9

Aligned Surface Normal

y x z

slide-10
SLIDE 10

After Alignment

slide-11
SLIDE 11

Find Major Planes by RANSAC

0.2454x+0.1918y+0.9503z-4.2327

slide-12
SLIDE 12

Reassign Pixels to Planes

slide-13
SLIDE 13

Watershed Segmentation

  • Force the over-segmentation to be consistent with

the previous planes 1614

slide-14
SLIDE 14

Hierarchical Grouping

  • Bottom-up grouping by boundary classifier


(Logistic regression AdaBoost)

309 145 85 78 77

slide-15
SLIDE 15

AdaBoost Decision Tree

  • Reweigh misclassified regions
  • Optimize new tree with reweighed regions
  • Score the tree
  • Weighted sum over all trees optimized in each

iteration merge?

slide-16
SLIDE 16

Final Regions

77

Ground truth

slide-17
SLIDE 17

Outline

  • Run the segmentation pipeline
  • Experiment on the segmentation pipeline
  • Run the support inference pipeline
  • Address strength and weakness
slide-18
SLIDE 18

Experiment on Segmentation Pipeline

  • NYU Depth Dataset V2
  • Images 909~1200
  • Assign pixels to major planes
  • AdaBoost decision tree as boundary classifier
slide-19
SLIDE 19

Hypothesis

  • The trade-off between matching to 3D values,

normals, and gradient smoothing

  • If alpha is small, neighbor pixels with similar RGB

tend to be assigned to a same plane

  • If alpha is large, match pixels to planes based on

3D points and normals, regardless gradient smoothing

slide-20
SLIDE 20

Result of Plane Labeling

alpha=0

slide-21
SLIDE 21

Result of Plane Labeling

alpha=2500

slide-22
SLIDE 22

Result of Plane Labeling

alpha=0.25

slide-23
SLIDE 23

Result of Plane Labeling

alpha=0.25 alpha=0 alpha=2500

slide-24
SLIDE 24

Segmentation Score

alpha=0.25 alpha=2.5 alpha=0.25e-12

slide-25
SLIDE 25

Hypothesis

  • Number of iteration of an AdaBoost decision forest

boundary classifier (underfit vs. overfit)

  • At higher stage, the number of training

example(boundary) decreases, causing lower accuracy and overfitting

  • Accuracy at lower stage is more important because
  • f error propagation
slide-26
SLIDE 26

stage 1 stage 2 stage 3 stage 4 stage 5

slide-27
SLIDE 27

ROC Curve at Stage 1

  • iteration = 30
  • iteration = 5

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Train AUC: 0.904903, Test AUC: 0.894981

training testing

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Train AUC: 0.917977, Test AUC: 0.903215

training testing

true positive rate false positive rate

slide-28
SLIDE 28

ROC Curve at Stage 2

  • iteration = 30
  • iteration = 5

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Train AUC: 0.796447, Test AUC: 0.780641

training testing

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Train AUC: 0.816867, Test AUC: 0.777968

training testing

slide-29
SLIDE 29

ROC Curve at Stage 3

  • iteration = 30
  • iteration = 5

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Train AUC: 0.762504, Test AUC: 0.746715

training testing

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Train AUC: 0.802231, Test AUC: 0.737996

training testing

slide-30
SLIDE 30

ROC Curve at Stage 4

  • iteration = 30
  • iteration = 5

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Train AUC: 0.727054, Test AUC: 0.718036

training testing

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Train AUC: 0.773312, Test AUC: 0.718135

training testing

slide-31
SLIDE 31

ROC Curve at Stage 5

  • iteration = 30
  • iteration = 5

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Train AUC: 0.727807, Test AUC: 0.720329

training testing

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Train AUC: 0.774677, Test AUC: 0.713322

training testing

slide-32
SLIDE 32

Accuracy versus Iteration at Stage 1

5 10 15 20 25 30

iteration

0.83 0.84 0.85 0.86 0.87 0.88 0.89 0.9 0.91 0.92

accuracy

training testing

slide-33
SLIDE 33

Accuracy versus Iteration at Stage 2

5 10 15 20 25 30

iteration

0.77 0.78 0.79 0.8 0.81 0.82 0.83 0.84 0.85

accuracy

training testing

slide-34
SLIDE 34

Accuracy versus Iteration at Stage 3

5 10 15 20 25 30

iteration

0.73 0.74 0.75 0.76 0.77 0.78 0.79 0.8 0.81

accuracy

training testing

slide-35
SLIDE 35

Accuracy versus Iteration at Stage 4

5 10 15 20 25 30

iteration

0.71 0.72 0.73 0.74 0.75 0.76 0.77 0.78

accuracy

training testing

slide-36
SLIDE 36

Accuracy versus Iteration at Stage 5

5 10 15 20 25 30

iteration

0.71 0.72 0.73 0.74 0.75 0.76 0.77 0.78

accuracy

training testing

slide-37
SLIDE 37

Segmentation Score

iteration = [5 5 5 5 5]

iteration = [10 10 10 10 10]

iteration = [30 30 30 30 30]

Accuracy at lower stage is more important!

slide-38
SLIDE 38

Segmentation Score

iteration = [1 1 1 1 1]

Accuracy at lower stage is more important!

5 10 15 20 25 30

iteration

0.83 0.84 0.85 0.86 0.87 0.88 0.89 0.9 0.91 0.92

accuracy

training testing

slide-39
SLIDE 39

Outline

  • Run the segmentation pipeline
  • Experiment on the segmentation pipeline
  • Run the support inference pipeline
  • Address strength and weakness
slide-40
SLIDE 40

Support Inference Pipeline

slide-41
SLIDE 41

Structure Class Classifier

slide-42
SLIDE 42

Structure Class Classifier

slide-43
SLIDE 43

Support Classifier

containment, geometry, and horz feature take ~1 day to extract features for 292 images!

slide-44
SLIDE 44

Support Classifier

slide-45
SLIDE 45

Infer by Linear Program

6 minutes for an image!

slide-46
SLIDE 46

Structure and Support Inference

stripe: incorrect structure prediction

ground furniture props structure

50 100 150 200 250 300 350 400 450 500 550 50 100 150 200 250 300 350 400

slide-47
SLIDE 47

Structure and Support Inference

50 100 150 200 250 300 350 400 450 500 550 50 100 150 200 250 300 350 400

slide-48
SLIDE 48

50 100 150 200 250 300 350 400 450 500 550 50 100 150 200 250 300 350 400

Structure and Support Inference

  • ver-segmentation(color variance in an object)

clutter, small objects

  • ut of 4 classes
slide-49
SLIDE 49

Outline

  • Run the segmentation pipeline
  • Experiment on the segmentation pipeline
  • Run the support inference pipeline
  • Address strength and weakness
slide-50
SLIDE 50

Strength

  • Reason joint assignment for structure and support
  • ~73% accuracy if ground truth segmentation is

given

slide-51
SLIDE 51

Weakness

  • Slow in testing time

  • 5 minutes for feature extraction

  • 6 minutes for inference by linear programming
  • Clutters, small(thin) objects, color variance in
  • bjects
  • Only 4 structure classes(no human, pet,…etc)
  • ~55% accuracy if bottom up segmentation followed

by support inference

slide-52
SLIDE 52

Reference

  • Code:


http://cs.nyu.edu/~silberman/projects/ indoor_scene_seg_sup.html