Optimizing the Relevance-Redundancy Tradeoff for Efficient Semantic - - PowerPoint PPT Presentation

▶
optimizing the relevance redundancy tradeoff for
SMART_READER_LITE
LIVE PREVIEW

Optimizing the Relevance-Redundancy Tradeoff for Efficient Semantic - - PowerPoint PPT Presentation

May 26, 2023 569 likes •796 views

Optimizing the Relevance-Redundancy Tradeoff for Efficient Semantic Segmentation Caner Hazrba Joint work with Julia Diebold and Daniel Cremers Optimizing the Relevance-Redundancy Tradeoff for Efficient Semantic Segmentation Caner Hazrba

slide-1
SLIDE 1

Optimizing the Relevance-Redundancy Tradeoff for Efficient Semantic Segmentation

Joint work with Julia Diebold and Daniel Cremers

Caner Hazırbaş

slide-2
SLIDE 2

Optimizing the Relevance-Redundancy Tradeoff for Efficient Semantic Segmentation

Joint work with Julia Diebold and Daniel Cremers

Runtime Accuracy

min E = min {λ · D + R}

Caner Hazırbaş

slide-3
SLIDE 3

Caner Hazırbaş Optimizing the Relevance-Redundancy Tradeoff for Efficient Semantic Segmentation

Optimizing the Relevance-Redundancy Tradeoff

I

x

I

y

h

Train Validation Test Feature Ranking Feature Selection Feature Set Classification with
 selected features Segmentation

3

min E = min {λ · D + R}

slide-4
SLIDE 4

Caner Hazırbaş Optimizing the Relevance-Redundancy Tradeoff for Efficient Semantic Segmentation

Feature Set

h

4

Haar-Like Texton Color Depth Location

I

x y

I

y

slide-5
SLIDE 5

Caner Hazırbaş Optimizing the Relevance-Redundancy Tradeoff for Efficient Semantic Segmentation

Feature Analysis

I

x

I

y

h

Train Validation Test Feature Ranking Feature Selection Feature Set Classification with
 selected features Segmentation

5

min E = min {λ · D + R}

slide-6
SLIDE 6

Caner Hazırbaş Optimizing the Relevance-Redundancy Tradeoff for Efficient Semantic Segmentation

Objective:

  • maximize the relevance between the feature and its class
  • minimize the redundancy between the feature pairs

Ranking:

  • iteratively rank the features, select one feature at a time
  • maximize the objective function at each iteration:

Feature Ranking

6

MI (X; Y ) =

Z

Y

Z

X

log p (x, y) p (x) p (y)dxdy

fm = arg max

fi∈F\Fm−1

"

MI (fi; c) −

1 m − 1 X

fj∈Fm−1

MI (fi; fj)

# max Φ (Rel, Red) , Φ = Relevance − Redundancy = MI (fi; class) − 1 m − 1 X

i6=j

MI (fi; fj)

slide-7
SLIDE 7

Caner Hazırbaş Optimizing the Relevance-Redundancy Tradeoff for Efficient Semantic Segmentation

Incremental Feature Analysis

C C T H C C H C T T H T H T T C H T T T T T T L L T H T T T H T H H H H H H H H H H H D T T C T C T D D H T T T T T C L T H L T T C T H T T T T H H H H C H H H H C H H H H H H H C C H T C C T C H H T C H T T T T H H T T T H T T L L H T H T T T H T H H H H H H H H C H C C T T H C T C T C T T T T T T H L T T L H T T T T T H H H H H H H H H H H H H H D C D H C D H C H T H T T T H C T H T C T H T T C L L H T T H T T H T T T T H H H H H H H H 84.1 75.9 66.3 44.4 65.1 Selected Sowerby eTrims Corel NYUv1 NYUv2 5 10 15 20 25 30 35 40 45 20 30 40 50 60 70 80 90 100 Size of Feature Set Classification Accuracy (%) 7

slide-8
SLIDE 8

Caner Hazırbaş Optimizing the Relevance-Redundancy Tradeoff for Efficient Semantic Segmentation

Relevance of Haar-Like Features

8 C C C C HC T T HT H

HT T T T T T L L T HT HHHHHHHHHHH

D T T C T C T

HT

T T C L T HL T T C T HT T

HHHH HH HH H

C C

HT C

C T C HH T C HT T T T HHT T T H T T L L HT HT T T H T HHHHHHHH C H C C

HC T C T C T T T T T T HL T T

L HT T T

HHHHHHHH H H

D C D HC D HC HT H T HC T

HT T C L L HT T HT T HT T T T HHHHHH

84.1 75.9 66.3 44.4 65.1 Selected Sowerby eTrims Corel NYUv1 NYUv2 5 10 15 20 25 30 35 40 45 20 30 40 50 60 70 80 90 100 Size of Feature Set Classification Accuracy (%) T H T T C T T HT

HT C T

C H

HC HH HH

T T T T D D T T T T

HH HH HH

T T

slide-9
SLIDE 9

Caner Hazırbaş Optimizing the Relevance-Redundancy Tradeoff for Efficient Semantic Segmentation

Relevance of Texton Features

9 C C C C H C TTH TH H TTTTTTL L TH T H H H H H H H H H H H D

TTC TC T

H T

TTC L TH L TTC TH TT

H H H H H H H H H C C H TC C TC H H TC H TTTTH H TTT H

T TL L H TH TTTH TH H H H

H H H H C H C C H C TC TC TTTTTTH L TT L H TTT H H H H H H H H H H D C D H C D H C H TH

TH C T

H TTC L L H TTH TTH TTTT H H H H H H 84.1 75.9 66.3 44.4 65.1 Selected Sowerby eTrims Corel NYUv1 NYUv2 5 10 15 20 25 30 35 40 45 20 30 40 50 60 70 80 90 100 Size of Feature Set Classification Accuracy (%)

TH TTC TTH T

H TC T C H H C H H H H

TT TT

D D

TT TT

H H H H H H

TT

slide-10
SLIDE 10

Caner Hazırbaş Optimizing the Relevance-Redundancy Tradeoff for Efficient Semantic Segmentation

Relevance of Color Features

10

C C CCH CT T H T H

H T T T T T T L L T H T H H H H H H H H H H H D T T CT CT H T T T CL T H L T T CT H T T H H H H H H H H H

CC

H T CCT CH H T CH T T T T H H T T T H T T L L H T H T T T H T H H H H H H H H

CH CC

H CT CT CT T T T T T H L T T L H T T T H H H H H H H H H H D C D H C D H CH T H T H CT H T T CL L H T T H T T H T T T T H H H H H H 84.1 75.9 66.3 44.4 65.1 Selected Sowerby eTrims Corel NYUv1 NYUv2 5 10 15 20 25 30 35 40 45 20 30 40 50 60 70 80 90 100 Size of Feature Set Classification Accuracy (%) T H T T C T T H T H T CT

CH

H C H H H H T T T T D D T T T T H H H H H H T T

slide-11
SLIDE 11

Caner Hazırbaş Optimizing the Relevance-Redundancy Tradeoff for Efficient Semantic Segmentation

Relevance of Location Features

11 C C C C H C T T H T H H T T T T T T LLT H T H H H H H H H H H H H D T T C T C T H T T T C LT H LT T C T H T T H H H H H H H H H C C H T C C T C H H T C H T T T T H H T T T H T T LLH T H T T T H T H H H H H H H H C H C C H C T C T C T T T T T T H LT T

LH T T T

H H H H H H H H H H D C D H C D H C H T H T H C T H T T C LLH T T H T T H T T T T H H H H H H 84.1 75.9 66.3 44.4 65.1 Selected Sowerby eTrims Corel NYUv1 NYUv2 5 10 15 20 25 30 35 40 45 20 30 40 50 60 70 80 90 100 Size of Feature Set Classification Accuracy (%) T H T T C T T H T H T C T C H H C H H H H T T T T D D T T T T H H H H H H T T

slide-12
SLIDE 12

Caner Hazırbaş Optimizing the Relevance-Redundancy Tradeoff for Efficient Semantic Segmentation

Relevance of Depth Features

12 C C C C H C T T H T H H T T T T T T L L T H T H H H H H H H H H H H

D

T T C T C T H T T T C L T H L T T C T H T T H H H H H H H H H C C H T C C T C H H T C H T T T T H H T T T H T T L L H T H T T T H T H H H H H H H H C H C C H C T C T C T T T T T T H L T T L H T T T H H H H H H H H H H

DC DH C DH C H T H

T H C T H T T C L L H T T H T T H T T T T H H H H H H 84.1 75.9 66.3 44.4 65.1 Selected Sowerby eTrims Corel NYUv1 NYUv2 5 10 15 20 25 30 35 40 45 20 30 40 50 60 70 80 90 100 Size of Feature Set Classification Accuracy (%) T H T T C T T H T H T C T C H H C H H H H T T T T

DD

T T T T H H H H H H T T

slide-13
SLIDE 13

Caner Hazırbaş Optimizing the Relevance-Redundancy Tradeoff for Efficient Semantic Segmentation

Feature Selection

13 C C T H C C H C T T H T H T T C H T T T T T T L L T H T T T H T H H H H H H H H H H H D T T C T C T D D H T T T T T C L T H L T T C T H T T T T H H H H C H H H H C H H H H H H H C C H T C C T C H H T C H T T T T H H T T T H T T L L H T H T T T H T H H H H H H H H C H C C T T H C T C T C T T T T T T H L T T L H T T T T T H H H H H H H H H H H H H H D C D H C D H C H T H T T T H C T H T C T H T T C L L H T T H T T H T T T T H H H H H H H H 84.1 75.9 66.3 44.4 65.1 Selected Sowerby eTrims Corel NYUv1 NYUv2 5 10 15 20 25 30 35 40 45 20 30 40 50 60 70 80 90 100 Size of Feature Set Classification Accuracy (%)

n∗ = arg max

n∈{1,...,N}

  • Acc (n)

α (N + 1 − n)

1 β

α = 5 β = 2

slide-14
SLIDE 14

Caner Hazırbaş Optimizing the Relevance-Redundancy Tradeoff for Efficient Semantic Segmentation

8

Selected Sowerby eTrims Corel NYUv1 NYUv2

Size of Feature Set

5 10 15 20 25 30 35 40 45

Acc(n)α (N+1-n)1/β

0.5 1 1.5 2 2.5 3

8 10 23 27

Number of Selected Features

14

n∗ = arg max

n∈{1,...,N}

  • Acc (n)

α (N + 1 − n)

1 β

slide-15
SLIDE 15

Caner Hazırbaş Optimizing the Relevance-Redundancy Tradeoff for Efficient Semantic Segmentation

Semantic Image Segmentation

I

x

I

y

h

Train Validation Test Feature Ranking Feature Selection Feature Set Classification with
 selected features Segmentation

15

min E = min {λ · D + R}

slide-16
SLIDE 16

Caner Hazırbaş Optimizing the Relevance-Redundancy Tradeoff for Efficient Semantic Segmentation

Classification & Segmentation

Rhino

  • P. Bear

Water Snow

Vegetation

Ground Sky

16

min E = min {λ · D + R}

arg max D

slide-17
SLIDE 17

Caner Hazırbaş Optimizing the Relevance-Redundancy Tradeoff for Efficient Semantic Segmentation

Improved Runtime

17

Training Time (in seconds) Testing Time (in seconds) eTrims Corel Sowerby NYUv1 NYUv2 eTrims Corel Sowerby NYUv1 NYUv2 Shotton et al. — 1800 1200 — — — 1.10 2.50 — — Fröhlich et al. — — — — — 17.0 — — — — Couprie et al. — — — — 172800 — — — — 0.70 Hermans et al. — — — — — — — — 0.38 0.38 Proposed 143 20 2 133 183 6.6 0.27 0.07 0.32 0.26

On average we improve the runtime by a factor of 7.7

slide-18
SLIDE 18

Caner Hazırbaş Optimizing the Relevance-Redundancy Tradeoff for Efficient Semantic Segmentation

Competitive Results

18

Classification Segmentation eTrims Corel Sowerby NYUv1 NYUv2 eTrims Corel Sowerby NYUv1 NYUv2 Shotton et al. — 68.4 85.6 — — — 74.6 88.6 — — Fröhlich et al. — — — — — 77.2 — — — — Couprie et al. — — — — — — — — — 52.4 Hermans et al. — — — 65.0 — — — — 71.5 54.2 Proposed 77.1 74.4 87.1 65.0 44.0 77.9 78.2 88.8 66.5 45.0

Image Class. Segm. GT Others

slide-19
SLIDE 19

I

x

I

y

h

Train Validation Test Feature Ranking Feature Selection Feature Set Classification with
 selected features Segmentation

Caner Hazırbaş | hazirbas@cs.tum.edu

Optimizing the Relevance-Redundancy Tradeoff for Efficient Semantic Segmentation

min E = min {λ · D + R}

slide-20
SLIDE 20

I

x

I

y

h

Train Validation Test Feature Ranking Feature Selection Feature Set Classification with
 selected features Segmentation

Caner Hazırbaş | hazirbas@cs.tum.edu

Optimizing the Relevance-Redundancy Tradeoff for Efficient Semantic Segmentation

min E = min {λ · D + R}

slide-21
SLIDE 21

Caner Hazırbaş Optimizing the Relevance-Redundancy Tradeoff for Efficient Semantic Segmentation

  • TextonBoost for Image Understanding: Multi-Class Object Recognition

and Segmentation by Jointly Modeling Texture, Layout, and Context


Jamie Shotton, John Winn, Carsten Rother, and Antonio Criminisi IJCV’07

  • Semantic Segmentation with Millions of Features: Integrating Multiple

Cues in a Combined Random Forest Approach


Björn Fröhlich, Erik Rodner, and Joachim Denzler ACCV’12

  • Indoor Semantic Segmentation using depth information


Camille Couprie, Clément Farabet, Laurent Najman, and Yann LeCun ICLR’13

  • Dense 3D Semantic Mapping of Indoor Scenes from RGB-D Images


Alexander Hermans, Georgios Floros and Bastian Leibe ICRA’14

References

21