Object Detection and Segmentation from Joint Embedding of Parts and - - PowerPoint PPT Presentation

▶
object detection and segmentation from joint embedding of
SMART_READER_LITE
LIVE PREVIEW

Object Detection and Segmentation from Joint Embedding of Parts and - - PowerPoint PPT Presentation

Jan 15, 2023 131 likes •742 views

Object Detection and Segmentation from Joint Embedding of Parts and Pixels Michael Maire 1 , Stella X. Yu 2 , Pietro Perona 1 1 California Institute of Technology - Pasadena, CA 91125 2 Boston College - Chestnut Hill, MA 02467 Segmentation

slide-1
SLIDE 1

Object Detection and Segmentation from Joint Embedding of Parts and Pixels

Michael Maire1, Stella X. Yu2, Pietro Perona1

1California Institute of Technology - Pasadena, CA 91125 2Boston College - Chestnut Hill, MA 02467

slide-2
SLIDE 2

Segmentation Detection

slide-3
SLIDE 3

Segmentation Detection

  • Perceptual Grouping Framework
slide-4
SLIDE 4

Ingredients

Plug in state-of-the-art components:

slide-5
SLIDE 5

Ingredients

Plug in state-of-the-art components: low-level cues: color, texture, edges

[Arbel´ aez, Maire, Fowlkes, Malik, PAMI 2011]

slide-6
SLIDE 6

Ingredients

Plug in state-of-the-art components: low-level cues: color, texture, edges

[Arbel´ aez, Maire, Fowlkes, Malik, PAMI 2011]

top-down parts: poselets for person detection

[Bourdev, Maji, Brox, Malik, ECCV 2010]

slide-7
SLIDE 7

Ingredients

Plug in state-of-the-art components: low-level cues: color, texture, edges

[Arbel´ aez, Maire, Fowlkes, Malik, PAMI 2011]

top-down parts: poselets for person detection

[Bourdev, Maji, Brox, Malik, ECCV 2010]

PASCAL VOC 2010 Person Category: Improved Detection and Segmentation

slide-8
SLIDE 8

Grouping Relationships

slide-9
SLIDE 9

Grouping Relationships

slide-10
SLIDE 10

Pixel Affinity: Color, Texture Similarity

slide-11
SLIDE 11

Pixel Affinity: Color, Texture Similarity

b

slide-12
SLIDE 12

Pixel Affinity: Color, Texture Similarity

b

slide-13
SLIDE 13

Part Affinity: Geometric Compatibility

slide-14
SLIDE 14

Part Affinity: Geometric Compatibility

slide-15
SLIDE 15

pixels

b b

slide-16
SLIDE 16

pixels

b b

parts

slide-17
SLIDE 17

pixels

b b

parts surround

slide-18
SLIDE 18

pixels

b b

parts surround

slide-19
SLIDE 19

pixels

b b

parts surround

slide-20
SLIDE 20

pixels

b b

parts surround figure/ground prior

b C

slide-21
SLIDE 21

pixels

b b

parts surround figure/ground prior

bC

⇒

Angular Embedding

⇒ ⇒ ⇒

  • bjects

figure/ground segmentation

slide-22
SLIDE 22

Angular Embedding

slide-23
SLIDE 23

Angular Embedding p q

slide-24
SLIDE 24

Angular Embedding p q

slide-25
SLIDE 25

Angular Embedding p q

Given:

◮ Relative ordering Θ(·, ·) ◮ Confidence on relationships C(·, ·)

slide-26
SLIDE 26

Angular Embedding p q

Given:

◮ Relative ordering Θ(·, ·) ◮ Confidence on relationships C(·, ·)

Compute:

◮ Global ordering θ(·) ◮ Embed into unit circle:

p → z(p) = eiθ(p)

θ

  • p
  • q
slide-27
SLIDE 27

Angular Embedding p q

Given:

◮ Relative ordering Θ(·, ·) ◮ Confidence on relationships C(·, ·)

Compute:

◮ Global ordering θ(·) ◮ Embed into unit circle:

p → z(p) = eiθ(p)

θ

  • p
  • q

Subject to:

◮ Linear constraints on embedding solution in columns of U

slide-28
SLIDE 28

i 1 −1

z(p) z(q) z(r) minimize: ε =

p

  • q C(p,q)
  • p,q C(p,q) · |z(p) − ˜

z(p)|2

[Yu, PAMI 2011]

slide-29
SLIDE 29

i 1 −1

z(p) z(q) z(r) z(r)eiΘ(p,r) z(q)eiΘ(p,q) C(p, r) C ( p , q ) Θ(p, r) Θ(p, q) minimize: ε =

p

  • q C(p,q)
  • p,q C(p,q) · |z(p) − ˜

z(p)|2

[Yu, PAMI 2011]

slide-30
SLIDE 30

i 1 −1

z(p) z(q) z(r) z(r)eiΘ(p,r) z(q)eiΘ(p,q) C(p, r) C ( p , q ) Θ(p, r) Θ(p, q)

˜ z(p)

minimize: ε =

p

  • q C(p,q)
  • p,q C(p,q) · |z(p) − ˜

z(p)|2

[Yu, PAMI 2011]

slide-31
SLIDE 31

b b

b C

Cp Cq (Cs, Θs) (Cf , Θf ) U

slide-32
SLIDE 32

pixels

  • parts

surround prior

  • C =

    Cp α · Cq β · Cs γ · Cf β · C T

s

γ · C T

f

    Θ = Σ−1     Θs Θf −ΘT

s

−ΘT

f

   

slide-33
SLIDE 33

Angular Embedding

Relax to generalized eigenproblem QPQz = λz: P = D−1W Q = I − D−1U(UTD−1U)−1UT with D and W defined as: D = Diag(C1n) W = C • eiΘ Eigenvectors {z0, z1, ..., zm−1} embed pixels and parts into Cm

slide-34
SLIDE 34

Angular Embedding

∠z0 encodes global ordering z1, z2, ..., zm−1 encode grouping

slide-35
SLIDE 35

Angular Embedding

∠z0 encodes global ordering z1, z2, ..., zm−1 encode grouping if Θ = 0 ⇒ Normalized Cuts (grouping without ordering)

slide-36
SLIDE 36

Decoding Eigenvectors: Object Detection

ℜ(z0) ℜ(z1) ℜ(z2) ℑ(z0) ℑ(z1) ℑ(z2) Ordering Grouping

slide-37
SLIDE 37

Decoding Eigenvectors: Object Detection

ℜ(z0) ℜ(z1) ℜ(z2) ℑ(z0) ℑ(z1) ℑ(z2)

b b b

Ordering Grouping

slide-38
SLIDE 38

Decoding Eigenvectors: Object Detection

ℜ(z0) ℜ(z1) ℜ(z2) ℑ(z0) ℑ(z1) ℑ(z2)

b b b b b b

Ordering Grouping

slide-39
SLIDE 39

Decoding Eigenvectors: Object Detection

ℜ(z0) ℜ(z1) ℜ(z2) ℑ(z0) ℑ(z1) ℑ(z2)

b b b b b b b b b

Ordering Grouping

slide-40
SLIDE 40

Decoding Eigenvectors: Object Detection

ℜ(z0) ℜ(z1) ℜ(z2) ℑ(z0) ℑ(z1) ℑ(z2)

b b b b b b b b b

Ordering Grouping

slide-41
SLIDE 41

Decoding Eigenvectors: Object Detection

ℜ(z0) ℜ(z1) ℜ(z2) ℑ(z0) ℑ(z1) ℑ(z2)

b b b b b b b b b

Ordering Grouping

slide-42
SLIDE 42

Decoding Eigenvectors: Object Detection

ℜ(z0) ℜ(z1) ℜ(z2) ℑ(z0) ℑ(z1) ℑ(z2)

b b b b b b b b b

Ordering Grouping

slide-43
SLIDE 43

Decoding Eigenvectors: Object Detection

ℜ(z0) ℜ(z1) ℜ(z2) ℑ(z0) ℑ(z1) ℑ(z2)

b b b b b b b b b

Ordering Grouping

slide-44
SLIDE 44

Decoding Eigenvectors: Figure/Ground

ℜ(z) ℑ(z) z0 z1 z2 z3 z4

slide-45
SLIDE 45

Decoding Eigenvectors: Figure/Ground

ℜ(z) ℑ(z) z0 z1 z2 z3 z4 ⇐ ℑℜ(z) ∠z0 ∇z1 ∇z2 ∇z3 ∇z4

slide-46
SLIDE 46

Decoding Eigenvectors: Segmentation

ℑℜ(z) ∠z0 ∇z1 ∇z2 ∇z3 ∇z4 Figure/Ground

  • Hierarchical Segmentation

[Arbel´ aez, Maire, Fowlkes, Malik, PAMI 2011]

slide-47
SLIDE 47

Decoding Eigenvectors: Object Segmentation

Assign pixels pk to objects Qi via parts qj: pk → argmin

Qi

  • min

qj∈Qi{Dist(pk, qj)}

slide-48
SLIDE 48

Decoding Eigenvectors: Object Segmentation

Assign pixels pk to objects Qi via parts qj: pk → argmin

Qi

  • min

qj∈Qi{Dist(pk, qj)}

slide-49
SLIDE 49

Decoding Eigenvectors

slide-50
SLIDE 50

Results: PASCAL 2010 Person Category

Detections Poselet Mask F/G Mask Segmentation

slide-51
SLIDE 51

Results: PASCAL 2010 Person Category

Detections Poselet Mask F/G Mask Segmentation

slide-52
SLIDE 52

Results: PASCAL 2010 Person Category

◮ Segmentation task score: 41.1 (35.5 for poselet baseline)

slide-53
SLIDE 53

Results: PASCAL 2010 Person Category

◮ Segmentation task score: 41.1 (35.5 for poselet baseline) ◮ 11% relative improvement due to better detection

slide-54
SLIDE 54

Summary

◮ Simultaneous segmentation and detection:

◮ Part detectors → figure pop-out, object grouping ◮ Color, texture → pixel grouping

slide-55
SLIDE 55

Summary

◮ Simultaneous segmentation and detection:

◮ Part detectors → figure pop-out, object grouping ◮ Color, texture → pixel grouping

◮ Graph:

◮ Parts and pixels as nodes ◮ Links encode multiple relationship types

slide-56
SLIDE 56

Summary

◮ Simultaneous segmentation and detection:

◮ Part detectors → figure pop-out, object grouping ◮ Color, texture → pixel grouping

◮ Graph:

◮ Parts and pixels as nodes ◮ Links encode multiple relationship types

◮ Embedding: graph nodes → Cm

slide-57
SLIDE 57

Summary

◮ Simultaneous segmentation and detection:

◮ Part detectors → figure pop-out, object grouping ◮ Color, texture → pixel grouping

◮ Graph:

◮ Parts and pixels as nodes ◮ Links encode multiple relationship types

◮ Embedding: graph nodes → Cm ◮ Decode:

◮ Figure/ground ◮ Image segmentation ◮ Detected objects ◮ Segmentation of each object instance

slide-58
SLIDE 58

Summary

◮ Simultaneous segmentation and detection:

◮ Part detectors → figure pop-out, object grouping ◮ Color, texture → pixel grouping

◮ Graph:

◮ Parts and pixels as nodes ◮ Links encode multiple relationship types

◮ Embedding: graph nodes → Cm ◮ Decode:

◮ Figure/ground ◮ Image segmentation ◮ Detected objects ◮ Segmentation of each object instance

◮ Better person detection and segmentation on PASCAL

slide-59
SLIDE 59

Thank You