The DPM Detector P. Felzenszwalb, R. Girshick, D. McAllester, D. - - PowerPoint PPT Presentation

▶
the dpm detector
SMART_READER_LITE
LIVE PREVIEW

The DPM Detector P. Felzenszwalb, R. Girshick, D. McAllester, D. - - PowerPoint PPT Presentation

Dec 02, 2022 124 likes •574 views

The DPM Detector P. Felzenszwalb, R. Girshick, D. McAllester, D. Ramanan Object Detection with Discriminatively Trained Part Based Models T-PAMI, 2010 Paper: http://cs.brown.edu/~pff/papers/lsvm-pami.pdf Code:

slide-1
SLIDE 1

The DPM Detector

  • P. Felzenszwalb, R. Girshick, D. McAllester, D. Ramanan

Object Detection with Discriminatively Trained Part Based Models T-PAMI, 2010 Paper: http://cs.brown.edu/~pff/papers/lsvm-pami.pdf

Code: http://www.cs.berkeley.edu/~rbg/latent/

Sanja Fidler CSC420: Intro to Image Understanding 1 / 37

slide-2
SLIDE 2

The HOG Detector

The HOG detector models an object class as a single rigid template Figure: Single HOG template models people in upright pose.

Sanja Fidler CSC420: Intro to Image Understanding 2 / 37

slide-3
SLIDE 3

But Objects Are Composed of Parts

Sanja Fidler CSC420: Intro to Image Understanding 3 / 37

slide-4
SLIDE 4

Even Rigid Objects Are Composed of Parts

Sanja Fidler CSC420: Intro to Image Understanding 4 / 37

slide-5
SLIDE 5

Objects Are Composed of Deformable Parts

Revisit the old idea by Fischler & Elschlager 1973 Objects are composed of parts at specific relative locations. Our model should probably also model object parts. Different instances of the same object class have parts in slightly different

  • locations. Our object model should thus allow slight slack in part position.

Figure: Objects are a collection of deformable parts

[Pic from: R. Girshik]

Sanja Fidler CSC420: Intro to Image Understanding 5 / 37

slide-6
SLIDE 6

The DPM Model

The DPM model starts by borrowing the idea of the HOG detector. It takes a HOG template for the full object. (If you take something that works, things can only get better, right?)

Sanja Fidler CSC420: Intro to Image Understanding 6 / 37

slide-7
SLIDE 7

The DPM Model

DPM now wants to add parts. It wants to add them at locations relative to the location of the root filter. Relative makes sense: if we move, we take our parts with us.

Sanja Fidler CSC420: Intro to Image Understanding 7 / 37

slide-8
SLIDE 8

The DPM Model

Add a part at a relative location and scale.

Sanja Fidler CSC420: Intro to Image Understanding 8 / 37

slide-9
SLIDE 9

The DPM Model

Give some slack to the location of the part. Why is this a good idea?

Sanja Fidler CSC420: Intro to Image Understanding 9 / 37

slide-10
SLIDE 10

The DPM Model

People are of different heights, thus have feet at different locations relative to the head. And we want to detect all people, not just the average ones.

Sanja Fidler CSC420: Intro to Image Understanding 10 / 37

slide-11
SLIDE 11

The DPM Model

People are of different heights, thus have feet at different locations relative to the head. And we want to detect all people, not just the average ones.

Sanja Fidler CSC420: Intro to Image Understanding 11 / 37

slide-12
SLIDE 12

The DPM Model

People are of different heights, thus have feet at different locations relative to the head. And we want to detect all people, not just the average ones.

Sanja Fidler CSC420: Intro to Image Understanding 12 / 37

slide-13
SLIDE 13

The DPM Model

We will, however, trust less detections where parts are not exactly in their expected location. DPM penalizes part shifts with a quadratic function: a(x − vx)2 + b(x − vx) + c(y − vy)2 + d(y − vy)

Sanja Fidler CSC420: Intro to Image Understanding 13 / 37

slide-14
SLIDE 14

The DPM Model

Each part also has an appearance, which is modeled with a HOG template Each part’s template is at twice the resolution as the root filter

Sanja Fidler CSC420: Intro to Image Understanding 14 / 37

slide-15
SLIDE 15

The DPM Model

And finally, DPM has a few parts. Typically 6 (but it’s a parameter you can play with). How many weights does a 6-part DPM model have? How shall we score this part-model guy in an image (how to do detection)?

Sanja Fidler CSC420: Intro to Image Understanding 15 / 37

slide-16
SLIDE 16

Remember the HOG Detector

HOG detector computes image pyramid, HOG features, and scores each window with a learned linear classifier

[Pic from: R. Girshik]

Sanja Fidler CSC420: Intro to Image Understanding 16 / 37

slide-17
SLIDE 17

DPM Detector

For DPM the story is quite similar (pyramid, HOG, score window with a learned linear classifier), but now we also need to score the parts.

[Pic from: R. Girshik]

Sanja Fidler CSC420: Intro to Image Understanding 17 / 37

slide-18
SLIDE 18

Scoring

Sanja Fidler CSC420: Intro to Image Understanding 18 / 37

slide-19
SLIDE 19

Scoring

More specifically, we will score a location (window) in the image as follows: score(l, p0) = max

p1,...,pn

  • n
  • i=0

Fi · HOG(l, pi) −

n

  • i=1

wdef

i · (dx, dy, dx2, dy 2)

  • where

F0 is the (learned) HOG template for root filter Fi is the (learned) HOG template for part i HOG(l, pi) means a HOG feature cropped in window defined by part location pi at level l of the HOG pyramid wdefi are (learned) weights for the deformation penalty (dx, dy, dx2, dy 2) with (dx, dy) = (xi, yi) − ((x0, y0) + vi) tell us how far the part i is from its expected position (x0, y0) + vi) Main question: How shall we compute that nasty maxp1,...,pn?

Sanja Fidler CSC420: Intro to Image Understanding 19 / 37

slide-20
SLIDE 20

Scoring

More specifically, we will score a location (window) in the image as follows: score(l, p0) = max

p1,...,pn

  • n
  • i=0

Fi · HOG(l, pi) −

n

  • i=1

wdef

i · (dx, dy, dx2, dy 2)

  • where

F0 is the (learned) HOG template for root filter Fi is the (learned) HOG template for part i HOG(l, pi) means a HOG feature cropped in window defined by part location pi at level l of the HOG pyramid wdefi are (learned) weights for the deformation penalty (dx, dy, dx2, dy 2) with (dx, dy) = (xi, yi) − ((x0, y0) + vi) tell us how far the part i is from its expected position (x0, y0) + vi) Main question: How shall we compute that nasty maxp1,...,pn?

Sanja Fidler CSC420: Intro to Image Understanding 19 / 37

slide-21
SLIDE 21

Scoring

Push the max inside (why can we do that?): score(l, p0) = F0 ·HOG(l, p0)+

n

  • i=1

max

pi

  • Fi ·HOG(l, pi)−wdef

i ·φdef (xi, yi)

  • Sanja Fidler

CSC420: Intro to Image Understanding 20 / 37

slide-22
SLIDE 22

Scoring

Push the max inside: score(l, p0) = F0 ·HOG(l, p0)+

n

  • i=1

max

pi

  • Fi ·HOG(l, pi)−wdef

i ·φdef (xi, yi)

  • We can compute this with dynamic programming. Any idea how?

Sanja Fidler CSC420: Intro to Image Understanding 20 / 37

slide-23
SLIDE 23

Computing the Score with Dynamic Programming

Figure: We can compute Fi · HOG(l, pi) for the full level l via cross-correlation of the HOG feature matrix at level l with the template (filter) Fi

Sanja Fidler CSC420: Intro to Image Understanding 21 / 37

slide-24
SLIDE 24

Computing the Score with Dynamic Programming

Sanja Fidler CSC420: Intro to Image Understanding 22 / 37

slide-25
SLIDE 25

Computing the Score with Dynamic Programming

Sanja Fidler CSC420: Intro to Image Understanding 23 / 37

slide-26
SLIDE 26

Computing the Score with Dynamic Programming

Sanja Fidler CSC420: Intro to Image Understanding 24 / 37

slide-27
SLIDE 27

Computing the Score with Dynamic Programming

Sanja Fidler CSC420: Intro to Image Understanding 25 / 37

slide-28
SLIDE 28

Computing the Score with Dynamic Programming

Figure: We can compute these scores efficiently with something called distance transforms

(this is exact). But works equally well: Simply limit the scope of where each part could be to a small area, e.g., a few HOG cells up,down,left,right relative to yellow spot (this is approx).

Sanja Fidler CSC420: Intro to Image Understanding 26 / 37

slide-29
SLIDE 29

Computing the Score with Dynamic Programming

Sanja Fidler CSC420: Intro to Image Understanding 27 / 37

slide-30
SLIDE 30

Computing the Score with Dynamic Programming

Sanja Fidler CSC420: Intro to Image Understanding 28 / 37

slide-31
SLIDE 31

Detection

[Pic from: Felzenswalb et al., 2010] Sanja Fidler CSC420: Intro to Image Understanding 29 / 37

slide-32
SLIDE 32

Training

You can’t train this model as simple as the HOG detector, via SVM. For those taking CSC411: Why not?

Sanja Fidler CSC420: Intro to Image Understanding 30 / 37

slide-33
SLIDE 33

Training

You can’t train this model as simple as the HOG detector, via SVM. For those taking CSC411: Why not? Because the part positions are not annotated (we don’t have ground-truth, and SVM needs ground-truth). We say that the parts are latent. You can train the model with something called latent SVM. For ML buffs: Check the Felzenswalb paper For those with even stronger ML stomach: Yu, Joachims, Learning Structural SVMs with Latent Variables, ICML’09.

Sanja Fidler CSC420: Intro to Image Understanding 30 / 37

slide-34
SLIDE 34

Results

Figure: Performance of the HOG detector on person class on PASCAL VOC

[Pic from: R. Girshik] Sanja Fidler CSC420: Intro to Image Understanding 31 / 37

slide-35
SLIDE 35

Results

Figure: DPM version 1: adds the parts

[Pic from: R. Girshik] Sanja Fidler CSC420: Intro to Image Understanding 31 / 37

slide-36
SLIDE 36

Results

Figure: DPM version 2: adds another template (called mixture or component). Supposed to detect also people sitting down (e.g., occluded by desk).

[Pic from: R. Girshik] Sanja Fidler CSC420: Intro to Image Understanding 31 / 37

slide-37
SLIDE 37

Results

Figure: DPM version 3: adds multiple mixtures (components)

[Pic from: R. Girshik] Sanja Fidler CSC420: Intro to Image Understanding 31 / 37

slide-38
SLIDE 38

Results

[Pic from: R. Girshik] Sanja Fidler CSC420: Intro to Image Understanding 31 / 37

slide-39
SLIDE 39

Learned Models

[Pic from: Felzenswalb et al., 2010] Sanja Fidler CSC420: Intro to Image Understanding 32 / 37

slide-40
SLIDE 40

Learned Models

[Pic from: Felzenswalb et al., 2010] Sanja Fidler CSC420: Intro to Image Understanding 33 / 37

slide-41
SLIDE 41

Learned Models

(Takes some imagination to see a cat...)

[Pic from: Felzenswalb et al., 2010] Sanja Fidler CSC420: Intro to Image Understanding 34 / 37

slide-42
SLIDE 42

Results

[Pic from: Felzenswalb et al., 2010] Sanja Fidler CSC420: Intro to Image Understanding 35 / 37

slide-43
SLIDE 43

Results

[Pic from: Felzenswalb et al., 2010] Sanja Fidler CSC420: Intro to Image Understanding 36 / 37

slide-44
SLIDE 44

DPM

As you already know, the code is available: http://www.cs.berkeley.edu/~rbg/latent/ Trivia: Takes about 20-30 seconds per image per class. Speed-ups exist. Depending on the size of the dataset, training takes around 12 hours (for most PASCAL classes). Has some cool post-processing tricks: bounding box prediction and context re-scoring. Each typically results in around 2% improvement in AP. In the code, if you switch off the parts, you get the Dalal & Triggs’ HOG detector.

Sanja Fidler CSC420: Intro to Image Understanding 37 / 37