People Tracking and Pose Es5ma5on by Graph Decomposi5on Siyu Tang - - PowerPoint PPT Presentation

Siyu Tang

Holis&c Vision Group Max Planck Ins&tute for Intelligent Systems 04 July 2018

People Tracking and Pose Es5ma5on by Graph Decomposi5on

• Graph Decomposition and Multicut
• Multi-person Tracking
• Minimum cost lifted multicut problem
• Multi-person Pose Estimation
• End-to-end Learning for Graph Decompostion

Overview

2

Decomposi5on and Mul5cut

3

• A decomposition of a graph is a partition of the node set into

connected components.

• The set of edges that straddle distinct components are precisely

the multicut of the graph.

Decomposi5on and Mul5cut

4

y ∈ {0, 1}E

d ∈ RE

• Minimum Cost Multicut Problem [Groetschel et al @Mathematical Programming’1989]

min

y∈{0,1}E

X

e∈E

deye subject to ∀C ∈ cycles(G) ∀e ∈ C : (1 − ye) ≤ X

e0∈C\{e}

(1 − ye0)

• Graph Decomposition and Multicut
• Multi-person Tracking
• Minimum cost lifted multicut problem
• Multi-person Pose Estimation
• End-to-end Learning for Graph Decompostion

Overview

5

6

Input video Our tracking result People detec&on

Mul5-person Tracking

7

• Tracking as a global data associa&on problem
• Typically addressed as finding disjoint paths in the graph
• Disjoint paths do not merge or branch

Mul5-person Tracking

[Andriluka et al@CVPR’08; Zhang et al @CVPR’08; Shitrit et al @ ICCV’11; Pirsiavash et al@CVPR’11; Kuo et al @CVPR’11; Chen et al @CVPR’14; Wang et al@ECCV’16; Schulter at al @CVPR’17]

8

• Tracking as a global data associa&on problem

Mul5-person Tracking

• Typically addressed as finding disjoint paths in the graph
• Disjoint paths do not merge or branch
• Pre-processing: spa&o-NMS per frame
• Post-processing: merge tracks cross frames

9

• Tracking as a global data associa&on problem
• Typically addressed as finding disjoint paths in the graph
• Disjoint paths do not merge or branch
• Pre-processing: spa&o-NMS per frame
• Post-processing: merge tracks cross frames

Mul5-person Tracking

10

• Graph decomposi&on for mul&-person tracking
• Desired property of “tracking by graph decomposi5on”
• Joint spa&o-temporal associa&on
• Short- and long-range edges complement each other
• The number of people is op&mized

Mul5-person Tracking

Within-frame edges Long-range edges

11

• Graph decomposi&on for mul&-person tracking
• Desired property of “tracking by graph decomposi&on”
• Joint spa&o-temporal associa&on
• Short- and long-range edges complement each other
• The number of people is op&mized

Mul5-person Tracking

The Underlying Graph in Space-Time Domain: Visualizing Disjoint Paths Solu&on

Disjoint Paths x y time

Red dots: detec&on hypotheses

12

Red lines: linking hypotheses

Decompositions (clusters)

The Underlying Graph in Space-Time Domain: Visualizing Mul&Cut Solu&on

x y time

13

14

Tracking Result by Graph Decomposi5on

Detec5ons Tracklets Decomposi5on Final tracks

15

• blem.

min

x2{0,1}V y2{0,1}E

X

v2V

cvxv + X

e2E

deye

∀e = vw ∈ E : yvw ≤ xv ∀e = vw ∈ E : yvw ≤ xw ∀C ∈ cycles(G) ∀e ∈ C : (1 − ye) ≤ X

e02C\{e}

(1 − ye0)

Minimum Cost Mul5cut Problem

Consistency Transi5vity

16

• blem.

min

x2{0,1}V y2{0,1}E

X

v2V

cvxv + X

e2E

deye

• Op&miza&on
• ILP solver (Branch-and-Cut)
• Heuris&c solver (Kernighan-Lin heuris&c)

Minimum Cost Mul5cut Problem

[Kernighan&Lin@Bell System Technical Journal’1970]

∀e = vw ∈ E : yvw ≤ xv ∀e = vw ∈ E : yvw ≤ xw ∀C ∈ cycles(G) ∀e ∈ C : (1 − ye) ≤ X

e02C\{e}

(1 − ye0)

Consistency Transi5vity

17

• blem.

min

x2{0,1}V y2{0,1}E

X

v2V

cvxv + X

e2E

deye

Minimum Cost Mul5cut Problem

• Pairwise feature
• Spa&o-temporal rela&on
• Local image patch matching (DeepMatching [Weinzaepfel et al @ICCV’13])

f (e)

e = hθ, f (e)i

dey

∀e = vw ∈ E : yvw ≤ xv ∀e = vw ∈ E : yvw ≤ xw ∀C ∈ cycles(G) ∀e ∈ C : (1 − ye) ≤ X

e02C\{e}

(1 − ye0)

Consistency Transi5vity

18

How do we model the long-term connec5ons?

Time

• Deep Person Re-iden&fica&on Network

19

How do we model the long-term connec5ons?

Siamese Net Stack Net Accuracy: 84.7% 86.9% 90.0% StackPose Net

20

How do we model the long-term connec5ons?

• Deep Person Re-iden&fica&on Network
• Compare with DeepMatching feature (DM) and

Spa&o-temporal rela&on feature (ST) 0 10 30 50 100 150 200 0.4 0.5 0.6 0.7 0.8 0.9 1 Temporal distance (frames) Accuracy ST DM Re-ID Comb

• Minimum Cost LiWed Mul5cut

min

y2{0,1}E0

X

e2E0

deye

X

2

• ∀C ∈ cycles(G) ∀e ∈ C : (1 − ye) ≤

X

e02C\{e}

(1 − ye0) ∀vw ∈ E0 \ E ∀P ∈ vw-paths(G) : (1 − yvw) ≤ X

e2P

(1 − ye) ∀vw ∈ E0 \ E ∀C ∈ vw-cuts(G) : yvw ≤ X

e2C

ye

21

How to cope with and expose the uncertainty?

x x x

v1 v2 v3 v4

• 3

1

• 3

3

Mul5cut

x x x x

v1 v2 v3 v4

• 3

1

• 3

3

LiWed Mul5cut

LiWed edges

Transi5vity

3 3

• 3
• 1

3 3

• 3
• 1

Results on the MOT Benchmark

22

CVPR 17 CVPR 17 arXiv’ 16 ECCVW' 16

23

Liied Mul&cut (Ours) SenseTime

Tracking Result

• Graph Decomposition and Multicut
• Multi-person Tracking
• Minimum cost lifted multicut problem [CVPR15 CVPR17]
• Multi-person Pose Estimation
• End-to-end Learning for Graph Decomposition

Overview

24

• Rapid progresses in the recent two years
• Open pose from CMU/DeepCut from MPII


Mul5-person Pose Es5ma5on

25

26

• A node labeling of a graph

min

x

X

d∈D

X

c∈C

αdc xdc

• A mul5cut of a graph

min

y

X

dd0∈E

βdd0 ydd0

• A joint mul5cut and node labeling problem

edge variable node variable

DeepCut recap

27

body joint labels pair of detec5ons

Consistency Transi5vity Uniqueness

DeepCut recap

28

Graph Decomposi5on Problems

• Detec&on Non-maximum

suppression

• Mul& person pose es&ma&on
• Instance Segmenta&on

Patch-based CNN

• KITTI dataset, 375 x 1242
• Extract patches of different sizes: 270 x 432, 180 x 288, and 120 x 192
• Run the extracted patches to obtain local instance predictions
• There are less number of instances in the patch, so easier for CNN to assign

instance labels.

• The instance ID is not guaranteed to be consistent across different patches.

(Image from Zhang et al. 2015)

• Mul& person tracking

[Tang et al. CVPR 15, CVPR17] [Tang et al. CVPR 15] [Insafutdinov et al. CVPR17] [Kirillov et al. CVPR17]

An end-to-end learning approach?

29

• How to jointly op&mize the model parameters and the

weights of the front end CNNs?

• How to u&lize the cycle consistency constraints as

supervisory signals?


A binary cubic program

30

• The minimum cost multicut problem

min

x∈{0,1}E

X

e∈E

ce xe subject to ∀C ∈ cc(G) ∀e ∈ C : xe ≤ X

e0∈C\{e}

xe0 .

• It can be equivalently stated as an unconstrained binary multilinear

problem with a large enough C

min

x∈{0,1}E

X

e∈E

ce xe + C X

C∈cc(G)

X

e∈C

xe Y

e0∈C\{e}

(1 − xe0) .

• In the special case where the graph is complete, the above problem

is specialised to a binary cubic problem

min

x∈{0,1}E

X

e∈E

ce xe + C X

{u,v,w}∈

V

3

• (xuv¯

xvw¯ xuw + ¯ xuvxvw¯ xuw + ¯ xuv¯ xvwxuw) .

31

min

x∈{0,1}E

X

e∈E

ce xe + C X

{u,v,w}∈

V

3

• (xuv¯

xvw¯ xuw + ¯ xuvxvw¯ xuw + ¯ xuv¯ xvwxuw) .

• The new binary cubic problem.
• The corresponding CRF formulation.

E(x) = X

i

ψU

i (xi) +

X

c

ψCycle

c

(xc)

• Pattern-based Potential [Vineet et. al. @eccv’12]

ψpat

c

(xc) = ( γxc if xc ∈ Pc γmax

• therwise

End-to-End Learning for Mul5cut

End-to-End Learning for Mul5cut

32

• A better feature map.

without CRF inference with CRF inference

33

• A better pose estimation.

without CRF inference with CRF inference

End-to-End Learning for Mul5cut

34

End-to-End Learning for Mul5cut

Head-Neck-Shou Shou-Elbo-Wris Neck-LHip-RHip Hip-Knee-Ankl Mean

• rigin

1.68 3.40 1.41 3.83 2.60 Iter 1 1.12 2.79 1.06 3.17 2.04 Iter 2 1.01 2.58 0.89 2.82 1.81 Iter 3 0.96 2.47 0.87 2.79 1.76 Table 2: Ratio of non valid cycle. Numbers (%) represent the ratio of non valid cycle for four different types of cliques that are defined for adjacent body joints.

Method Head Shou Elbo Wris Hip Knee Ankl Mean Iqbal et al., [28] 58.4 53.9 44.5 35.0 42.2 36.7 31.1 43.1 pishchulin et al., [1] 89.4 84.5 70.4 59.3 68.9 62.7 54.6 70.0 Insafutdinov et al., [10] 78.4 72.5 60.2 51.0 57.2 52.0 45.4 59.5 Insafutdinov et al., [7] 88.8 87.0 75.9 64.9 74.2 68.8 60.5 74.3 Cao et al., [12] 91.2 87.6 77.7 66.8 75.4 68.9 61.7 75.6 Fang et al., [21] 88.4 86.5 78.6 70.4 74.4 73.0 65.8 76.7 Newell et al., [17] 92.1 89.3 78.9 69.8 76.2 71.6 64.7 77.5 Baseline 90.7 87.4 77.3 66.5 75.7 69.0 60.9 75.2 Our Method 90.8 87.7 78.2 67.2 76.0 69.4 61.5 75.9

Table 3: Comparison with the state-of-the-art on the MPII Human Pose dataset. 7

35

Conclusion

• The talk focuses on two visual recogni&on tasks: tracking

and pose es&ma&on.

• Towards unconstrained, realis&c visual scenes.
• Towards robust computa&onal models.

36

Open Posi5ons - Welcome to Tübingen, Germany

Max Planck Ins&tute for Intelligent Systems, Tübingen, Germany Perceiving System Department (Michael Black)

• Autonomous Vision Group (Andreas Geiger)
• Holis'c Vision Group (Siyu Tang) stang@tuebingen.mpg.de

37

Thanks.