Chirality Nets for Human Pose Regression Raymond A. Yeh*, Yuan-Ting - - PowerPoint PPT Presentation

Yeh et al. NeurIPS2019

Raymond A. Yeh*, Yuan-Ting Hu*, Alexander G. Schwing University of Illinois at Urbana-Champaign December 14, 2019

1

Chirality Nets for Human Pose Regression

Yeh et al. NeurIPS2019

Motivation

2

Human Pose Regression Prior works:

• Concatenation of joint coordinates
• Regression via deep-nets

Can we take advantage of structure in human pose?

Yeh et al. NeurIPS2019

Left Right Symmetry

Motivation

3

Yeh et al. NeurIPS2019

Respecting Left Right Symmetry

4

Yeh et al. NeurIPS2019

Chirality Chiral Pairs for Human Pose

Image Credit: Solomons & Fryhle

Chirality

5

Reflected Pose Chirality Transformed T(x) Input Pose x

Yeh et al. NeurIPS2019

Equivariance w.r.t. Chirality Transform

6

A function is chirality equivariant w.r.t. if

Fθ (T𝚓𝚘, T𝚙𝚟𝚞) T𝚙𝚟𝚞(Fθ(x)) = Fθ(T𝚓𝚘(x)) ∀x

Equivariance

Chirality Transformed Input T𝚓𝚘(x)

T𝚓𝚘

Chirality Transformed Output T𝚙𝚟𝚞(y)

T𝚙𝚟𝚞

Yeh et al. NeurIPS2019

Human Pose Representation

7

u𝚒𝚏𝚋𝚎,v𝚒𝚏𝚋𝚎 u𝚐𝚙𝚙𝚞,v𝚐𝚙𝚙𝚞

Notation:

• denote the set of right, left and

center joints

• denotes the set of dimensions per joint
• for 2D key-points
• Split

into and \

• Indicates whether to negate (reflect)

the coordinates

x ∈ ℝ(|J𝚓𝚘

𝚖 |+|J𝚓𝚘 𝚜 |+|J𝚓𝚘 𝚍 |)⋅|D𝚓𝚘|

J𝚓𝚘

𝚜 , J𝚓𝚘 𝚖 , J𝚓𝚘 𝚍

D𝚓𝚘 (u, v) D𝚓𝚘 D𝚓𝚘

n

D𝚓𝚘

p := D𝚓𝚘 D𝚓𝚘 n

Yeh et al. NeurIPS2019

Chirality Transform

8

Chirality Transform:

• Step (1): Negating dimensions for all joints
• Step (2): Switch the right and left joints’ labels

Achieve Equivariance:

• Multiply with a diagonal matrix with entries -1 and 1
• Multiply with a permutation matrix
• (1) Enforce odd symmetry in parameters
• (2) Sharing parameters (Ravanbakhsh et al. 2017)

Yeh et al. NeurIPS2019

Bias Layer:

• Consider only step (2), the switch between right & left joints

y = f𝚌𝚓𝚋𝚝(x; b) := x + b

Chiral Bias Layer

9

=

y𝚜𝚘 y𝚜𝚚 y𝚖𝚘 y𝚖𝚚 y𝚍𝚘 y𝚍𝚚

= + x b + x + b + +

Yeh et al. NeurIPS2019

Bias Layer:

• Consider the full chirality transform

y = f𝚌𝚓𝚋𝚝(x; b) := x + b

Chiral Bias Layer

10

=

y𝚜𝚘 y𝚜𝚚 y𝚖𝚘 y𝚖𝚚 y𝚍𝚘 y𝚍𝚚

= + x b + x + b + +

Darker color → multiply by -1, and white → multiply by 0

Yeh et al. NeurIPS2019

Fully Connected Layer:

• Permutation between left/right joints → Parameter sharing
• Reflection in pose → Odd symmetry
• Generalize to other layers, e.g., Conv1D, LSTM/GRU, Batch-Normalization, etc.

y = f𝙶𝙳(x; W, b) := Wx + b

Chiral Fully Connected Layer

11

y𝚜𝚘 y𝚜𝚚 y𝚖𝚘 y𝚖𝚚 y𝚍𝚘 y𝚍𝚚

= + W x b

Yeh et al. NeurIPS2019

Benefits of Chiral Layers

12

Reduction in Parameters:

• Better data efficiency, i.e., performs better with less training data

Reduction in FLOPs:

• Consider

, to compute :

• Naive:
• Exploit symmetry:
• Removes one multiplication operation for every shared weight

w := [w1, w1], x := [x1, x2] w⊺x w1 ⋅ x1+w1 ⋅ x2 w1 ⋅ (x1 + x2)

Yeh et al. NeurIPS2019

Video 2D to 3D Pose Estimation

13

2D to 3D pose estimation

x y x y z y

Image Credit: Pavllo et al.

Task:

• Predict 3D pose given 2D pose

Metric:

• Mean per-joint position error between

prediction and ground-truth

Models:

• 3D Human Pose Estimation in Video

with Temporal Convolutions and Semi- Supervised Training (Pavllo et al. CVPR, 2019)

• Ours: replace all layers with their chiral

equivariant version

Yeh et al. NeurIPS2019

Human3.6M Results

14

MPJPE (mm)

50 65 80 95 110

Training Splits (Less data → More data)

S1 .1% S1 1% S1 5% S1 10% S1 50% S1 100% S15 S156 108.9 93 85 77.8 68.2 63.9 62 56.6 105.6 99.5 93.7 85.4 71.9 67.1 65.1 59.9 Pavllo et al. Ours

Yeh et al. NeurIPS2019

HumanEva-I Results

15

Yeh et al. NeurIPS2019

Summary

16

http://chiralitynets.web.illinois.edu/

Chirality Nets:

• A family of networks built from chiral layers
• Equivariance guarantees
• Data efficiency
• Reduction in computation

Applications on human pose regression tasks