Chirality Nets for Human Pose Regression Raymond A. Yeh*, Yuan-Ting - PowerPoint PPT Presentation

Chirality Nets for Human Pose Regression Raymond A. Yeh*, Yuan-Ting Hu*, Alexander G. Schwing University of Illinois at Urbana-Champaign December 14, 2019 Yeh et al. 1 NeurIPS2019

Motivation Human Pose Regression Prior works: • Concatenation of joint coordinates • Regression via deep-nets Can we take advantage of structure in human pose? Yeh et al. NeurIPS2019 2

Motivation Left Right Symmetry Yeh et al. NeurIPS2019 3

Respecting Left Right Symmetry Yeh et al. NeurIPS2019 4

Chirality Chirality Image Credit: Solomons & Fryhle Chiral Pairs for Human Pose Chirality Transformed T ( x ) Input Pose x Reflected Pose Yeh et al. NeurIPS2019 5

Equivariance w.r.t. Chirality Transform T 𝚓𝚘 T 𝚙𝚟𝚞 Chirality Transformed Input T 𝚓𝚘 ( x ) Chirality Transformed Output T 𝚙𝚟𝚞 ( y ) Equivariance ( T 𝚓𝚘 , T 𝚙𝚟𝚞 ) A function is chirality equivariant w.r.t. if F θ T 𝚙𝚟𝚞 ( F θ ( x )) = F θ ( T 𝚓𝚘 ( x )) ∀ x Yeh et al. NeurIPS2019 6

Human Pose Representation Notation: x ∈ ℝ ( | J 𝚓𝚘 𝚖 | + | J 𝚓𝚘 𝚜 | + | J 𝚓𝚘 𝚍 | ) ⋅ | D 𝚓𝚘 | u 𝚒𝚏𝚋𝚎 , v 𝚒𝚏𝚋𝚎 • J 𝚓𝚘 𝚜 , J 𝚓𝚘 𝚖 , J 𝚓𝚘 denote the set of right, left and • 𝚍 center joints D 𝚓𝚘 denotes the set of dimensions per joint • for 2D key-points • ( u , v ) p := D 𝚓𝚘 D 𝚓𝚘 D 𝚓𝚘 D 𝚓𝚘 D 𝚓𝚘 • Split into and \ n n u 𝚐𝚙𝚙𝚞 , v 𝚐𝚙𝚙𝚞 • Indicates whether to negate (reflect) the coordinates Yeh et al. NeurIPS2019 7

Chirality Transform Chirality Transform: • Step (1): Negating dimensions for all joints • Multiply with a diagonal matrix with entries -1 and 1 • Step (2): Switch the right and left joints’ labels • Multiply with a permutation matrix Achieve Equivariance: • (1) Enforce odd symmetry in parameters • (2) Sharing parameters (Ravanbakhsh et al. 2017) Yeh et al. NeurIPS2019 8

Chiral Bias Layer Bias Layer: y = f 𝚌𝚓𝚋𝚝 ( x ; b ) := x + b • Consider only step (2), the switch between right & left joints y 𝚜𝚘 + y 𝚜𝚚 y 𝚖𝚘 = + = + y 𝚖𝚚 y 𝚍𝚘 + y 𝚍𝚚 x + b b x Yeh et al. NeurIPS2019 9

Chiral Bias Layer Bias Layer: y = f 𝚌𝚓𝚋𝚝 ( x ; b ) := x + b • Consider the full chirality transform y 𝚜𝚘 + y 𝚜𝚚 y 𝚖𝚘 = + = + y 𝚖𝚚 y 𝚍𝚘 + y 𝚍𝚚 x + b b x Darker color → multiply by -1, and white → multiply by 0 Yeh et al. NeurIPS2019 10

Chiral Fully Connected Layer Fully Connected Layer: y = f 𝙶𝙳 ( x ; W , b ) := W x + b y 𝚜𝚘 y 𝚜𝚚 y 𝚖𝚘 = + y 𝚖𝚚 y 𝚍𝚘 y 𝚍𝚚 W b x • Permutation between left/right joints → Parameter sharing • Reflection in pose → Odd symmetry • Generalize to other layers, e.g., Conv1D, LSTM/GRU, Batch-Normalization, etc . Yeh et al. NeurIPS2019 11

Benefits of Chiral Layers Reduction in Parameters: • Better data efficiency, i.e., performs better with less training data Reduction in FLOPs: w ⊺ x • Consider , to compute : w := [ w 1 , w 1 ], x := [ x 1 , x 2 ] • Naive: w 1 ⋅ x 1 + w 1 ⋅ x 2 • Exploit symmetry: w 1 ⋅ ( x 1 + x 2 ) • Removes one multiplication operation for every shared weight Yeh et al. NeurIPS2019 12

Video 2D to 3D Pose Estimation Task: y y • Predict 3D pose given 2D pose x 2D to 3D pose estimation Metric: • Mean per-joint position error between y z prediction and ground-truth x Models: • 3D Human Pose Estimation in Video with Temporal Convolutions and Semi- Supervised Training (Pavllo et al. CVPR, 2019) • Ours: replace all layers with their chiral Image Credit: Pavllo et al. equivariant version Yeh et al. NeurIPS2019 13

Human3.6M Results 105.6 110 Pavllo et al. Ours 99.5 108.9 93.7 95 MPJPE (mm) 85.4 93 80 85 71.9 77.8 67.1 65.1 65 59.9 68.2 63.9 62 56.6 50 S1 .1% S1 1% S1 5% S1 10% S1 50% S1 100% S15 S156 Training Splits (Less data → More data) Yeh et al. NeurIPS2019 14

HumanEva-I Results Yeh et al. NeurIPS2019 15

Summary Chirality Nets: • A family of networks built from chiral layers • Equivariance guarantees • Data efficiency • Reduction in computation http://chiralitynets.web.illinois.edu/ Applications on human pose regression tasks Yeh et al. NeurIPS2019 16