SLIDE 1
We introduce a novel approach to solve the problem of 3D human pose estimation from a single RGB image
Input image Output 3D Pose
Lifting from the Deep: Convolutional 3D Pose Estimation from a - - PowerPoint PPT Presentation
Lifting from the Deep: Convolutional 3D Pose Estimation from a - - PowerPoint PPT Presentation
Lifting from the Deep: Convolutional 3D Pose Estimation from a Single Image Denis Tom Chris Russell Lourdes Agapito We introduce a novel approach to solve the problem of 3D human pose estimation from a single RGB image Input
SLIDE 2
SLIDE 3
Our method reasons jointly about 2D joint estimation and 3D pose reconstruction to improve both tasks.
SLIDE 4
Our approach
- First, we learn a probabilistic model of 3D human pose from
3D mocap data
- We integrate this model within a novel end-to-end CNN
architecture for joint 2D and 3D human pose estimation
- Our method achieves state-of-the-art results on the Human3.6M
dataset This model lifts 2D joint positions (landmarks) into 3D
2D landmarks 3D pose probabilistic 3D pose model
SLIDE 5
Our approach
- Next, we train a novel end-to-end multi-stage CNN for 2D
landmark estimation
Stage6 Stage2 Stage1
SLIDE 6
Our approach
- Next, we train a novel end-to-end multi-stage CNN for 2D
landmark estimation
- Each stage includes a new layer based on our probabilistic 3D
pose model of human poses to enforce 3D pose constraints
Stage6 Stage2 Stage1
SLIDE 7
Detailed architecture
SLIDE 8
Feature extraction 2D Joint prediction
Convolutional layers
C
5 x 5
C
9 x 9
C
9 x 9
C
9 x 9
P
x 2
P
x 2
P
x 2
feature extraction
C
1 x 1
C
1 x 1
C
9 x 9
2D joint prediction
SLIDE 9
Feature extraction 2D Joint prediction
For each landmark, a 2D belief map is generated This defines how confident the architecture is that a specific landmark occurs at any given pixel (u,v) of the input image
belief maps
SLIDE 10
Feature extraction 2D Joint prediction
belief maps
Probabilistic 3D pose model 3D pose
Our pre-learned probabilistic model lifts 2D landmarks into 3D and injects 3D pose information
SLIDE 11
Feature extraction 2D Joint prediction
belief maps
Probabilistic 3D pose model 3D pose
belief maps
The 3D pose is used to generate a new set of 2D belief maps
SLIDE 12
Feature extraction 2D Joint prediction
belief maps
Probabilistic 3D pose model 3D pose
belief maps
2D FUSION
Belief maps are fused together
SLIDE 13
Feature extraction 2D Joint prediction
belief maps
Probabilistic 3D pose model 3D pose
belief maps
2D FUSION
SLIDE 14
Feature extraction 2D Joint prediction
belief maps
Probabilistic 3D pose model 3D pose
2D FUSION
STAGE t=1 belief maps
SLIDE 15
STAGE t=6 STAGE t=3
Feature extraction 2D Joint prediction
belief maps
Probabilistic 3D pose model 3D pose
2D FUSION
STAGE t=1 belief maps
Shared feature extraction 2D Joint prediction Probabilistic 3D pose model
2D FUSION
STAGE t=2 belief maps belief maps
3D pose
The accuracy of the belief maps increases progressively through the stages The 2D belief maps from each stage are used as input to the next stage
SLIDE 16
STAGE t=6 STAGE t=3
Feature extraction 2D Joint prediction
belief maps
Probabilistic 3D pose model 3D pose
2D FUSION
STAGE t=1 belief maps
Shared feature extraction 2D Joint prediction Probabilistic 3D pose model
2D FUSION
STAGE t=2 belief maps belief maps
3D pose
End-to-end learning
Probabilistic 3D pose model
Output 3D pose
Output 3D Pose
Final lifting
Output Belief maps
SLIDE 17
Our approach achieves state-of-the-art results
- n the Human3.6M dataset
SLIDE 18
Example results on the Human3.6M dataset