Hao Su Image world Shape world How humans represent 3D in mind? - - PowerPoint PPT Presentation

Synthesize for Learning:

Joint analysis of 2D images and 3D shapes

Hao Su

Image world Shape world

How humans represent 3D in mind?

Mental rotation

by Roger N. Shepard, National Science Medal Laurate, Stanford and Lynn Cooper, Professor at Columbia University

Shape constancy

3D Perception is important for robots

Cosimo Alfredo Pina, “The domestic robots are getting closer”

3D Perception is important for robots

3D Perception is important for robots

3D Perception is important for robots

3D Perception is important for robots

contrast color motion texture symmetry category-specific 3D knowledge part ……

2D-3D lifting by machine learning

Synthesize for learning: from virtual world to real world

Shape Database

A shape repository with rich annotation

• First build & learn in a 3D Virtual Environment,

Synthesize for learning: from virtual world to real world

Simulator

Synthetic sensory data

Shape Database

Object attributes

Class, Viewpoint, Material, Symmetry, …

A shape repository with rich annotation

• First build & learn in a 3D Virtual Environment,

…

Synthesize for learning: from virtual world to real world

Simulator

Training

Shape Database

Object attributes

Class, Viewpoint, Material, Symmetry, …

• First build & learn in a 3D Virtual Environment,

A shape repository with rich annotation

Synthetic sensory data

…

Synthesize for learning: from virtual world to real world

Object attributes

Real data Testing

• Then adapt to 2D Real World

Machine learning is data hungry

2000 2002 2008 2010 2004 2006

Caltech 101 Caltech 256 LabelMe CIFAR

10# 10$ 10% 10& 10'

ImageNet

Review: image classification dataset

Status review of 3D datasets

<= 60 models per class (average) <= 10,000 models in total <= 100 models in total

Status review of 3D datasets

2000 2002 2008 2010 2004 2006

Caltech 101 Caltech 256 LabelMe CIFAR

10# 10$ 10% 10& 10'

ImageNet

State-of-the-art 3D shape dataset

# images

Limited in

• scale
• bject classes
• diversity

My work: Build large-scale 3D datasets of objects

…

~3 million models in total ~2,000 classes Rich annotations

(in progress)

An object-centric 3D knowledge-base

Physical properties Part decomposition Symmetry Affordance Material Semantics Images

ShapeNet: a large-scale 3D datasets of objects

# models # models per classes

10( 10 10) 10# 10$ 10% 10& 10 10( 10) 10#

SHREC14 TSB PSB CCCC WMB SHREC12 MSB BAB ESB

ShapeNet

My work: Develop data-driven 3D learning algorithms

Simulator

Synthetic sensory data Training

ShapeNet

Object attributes

Class, Viewpoint, Material, Symmetry, …

A shape repository with rich annotation …

Application 1: 3D viewpoint estimation

car

3D Viewpoint Estimation

azimuth elevatio n in-plane rotation

ICCV 2015 oral: Render for CNN: Viewpoint Estimation in Images Using CNNs Trained with Rendered 3D Model Views

PASCAL3D+ dataset [Xiang et al.]

Accurate viewpoint label acquisition is expensive

Annotation takes ~1 min per object

30K images with viewpoint labels in PASCAL3D+ dataset [Xiang et al.]

High-cost Label Acquisition High-capacity Model

60M parameters. AlexNet [Krizhevsky et al.]

How to get MORE images with ACCURATE viewpoint labels?

Manual alignment by annotators Auto alignment through rendering

A “Data Engineering” journey ConvNet: Ah ha, I know! Viewpoint is just the brightness pattern!

47% on real test set L 95% on synthetic val set

A “Data Engineering” journey ConvNet: Ah ha, I know! Viewpoint is just the brightness pattern!

47% on real test set L 95% on synthetic val set

47% -> 74%

A “Data Engineering” journey

Randomize lighting

ConvNet: hmm.. viewpoint is not the brightness

• pattern. Maybe it’s the contour?

47% -> 74%

A “Data Engineering” journey

Randomize lighting

ConvNet: hmm.. viewpoint is not the brightness

• pattern. Maybe it’s the contour?

A “Data Engineering” journey

74% -> 86% Add backgrounds

ConvNet: It becomes really hard! Let me look more into the picture.

A “Data Engineering” journey

bbox crop texture 86% -> 93%

A “Data Engineering” journey

bbox crop texture

ConvNet: the mapping becomes hard. I have to learn harder to get it right!

86% -> 93%

Key Lesson: Don’t give CNN a chance to “cheat” - it’s very good

at it. When there is no way to cheat, true learning starts.

Render for CNN Image Synthesis Pipeline

3D model Rendering Add bkg Crop

Hyper-parameters estimation from real images

2.4M synthesized images for 12 categories

• High scalability
• High quality
• Overfit-resistant
• Accurate labels

Metric: viewpoint accuracy and median angle error (lower the better) Real test images from PASCAL3D+ dataset Our model trained on rendered images outperforms state-of-the-art model trained on real images in PASCAL3D+.

8 9 10 11 12 13 14 15 16 Vps&Kps (CVPR15) RenderForCNN (Ours) Viewpoint Median Error

Results

Application 2: 3D human pose estimation

3DV 2015 oral: Synthesizing Training Images for Boosting Human 3D Pose Estimation

Challenge: clothing variation

3DV 2015 oral: Synthesizing Training Images for Boosting Human 3D Pose Estimation

Automatic texture transfer from images to shapes

3DV 2015 oral: Synthesizing Training Images for Boosting Human 3D Pose Estimation

Effectiveness of texture augmentation

Texture transfer for rigid objects

SIGGRAPH Asia 16: Unsupervised Texture Transfer from Images to Model Collections

Product photos Automatically textured shapes

Domain adaptation between Virtual and Reality

Map features from real and synthetic images to the same domain 3DV 2015 oral: Synthesizing Training Images for Boosting Human 3D Pose Estimation

Adversarial learning based domain adaptation

3DV 2015 oral: Synthesizing Training Images for Boosting Human 3D Pose Estimation

Domain adaptation between Virtual and Reality

Results: 3D human pose estimation

Application 3: Attention-based object identification

SIGGRAPH Asia 2016: 3D Attention-Driven Depth Acquisition for Object Identification

Background

• 1. How is the scene

composited?

• 2. What are these?

Background

49

Object identification

ShapeNet

Autonomous object identification

The main challenge – next-best-view problem

• Observation is partial and progressive

à View planning

• Assessing views whose observation is unknown

51

Observed view

Unobserve d views

? ? ?

How can you know which view is better without knowing its observation?

Simulate For Reinforcement learning

• Train from virtual scanned ShapeNet models using Reinforcement Learning
• Test in a real environment

The general framework

Goal Belief

Observe

Action

Recognition:

• Incremental

classification based on history View planning:

• Evaluate a

view based

• n history

The general framework

Attention mechanism

• Goal-oriented and stimulus-driven

55

Glimpse Task

Perform Supervision

• r reward

Internal Representation

Stores the info.

• f history

Control of goal oriented and stimulus driven attention mechanisms in the brain, Nature Review Neuroscience. 2002

3D Recurrent Attention Model

ℎ(

(,)

ℎ(

(()

𝐽(/)

𝜄 , , 𝜚(,) 𝜄 ( , 𝜚(() 𝜄 ) , 𝜚())

…

𝐽(,) 𝐽(()

ℎ(

())

NBV emission NBV emission NBV emission

…

ℎ,

(,)

ℎ,

(()

ℎ,

())

Feature extraction Feature extraction Feature extraction

𝜄 / , 𝜚(/)

𝜄 , , 𝜚(,) 𝜄 ( , 𝜚(() initial view classify classify classify

Discriminative view selection View aggregation

Reinforcement learning needs LOTS of data to train!

• Simulate many many scan sequences in virtual environment

Results

Results

59

Quantitative results

Reconstructed 3D scene

SIGGRAPH Asia 2016: 3D Attention-Driven Depth Acquisition for Object Identification

Summary

• Key theme: learn in a virtual environment of

3D shapes, test in real scenes of 2D RGB(D) images

• Data: build a large-scale 3D database

(ShapeNet) with rich annotations

• Synthesize training data for deep learning,

applicable for many tasks

CV CG ML

Thank you!