04/03/2014 3D Reconstruction: to Recover Shape from Images [Gibson ~ - - PDF document

04/03/2014 1

Shape-from-Template

Adrien Bartoli

ALCoV-ISIT, Clermont-Ferrand

Florent Brunet, Toby Collins, Vincent Gay-Bellile, Mathieu Perriollat, Daniel Pizarro, Richard Hartley Keynote at the European Workshop on Deformable Object Manipulation Lyon, March 20, 2014

3D Reconstruction: to Recover Shape from Images

Active methods Passive methods

Image from [Crandall et al, CVPR’11] Using Shape-from-Motion Image by Visnjic, 2010 Using Kinect

Escaping Criticism by del Caso, 1874 Model Town by Matt West, 2006

Blur Shading Motion Occlusions Stereoscopy Texture

[Gibson ~ 1960 ; Marr ~ 1970]

Passive Single-View Reconstruction: Visual Cues

Blur Shading Motion Occlusions Stereoscopy Unapplicable to single-view Very weak here Unapplicable to single-view Weak in general Very weak here Very weak here Texture

+ database [Hoeim et al, SIGGRAPH’05]

Passive Single-View Reconstruction: Shape Priors

Low dimensional shape space 3D Morphable Face Model [Blanz and Vetter, PAMI’03] High dimensional shape space… … but simple deformation model! Object level High dimensional shape space Scene level Faces Newspapers Detection Recognition

Shape-from-Template

Shape Image

Shape-from-Template

Material Shape Appearance

Template

Scope: object-specific passive single-view reconstruction using simple physics- based deformation from a known reference shape and a matchable appearance

[Gumerov et al, ECCV’04 ; Salzmann et al, BMVC’05 ; Perriollat et al, BMVC’08]

Detection

Shape-from-Template

Template not found Template not found

Shape-from-Template Shape-from-Template Shape-from-Template Shape-from-Template Shape-from-Template

Template not found

Shape Appearance

Template

Material

04/03/2014 2

Shape-from-Template

Template not found Template not found Template not found

Shape-from-Template

Shape Appearance

Template

Material

Template not found

Augmented Reality for Deformable Surfaces Differential Geometric Setup

3D 2D

𝑣𝑤-map - Ω ⊂ ℝ2 Image Shape Camera projection Π 𝜒 ∈ 𝐷2 Ω,ℝ3 Π ∈ 𝐷∞ ℝ3, ℝ2 Shape Appearance

Template

Material ∘ Π ∘ 𝜒 =

⇒ ⇒

𝜒 = Image warp 𝜃

𝜃

Two-Step Approach

3D 2D

𝑣𝑤-map - Ω ⊂ ℝ2 Image Image warp 𝜃 Camera projection Π Shape 1 – Image registration 2 – Shape inference

Differential Problem Statement

Shape Appearance

Template

Material Embedding 𝜒 ∈ 𝐷2 Ω, ℝ3 Projection Π ∈ 𝐷∞ ℝ3, ℝ2 Find s.t. 𝜒 = ∘ Π ∘ 𝜒 =

Shape-from-Template

Problem Relaxation

Shape Appearance

Template

Material Embedding 𝜒 ∈ 𝐷2 Ω, ℝ3 Projection Π ∈ 𝐷∞ ℝ3, ℝ2 Find s.t. 𝜒 = ∘ Π ∘ 𝜒 =

Shape-from-Template

𝜽 ∈ ? ? 𝜽

𝑈𝑊

2𝑀2 𝜃 → min

04/03/2014 3

Image Registration

Shape Appearance

Template

Material Embedding 𝜒 ∈ 𝐷2 Ω, ℝ3 Projection Π ∈ 𝐷∞ ℝ3, ℝ2 Find s.t. 𝜒 = ∘ Π ∘ 𝜒 =

Shape-from-Template

𝜽 ∈ ? ? 𝜽

𝑈𝑊

2𝑀2 𝜃 → min

Find s.t. = ∘ 𝜃 Warp 𝜃 ∈ 𝐷2 Ω, ℝ2

1 – Image registration

Factoring the Warp into Embedding and Projection

Shape Appearance

Template

Material Embedding 𝜒 ∈ 𝐷2 Ω, ℝ3 Projection Π ∈ 𝐷∞ ℝ3, ℝ2 Find s.t. 𝜒 = ∘ Π ∘ 𝜒 =

Shape-from-Template

𝜽 = 𝚸 ∘ 𝝌

Shape Inference

Shape Appearance

Template

Material Embedding 𝜒 ∈ 𝐷2 Ω, ℝ3 Projection Π ∈ 𝐷∞ ℝ3, ℝ2 Find s.t. 𝜒 = ∘ Π ∘ 𝜒 =

Shape-from-Template

𝜽 = 𝚸 ∘ 𝝌

Find s.t.

2 – Shape inference

Embedding 𝜒 ∈ 𝐷2 Ω, ℝ3 Projection Π ∈ 𝐷∞ ℝ3, ℝ2 𝜒 = 𝜃 = Π ∘ 𝜒

Shape-from-Template

Two-Step Approach

Find s.t.

2 – Shape inference

Embedding 𝜒 ∈ 𝐷2 Ω, ℝ3 Projection Π ∈ 𝐷∞ ℝ3, ℝ2 𝜒 = 𝜃 = Π ∘ 𝜒

Template

Find s.t. = ∘ 𝜃 Warp 𝜃 ∈ 𝐷2 Ω, ℝ2

1 – Image registration

Appearance Shape Material

Step 1: Image Registration, in a Nutshell

Find s.t. = ∘ 𝜃 Warp 𝜃 ∈ 𝐷2 Ω, ℝ2

1 – Image registration

Putative keypoint matches [Lowe, IJCV’04] Densification [Bookstein, PAMI’89] Template not found Local consistency [Pizarro et al, IJCV’12] 1 2 [Schmid et al, PAMI’97]

Step 1: Image Registration, in a Nutshell

Local consistency [Pizarro et al, IJCV’12] Partial self-occlusion detection [Pizarro et al, IJCV’12] Densification [Bookstein, PAMI’89] Self-occlusion aware densification [Pizarro et al, IJCV’12] Color-based refinement [Gay-Bellile et al, PAMI’10] 2 3 4 5 6

04/03/2014 4

Step 1: Image Registration, Results Step 1: Image Registration, Results

Shape-from-Template

Two-Step Approach

Find s.t.

2 – Shape inference

Embedding 𝜒 ∈ 𝐷2 Ω, ℝ3 Projection Π ∈ 𝐷∞ ℝ3, ℝ2 𝜒 = 𝜃 = Π ∘ 𝜒

Template

Find s.t. = ∘ 𝜃 Warp 𝜃 ∈ 𝐷2 Ω, ℝ2

1 – Image registration

Appearance Shape Material

Step 2: Shape Inference

3D 2D

𝑣𝑤-map - Ω ⊂ ℝ2 Image Camera projection Π Image warp 𝜃 Find s.t.

2 – Shape inference

Embedding 𝜒 ∈ 𝐷2 Ω, ℝ3 Projection Π ∈ 𝐷∞ ℝ3, ℝ2 𝜒 = 𝜃 = Π ∘ 𝜒 Calibrated pinhole

Zeroth Order Methods

3D 2D

𝑣𝑤-map - Ω ⊂ ℝ2 Image Camera projection Π Image warp 𝜃 Find s.t.

2 – Shape inference

Embedding 𝜒 ∈ 𝐷2 Ω, ℝ3 Projection Π ∈ 𝐷∞ ℝ3, ℝ2 𝜒 = 𝜃 = Π ∘ 𝜒 𝛼𝜒⊤𝛼𝜒 = I Main result: shape-wise solution of a convex relaxation [Perriollat et al, IJCV’11; Salzmann et al, PAMI’11; Brunet et al, ACCV’10; Östlund et al, ECCV’12] Zeroth order reprojection [Perriollat et al, IJCV’11; Salzmann et al, PAMI’11]

First Order Methods

3D 2D

𝑣𝑤-map - Ω ⊂ ℝ2 Image Camera projection Π Image warp 𝜃 Find s.t.

2 – Shape inference

Embedding 𝜒 ∈ 𝐷2 Ω, ℝ3 Projection Π ∈ 𝐷∞ ℝ3, ℝ2 𝜒 = 𝜃 = Π ∘ 𝜒 𝛼𝜒⊤𝛼𝜒 = I Zeroth order reprojection 𝛼𝜃 = 𝛼Π ∘ 𝜒 𝛼𝜒 First order reprojection [Bartoli et al, CVPR’12]

04/03/2014 5

First Order Methods

Find s.t.

2 – Shape inference

Embedding 𝜒 ∈ 𝐷2 Ω, ℝ3 Projection Π ∈ 𝐷∞ ℝ3, ℝ2 𝜒 = 𝜃 = Π ∘ 𝜒 𝛼𝜒⊤𝛼𝜒 = I Zeroth order reprojection 𝛼𝜃 = 𝛼Π ∘ 𝜒 𝛼𝜒 First order reprojection

𝜃 2

2𝛼𝛿⊤𝛼𝛿 + 𝛿2𝛼𝜃⊤𝛼𝜃 + 𝛿 𝛼𝛿⊤𝜃⊤𝛼𝜃 + 𝛼𝜃⊤𝜃𝛼𝛿 = I

Let 𝛿 ∈ 𝐷1 Ω, ℝ be the depth function Shape inference is this first-order quadratic PDE in 𝜹

Main result: exact point-wise solution [Bartoli et al, CVPR’12]

Intuition: neglect the dependency between 𝛿 and 𝛼𝛿

First Order Methods

𝜃 2

2𝛼𝛿⊤𝛼𝛿 + 𝛿2𝛼𝜃⊤𝛼𝜃 + 𝛿 𝛼𝛿⊤𝜃⊤𝛼𝜃 + 𝛼𝜃⊤𝜃𝛼𝛿 = I

Isometric developable Conformal

𝜃 2

2𝛼𝛿⊤𝛼𝛿 + 𝛿2𝛼𝜃⊤𝛼𝜃 + 𝛿 𝛼𝛿⊤𝜃⊤𝛼𝜃 + 𝛼𝜃⊤𝜃𝛼𝛿 = 𝜉𝛼Δ⊤𝛼Δ

𝜃 2

2𝛼𝛿⊤𝛼𝛿 + 𝛿2𝛼𝜃⊤𝛼𝜃 + 𝛿 𝛼𝛿⊤𝜃⊤𝛼𝜃 + 𝛼𝜃⊤𝜃𝛼𝛿 = 𝛼Δ⊤𝛼Δ

Isometric non-developable object Isometric (infinitesimal) weak-perspective

𝛼𝛿 2

2 + 𝛿2 𝛼𝜃𝛼𝜃⊤ + 𝛽2𝛼𝛿⊤𝛼𝛿 = 𝜉𝛼Δ⊤𝛼Δ

Isometric, unknown focal length

𝑔2 𝛼𝛿 2

2 + 𝛿2 𝛼𝜃𝛼𝜃⊤ + 𝛽2𝛼𝛿⊤𝛼𝛿 = 𝜉𝛼Δ⊤𝛼Δ ‘true’ 𝑔 = 2040 pixels 3.8% relative error Estimated 𝑔 = 2118 pixels 𝑔 = 5870 pixels

Non-flattenable Objects

3D 2D

Image warp 𝜃 Camera projection Π Deformation Ψ Conformal flattening Δ−1 𝑣𝑤-map - Ω ⊂ ℝ2 Image Δ ∈ 𝐷2 Ω, ℝ3 Ψ = 𝜒 ∘ Δ−1

Step 2: Shape Inference, Results

Ground-truth (Rigid Shape-from-Motion) Shape-from-Template

Step 2: Shape Inference, Results Step 2: Shape Inference, Results

04/03/2014 6

Some Extensions

Shape Appearance

Template 1

Material Shape Appearance

Template 2

Material Multiobject shape-from-template [Alcantarilla et al, BMVC’12]

etc.

Linear elastic deformations [Malti et al, CVPR’13] template image shape Isowarp and Conwarp [Pizarro et al, BMVC’13]

Differential constraints on 𝜃 so that 𝜃 = Π ∘ 𝜒

Main Surface Types

Physical flattening Virtual flattening Isometric deformation Elastic deformation

Gynecologic Surgery

Procedures

• Benign conditions
• Cancer
• Infertility
• Incontinence

Organs of the female reproductive system Located in the pelvic cavity Scope Abdominal (open, laparotomy) Hysteroscopic (through vagina) Laparoscopic (small incisions) Types

Preoperative Imaging

MRI US

• Diagnosis
• Procedure
• Type of surgery

Uterine Fibroids or Myomas

• Microscopic to extremely large size
• Often several of them

Benign tumors from the myometrium May be invisible in laparoscopy (and hysteroscopy) Intramural myomas (type b) Clearly visible in MRI

Laparoscopic Augmented Reality

3 Displaying Augmenting Augmentation data 2

Registration

1 Filming

04/03/2014 7

Registration is Challenging

Blur Occlusions Bleeding Smoke

Augmentation data

Preoperative MRI Preparation

Axial Sagittal Coronal Uterus surface First fibroid Second fibroid

Augmented Reality Framework

1 Registration 2 Augmentation Current frame Current frame

Registration Φ𝑗:ℝ3 → ℝ2 Requirements: R1 – deformable R2 – multimodal R3 – realtime R4 – automatic

Two-Step Registration

Registration Φ𝑗:ℝ3 → ℝ2

1 Registration Current frame Reference frame 1b Reference shape 1a

Requirements:(R1), R3, R4 Requirements: R1 – deformable R2 – multimodal R3 – realtime R4 – automatic Relax requirements with Φ𝑗 = Γ𝑗 ∘ Γ0 Preoperative to intraoperative reference Intraoperative reference to current frame

WBMTR (Wide-Baseline Multi-Texturemap Registration) [Collins et al, MIAR@MICCAI’13]

Registration Φ𝑗:ℝ3 → ℝ2

1 Registration Reference frame

Requirements: R1 – deformable R2 – multimodal R3 – realtime R4 – automatic Relax requirements with Φ𝑗 = Γ𝑗 ∘ Γ0

Current frame

Shape-from-Motion Pose with keypoints from best reference frame Reference frames Reference shape

WBMTR Registration Results

04/03/2014 8

WBMTR Registration Results Generalizing Rigid Pose to Deformations

Registration Φ𝑗:ℝ3 → ℝ2

1 Registration Current frame Reference shape

Requirements: R1 – deformable R2 – multimodal R3 – realtime R4 – automatic Relax requirements with Φ𝑗 = Γ𝑗 ∘ Γ0 Ψ𝑗:ℝ3 → ℝ3 Π: ℝ3 → ℝ2 Deformation Projection Shape-from-Template To recover Ψ𝑗 (and Π)

Uterine Shape-from-Template Results

First order, isometric Zeroth order, isometric [Salzmann et al, PAMI’09] First order, conformal

Shape-from-Template

Adrien Bartoli

ALCoV-ISIT, Clermont-Ferrand

Florent Brunet, Toby Collins, Vincent Gay-Bellile, Mathieu Perriollat, Daniel Pizarro, Richard Hartley Keynote at the European Workshop on Deformable Object Manipulation Lyon, March 20, 2014