04/03/2014 3D Reconstruction: to Recover Shape from Images [Gibson ~ - PDF document

04/03/2014 3D Reconstruction: to Recover Shape from Images [Gibson ~ 1960 ; Marr ~ 1970] Shape-from-Template Motion Stereoscopy Blur Occlusions Shading Texture Escaping Criticism by Model Town del Caso, 1874 by Matt West, 2006 Adrien Bartoli ALCoV-ISIT, Clermont-Ferrand Active methods Passive methods Florent Brunet, Toby Collins, Vincent Gay-Bellile, Mathieu Perriollat, Daniel Pizarro, Richard Hartley Keynote at the European Workshop on Deformable Object Manipulation Lyon, March 20, 2014 Image by Visnjic, 2010 Image from [Crandall et al, CVPR’11] Using Kinect Using Shape-from-Motion Passive Single-View Reconstruction: Visual Cues Passive Single-View Reconstruction: Shape Priors High dimensional shape space Scene level Detection Recognition Faces Newspapers Unapplicable Unapplicable Weak in Very weak here Very weak here Very weak here Low dimensional shape space High dimensional shape space … to single-view to single-view general Object level Occlusions Shading Texture Motion Stereoscopy Blur + database 3D Morphable Face Model … but simple deformation model! [Hoeim et al, SIGGRAPH’05] [Blanz and Vetter, PAMI’03] Shape-from-Template Shape-from-Template Template [Gumerov et al, ECCV’04 ; Salzmann et al, BMVC’05 ; Perriollat et al, BMVC’08] Scope: object-specific passive single-view reconstruction using simple physics- based deformation from a known reference shape and a matchable appearance Appearance Shape Material Shape-from-Template Template Shape-from-Template Appearance Shape Material Template Shape-from-Template not found Shape-from-Template Template Shape Shape-from-Template Detection not found Image Template Shape-from-Template not found 1

04/03/2014 Shape-from-Template Augmented Reality for Deformable Surfaces Template Appearance Shape Material Template not found Template not found Shape-from-Template Template not found Template not found Differential Geometric Setup Two-Step Approach Template 3D 3D Appearance Shape Material 𝜒 ∈ 𝐷 2 Ω,ℝ 3 Shape Shape Π ∈ 𝐷 ∞ ℝ 3 , ℝ 2 Camera Camera 2 – Shape inference projection Π projection Π ⇒ 𝜒 = Image warp 𝜃 Image warp 𝜃 1 – Image registration 𝜃 𝑣𝑤 -map - Ω ⊂ ℝ 2 𝑣𝑤 -map - Ω ⊂ ℝ 2 ⇒ Image Image = ∘ Π ∘ 𝜒 2D 2D Differential Problem Statement Problem Relaxation Template Template Appearance Shape Material Appearance Shape Material Shape-from-Template Shape-from-Template 𝜽 ? 𝜽 ∈ ? = ∘ Π ∘ 𝜒 = ∘ Π ∘ 𝜒 Embedding 𝜒 ∈ 𝐷 2 Ω, ℝ 3 Embedding 𝜒 ∈ 𝐷 2 Ω, ℝ 3 Find Π ∈ 𝐷 ∞ ℝ 3 , ℝ 2 s.t. Find Π ∈ 𝐷 ∞ ℝ 3 , ℝ 2 s.t. Projection Projection 𝜒 = 𝜒 = 𝑈𝑊 2 𝑀 2 𝜃 → min 2

04/03/2014 Image Registration Factoring the Warp into Embedding and Projection Template Template Appearance Shape Material Appearance Shape Material 𝜽 = 𝚸 ∘ 𝝌 Shape-from-Template Shape-from-Template 𝜽 ? 𝜽 ∈ ? = ∘ Π ∘ 𝜒 = ∘ Π ∘ 𝜒 Embedding 𝜒 ∈ 𝐷 2 Ω, ℝ 3 Embedding 𝜒 ∈ 𝐷 2 Ω, ℝ 3 Find Π ∈ 𝐷 ∞ ℝ 3 , ℝ 2 s.t. Find Π ∈ 𝐷 ∞ ℝ 3 , ℝ 2 s.t. Projection Projection 𝜒 = 𝜒 = 𝑈𝑊 2 𝑀 2 𝜃 → min 1 – Image registration 𝜃 ∈ 𝐷 2 Ω, ℝ 2 Find Warp s.t. = ∘ 𝜃 Shape Inference Two-Step Approach Template Template Appearance Shape Material Appearance Shape Material 𝜽 = 𝚸 ∘ 𝝌 Shape-from-Template Shape-from-Template 1 – Image registration = ∘ Π ∘ 𝜒 Embedding 𝜒 ∈ 𝐷 2 Ω, ℝ 3 Find Π ∈ 𝐷 ∞ ℝ 3 , ℝ 2 s.t. 𝜃 ∈ 𝐷 2 Ω, ℝ 2 Projection Find Warp s.t. = ∘ 𝜃 𝜒 = 2 – Shape inference 2 – Shape inference 𝜃 = Π ∘ 𝜒 𝜃 = Π ∘ 𝜒 Embedding 𝜒 ∈ 𝐷 2 Ω, ℝ 3 Embedding 𝜒 ∈ 𝐷 2 Ω, ℝ 3 Find s.t. Π ∈ 𝐷 ∞ ℝ 3 , ℝ 2 Find s.t. Π ∈ 𝐷 ∞ ℝ 3 , ℝ 2 Projection 𝜒 = Projection 𝜒 = Step 1: Image Registration, in a Nutshell Step 1: Image Registration, in a Nutshell Local consistency 1 – Image registration 2 [Pizarro et al, IJCV’12] 𝜃 ∈ 𝐷 2 Ω, ℝ 2 Find Warp s.t. = ∘ 𝜃 Densification 3 [Bookstein, PAMI’89] Putative keypoint matches 1 [Lowe, IJCV’04] Partial self-occlusion detection 4 [Pizarro et al, IJCV’12] Densification [Bookstein, PAMI’89] Self-occlusion aware densification 5 [Pizarro et al, IJCV’12] Local consistency 2 [Pizarro et al, IJCV’12] Color-based refinement 6 Template [Schmid et al, PAMI’97] [Gay-Bellile et al, PAMI’10] not found 3

04/03/2014 Step 1: Image Registration, Results Step 1: Image Registration, Results Two-Step Approach Step 2: Shape Inference Template 2 – Shape inference 𝜃 = Π ∘ 𝜒 Embedding 𝜒 ∈ 𝐷 2 Ω, ℝ 3 Find s.t. Π ∈ 𝐷 ∞ ℝ 3 , ℝ 2 3D Projection 𝜒 = Appearance Shape Material Calibrated pinhole Shape-from-Template 1 – Image registration Camera 𝜃 ∈ 𝐷 2 Ω, ℝ 2 Find Warp s.t. = ∘ 𝜃 projection Π Image warp 𝜃 2 – Shape inference 𝜃 = Π ∘ 𝜒 Embedding 𝜒 ∈ 𝐷 2 Ω, ℝ 3 𝑣𝑤 -map - Ω ⊂ ℝ 2 Image Find s.t. Π ∈ 𝐷 ∞ ℝ 3 , ℝ 2 Projection 𝜒 = 2D [Perriollat et al, IJCV’11; Salzmann et al, PAMI’11] [Bartoli et al, CVPR’12] Zeroth Order Methods First Order Methods Zeroth order First order Zeroth order reprojection reprojection reprojection 2 – Shape inference 2 – Shape inference 𝜃 = Π ∘ 𝜒 𝜃 = Π ∘ 𝜒 𝛼𝜃 = 𝛼Π ∘ 𝜒 𝛼𝜒 Embedding 𝜒 ∈ 𝐷 2 Ω, ℝ 3 Embedding 𝜒 ∈ 𝐷 2 Ω, ℝ 3 Find s.t. Find s.t. Π ∈ 𝐷 ∞ ℝ 3 , ℝ 2 Π ∈ 𝐷 ∞ ℝ 3 , ℝ 2 Projection Projection 3D 𝜒 = 𝛼𝜒 ⊤ 𝛼𝜒 = I 3D 𝜒 = 𝛼𝜒 ⊤ 𝛼𝜒 = I Camera Camera projection Π projection Π Image warp 𝜃 Image warp 𝜃 𝑣𝑤 -map - Ω ⊂ ℝ 2 𝑣𝑤 -map - Ω ⊂ ℝ 2 Image Image 2D Main result: shape-wise solution of a convex relaxation 2D [Perriollat et al, IJCV’11; Salzmann et al, PAMI’11; Brunet et al, ACCV’10; Östlund et al, ECCV’12] 4

04/03/2014 [Bartoli et al, CVPR’12] First Order Methods First Order Methods Zeroth order First order reprojection reprojection 2 – Shape inference Isometric developable 𝜃 = Π ∘ 𝜒 𝛼𝜃 = 𝛼Π ∘ 𝜒 𝛼𝜒 Embedding 𝜒 ∈ 𝐷 2 Ω, ℝ 3 𝜃 2 2 𝛼𝛿 ⊤ 𝛼𝛿 + 𝛿 2 𝛼𝜃 ⊤ 𝛼𝜃 + 𝛿 𝛼𝛿 ⊤ 𝜃 ⊤ 𝛼𝜃 + 𝛼𝜃 ⊤ 𝜃𝛼𝛿 = I Find s.t. Π ∈ 𝐷 ∞ ℝ 3 , ℝ 2 Projection 𝜒 = 𝛼𝜒 ⊤ 𝛼𝜒 = I Isometric non-developable object 2 𝛼𝛿 ⊤ 𝛼𝛿 + 𝛿 2 𝛼𝜃 ⊤ 𝛼𝜃 + 𝛿 𝛼𝛿 ⊤ 𝜃 ⊤ 𝛼𝜃 + 𝛼𝜃 ⊤ 𝜃𝛼𝛿 = 𝛼Δ ⊤ 𝛼Δ 𝜃 2 Let 𝛿 ∈ 𝐷 1 Ω, ℝ be the depth function Conformal Shape inference is this first-order quadratic PDE in 𝜹 2 𝛼𝛿 ⊤ 𝛼𝛿 + 𝛿 2 𝛼𝜃 ⊤ 𝛼𝜃 + 𝛿 𝛼𝛿 ⊤ 𝜃 ⊤ 𝛼𝜃 + 𝛼𝜃 ⊤ 𝜃𝛼𝛿 = 𝜉𝛼Δ ⊤ 𝛼Δ 𝜃 2 2 𝛼𝛿 ⊤ 𝛼𝛿 + 𝛿 2 𝛼𝜃 ⊤ 𝛼𝜃 + 𝛿 𝛼𝛿 ⊤ 𝜃 ⊤ 𝛼𝜃 + 𝛼𝜃 ⊤ 𝜃𝛼𝛿 = I 𝜃 2 𝑔 = 5870 pixels Isometric (infinitesimal) weak-perspective Main result: exact point-wise solution 2 + 𝛿 2 𝛼𝜃𝛼𝜃 ⊤ + 𝛽 2 𝛼𝛿 ⊤ 𝛼𝛿 = 𝜉𝛼Δ ⊤ 𝛼Δ 𝛼𝛿 2 Intuition: neglect the dependency between 𝛿 and 𝛼𝛿 ‘ true ’ 𝑔 = 2040 pixels Estimated 𝑔 = 2118 pixels Isometric, unknown focal length 𝑔 2 𝛼𝛿 2 2 + 𝛿 2 𝛼𝜃𝛼𝜃 ⊤ + 𝛽 2 𝛼𝛿 ⊤ 𝛼𝛿 = 𝜉𝛼Δ ⊤ 𝛼Δ 3.8% relative error Non-flattenable Objects Step 2: Shape Inference, Results 3D Deformation Ψ Ψ = 𝜒 ∘ Δ −1 Conformal Camera flattening Δ −1 projection Π Δ ∈ 𝐷 2 Ω, ℝ 3 Image warp 𝜃 𝑣𝑤 -map - Ω ⊂ ℝ 2 Image Ground-truth Shape-from-Template 2D (Rigid Shape-from-Motion) Step 2: Shape Inference, Results Step 2: Shape Inference, Results 5

04/03/2014 Some Extensions Main Surface Types Multiobject shape-from-template [Alcantarilla et al, BMVC’12] Physical flattening Virtual flattening Template 1 Template 2 Isometric deformation etc. Appearance Shape Material Appearance Shape Material Linear elastic deformations [Malti et al, CVPR’13] Elastic deformation image template shape Isowarp and Conwarp [Pizarro et al, BMVC’13] Differential constraints on 𝜃 so that 𝜃 = Π ∘ 𝜒 Gynecologic Surgery Preoperative Imaging Scope Organs of the female Located in the Procedures reproductive system pelvic cavity • Benign conditions • Cancer • Infertility • Incontinence MRI US Types • Diagnosis Abdominal Hysteroscopic Laparoscopic • Procedure (open, laparotomy) (through vagina) (small incisions) • Type of surgery Uterine Fibroids or Myomas Laparoscopic Augmented Reality Benign tumors from the myometrium Augmentation data • Microscopic to extremely large size 3 Displaying • Often several of them 2 Augmenting Registration Intramural myomas (type b) 1 Filming May be invisible in laparoscopy (and hysteroscopy) Clearly visible in MRI 6