Variational methods for photometric 3D-reconstruction Yvain Q UAU - PowerPoint PPT Presentation

Variational methods for photometric 3D-reconstruction Yvain Q UÉAU CNRS, GREYC laboratory, University of Caen, France Institut Henri Poincaré, Paris October 3rd, 2019 Yvain Q UÉAU Variational methods for photometric 3D-reconstruction 1 / 38

Outline Shape-from-Shading 1 Variational Solving of Shape-from-Shading 2 Photometric Depth Super-Resolution for RGBD Sensors 3 Combinging Variational Methods with Deep Learning 4 Uncalibrated Photometric Stereo 5 Yvain Q UÉAU Variational methods for photometric 3D-reconstruction 2 / 38

Outline Shape-from-Shading 1 Variational Solving of Shape-from-Shading 2 Photometric Depth Super-Resolution for RGBD Sensors 3 Combinging Variational Methods with Deep Learning 4 Uncalibrated Photometric Stereo 5 Yvain Q UÉAU Variational methods for photometric 3D-reconstruction 3 / 38

Shape-from-shading: A Classic Ill-posed Problem Given an image I : Ω ⊂ R 2 → R m , Shape-from-Shading (SfS) consists in inverting the forward photometric model ( image irradiance equation ) (1) I = R ( z , ρ , ℓ ) with R a radiance function depending on the unknown depth z : Ω → R , surface reflectance ρ : Ω → R m , and incident lighting ℓ : Ω → S 2 . Painter’s Sculptor’s Gaffer’s RGB image: explanation: explanation: explanation: I ρ z ℓ Yvain Q UÉAU Variational methods for photometric 3D-reconstruction 4 / 38 Impossible to tell reflectance from shape and lighting

Illustration of SfS’s Ill-posedness Even with known surface reflectance ρ and incident lighting ℓ , shape estimation by SfS is an ill-posed inverse problem (Horn, 1970). Example: two solutions of I = R ( z , ρ , ℓ ) with I := Lena, white reflectance ( ρ ≡ 1) and frontal lighting ( ℓ ≡ [ 0 , 0 , − 1 ] ⊤ ): Yvain Q UÉAU Variational methods for photometric 3D-reconstruction 5 / 38

Illustration of SfS’s Ill-posedness Even with known surface reflectance ρ and incident lighting ℓ , shape estimation by SfS is an ill-posed inverse problem (Horn, 1970). Example: two solutions of I = R ( z , ρ , ℓ ) with I := Lena, white reflectance ( ρ ≡ 1) and frontal lighting ( ℓ ≡ [ 0 , 0 , − 1 ] ⊤ ): Yvain Q UÉAU Variational methods for photometric 3D-reconstruction 5 / 38

Illustration of SfS’s Ill-posedness Even with known surface reflectance ρ and incident lighting ℓ , shape estimation by SfS is an ill-posed inverse problem (Horn, 1970). Example: two solutions of I = R ( z , ρ , ℓ ) with I := Lena, white reflectance ( ρ ≡ 1) and frontal lighting ( ℓ ≡ [ 0 , 0 , − 1 ] ⊤ ): Yvain Q UÉAU Variational methods for photometric 3D-reconstruction 5 / 38

Illustration of SfS’s Ill-posedness Even with known surface reflectance ρ and incident lighting ℓ , shape estimation by SfS is an ill-posed inverse problem (Horn, 1970). Example: two solutions of I = R ( z , ρ , ℓ ) with I := Lena, white reflectance ( ρ ≡ 1) and frontal lighting ( ℓ ≡ [ 0 , 0 , − 1 ] ⊤ ): Yvain Q UÉAU Variational methods for photometric 3D-reconstruction 5 / 38

Illustration of SfS’s Ill-posedness Even with known surface reflectance ρ and incident lighting ℓ , shape estimation by SfS is an ill-posed inverse problem (Horn, 1970). Example: two solutions of I = R ( z , ρ , ℓ ) with I := Lena, white reflectance ( ρ ≡ 1) and frontal lighting ( ℓ ≡ [ 0 , 0 , − 1 ] ⊤ ): Yvain Q UÉAU Variational methods for photometric 3D-reconstruction 5 / 38

Illustration of SfS’s Ill-posedness Even with known surface reflectance ρ and incident lighting ℓ , shape estimation by SfS is an ill-posed inverse problem (Horn, 1970). Maximal viscosity solution Variational solution [Cristiani and Falcone 2007] [Quéau et al. 2017] Yvain Q UÉAU Variational methods for photometric 3D-reconstruction 5 / 38

Parameterization the irradiance equation I = R ( z , ρ , ℓ ) Basic Lambertian model: = ⊙ RGB image Albedo Shading I : Ω → R 3 ρ : Ω → R 3 S ( z , ℓ ) : Ω → R where albedo (Lambertian reflectance) ≡ color, and shading ≡ lighting-geometry interaction. Yvain Q UÉAU Variational methods for photometric 3D-reconstruction 6 / 38

Parameterization the irradiance equation I = R ( z , ρ , ℓ ) Shading ≡ lighting-geometry interaction: = ⊙ · RGB image Albedo Lighting Normals I : Ω → R 3 ρ : Ω → R 3 ℓ ∈ S 2 n ( z ) : Ω → S 2 where the surface normal n relates to the depth map z in a nonlinear way: 1 � � f ∇ z n ( z ) = − z − < p , ∇ z > � | f ∇ z | 2 + ( − z − < p , ∇ z > ) 2 ( f > 0: length, and p : Ω → R 2 : centered pixel coordinates). Yvain Q UÉAU Variational methods for photometric 3D-reconstruction 6 / 38

Parameterization the irradiance equation I = R ( z , ρ , ℓ ) Extension to first-order spherical harmonics lighting ℓ ∈ R 4 : = ⊙ · Geometry RGB image Albedo Lighting � n � I : Ω → R 3 ρ : Ω → R 3 ℓ ∈ R 4 ( z ) : Ω → R 4 1 � n ( z ) � I = R ( z , ρ , ℓ ) := ρ � ℓ , � 1 where the surface normal n relates to the depth map z in a nonlinear way: 1 � � f ∇ z n ( z ) = − z − < p , ∇ z > � | f ∇ z | 2 + ( − z − < p , ∇ z > ) 2 ( f > 0: length, and p : Ω → R 2 : centered pixel coordinates). Yvain Q UÉAU Variational methods for photometric 3D-reconstruction 6 / 38

Outline Shape-from-Shading 1 Variational Solving of Shape-from-Shading 2 Photometric Depth Super-Resolution for RGBD Sensors 3 Combinging Variational Methods with Deep Learning 4 Uncalibrated Photometric Stereo 5 Yvain Q UÉAU Variational methods for photometric 3D-reconstruction 7 / 38

Variational Solving of SfS Equation I = R ( z , ρ , ℓ ) Assume (for now) that ρ and ℓ are known. Problem reduces to a nonlinear PDE I := R ( ∇ z ) . Horn and Brooks 1986: Regularization Set ( p , q ) := ∇ z over Ω ⊂ R 2 1) Estimate gradient components satisfying integrability: Ω ( I − R ( p , q )) 2 + λ ( ∂ y p − ∂ x q ) 2 ❞ x ❞ y �� min p , q Ω � ( p , q ) − ∇ z � 2 ❞ x ❞ y 2) Integrate: min z �� Quéau et al. 2017 (EMMCVPR): Hard constraint ( p , q ) is conservative by construction → Integrated estimation of gradient and depth: �� ( I − R ( p , q )) 2 ❞ x ❞ y min p , q , z s.t. ( p , q ) = ∇ z Yvain Q UÉAU Variational methods for photometric 3D-reconstruction 8 / 38

Regularized SfS Model Minimal surface regularization over Ω (Incomplete) depth prior over Ω ′ ⊂ Ω �� � λ ( I − R ( p , q )) 2 + ν 1 + p 2 + q 2 ❞ x ❞ y min p , q , z Ω z − z 0 � 2 �� � + Ω ′ µ ❞ x ❞ y s.t. ( p , q ) = ∇ z By tuning λ , µ and ν , we may achieve SfS, depth denoising and inpainting, or shading-based depth refinement. Yvain Q UÉAU Variational methods for photometric 3D-reconstruction 9 / 38

Solving the Regularized SfS Model using ADMM ( p ( k + 1 ) , q ( k + 1 ) ) solution of the local , nonlinear least-squares problem: use parallel BFGS iterations � � � ( p , q ) λ � I − R ( p , q ) � 2 1 + p 2 + q 2 � � min ℓ 2 (Ω) + ν � � � � ℓ 1 (Ω) + 1 2 � � ( p , q ) − ∇ z ( k ) + θ ( k ) � � � 2 β ℓ 2 (Ω) � z ( k + 1 ) solution of the global, linear least-squares problem: use preconditioned conjugate gradient iterations 2 ℓ 2 (Ω ′ ) + 1 2 � � z − z 0 � � � ( p ( k + 1 ) , q ( k + 1 ) ) − ∇ z + θ ( k ) � min z µ � � � � 2 β � � ℓ 2 (Ω) Auxiliary variable update θ ( k + 1 ) = θ ( k ) + ( p ( k + 1 ) , q ( k + 1 ) ) − ∇ z ( k + 1 ) Yvain Q UÉAU Variational methods for photometric 3D-reconstruction 10 / 38

Application 1: Depth Refinement for MVS Techniques Input depth map z 2 Shading-based refinement Input images I 1 and I 2 Estimated lighting Yvain Q UÉAU Variational methods for photometric 3D-reconstruction 11 / 38

Application 2: SfS under Natural Illumination Input RGB Calibrated SfS 3D-reconstruction image lighting Yvain Q UÉAU Variational methods for photometric 3D-reconstruction 12 / 38

Application 3: Depth Refinement for RGB-D Sensors Denoised (minimal surface) Input depth map Input RGB Estimated image lighting Shading-based refinement Yvain Q UÉAU Variational methods for photometric 3D-reconstruction 13 / 38

Outline Shape-from-Shading 1 Variational Solving of Shape-from-Shading 2 Photometric Depth Super-Resolution for RGBD Sensors 3 Combinging Variational Methods with Deep Learning 4 Uncalibrated Photometric Stereo 5 Yvain Q UÉAU Variational methods for photometric 3D-reconstruction 14 / 38

Problem with RGB-D Sensors Depth Shape RGB image image Depth image has RGB image has noise and quantization, less noise and quantization, missing areas, no missing area, coarse resolution . high resolution . Goal: Combine data to get high-resolution shape Yvain Q UÉAU Variational methods for photometric 3D-reconstruction 15 / 38

Problem with RGB-D Sensors Depth High-resolution Shape RGB image image shape Depth image has RGB image has noise and quantization, less noise and quantization, missing areas, no missing area, coarse resolution . high resolution . Goal: Combine data to get high-resolution shape Yvain Q UÉAU Variational methods for photometric 3D-reconstruction 15 / 38