Solving the Uncalibrated Photometric Stereo problem using Total - PowerPoint PPT Presentation

Solving the Uncalibrated Photometric Stereo problem using Total Variation eau 1 cois Lauze 2 Jean-Denis Durou 1 Yvain Qu´ Fran¸ IRIT 1 DIKU 2 Toulouse, France Copenhagen, Denmark Yvain Qu´ eau TV - Uncalibrated Photometric Stereo 1 / 26

Table of contents 1 Photometric Stereo Uncalibrated Photometric Stereo 2 Solving the GBR ambiguity using Total Variation 3 Overview 4 Yvain Qu´ eau TV - Uncalibrated Photometric Stereo 2 / 26

Photometric Stereo Table of contents 1 Photometric Stereo Uncalibrated Photometric Stereo 2 Solving the GBR ambiguity using Total Variation 3 Overview 4 Yvain Qu´ eau TV - Uncalibrated Photometric Stereo 3 / 26

Photometric Stereo Photometric Stereo setup Fixed camera n different illuminations ⇒ n images Yvain Qu´ eau TV - Uncalibrated Photometric Stereo 4 / 26

Photometric Stereo Photometric Stereo setup Fixed camera n different illuminations ⇒ n images Yvain Qu´ eau TV - Uncalibrated Photometric Stereo 4 / 26

Photometric Stereo Photometric Stereo setup Fixed camera n different illuminations ⇒ n images Yvain Qu´ eau TV - Uncalibrated Photometric Stereo 4 / 26

Photometric Stereo Photometric Stereo setup Fixed camera n different illuminations ⇒ n images Yvain Qu´ eau TV - Uncalibrated Photometric Stereo 4 / 26

Photometric Stereo Why use Photometric Stereo? Advantages No matching between images “Dense” reconstruction No geometric calibration needed Cheap materials for industrial applications Drawback 2.5D reconstruction Yvain Qu´ eau TV - Uncalibrated Photometric Stereo 5 / 26

Photometric Stereo Photometric Stereo framework I 1 I 2 I 3 Yvain Qu´ eau TV - Uncalibrated Photometric Stereo 6 / 26

Photometric Stereo Photometric Stereo framework ⇒ (1) I 1 I 2 I 3 N ρ (1) Estimate normal field N and albedo ρ Yvain Qu´ eau TV - Uncalibrated Photometric Stereo 6 / 26

Photometric Stereo Photometric Stereo framework ⇒ (1) I 1 I 2 I 3 N ρ ⇓ (2) (1) Estimate normal field N and albedo ρ (2) Integrate N into a surface Yvain Qu´ eau TV - Uncalibrated Photometric Stereo 6 / 26

Photometric Stereo Photometric Stereo framework ⇒ (1) I 1 I 2 I 3 N ρ ⇓ (2) ⇓ (3) (1) Estimate normal field N and albedo ρ ⇒ (2) Integrate N into a surface (3) (3) Possibly, warp ρ onto the surface Yvain Qu´ eau TV - Uncalibrated Photometric Stereo 6 / 26

Photometric Stereo Assumptions Lighting Distant point-light source: parallel lighting Reflectance Lambertian (diffuse) reflectance No highlight, no shadow, no inter-reflection Yvain Qu´ eau TV - Uncalibrated Photometric Stereo 7 / 26

Photometric Stereo Image formation model Intensity in p (Lambert’s law): I 1 p = ρ p � S 1 , N p � . . . I n p = ρ p � S n , N p � S 1 = [ S 1 x , S 1 y , S 1 z ] Matrix formulation I p = [ I 1 p . . . I n p ] T , M p = ρ p N p S 2 = [ S 2 x , S 2 y , S 2 z ] p I = [ I 1 . . . I | Ω | ], M = [ M 1 . . . M | Ω | ] S = [ S 1 T . . . S nT ] T N p = [ N x , N y , N z ] T I = SM S 3 = [ S 3 x , S 3 y , S 3 z ] Dimensions I ∈ R n ×| Ω | , S ∈ R n × 3 , M ∈ R 3 ×| Ω | Yvain Qu´ eau TV - Uncalibrated Photometric Stereo 8 / 26

Photometric Stereo Calibrated Photometric Stereo Solution (Woodham 1980) Lighting matrix S is known The system I = SM is well-constrained if n � 3 A least-square solution in M is given by the Moore-Penrose pseudo-inverse: M = S + I Notes “Dense” reconstruction: the normal is estimated in every pixel Size of S is “small” ⇒ Very fast estimation Estimation of S may be difficult to set up... Yvain Qu´ eau TV - Uncalibrated Photometric Stereo 9 / 26

Uncalibrated Photometric Stereo Table of contents 1 Photometric Stereo Uncalibrated Photometric Stereo 2 Solving the GBR ambiguity using Total Variation 3 Overview 4 Yvain Qu´ eau TV - Uncalibrated Photometric Stereo 10 / 26

Uncalibrated Photometric Stereo SVD factorization Goal Estimate S ∈ R n × 3 and M ∈ R 3 ×| Ω | such that: I = SM Method [Hayakawa 94] Apply SVD to I : I = U Σ 3 × 3 V T ⇒ � S = U (Σ 3 × 3 ) 1 / 2 , � M = (Σ 3 × 3 ) 1 / 2 V T Family of solutions Solution is not unique: I = SM = SAA − 1 M , A ∈ GL (3) M and S are recovered up to a 3 × 3 invertible linear transformation How to estimate the “best” A ? Yvain Qu´ eau TV - Uncalibrated Photometric Stereo 11 / 26

Uncalibrated Photometric Stereo Integrability constraint Integrability Integration step assumes the normals are “integrable”, i.e. curl( N ) = 0 It restrains A to a GBR [Yuille and Snow 97] Disambiguation Imposing integrability reduces the ambiguity: I = SM = SGG − 1 M , with:   1 0 0   G = 0 1 0 µ ν λ G is called a “Generalized Bas-Relief” (GBR) [Belhumeur et al. 97] transformation [Belhumeur et al. 97] ⇒ Estimating G disambiguates the Uncalibrated PS (UPS) problem Yvain Qu´ eau TV - Uncalibrated Photometric Stereo 12 / 26

Uncalibrated Photometric Stereo Generalized Bas-Relief transformation I = SGG − 1 M   1 0 0   0 1 0 G = µ ν λ z = f ( x , y ) ⇒ z = λ f ( x , y )+ µ x + ν y Yvain Qu´ eau TV - Uncalibrated Photometric Stereo 13 / 26

Uncalibrated Photometric Stereo Disambiguation methods Additional constraints to estimate µ , ν and λ : Uniform albedo [Hayakawa 94] Constant light magnitude [Yuille and Snow 97] Highlights [Georghiades 03] Inter-reflections [Chandraker et al. 05] Minimum Entropy of albedo [Alldrin et al. 07] Maxima of intensity indicate parallel normals and lightings [Favaro and Papadhimitri 12] Perspective camera [Papadhimitri and Favaro 13] ... Detection of outliers might be difficult or impossible Entropy is not easy to minimize Why not take into account both albedo and normals? Yvain Qu´ eau TV - Uncalibrated Photometric Stereo 14 / 26

Solving the GBR ambiguity using Total Variation Table of contents 1 Photometric Stereo Uncalibrated Photometric Stereo 2 Solving the GBR ambiguity using Total Variation 3 Overview 4 Yvain Qu´ eau TV - Uncalibrated Photometric Stereo 15 / 26

Solving the GBR ambiguity using Total Variation Idea Constraints to solve for GBR ambiguity: Objects are made of a small set of colors [Alldrin et al. 2007] Objects are usually compact 1D-cuts GBR spreads both along a albedo and normal line: distributions ⇒ Minimize the variations of N and ρ solves the GBR ⇒ Minimize the variations of M solves the GBR Yvain Qu´ eau TV - Uncalibrated Photometric Stereo 16 / 26

Solving the GBR ambiguity using Total Variation TV-regularized model E ( S , M | I ) = D ( S , M | I ) + α TV ( M ) D : Data term TV ( M ): Regularization α : Hyper-parameter Data term � � n ( I i p − R ( S i , M p )) 2 Reprojection error: D ( S , M | I ) = i =1 p ∈ Ω Reflectance R for Lambertian objects: R ( S i , M p ) = � S i , M p � Regularization �� M : R 2 → R 3 ⇒ TV ( M ) = Ω � Jac ( M ( x , y )) � F dxdy Yvain Qu´ eau TV - Uncalibrated Photometric Stereo 17 / 26

Solving the GBR ambiguity using Total Variation About the TV-regularized problem Why use total variation? Edge-preserving properties Efficient algorithms applied to images ⇒ Adapt such algorithms for the UPS problem? Minimizing E ( S , M | I ) = D ( S , M | I ) + α TV ( M ) How to choose α ? Resolution: iterative methods Lambertian case Closed-form minimizer of the data term, up to a GBR transformation �� Ω � Jac ( M ( x , y )) � 2 Closed-form minimizer of F dxdy A few Gauss-Newton steps away from �� TV ( M ) = Ω � Jac ( M ( x , y )) � F dxdy ... Yvain Qu´ eau TV - Uncalibrated Photometric Stereo 18 / 26

Solving the GBR ambiguity using Total Variation Our model Resolution Formal definition  1 Solve the UPS problem � ( � S , � p ∈ Ω � I p − SM p � 2  M ) = argmin  using SVD and   ( S , M )   integrability  s.t. curl( M ) = 0    2 Solve for the GBR   TV ( G ( µ, ν, λ ) − 1 � ν, �  ( � µ, � λ ) = argmin M )  ambiguity using Total ( µ,ν,λ ) Variation ⇒ Yvain Qu´ eau TV - Uncalibrated Photometric Stereo 19 / 26

Solving the GBR ambiguity using Total Variation Algorithm overview 1 Solve the Lambertian UPS problem using SVD and integrability (i.e. estimate � S and � M ) 2 Search for the GBR transformation G ( µ, ν, λ ) which minimizes �� Ω � Jac ( G ( µ, ν, λ ) − 1 � M ( x , y ) � 2 F dxdy 3 Use this minimum as initialization for minimizing TV ( G ( µ, ν, λ ) − 1 � M ) G − 1 � 4 Finally recover S = � S � G and M = � M Image (among 12) UPS result L2 result TV result (RMSE = 21 . 40) (RMSE = 8 . 21) (RMSE = 5 . 26) RMSE: Mean angular error between UPS and CPS estimated normals Yvain Qu´ eau TV - Uncalibrated Photometric Stereo 20 / 26

Solving the GBR ambiguity using Total Variation Experimental Datasets Face datasets: YaleDataFace B Lambertian objects: Washington University, Neil Alldrin’s homepage Validation Height ground truth is usually unavailable Light sources are calibrated: comparison between UPS and CPS Yvain Qu´ eau TV - Uncalibrated Photometric Stereo 21 / 26

Solving the GBR ambiguity using Total Variation Dataface 1 n = 22 192 × 168 RMSE = 12 . 1 CPU < 1 s Yvain Qu´ eau TV - Uncalibrated Photometric Stereo 22 / 26