Solving the Uncalibrated Photometric Stereo problem using Total - - PowerPoint PPT Presentation

Solving the Uncalibrated Photometric Stereo problem using Total Variation

Yvain Qu´ eau1 Fran¸ cois Lauze2 Jean-Denis Durou1 IRIT1 DIKU2 Toulouse, France Copenhagen, Denmark

Yvain Qu´ eau TV - Uncalibrated Photometric Stereo 1 / 26

Table of contents

1

Photometric Stereo

2

Uncalibrated Photometric Stereo

3

Solving the GBR ambiguity using Total Variation

4

Overview

Yvain Qu´ eau TV - Uncalibrated Photometric Stereo 2 / 26

Photometric Stereo

Table of contents

1

Photometric Stereo

2

Uncalibrated Photometric Stereo

3

Solving the GBR ambiguity using Total Variation

4

Overview

Yvain Qu´ eau TV - Uncalibrated Photometric Stereo 3 / 26

Photometric Stereo

Photometric Stereo setup

Fixed camera n different illuminations ⇒ n images

Yvain Qu´ eau TV - Uncalibrated Photometric Stereo 4 / 26

Photometric Stereo

Photometric Stereo setup

Fixed camera n different illuminations ⇒ n images

Yvain Qu´ eau TV - Uncalibrated Photometric Stereo 4 / 26

Photometric Stereo

Photometric Stereo setup

Fixed camera n different illuminations ⇒ n images

Yvain Qu´ eau TV - Uncalibrated Photometric Stereo 4 / 26

Photometric Stereo

Photometric Stereo setup

Fixed camera n different illuminations ⇒ n images

Yvain Qu´ eau TV - Uncalibrated Photometric Stereo 4 / 26

Photometric Stereo

Why use Photometric Stereo?

Advantages No matching between images “Dense” reconstruction No geometric calibration needed Cheap materials for industrial applications Drawback 2.5D reconstruction

Yvain Qu´ eau TV - Uncalibrated Photometric Stereo 5 / 26

Photometric Stereo

Photometric Stereo framework

I 1 I 2 I 3

Yvain Qu´ eau TV - Uncalibrated Photometric Stereo 6 / 26

Photometric Stereo

Photometric Stereo framework

⇒ (1) I 1 I 2 I 3 N ρ (1) Estimate normal field N and albedo ρ

Yvain Qu´ eau TV - Uncalibrated Photometric Stereo 6 / 26

Photometric Stereo

Photometric Stereo framework

⇒ (1) I 1 I 2 I 3 N ρ ⇓ (2) (1) Estimate normal field N and albedo ρ (2) Integrate N into a surface

Yvain Qu´ eau TV - Uncalibrated Photometric Stereo 6 / 26

Photometric Stereo

Photometric Stereo framework

⇒ (1) I 1 I 2 I 3 N ρ ⇓ (2) ⇓ (3) (1) Estimate normal field N and albedo ρ (2) Integrate N into a surface (3) Possibly, warp ρ onto the surface ⇒ (3)

Yvain Qu´ eau TV - Uncalibrated Photometric Stereo 6 / 26

Photometric Stereo

Assumptions

Lighting Distant point-light source: parallel lighting Reflectance Lambertian (diffuse) reflectance No highlight, no shadow, no inter-reflection

Yvain Qu´ eau TV - Uncalibrated Photometric Stereo 7 / 26

Photometric Stereo

Image formation model

Np = [Nx, Ny, Nz]T p S1 = [S1

x , S1 y , S1 z ]

S2 = [S2

x , S2 y , S2 z ]

S3 = [S3

x , S3 y , S3 z ]

Intensity in p (Lambert’s law): I 1

p = ρpS1, Np . . . I n p = ρpSn, Np

Matrix formulation Ip = [I 1

p . . . I n p ]T, Mp = ρpNp

I = [I1 . . . I|Ω|], M = [M1 . . . M|Ω|] S = [S1T . . . SnT]T I = SM Dimensions I ∈ Rn×|Ω|, S ∈ Rn×3, M ∈ R3×|Ω|

Yvain Qu´ eau TV - Uncalibrated Photometric Stereo 8 / 26

Photometric Stereo

Calibrated Photometric Stereo

Solution (Woodham 1980) Lighting matrix S is known The system I = SM is well-constrained if n 3 A least-square solution in M is given by the Moore-Penrose pseudo-inverse: M = S+I Notes “Dense” reconstruction: the normal is estimated in every pixel Size of S is “small” ⇒ Very fast estimation Estimation of S may be difficult to set up...

Yvain Qu´ eau TV - Uncalibrated Photometric Stereo 9 / 26

Uncalibrated Photometric Stereo

Table of contents

1

Photometric Stereo

2

Uncalibrated Photometric Stereo

3

Solving the GBR ambiguity using Total Variation

4

Overview

Yvain Qu´ eau TV - Uncalibrated Photometric Stereo 10 / 26

Uncalibrated Photometric Stereo

SVD factorization

Goal Estimate S ∈ Rn×3 and M ∈ R3×|Ω| such that: I = SM Method [Hayakawa 94] Apply SVD to I: I = UΣ3×3V T ⇒ S = U(Σ3×3)1/2, M = (Σ3×3)1/2V T Family of solutions Solution is not unique: I = SM = SAA−1M, A ∈ GL(3) M and S are recovered up to a 3 × 3 invertible linear transformation How to estimate the “best” A?

Yvain Qu´ eau TV - Uncalibrated Photometric Stereo 11 / 26

Uncalibrated Photometric Stereo

Integrability constraint

Integrability Integration step assumes the normals are “integrable”, i.e. curl(N) = 0 It restrains A to a GBR [Yuille and Snow 97] Disambiguation Imposing integrability reduces the ambiguity: I = SM = SGG −1M, with: G =   1 1 µ ν λ   G is called a “Generalized Bas-Relief” (GBR) transformation [Belhumeur et al. 97] [Belhumeur et al. 97] ⇒ Estimating G disambiguates the Uncalibrated PS (UPS) problem

Yvain Qu´ eau TV - Uncalibrated Photometric Stereo 12 / 26

Uncalibrated Photometric Stereo

Generalized Bas-Relief transformation

I = SGG −1M G =   1 1 µ ν λ   z = f (x, y) ⇒ z = λf (x, y)+µx +νy

Yvain Qu´ eau TV - Uncalibrated Photometric Stereo 13 / 26

Uncalibrated Photometric Stereo

Disambiguation methods

Additional constraints to estimate µ, ν and λ: Uniform albedo [Hayakawa 94] Constant light magnitude [Yuille and Snow 97] Highlights [Georghiades 03] Inter-reflections [Chandraker et al. 05] Minimum Entropy of albedo [Alldrin et al. 07] Maxima of intensity indicate parallel normals and lightings [Favaro and Papadhimitri 12] Perspective camera [Papadhimitri and Favaro 13] ... Detection of outliers might be difficult or impossible Entropy is not easy to minimize Why not take into account both albedo and normals?

Yvain Qu´ eau TV - Uncalibrated Photometric Stereo 14 / 26

Solving the GBR ambiguity using Total Variation

Table of contents

1

Photometric Stereo

2

Uncalibrated Photometric Stereo

3

Solving the GBR ambiguity using Total Variation

4

Overview

Yvain Qu´ eau TV - Uncalibrated Photometric Stereo 15 / 26

Solving the GBR ambiguity using Total Variation

Idea

Constraints to solve for GBR ambiguity: Objects are made

• f a small set of

colors [Alldrin et

• al. 2007]

Objects are usually compact GBR spreads both albedo and normal distributions 1D-cuts along a line: ⇒ Minimize the variations of N and ρ solves the GBR ⇒ Minimize the variations of M solves the GBR

Yvain Qu´ eau TV - Uncalibrated Photometric Stereo 16 / 26

Solving the GBR ambiguity using Total Variation

TV-regularized model

E(S, M|I) = D(S, M|I) + αTV (M) D: Data term TV (M): Regularization α: Hyper-parameter Data term Reprojection error: D(S, M|I) =

n

• i=1
• p∈Ω

(I i

p − R(Si, Mp))2

Reflectance R for Lambertian objects: R(Si, Mp) = Si, Mp Regularization M : R2 → R3 ⇒ TV (M) =

• Ω Jac(M(x, y))Fdxdy

Yvain Qu´ eau TV - Uncalibrated Photometric Stereo 17 / 26

Solving the GBR ambiguity using Total Variation

About the TV-regularized problem

Why use total variation? Edge-preserving properties Efficient algorithms applied to images ⇒ Adapt such algorithms for the UPS problem? Minimizing E(S, M|I) = D(S, M|I) + αTV (M) How to choose α? Resolution: iterative methods Lambertian case Closed-form minimizer of the data term, up to a GBR transformation Closed-form minimizer of

• Ω Jac(M(x, y))2

Fdxdy

A few Gauss-Newton steps away from TV (M) =

• Ω Jac(M(x, y))Fdxdy...

Yvain Qu´ eau TV - Uncalibrated Photometric Stereo 18 / 26

Solving the GBR ambiguity using Total Variation

Our model

Formal definition                ( S, M) = argmin

(S,M)

• p∈Ω Ip − SMp2

s.t. curl(M) = 0 ( µ, ν, λ) = argmin

(µ,ν,λ)

TV (G(µ, ν, λ)−1 M) Resolution

1 Solve the UPS problem

using SVD and integrability

2 Solve for the GBR

ambiguity using Total Variation ⇒

Yvain Qu´ eau TV - Uncalibrated Photometric Stereo 19 / 26

Solving the GBR ambiguity using Total Variation

Algorithm overview

1 Solve the Lambertian UPS problem using SVD and integrability

(i.e. estimate S and M)

2 Search for the GBR transformation G(µ, ν, λ) which minimizes

• Ω Jac(G(µ, ν, λ)−1

M(x, y)2

Fdxdy 3 Use this minimum as initialization for minimizing TV (G(µ, ν, λ)−1

M)

4 Finally recover S =

S G and M = G −1 M Image (among 12) UPS result (RMSE = 21.40) L2 result (RMSE = 8.21) TV result (RMSE = 5.26) RMSE: Mean angular error between UPS and CPS estimated normals

Yvain Qu´ eau TV - Uncalibrated Photometric Stereo 20 / 26

Solving the GBR ambiguity using Total Variation

Experimental Datasets

Face datasets: YaleDataFace B Lambertian objects: Washington University, Neil Alldrin’s homepage Validation Height ground truth is usually unavailable Light sources are calibrated: comparison between UPS and CPS

Yvain Qu´ eau TV - Uncalibrated Photometric Stereo 21 / 26

Solving the GBR ambiguity using Total Variation

Dataface 1

n = 22 192 × 168 RMSE = 12.1 CPU < 1s

Yvain Qu´ eau TV - Uncalibrated Photometric Stereo 22 / 26

Solving the GBR ambiguity using Total Variation

Korean doll

n = 5 1321 × 521 RMSE = 16.0 CPU = 5s

Yvain Qu´ eau TV - Uncalibrated Photometric Stereo 23 / 26

Solving the GBR ambiguity using Total Variation

Owl

n = 12 320 × 300 RMSE = 8.0 CPU = 1s

Yvain Qu´ eau TV - Uncalibrated Photometric Stereo 24 / 26

Overview

Table of contents

1

Photometric Stereo

2

Uncalibrated Photometric Stereo

3

Solving the GBR ambiguity using Total Variation

4

Overview

Yvain Qu´ eau TV - Uncalibrated Photometric Stereo 25 / 26

Overview

Overview

Contributions Novel constraint to solve the Uncalibrated PS problem

Both normals and albedo should have low total variation

New method to solve for the GBR ambiguity

By minimizing the total variation of M = ρN

Experimental validation Future work Non-Lambertian objects? Non-distant light source? This variational model can easily be adapted to more complicated reflectance models! [Chambolle and Pock 11]

Yvain Qu´ eau TV - Uncalibrated Photometric Stereo 26 / 26