Photometric 3D-reconstruction Jean-Denis D UROU , Yvain Q UAU , and - - PowerPoint PPT Presentation

Photometric 3D-reconstruction

Jean-Denis DUROU, Yvain QUÉAU, and Jean MÉLOU IRIT, Toulouse, France Computational Methods for Inverse Problems in Imaging Como, July 17, 2018

Jean-Denis DUROU (IRIT) Photometric 3D-reconstruction Como, July 17, 2018 1 / 35

Outline

1

Introduction

2

Shape-from-shading (SfS)

3

Photometric stereo

4

Multi-view shape-from-shading

5

Conclusion and perspectives

Jean-Denis DUROU (IRIT) Photometric 3D-reconstruction Como, July 17, 2018 2 / 35

Introduction

Outline

1

Introduction

2

Shape-from-shading (SfS)

3

Photometric stereo

4

Multi-view shape-from-shading

5

Conclusion and perspectives

Jean-Denis DUROU (IRIT) Photometric 3D-reconstruction Como, July 17, 2018 3 / 35

Introduction

3D-scanning ≡ 3D-reconstruction + Color estimation

Some applications of 3D-scanning Quality control Architecture Cultural heritage Augmented reality Metrology Different kinds of 3D-reconstruction techniques Palpation ≡ Mechanical process Kinect V1 ≡ Projection of an infra-red pattern Kinect V2 ≡ Time of flight of laser pulses Photographic techniques ≡ Shape-from-X

Jean-Denis DUROU (IRIT) Photometric 3D-reconstruction Como, July 17, 2018 4 / 35

Introduction

3D-scanning ≡ 3D-reconstruction + Color estimation

Some applications of 3D-scanning Quality control Architecture Cultural heritage Augmented reality Metrology Different kinds of 3D-reconstruction techniques Palpation ≡ Mechanical process Kinect V1 ≡ Projection of an infra-red pattern Kinect V2 ≡ Time of flight of laser pulses Photographic techniques ≡ Shape-from-X

Jean-Denis DUROU (IRIT) Photometric 3D-reconstruction Como, July 17, 2018 4 / 35

Introduction

Main shape-from-X techniques

Geometric techniques Photometric techniques N = 1 view Structured light Shape-from-shading (SfS) N = 1 image Shape-from-texture Shape-from-contour Shape-from-shadow N = 1 view Shape-from-defocus Photometric stereo N > 1 images N > 1 views Stereoscopy Multi-view SfS N > 1 images Shape-from-silhouettes Structure-from-motion Multi-view stereo

Jean-Denis DUROU (IRIT) Photometric 3D-reconstruction Como, July 17, 2018 5 / 35

Introduction

Main shape-from-X techniques

Geometric techniques Photometric techniques N = 1 view Structured light Shape-from-shading (SfS) N = 1 image Shape-from-texture Shape-from-contour Shape-from-shadow N = 1 view Shape-from-defocus Photometric stereo N > 1 images N > 1 views Stereoscopy Multi-view SfS N > 1 images Shape-from-silhouettes Structure-from-motion Multi-view stereo

Jean-Denis DUROU (IRIT) Photometric 3D-reconstruction Como, July 17, 2018 5 / 35

Introduction

Geometric techniques vs. Photometric techniques

Geometric techniques Goal: match feature points in several images Tools: linear algebra Pros: simple tools; robust Cons: sparse 3D-reconstruction; no color estimation Photometric techniques Goal: explain the color of each pixel in each image Tools: variational methods Pros: dense 3D-reconstruction; color can be estimated Cons: nonlinear models; heavy computations

Jean-Denis DUROU (IRIT) Photometric 3D-reconstruction Como, July 17, 2018 6 / 35

Shape-from-shading (SfS)

Outline

1

Introduction

2

Shape-from-shading (SfS)

3

Photometric stereo

4

Multi-view shape-from-shading

5

Conclusion and perspectives

Jean-Denis DUROU (IRIT) Photometric 3D-reconstruction Como, July 17, 2018 7 / 35

Shape-from-shading (SfS)

SfS: An intuitive technique to infer 3D-shape

This image suffices to infer that the wall is not perfectly flat

Jean-Denis DUROU (IRIT) Photometric 3D-reconstruction Como, July 17, 2018 8 / 35

Shape-from-shading (SfS)

SfS for a Lambertian surface (1/2)

= × Image Albedo Shading I

• Image

= ρ

• Albedo

× s · n

• Shading

s: lighting vector n: normal to the surface (3D-shape) s n ⇓ Shape-from-shading 3D-shape

Jean-Denis DUROU (IRIT) Photometric 3D-reconstruction Como, July 17, 2018 9 / 35

Shape-from-shading (SfS)

SfS for a Lambertian surface (2/2)

Classical assumptions Albedo ρ ≡ 1 Lighting s = [0, 0, 1]⊤ Data: graylevel I Unknown: normal n such that n = 1 I = s · n = cos θ θ s The model specifies only one out of the two degrees of freedom of n → Impossible to locally estimate the shape

Jean-Denis DUROU (IRIT) Photometric 3D-reconstruction Como, July 17, 2018 10 / 35

Shape-from-shading (SfS)

Non-differential approach vs. differential approach

Non-differential approach: unknown normal n min

n: Ω→R3

• Ω

[I(u, v) − s · n(u, v)]2 dudv + λ

• Ω

R(n(u, v))du dv Pros: boundary condition not required; easily discretized Cons: approximate solution; n integrated afterwards Differential approach: unknown depth z I(u, v) = s · 1

• ∇z(u, v)2 + 1

[−∇z(u, v)⊤, 1]⊤

• ≡ n(u,v) under orthographic projection

Pros: exact solution; direct estimation of the 3D-shape Cons: boundary condition required; solution not always realistic

Jean-Denis DUROU (IRIT) Photometric 3D-reconstruction Como, July 17, 2018 11 / 35

Shape-from-shading (SfS)

Shape-from-shading: Three major drawbacks

Knowledge of the albedo ρ ρ ≡ 1 → Arbitrary Difficult to get ρ Knowledge of the lighting vector s s = [0, 0, 1]⊤ → Arbitrary Possible to get s by calibration SfS is still ill-posed Non-differential approach: Regularization → Approximate solution Differential approach: Boundary condition → Arbitrary

Jean-Denis DUROU (IRIT) Photometric 3D-reconstruction Como, July 17, 2018 12 / 35

Photometric stereo

Outline

1

Introduction

2

Shape-from-shading (SfS)

3

Photometric stereo

4

Multi-view shape-from-shading

5

Conclusion and perspectives

Jean-Denis DUROU (IRIT) Photometric 3D-reconstruction Como, July 17, 2018 13 / 35

Photometric stereo

A well-posed extension of SfS

N > 1 lighting vectors s1, . . . sN − → Albedo ρ + Depth z

Jean-Denis DUROU (IRIT) Photometric 3D-reconstruction Como, July 17, 2018 14 / 35

Photometric stereo

Claude Monet: a precursor?

N ≈ 30 paintings of the cathedral of Rouen (1892-1894)

Jean-Denis DUROU (IRIT) Photometric 3D-reconstruction Como, July 17, 2018 15 / 35

Photometric stereo

A linear and local problem [Woodham, 1980]

I1 = ρ s1 · n = s1 · m I2 = ρ s2 · n = s2 · m I3 = ρ s3 · n = s3 · m In each pixel, denoting m = ρ n: Ii = si · m, i = 1 . . . N If lighting vectors si are non-coplanar: Unique exact solution if N = 3 Unique approximate solution if N > 3 Eventually: ρ = m and n =

m m

Jean-Denis DUROU (IRIT) Photometric 3D-reconstruction Como, July 17, 2018 16 / 35

Photometric stereo

Photometric stereo: Three major advantages vs. SfS

No assumption on the albedo ρ → Realistic ρ is now an additional unknown No other shape-from-X technique can estimate the albedo Well-posed problem → Accurate Requires that the N vectors si are known (by calibration) Neither regularization, nor boundary condition Linear and local problem → Quick resolution Problem to solve in each point: min

m N

• i=1
• Ii − si · m

2 Integrating the normal field n into a depth map z is global, yet

Jean-Denis DUROU (IRIT) Photometric 3D-reconstruction Como, July 17, 2018 17 / 35

Photometric stereo

First example: N = 3 images of a plaster bust

I1 = s1 · m I2 = s2 · m I3 = s3 · m Albedo ρ Normal field n Depth map z

Jean-Denis DUROU (IRIT) Photometric 3D-reconstruction Como, July 17, 2018 18 / 35

Photometric stereo

Second example: N = 2812 images of a ball of wire

3 images, out of N = 2812 [Wu et al., 2006] Least-squares normal estimation Robust normal estimation

Jean-Denis DUROU (IRIT) Photometric 3D-reconstruction Como, July 17, 2018 19 / 35

Photometric stereo

Historical landmarks

[Van Diggelen, 1951] First paper on photoclinometry (premise of shape-from-shading) [Horn, 1970] First explicit mention of shape-from-shading [Woodham, 1980] First paper on photometric stereo [Rouy, Tourin, Lions, 1988] First applied mathematics contribution to shape-from-shading (viscosity solutions) [Hayakawa, 1994] Renewed interest for photometric stereo (linked to the popularization of digital photography?) Since 2000: More papers on photometric stereo than on shape-from-shading Since 2010: First convincing applications of photometric stereo

Jean-Denis DUROU (IRIT) Photometric 3D-reconstruction Como, July 17, 2018 20 / 35

Photometric stereo

Historical landmarks

[Van Diggelen, 1951] First paper on photoclinometry (premise of shape-from-shading) [Horn, 1970] First explicit mention of shape-from-shading [Woodham, 1980] First paper on photometric stereo [Rouy, Tourin, Lions, 1988] First applied mathematics contribution to shape-from-shading (viscosity solutions) [Hayakawa, 1994] Renewed interest for photometric stereo (linked to the popularization of digital photography?) Since 2000: More papers on photometric stereo than on shape-from-shading Since 2010: First convincing applications of photometric stereo

Jean-Denis DUROU (IRIT) Photometric 3D-reconstruction Como, July 17, 2018 20 / 35

Photometric stereo

Application to 3D-scanning of faces (1/2)

Experimental setup in our lab Two images of a face, out of N = 8

Jean-Denis DUROU (IRIT) Photometric 3D-reconstruction Como, July 17, 2018 21 / 35

Photometric stereo

Application to 3D-scanning of faces (2/2)

(bastien.u3d) CPU time ≈ 1 minute

Jean-Denis DUROU (IRIT) Photometric 3D-reconstruction Como, July 17, 2018 22 / 35

Photometric stereo

Application to micrometric 3D-scanning (1/2)

Transformation of a dermoscope (skin camera) into a micrometric 3D-scanner

Jean-Denis DUROU (IRIT) Photometric 3D-reconstruction Como, July 17, 2018 23 / 35

Photometric stereo

Application to micrometric 3D-scanning (2/2)

1 euro (Italy) 50 cents (Spain) 1 yuan (China) 3D-reconstructions of metallic coins (accuracy ≈ 10 microns)

Jean-Denis DUROU (IRIT) Photometric 3D-reconstruction Como, July 17, 2018 24 / 35

Multi-view shape-from-shading

Outline

1

Introduction

2

Shape-from-shading (SfS)

3

Photometric stereo

4

Multi-view shape-from-shading

5

Conclusion and perspectives

Jean-Denis DUROU (IRIT) Photometric 3D-reconstruction Como, July 17, 2018 25 / 35

Multi-view shape-from-shading

Another well-posed extension of SfS

Geometric techniques Photometric techniques N = 1 view Structured light Shape-from-shading (SfS) N = 1 image Shape-from-texture Shape-from-contour Shape-from-shadow N = 1 view Shape-from-defocus Photometric stereo N > 1 images N > 1 views Stereoscopy Multi-view SfS N > 1 images Shape-from-silhouettes Structure-from-motion Multi-view stereo

Jean-Denis DUROU (IRIT) Photometric 3D-reconstruction Como, July 17, 2018 26 / 35

Multi-view shape-from-shading

Multi-view SfS vs. photometric stereo

Photometric stereo → Rather inside the lab Pro 1: No extrinsic calibration required (single camera pose) Pro 2: Estimation of the albedo → 3D-scanning Pro 3: No matching required Con 1: Restricted to depth maps → No full 3D-reconstruction Con 2: Need to vary the lighting → Difficult outside Multi-view SfS → Even outside the lab Con 1: Extrinsic calibration required (N > 1 camera poses) Con 2: No estimation of the albedo → No 3D-scanning Con 3: Matching required Pro 1: Not restricted to depth maps → Full 3D-reconstruction Pro 2: No need to vary the lighting → Possible outside

Jean-Denis DUROU (IRIT) Photometric 3D-reconstruction Como, July 17, 2018 27 / 35

Multi-view shape-from-shading

Jean Mélou’s PhD thesis (in progress)

3D-scanning for scene relighting applications (augmented reality) Collaboration with Mikros Image (Technicolor) Part 1: Lighting and albedo estimation from N > 1 views + coarse 3D-shape (obtained by depth-sensor or multi-view stereo) Part 2: Improvement of the coarse 3D-shape Part 1: Albedo estimation from N > 1 views + coarse 3D-shape

Jean-Denis DUROU (IRIT) Photometric 3D-reconstruction Como, July 17, 2018 28 / 35

Multi-view shape-from-shading

Jean Mélou’s PhD thesis (in progress)

3D-scanning for scene relighting applications (augmented reality) Collaboration with Mikros Image (Technicolor) Part 1: Lighting and albedo estimation from N > 1 views + coarse 3D-shape (obtained by depth-sensor or multi-view stereo) Part 2: Improvement of the coarse 3D-shape → Multi-view SfS Part 1: Albedo estimation from N > 1 views + coarse 3D-shape

Jean-Denis DUROU (IRIT) Photometric 3D-reconstruction Como, July 17, 2018 28 / 35

Multi-view shape-from-shading

Multi-view SfS: A practical implementation

Part 2: How to overcome the drawbacks of multi-view SfS Con 1: Extrinsic calibration required (N > 1 camera poses) → Use the structure-from-motion technique Con 2: No estimation of the albedo → No 3D-scanning → Apply Part 1 of Jean Mélou’s PhD thesis Con 3: Matching required → Add an error of reprojection to the model Model in the case of N = 2 images I1 and I2

min

z

• I1 − s · [−∇z⊤, 1]⊤
• ∇z2 + 1
• 2

ℓ2(Ω1)

• SfS error

+ λ

• I1 − I2 ◦ hz
• 2

ℓ2(Ω1,2)

• Error of reprojection

The homography hz reprojecting I1 towards I2 depends on z The set Ω1,2 contains the pixels of Ω1 which are seen in image I2

Jean-Denis DUROU (IRIT) Photometric 3D-reconstruction Como, July 17, 2018 29 / 35

Multi-view shape-from-shading

Multi-view SfS: A nonlinear and non-local problem

In comparison with photometric stereo

min

z

• I1 − s · [−∇z⊤, 1]⊤
• ∇z2 + 1
• 2

ℓ2(Ω1)

+ λ

• I1 − I2 ◦ hz
• 2

ℓ2(Ω1,2)

The problem is nonlinear in z It cannot be solved locally, due to ∇z Introducing an auxiliary function θ : Ω1 → R2

       min

θ,z

• I1 − s · [−θ⊤, 1]⊤
• θ2 + 1
• 2

ℓ2(Ω1)

+ λ

• I1 − I2 ◦ hz
• 2

ℓ2(Ω1,2)

s.t. θ = ∇z

Jean-Denis DUROU (IRIT) Photometric 3D-reconstruction Como, July 17, 2018 30 / 35

Multi-view shape-from-shading

Multi-view SfS: A nonlinear and non-local problem

In comparison with photometric stereo

min

z

• I1 − s · [−∇z⊤, 1]⊤
• ∇z2 + 1
• 2

ℓ2(Ω1)

+ λ

• I1 − I2 ◦ hz
• 2

ℓ2(Ω1,2)

The problem is nonlinear in z It cannot be solved locally, due to ∇z Introducing an auxiliary function θ : Ω1 → R2

       min

θ,z

• I1 − s · [−θ⊤, 1]⊤
• θ2 + 1
• 2

ℓ2(Ω1)

+ λ

• I1 − I2 ◦ hz
• 2

ℓ2(Ω1,2)

s.t. θ = ∇z

Jean-Denis DUROU (IRIT) Photometric 3D-reconstruction Como, July 17, 2018 30 / 35

Multi-view shape-from-shading

Multi-view SfS: A nonlinear and non-local problem

In comparison with photometric stereo

min

z

• I1 − s · [−∇z⊤, 1]⊤
• ∇z2 + 1
• 2

ℓ2(Ω1)

+ λ

• I1 − I2 ◦ hz
• 2

ℓ2(Ω1,2)

The problem is nonlinear in z It cannot be solved locally, due to ∇z Introducing an auxiliary function θ : Ω1 → R2

       min

θ,z

• I1 − s · [−θ⊤, 1]⊤
• θ2 + 1
• 2

ℓ2(Ω1)

+ λ

• I1 − I2 ◦ hz
• 2

ℓ2(Ω1,2)

s.t. θ = ∇z

Jean-Denis DUROU (IRIT) Photometric 3D-reconstruction Como, July 17, 2018 30 / 35

Multi-view shape-from-shading

Multi-view SfS: A nonlinear and non-local problem

In comparison with photometric stereo

min

z

• I1 − s · [−∇z⊤, 1]⊤
• ∇z2 + 1
• 2

ℓ2(Ω1)

+ λ

• I1 − I2 ◦ hz
• 2

ℓ2(Ω1,2)

The problem is nonlinear in z It cannot be solved locally, due to ∇z Introducing an auxiliary function θ : Ω1 → R2

       min

θ,z

• I1 − s · [−θ⊤, 1]⊤
• θ2 + 1
• 2

ℓ2(Ω1)

+ λ

• I1 − I2 ◦ hz
• 2

ℓ2(Ω1,2)

s.t. θ = ∇z

Jean-Denis DUROU (IRIT) Photometric 3D-reconstruction Como, July 17, 2018 30 / 35

Multi-view shape-from-shading

Multi-view SfS: Numerical resolution by ADMM

Introducing Lagrange multipliers µ : Ω1 → R2 and a parameter β > 0

min

θ,z,µ

• I1 − s · [−θ⊤, 1]⊤
• θ2 + 1
• 2

ℓ2(Ω1)

+λ

• I1 − I2 ◦ hz
• 2

ℓ2(Ω1,2)+µ|∇z−θ+β ∇z − θ2 ℓ2(Ω1)

Iterative scheme ADMM (alternating direction method of multipliers) Repeat until convergence (parameter β is automatically updated):

1

min

θ

• I1 − s · [−θ⊤, 1]⊤
• θ2 + 1
• 2

ℓ2(Ω1)

+ µ(k)|∇z(k) − θ + β(k) ∇z(k) − θ

• 2

ℓ2(Ω1)

2

min

z

λ

• I1 − I2 ◦ hz
• 2

ℓ2(Ω1,2)+ µ(k)|∇z − θ(k+1) + β(k)

∇z − θ(k+1) 2

ℓ2(Ω1)

3

µ(k+1) = µ(k) + β(k) (∇z(k+1) − θ(k+1))

Jean-Denis DUROU (IRIT) Photometric 3D-reconstruction Como, July 17, 2018 31 / 35

Multi-view shape-from-shading

Multi-view SfS: A preliminary result

N = 2 synthetic images of size 540 × 540 (|Ω1| = 63474 pixels) Knowledge of: camera poses, albedo (ρ ≡ 1), lighting vector s Hyper-parameter λ = 0.1 z(0) ≡ fronto-parallel plane Result obtained after 84 iterations (92s on an i5 processor)

I1 I2 3D-reconstruction

Jean-Denis DUROU (IRIT) Photometric 3D-reconstruction Como, July 17, 2018 32 / 35

Conclusion and perspectives

Outline

1

Introduction

2

Shape-from-shading (SfS)

3

Photometric stereo

4

Multi-view shape-from-shading

5

Conclusion and perspectives

Jean-Denis DUROU (IRIT) Photometric 3D-reconstruction Como, July 17, 2018 33 / 35

Conclusion and perspectives

Conclusion: Two well-posed extensions of SfS

Both extensions have very different features: Photometric stereo Single camera pose N > 1 lighting vectors Inside the lab Linear and local problem Multi-view SfS Single lighting vector N > 1 camera poses Outside the lab Nonlinear and non-local problem

Jean-Denis DUROU (IRIT) Photometric 3D-reconstruction Como, July 17, 2018 34 / 35

Conclusion and perspectives

Future work on multi-view SfS

Theory: Collaboration with Maurizio Falcone and Silvia Tozza Proof of uniqueness by Chambolle (Annales de l’IHP , 1994):

Smooth white Lambertian surface Frontal lighting Pure translation of the camera (N = 2 views) Depth z known on the boundary

Extension of the proof:

Any direction of the lighting Any pair of camera poses Absence of boundary condition

Practice: Jean Mélou’s PhD thesis (in progress) Extension to N > 2 images → Full 3D-reconstruction Use regularization or robust estimators in the model → Real data Towards a complete 3D-scanner?

Jean-Denis DUROU (IRIT) Photometric 3D-reconstruction Como, July 17, 2018 35 / 35