Photometric*Stereo* October*8,*2013* Dr.*Grant*Schindler* * - - PowerPoint PPT Presentation

Photometric*Stereo*

October*8,*2013* Dr.*Grant*Schindler* * schindler@gatech.edu*

Mul@ple*Images:*Different*Ligh@ng*

Mul@ple*Images:*Different*Ligh@ng*

Mul@ple*Images:*Different*Ligh@ng*

Religh@ng*the*Scene*

Credit:*Alex*Powell*

Credit:*Alex*Powell*

Credit:*Alex*Powell*

Credit:*Alex*Powell*

Credit:*Alex*Powell*

Credit:*Alex*Powell*

Demo*

• Froggies*

Geometry*of*Ligh@ng*

Lamber@an*Surface*

Infinitely*Distant*Light*Source*

Acquiring)the)Reflectance)Field)of)a) Human)Face:)Paul)Debevec)et)al*

hOp://gl.ict.usc.edu/LightStages/*

More*Info*

• hOp://www.pauldebevec.com/*
• hOp://www.pauldebevec.com/Research/LS/*
• hOp://gl.ict.usc.edu/Research/DigitalEmily/*
• hOp://gl.ict.usc.edu/Research/RHL/*

RealTTime*Photometric*Stereo*

Surface*Normals*

(Eigenvectors*via*Power*Itera@on)* 6*ms*

Height*Field*

(Integrate*Surface*Normals*via*GaussTSeidel)* 32*ms*

RealTTime*Photometric*Stereo*

Demo*

• Processing*Demo*

Reading **

• Szeliski*Book:*Sec@on*12.1.1*(page*580T583)*

3D*Scanner*in*Your*Pocket*

How*does*it*work?*

Emit*Light* Reflect*Light* Camera*(FrontTFacing)*

How*does*it*work?*

Emit*Light* Reflect*Light* Camera*(FrontTFacing)*

How*does*it*work?*

Emit*Light* Reflect*Light* Camera*(FrontTFacing)*

How*does*it*work?*

Emit*Light* Reflect*Light* Camera*(FrontTFacing)*

How*does*it*work?*

2D* 3D*

Lambertian Surface

a(u,v) is the albedo of the surface projecting to (u,v). n(u,v) is the direction of the surface normal. s0 is the light source intensity. s is the direction to the light source.

^ n ^ s a

I(u,v) The intensity of a pixel I(u,v) is:

Slide courtesy of David Kriegman

I(u,v) = [a(u,v) ˆ n (u,v)]•[s0 ˆ s ] = b(u,v)•s

b

Orthographic Projection

• Simplification for light

sources that are sufficiently far away from an object.

• All incoming light rays

are parallel.

• Thus, while b vectors

vary over the surface, s vector is constant.

s b s b s

Pixels:b1

Ts, b2 Ts, b3 Ts,...⇒ Bs

Images Live in a 3-D Subspace

• f All Possible Images

Images = Surface Normals #pixels #pixels 4 3 3 4 Light Directions

Images Live in a 3-D Subspace

• f All Possible Images

Images = Surface Normals #pixels #pixels 4 3 3 4 Light Directions

Background:*Linear*Algebra*

• hOp://www.cc.gatech.edu/~phlosob/

transforms/*

• Keywords:*Singular*Value*Decomposi@on*
• Related:*Image*Compression*

Synthetic Sphere Images

Five different lighting conditions

Slide courtesy of David Forsyth

Recovered Albedo

Slide courtesy of David Forsyth

Recovered Surface Normals

Slide courtesy of David Forsyth

Recovered Surface Shape

Recovery up to a constant depth error (not absolute depth)

Slide courtesy of David Forsyth

Integra@on*Methods*

TFrankotTChellappa*(FFT)* TGaussTSeidel*(Itera@ve*Condi@onal*Modes)* TPath*Integra@on*

High Quality Results

George Vogiatzis and Carlos Hernandez

Photometric Stereo

Photometric Stereo

• Given multiple images of the same surface

under different known lighting conditions, can we recover the surface shape?

– Yes! (Woodham, 1978)

b1 b2 b3

s1 s2 s3

Photometric Stereo

• Assume:

– A set of point sources that are infinitely distant – A set of pictures of an object, obtained in exactly the same camera/object configuration but using different sources – A Lambertian object (or the specular component has been identified and removed)

I1 = Bs1; I2 = Bs2; I3 = Bs3 ...

Stereo for a Pixel

b

s1 s2 s3 I

Solve an over-constrained linear system for b (with n>3)

For a pixel (x,y) we have n measurements: I1(x,y) =s1

Tb(x,y);

I2(x,y) =s2

Tb(x,y) ...⇒ I(x,y) = Sb(x,y)

What About Shadows?

• Shadowed pixels (e.g. attached shadows for

a given light source position) are outliers.

• Max trick can be adapted for this case too:

Pre-multiplying by a thresholded weight matrix zeros the contributions from shadowed pixels I1(x,y)  In(x,y) " # $ $ $ % & ' ' ' I1(x,y)  In(x,y) " # $ $ $ % & ' ' ' = I1(x,y)  In(x,y) " # $ $ $ % & ' ' ' s1

T

 sn

T

" # $ $ $ % & ' ' ' b(x,y)

Recovering the Albedo

• This gives a check on the normal

recovery at a pixel

– If the magnitude of a(x, y) is greater than 1, theres a problem

Recall that b(x,y) = a(x,y)ˆ n (x,y) ⇒ a(x,y) = b(x,y)

Then ˆ n (x,y) = b(x,y)/a(x,y)

Recovering the Surface Shape

Depth map model (also called Monge patch):

Graphic courtesy of David Forsyth

z = f (x,y) Surface is set of points {x,y, f (x,y)}

Recovering a surface from normals - 1

• Recall the surface

is written as

• This means the

normal has the form:

• If we write the known

vector g as

• Then we obtain values

for the partial derivatives of the surface:

(x,y, f (x, y))

g(x,y) = g1(x, y) g2(x, y) g3(x, y) " # $ $ % & ' '

fx(x,y) = g1(x, y) g3(x, y)

( )

fy(x, y) = g2(x,y) g3(x,y)

( )

N(x,y) = 1 fx

2 + fy 2 +1

" # $ % & ' − fx − fy 1 " # $ $ % & ' '

Slide courtesy of David Forsyth

Recovering a surface from normals - 2

• Recall that mixed

second partials are equal --- this gives us a check. We must have:

• (or they should be

similar, at least)

• We can now recover

the surface height at any point by integration along some path, e.g. ∂ g1(x, y) g3(x, y)

( )

∂y = ∂ g2(x, y) g3(x, y)

( )

∂x f (x, y) = fx(s, y)ds

x

∫

+ fy(x,t)dt

y

∫

+ c

Slide courtesy of David Forsyth

Light Sources

Lambertian Surface

a(u,v) is the albedo of the surface projecting to (u,v). n(u,v) is the direction of the surface normal. s0 is the light source intensity. s is the direction to the light source.

^ n ^ s a

I(u,v) The intensity of a pixel I(u,v) is:

Slide courtesy of David Kriegman

I(u,v) = [a(u,v) ˆ n (u,v)]•[s0 ˆ s ] = b(u,v)•s

b

Orthographic Projection

• Simplification for light

sources that are sufficiently far away from an object.

• All incoming light rays

are parallel.

• Thus, while b vectors

vary over the surface, s vector is constant.

s b s b s

Pixels:b1

Ts, b2 Ts, b3 Ts,...⇒ Bs