Polarimetric 3D Reconstruction and Image Separation Zhaopeng Cui ETH Zurich 12.19.2019 | |
Outline Polarization and Polarizer Polarimetric 3D Reconstruction Polarimetric Multiple- View Stereo [CVPR’2017] Poalrimetric Dense Monocular SLAM [CVPR’2018] Poalrimetric Relative Pose Estimation [ICCV’2019] Polarimetric Reflection Separation [NeurIPS’2019] Conclusion Zhaopeng Cui | | 12/20/2019 2
Polarization Polarization is a characteristic of all transverse waves. Oscillation which take places in a transverse wave in many different directions is said to be unpolarized. In an unpolarized transverse wave oscillations may take place in any direction at right angles to the direction in which the wave travels. Direction of propagation of wave Zhaopeng Cui | | 12/20/2019 3
Polarization by Reflection Unpolarized light can be polarized, either partially or completely, by reflection. The amount of polarization in the reflected beam depends on the angle of incidence. Zhaopeng Cui | | 12/20/2019 4
Polarizer Polarizer is made from long chain molecules oriented with their axis perpendicular to the polarizing axis; These molecules preferentially absorb light that is polarized along their length. Polarizing axis Polarizing axis Zhaopeng Cui | | 12/20/2019 5
Polarimetric Imaging Images with a Rotating Polarizer Pixel intensity varies with polarizer angles We can recover geometric information from polarized images Camera Polarizer ∅ Object Zhaopeng Cui | | 12/20/2019 6
Surface Normal from Polarization Camera Normal Light Estimation of the azimuth angle 𝜒 (diffuse reflection): 𝜚 𝑞𝑝𝑚 𝐽 𝜚 𝑞𝑝𝑚 = 𝐽 𝑛𝑏𝑦 + 𝐽 𝑛𝑗𝑜 + 𝐽 𝑛𝑏𝑦 − 𝐽 𝑛𝑗𝑜 cos 2 𝜚 𝑞𝑝𝑚 − 𝜚 2 2 Polarizer 𝜒 = 𝜚 or 𝜒 = 𝜚 + 𝜌 Estimation of the zenith angle 𝜄 (diffuse reflection): 𝑜 − 1/𝑜 2 sin 2 𝜄 𝜍 = 𝐽 𝑛𝑏𝑦 − 𝐽 𝑛𝑗𝑜 = 2 + 2𝑜 2 − 𝑜 + 1/𝑜 sin 2 𝜄 + 4 cos 𝜄 𝑜 2 − sin 2 𝜄 𝐽 𝑛𝑏𝑦 + 𝐽 𝑛𝑗𝑜 Estimation of the surface normal v : T = cos 𝜒 sin 𝜄 , − sin 𝜒 sin 𝜄 , − cos 𝜄 T v = 𝑤 𝑦 , 𝑤 𝑧 , 𝑤 𝑨 Zhaopeng Cui | | 12/20/2019 7
Polarimetric 3D Reconstruction Polarimetric Dense Monocular SLAM Polarimetric Multiple-View Stereo [R, t] c ′ c Polarimetric Relative Pose Estimation Zhaopeng Cui | | 12/20/2019 8
Traditional Multi-View Stereo Given several images of the same object or scene, compute a representation of its 3D shape. Traditional methods usually failed for featureless objects. Zhaopeng Cui | | Input Sample 12/20/2019 9 Traditional MVS Results
Shape from Surface Normal [Xie et al. CVPR’19] Zhaopeng Cui | | 12/20/2019 10
Challenges Surface normal estimation from polarization is hard: Refractive distortion: Zenith angle estimation requires the knowledge of the refractive index. Azimuthal ambiguity: The estimation of the azimuthal angle has 𝜌 -ambiuity. 𝐽 𝜚 𝑞𝑝𝑚 = 𝐽 𝑛𝑏𝑦 + 𝐽 𝑛𝑗𝑜 + 𝐽 𝑛𝑏𝑦 − 𝐽 𝑛𝑗𝑜 cos 2 𝜚 𝑞𝑝𝑚 − 𝜚 2 2 𝜒 = 𝜚 or 𝜒 = 𝜚 + 𝜌 Mixed reflection in real environment. Zhaopeng Cui | | 12/20/2019 11
Mixed Reflection Polarized Unpolarized Polarized Specularly Diffusely Diffusely Incident light reflected light reflected light reflected light Air Object Zhaopeng Cui | | 12/20/2019 12
Polarimetric Multiple View Stereo [CVPR’17] Proposition 1. Under unpolarized illumination, the measured scene radiance from a reflective surface through a linear polarizer at a polarization angle 𝜔 𝑞𝑝𝑚 is 𝐽 ∅ 𝑞𝑝𝑚 = 𝐽 𝑛𝑏𝑦 + 𝐽 𝑛𝑗𝑜 + 𝐽 𝑛𝑏𝑦 − 𝐽 𝑛𝑗𝑜 cos 2 ∅ 𝑞𝑝𝑚 − ∅ , 2 2 where 𝐽 𝑛𝑏𝑦 and 𝐽 𝑛𝑗𝑜 are the maximum and minimum measured radiance. The phase angle ∅ is related to the azimuth angle 𝜒 as follows: 𝜒 𝑗𝑔 𝑞𝑝𝑚𝑏𝑠𝑗𝑨𝑓𝑒 𝑒𝑗𝑔𝑔𝑣𝑡𝑓 𝑠𝑓𝑔𝑚𝑓𝑑𝑢𝑗𝑝𝑜 𝑒𝑝𝑛𝑗𝑜𝑏𝑢𝑓𝑡 𝜒 − 𝜌 ∅ = ቐ 𝑝𝑢ℎ𝑓𝑠𝑥𝑗𝑡𝑓 2 𝜌/2 -ambiguity * The azimuthal ( 𝜌 ) ambiguity still holds. Zhaopeng Cui | | 12/20/2019 13
Polarimetric Multiple View Stereo [CVPR’17] Exploit polarimetric information for dense reconstruction: Use geometric information to help resolve ambiguities of polarimetric information … View N Structure-from-Motion Initialization (Depth Estimation) View 1 … … Polarized Images from Multiple Resolving π /2 - Ambiguity Phase Angle Estimation Viewpoints (Per-View) Zhaopeng Cui | | 12/20/2019 14
Polarimetric Multiple View Stereo [CVPR’17] Use geometric information to help resolve 𝜌/2 -ambiguity Initial Depth after depth consistency Azimuth angle map check (after solving 𝜌/2 - ambiguity) Phase angle map 𝑞 = ቊ0, 𝑒𝑗𝑔𝑔𝑣𝑡𝑓𝑒 𝑒𝑝𝑛𝑗𝑜𝑏𝑢𝑓𝑡 𝑔 𝐹 𝑔 = 𝐸 𝑔 𝑞 + 𝜇 𝑇(𝑔 𝑞 , 𝑔 𝑟 ) 𝑞 1, 𝑡𝑞𝑓𝑑𝑣𝑚𝑏𝑠 𝑒𝑝𝑛𝑗𝑜𝑏𝑢𝑓𝑡 𝑞∈𝒬 𝑞,𝑟∈𝒪 𝐸 𝑔 𝑞 enforces consistency with MVS at well-textured regions. 𝑇(𝑔 𝑞 , 𝑔 𝑟 ) enforces neighboring pixels to have similar azimuth angles. Zhaopeng Cui | | 12/20/2019 15
Polarimetric Multiple View Stereo [CVPR’17] Exploit polarimetric information for dense reconstruction: Use geometric information to help resolve ambiguities of polarimetric information Use polarimetric information to improve geometric information … View N Structure-from-Motion Initialization … (Depth Estimation) View 1 Depth Depth Fusion Optimization … … Polarized Images from Multiple Resolving π /2 - Ambiguity Phase Angle Estimation Viewpoints (Per-View) Zhaopeng Cui | | 12/20/2019 16
Polarimetric Multiple View Stereo [ CVPR’17 ] Iso-depth contour tracing: Propagate reliable depth values along iso-depth contour 1. Phase angle determine the projected surface normal direction (with 𝜌 -ambiguity) 2. From the normal, we can get iso-depth contour on which the pixels have with the same depth 3. Propagate sparse depth values along iso-depth contour 𝜔 Projected Contour surface normals Zhaopeng Cui | | 12/20/2019 17
Polarimetric Multiple View Stereo [CVPR’17] Per-frame depth optimization 𝐹 𝑞 𝑒 𝑦, 𝑧 + 𝛿𝐹 𝑒 𝑒 𝑦, 𝑧 + |∆𝑒(𝑦, 𝑧)| (𝑦,𝑧)∈𝒬 constraint from smoothness constraint from azimuth angles known 3D points constraint Zhaopeng Cui | | 12/20/2019 18
Polarimetric multiple view stereo [CVPR17] Zhaopeng Cui | | 12/20/2019 19
Polarimetric Dense Monocular SLAM [CVPR’18] DSLR + Polarizer Filters Polarization camera Sensor Structure Rotate the polarizer filter video with multiple polarized image manually Zhaopeng Cui | | 12/20/2019 20
Polarimetric Dense Monocular SLAM [CVPR’18] Zhaopeng Cui | | 12/20/2019 21
Polarimetric Dense Monocular SLAM [CVPR’18] Phase angle disambiguation: Using rough depth to solve the 𝜌/2 -ambiguity Intuition: The correct iso-contour should have less depth variation. Strategy: Trace two local contours, select the one with less depth variance. Captured Disambiguation Results Phase Angle Map Polarized Images Zhaopeng Cui | | 12/20/2019 22
Polarimetric Dense Monocular SLAM [CVPR’18] Depth propagation along contours Issue: wrong propagation caused by noisy 3D points Solution: Two-View propagation and validation Phase map Inlier Points Propagated Points (Using Single-View) Zhaopeng Cui | | 12/20/2019 23
Polarimetric Dense Monocular SLAM [CVPR’18] Propagate depth in the current Keyframe ′ Φ 𝑢 Consistency ′ Φ 𝑠 Check Inlier New Inlier points 𝒀 𝒋−𝟐 points 𝒀 𝒋 iteration i -1 iteration i Propagate depth in the reference Keyframe Zhaopeng Cui | | 12/20/2019 24
Zhaopeng Cui | | 12/20/2019 25
Traditional Relative Pose Estimation 5-point algorithm: [R, t] c ′ c Zhaopeng Cui | | 12/20/2019 26
Challenges Surface normal estimation from polarization is hard: Refractive distortion: Zenith angle estimation requires the knowledge of the refractive index Azimuthal ambiguity: The estimation of the azimuthal angle has 𝜌 -ambiuity 𝐽 𝜚 𝑞𝑝𝑚 = 𝐽 𝑛𝑏𝑦 + 𝐽 𝑛𝑗𝑜 + 𝐽 𝑛𝑏𝑦 − 𝐽 𝑛𝑗𝑜 4 𝑜 possibilities given n cos 2 𝜚 𝑞𝑝𝑚 − 𝜚 2 2 pairs of correspondences. 𝜒 = 𝜚 or 𝜒 = 𝜚 + 𝜌 Mixed reflection in real environment. Zhaopeng Cui | | 12/20/2019 27
Polarimetric Relative Pose Estimation [ICCV’19] Two-point relative pose estimation: Step 1. Solve the relative rotation: 2 + Rv 2 − v 2 ′ ′ 2 R∈𝑇𝑃(3) Rv 1 − v 1 m𝑗𝑜 R = U diag 1,1, det UV T V T T + v 2 ′ v 2 UΣV T = v 1 ′ v 1 T Step 2. Solve the relative translation: ′ ∙ t × Rx 𝑗 = t ∙ Rx 𝑗 × x 𝑗 ′ = 0, 𝑗 = 1,2 x 𝑗 ′ ) × (Rx 2 × x 2 ′ ) [R, t] c ′ t = (Rx 1 × x 1 c Step 3. Hypothesis validation to choose the one which has the largest consensus. Zhaopeng Cui | | 12/20/2019 28
Recommend
More recommend
