Light Field Vision for Transparent Object Categorization and Segmentation 光 场视觉 在透明物体分类和分割中的 应 用 Yichao Xu 徐 轶 超 xuyichao.cn Jan. 6, 2016
Just a reminder – Last day P4A-04 1
About me • A: Hometown in Zhejiang - Jiaxing • B: Undergraduate in Beijing - BESTI • C: Master 1 in Anhui - USTC, Hefei • D: Master 2-3 in Shanghai - SINAP , CAS • E: PhD in Fukuoka, Japan - Kyushu University 2
Outline • Introduction of Light Field Vision • Transcat: Transparent Object Categorization • Transcut: Transparent Object Segmentation 3
Light field Scene Light field describes all the light rays in the space 4
Sensors for visual perception Cameras with CCD and CMOS sensors 5
Regular camera sensing Scene Image Only a few light rays can be captured 6
Light field parameterization Scene Position (s, t) Angle (u, v) 4D light field Each light ray can be represented by L(s, t, u, v) 7
Light field sensing Scene Viewpoint plane Sensor plane 𝑡 u Light field camera can capture richer information 8
Light field sampling in phase space Scene Viewpoint plane Sensor plane 𝑡 u u 𝑡 𝑡u phase space Regular camera can only sample sub light field space 9
Light field sampling in phase space Scene Viewpoint plane Sensor plane 𝑡 u u 𝑡 𝑡u phase space Light field camera can capture richer information 10
Computational Photography Multi-focus Multi-view Light Field is widely used for Image-based Rendering 11
Light field cameras Raytrix Lytro PiCam Stanford Viewplus Simultaneously record positional and angular information of ray Obtain rich information with single-shot 12
Light field vision Capture To solve computer vision problems 13
Computer vision makes our life better Free our hands Help us know more 14
Visual recognition makes it possible France Prešeren , Poet Visual recognition is important in these applications 15
Advantage of light field vision Regular Computer vision Light Field Vision Redundant information makes it easier to understand the 3D world 16
Light field vision applications • Surveillance - Accurately detect desired foreground LF method Conventional [A.Shimada et al., IPSJ CVA 2013] • Depth estimation - Accurate and consistent LF method Conventional [S. Wanner et al., PAMI2014] • Salience detection - Accurate in challenge scenes LF method Conventional GT [N. Li et al., CVPR2014] 17
Light Field Vision Application -- transparent object recognition 18
Transparent object recognition Which type is the object? Where is the object? 19
Challenge of the target object Appearance of transparent objects drastically varies with background 20
Transparent object causes distortion Different objects produce different image of the same scene Regular computer vision methods cannot understand whether the image is distorted or not without prior knowledge 21
Know light field from background Transparent object [Ben-Ezra and Nayar, ICCV2003] Known motion, Manually tagged feature [G. Wetzstein et al, ICCV2011] Known background 22
Features from Light Field for Transparent Object Recognition 23
Distortion modeled by light field vision Background distortion changes with viewpoint Background distortion is modeled as the correspondences between the viewpoints 24
Background invariant distortion Modeled distortion is independent of background textures 25
Light Field Distortion (LFD) feature ∆ v ∆ u 26
LFD feature visualization ∆ v ∆ u 24x2D feature vector for each pixel 2D vectors on different viewpoints 27
Light Field Linearity (LF-linearity) Background Viewpoint plane Sensor plane u 𝑡 u 𝑡 𝑡u phase space Rays from background are linear distributed 28
Light Field Linearity (LF-linearity) Background Viewpoint plane Sensor plane u 𝑡 u 𝑡 𝑡u phase space Transparent object Rays from transparent object are not linear distributed 29
Extract LF-linearity ∆𝑣 ∆𝑣 Disparity Euclidean Distance 𝑡 𝑡 Hyper-plane 30
LF-linearity visualization Central view LF-linearity 31
Light Field Consistency (LF-consistency) Poor consistency backward matching forward matching (0,0) ( , ) view view s t Good consistency backward matching forward matching (0,0) ( , ) view view s t LF-consistency is used for detecting the depth discontinuity 32
Occlusion in light field Occlusion detector Occlusion is caused by depth discontinuity 33
Occlusion detectors 34
Detect occlusion point 0 0 0 1 1 0 0 0 1 1 = 0.7 = 1 X 0 0 0 0 0 0 0 1 1 1 0 0 0 1 1 0 0 0 1 1 The detected occlusion point is from θ = 0 35
Detected occlusion visualization Central view Occlusion response 36
Feature and descriptor • LFD Feature ( 光 场 扭曲特征 ) - 2x24 Dimensional vector - Describe the distortion pattern • LF-linearity （光 场线 性度） - A metric to describe how much is the distortion • Occlusion detector （遮 挡检测 ） - Describe the probability of a point to be in the occlusion - Occlusion in which direction 37
Outline • Introduction of Light Field Vision • Transcat: Transparent Object Categorization • Transcut: Transparent Object Segmentation 38
TransCat: Transparent Object Categorization Which object ？ 39
Training pipeline Filtering the background Estimate background by LF-linearity Extracting the LFD feature Extracting the LFD feature Non-linear Linear Linear Non-linear Non-linear Training based on Bag of Training based on Bag of Linear features features Linear Non-linear 40
Training pipeline Filtering the background Estimate relative differences by Extracting the LFD feature optical flow Training based on Bag of features 41
Training pipeline Extracting the LFD feature Filtering the background by LF-linearity Representative LFD Feature space Training based on Bag of features 42
Testing for transparent objects categorization Categorization based on Bag of features Representative LFD result 43
Experimental setting 18 objects 10 backgrounds Background scenes can be dynamic! 44
Categorization result Evaluation by leave-one-out cross-validation 1 0.9 0.8 Recognition ratio 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 A B C D E F G H I J K L M N O P Q R Object Average categorization accuracy: 84% 45
Analysis • Applicable conditions 1 1 Recognition ratio Recognition ratio 0.8 0.8 0.6 0.6 0.4 0.4 0.2 0.2 0 0 30 35 40 45 50 50 100 150 200 250 Camera position [cm] Background position [cm] 0.8 0.8 Recognition ratio 0.6 Recognition ratio 0.6 0.4 0.2 0.4 0 0.2 0 Lighting angle [degree] 0 0.1 0.2 0.3 Noise standard deviation 46
Analysis 1 Recognition ratio • Applicable conditions 0.8 0.6 0.4 0.2 0 -10 -5 0 5 10 Rotation along x-axis [degree] 0.8 Recognition ratio 0.6 Recognition ratio 0.8 0.6 0.4 Overall 0.4 0.2 Symmetric objects 0.2 Asymmetric objects 0 0 0 10 20 30 40 0 10 20 30 40 Rotation along y-axis [degree] Rotation along z-axis [degree] 47
Results for real scene 48
Outline • Introduction of Light Field Vision • Transcat: Transparent Object Categorization • Transcut: Transparent Object Segmentation 49
Transcut: Transparent Object Segmentation 50
Properties of different components Transparent Object Background Poor LF-linearity Good LF-linearity exclude the occlusion Trans Obj Occlusion Extracted by occlusion detector Transparent object segmentation formulated as labeling problem 51
Regional term large penalty assigns to pixels that have poor LF-linearity exclude the occlusion area Background penalty large penalty assigns to Central pixels with poor LF-linearity in view of the occlusion or pixels with input light good LF-linearity Foreground penalty field image 52
Boundary term … q 2 … If is from 𝑃 𝑞 θ = 0 , q 3 p q 1 … … q 4 Central Detected view of occlusion point input light 𝑃 𝑞 field image 53
Energy minimization Regional term Graph Boundary term Cut Central view of input light field image 54
Experiments Object 1 Object 2 Object 3 Object 4 Object 5 Object 6 Object 7 Scene 1 Scene 2 Scene 3 Scene 4 Scene 5 Scene 6 Scene 7 Background scenes can be dynamic! 55
Comparison with related work 6 features from appearance Single feature from LF Finding glass LF-linearity thresholding McHenry et al., CVPR2005 56
Visual comparison Images from the central viewpoint Results from Finding glass Results from LF-linearity thresholding Results from TransCut Ground Truth 57
Recommend
More recommend
