1 Basic Stereo Derivations Correspondence It is fundamentally - PDF document

Stereo Vision Reading: Chapter 11 • Stereo matching computes depth from two or more images • Subproblems: – Calibrating camera positions. – Finding all corresponding points (hardest part) – Computing depth or surfaces. Slide credits for this chapter: David Jacobs, Frank Dellaert, Octavia Camps, Steve Seitz The epipolar constraint Stereo vision Triangulate on two images of the epipolar line epipolar line epipolar line epipolar line same point to recover depth. epipolar plane – Feature matching across views depth – Calibrated cameras baseline • Epipolar Constraint Left Right – Matching points lie along corresponding epipolar lines �������������������� – Reduces correspondence problem to 1D search along ������������������������� conjugate epipolar lines – Greatly reduces cost and ambiguity of matching Slide credit: Steve Seitz We can always achieve this Simplest Case: Rectified Images geometry with image rectification • Image planes of cameras are parallel. • Focal points are at same height. • Focal lengths same. • Then, epipolar lines fall along the horizontal scan lines of the images • We will assume images have been rectified so that epipolar • Image Reprojection lines correspond to scan lines – reproject image planes onto common – Simplifies algorithms plane parallel to line between optical centers – Improves efficiency • Notice, only focal point of camera really matters (Seitz) 1

Basic Stereo Derivations Correspondence • It is fundamentally ambiguous, even with stereo constraints B e n z O L i (u L ,v L ) P L = (X,Y,Z) l e s a b x y (u R ,v R ) z O R x Disparity: y �������������������� ���������������� Correspondence: What should we match? • Objects? • Edges? • Pixels? • Collections of pixels? Julesz: showed that recognition is not needed for stereo. Correspondence: Epipolar constraint. Correspondence: Photometric constraint The epipolar constraint helps, but much ambiguity remains. • Same world point has same intensity in both images. – True for Lambertian surfaces • A Lambertian surface has a brightness that is independent of viewing angle – Violations: • Noise • Specularity • Non-Lambertian materials • Pixels that contain multiple surfaces 2

Pixel matching Correspondence Using Correlation Left Right scanline For each epipolar line For each pixel in the left image • compare with every pixel on same epipolar line in right image SSD error • pick pixel with minimum match cost This leaves too much ambiguity, so: disparity Improvement: match windows (Seitz) Sum of Squared (Pixel) Image Normalization Differences • Even when the cameras are identical models, there can be Left Right differences in gain and sensitivity. w w L R • For these reason and more, it is a good idea to normalize the m pixels in each window: w w m L R x L y x − d y ∑ ( , ) ( , ) I = I u v 1 ( , ) Average pixel L L L I I W x y ( , ) L R m u v ∈ W x y ( , ) ( , ) m w w m m ∑ and are correspond ing by windows of pixels. I = I u v 2 L R [ ( , )] Window magnitude W x y ( , ) We define the window function : m u v ∈ W x y ( , ) ( , ) m W x y = u v x − m ≤ u ≤ x + m y − m ≤ v ≤ y + m ( , ) { , | , } − I x y I m ( , ) 2 2 2 2 I ˆ x y = ( , ) Normalized pixel The SSD cost measures the intensity difference as a function of disparity : I − I ∑ C x y d = I u v − I u − d v 2 W x y ( , , ) [ ( , ) ( , )] ( , ) m r L R u v ∈ W x y ( , ) ( , ) m Images as Vectors Image Metrics Left Right “Unwrap” (Normalized) Sum of Squared Differences w R ( d ) ∑ image to form C d = I ˆ u v − I ˆ u − d v 2 ( ) [ ( , ) ( , )] L R w SSD vector, using u v ∈ W x y L ( , ) ( , ) m raster scan order = w − w d 2 ( ) L R w L Normalized Correlation w m row 1 R ∑ = ˆ ˆ − C d I u v I u d v ( ) ( , ) ( , ) w m L R NC L ∈ ( u , v ) W ( x , y ) m m m row 2 = w ⋅ w d = θ ( ) cos Each window is a vector L R in an m 2 dimensional w vector space. L = − 2 = ⋅ d * w w d w w d arg min ( ) arg max ( ) m row 3 Normalization makes d L R d L R them unit length. 3

Window size Stereo Results W = 3 W = 20 • Effect of window size • Some approaches have been developed to use an adaptive window size (try multiple sizes and select best match) Images courtesy of Point Grey Research (Seitz) Stereo testing and comparisons D. Scharstein and R. Szeliski. "A Taxonomy and Evaluation of Dense Two- Frame Stereo Correspondence Algorithms," International Journal of Computer Vision, 47 (2002), pp. 7-42. Scene Ground truth Scharstein and Szeliski Results with better method Results with window correlation Window-based matching Ground truth State of the art method: Graph cuts Ground truth (best window size) (Seitz) (Seitz) 4

Stereo Correspondences Stereo Correspondences Left scanline Right scanline Left scanline Right scanline … … … … Match Match Match Occlusion Disocclusion Stereo Matching with Dynamic Search Over Correspondences Programming Occluded Pixels Occluded Pixels Left scanline Left scanline Start Dynamic programming yields the optimal path through grid. This is Dis-occluded Pixels the best set of matches that Right scanline satisfy the ordering constraint Right scanline Disoccluded Pixels Three cases: – Sequential – add cost of match (small if intensities agree) – Occluded – add cost of no match (large cost) – Disoccluded – add cost of no match (large cost) End Dynamic Programming Dynamic Programming • Efficient algorithm for solving sequential decision (optimal path) problems. = i 1 Π 1 1 1 12 Π = i = 22 i 1 1 1 1 1 2 2 2 2 States: Π 32 … i = 2 2 2 2 2 = i 3 3 3 3 i = C C C 3 3 3 3 3 − + t t t 1 1 = = = t = t t t T 1 2 3 Suppose cost can be decomposed into stages: Π = i j T Cost of going from state to state How many paths through this trellis? 3 ij 5

Dynamic Programming Dynamic Programming i = i = 1 Π 1 1 1 1 1 1 1 12 Π b = ( 2 ) 2 = t = = 22 j = j i 2 i 2 2 2 2 2 2 2 2 2 Π 32 = = i i 3 3 3 3 3 3 3 3 C C C C C C t − t t + t − t t + 1 1 1 1 Principle of Optimality for an n-stage assignment problem: C j = Π + C i ( ) min ( ( )) − t i ij t 1 C j = Π + C 1 i ( ) min ( ( )) b j = Π + C i ( ) arg min ( ( )) − t i ij t − t i ij t 1 Stereo Matching with Dynamic Stereo Matching with Dynamic Programming Programming Occluded Pixels Occluded Pixels Left scanline Left scanline Scan across grid computing optimal Scan across grid computing optimal cost for each node given its cost for each node given its Dis-occluded Pixels Dis-occluded Pixels upper-left neighbors. upper-left neighbors. Backtrack from the terminal to Backtrack from the terminal to Right scanline Right scanline get the optimal path. get the optimal path. Terminal Terminal Stereo Matching with Dynamic Programming Occluded Pixels Left scanline Scan across grid computing optimal cost for each node given its Dis-occluded Pixels upper-left neighbors. Backtrack from the terminal to Right scanline get the optimal path. Terminal Scharstein and Szeliski 6

Segmentation-based Stereo Another Example Hai Tao and Harpreet W. Sawhney Result using a good technique View Interpolation Right Image Left Image Disparity Summary of different stereo methods Computing Correspondence • Constraints: – Geometry, epipolar constraint. Another approach is to match edges rather than windows of pixels: • – Photometric: Brightness constancy, only partly true. – Ordering: only partly true. – Smoothness of objects: only partly true. • Algorithms: • Which method is better? – What you compare: points, regions, features? – Edges tend to fail in dense texture (outdoors) – Correlation tends to fail in smooth featureless areas • How you optimize: – Local greedy matches. – 1D search. – 2D search. 7