Today Recap: epipolar constraint Stereo image rectification - PDF document

3/29/2011 Today – Recap: epipolar constraint – Stereo image rectification Stereo: – Stereo solutions Correspondence and Calibration • Computing correspondences • Non-geometric stereo constraints Mon, March 28 – Calibration Prof. Kristen Grauman UT-Austin – Example stereo applications Last time: Last time: Estimating depth with stereo Epipolar geometry • Stereo : shape from “motion” between two views • We need to consider: • Info on camera pose (“calibration”) • Image point correspondences Epipolar Line • Epipolar Plane scene point Baseline Epipole Epipole image plane optical center Last time: Example: converging cameras Epipolar constraint • Potential matches for p have to lie on the corresponding epipolar line l’ . • Potential matches for p’ have to lie on the corresponding epipolar line l . Slide credit: M. Pollefeys Figure from Hartley & Zisserman CS 376 Lecture 17 Stereo 1

3/29/2011 Example: parallel cameras An audio camera & epipolar geometry Spherical microphone array Adam O' Donovan, Ramani Duraiswami and Jan Neumann Microphone Arrays as Generalized Cameras for Integrated Audio Visual Processing, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Minneapolis, 2007 Figure from Hartley & Zisserman Last time: An audio camera & epipolar geometry Essential matrix       X T RX 0      X RX [T ] 0 x  Let E [T x ] R  EX  X T 0 E is called the essential matrix , and it relates corresponding points between both cameras, given the rotation and translation. If we observe a point in one image, its position in other image is constrained to lie on line defined by above. Note: these points are in camera coordinate systems . Essential matrix example: parallel cameras   R I p [ x , y , f ]     image I´(x´,y´) image I(x,y) p' [ x ' , y ' , f ] Disparity map D(x,y) T [ d , 0 , 0 ] 0 0 0   E [ T ]R x 0 0 d 0 – d 0   Ep  p 0 (x´,y´)=(x+D(x,y),y) For the parallel cameras, image of any point must lie What about when cameras’ optical axes are not parallel? on same horizontal line in each image plane. CS 376 Lecture 17 Stereo 2

3/29/2011 Stereo image rectification Today In practice, it is convenient if image – Recap: epipolar constraint scanlines (rows) are the – Stereo image rectification epipolar lines. – Stereo solutions • Computing correspondences • Non-geometric stereo constraints – Calibration reproject image planes onto a common plane parallel to the line between optical – Example stereo applications centers pixel motion is horizontal after this transformation two homographies (3x3 transforms), one for each input image reprojection Slide credit: Li Zhang Stereo image rectification: example Today – Recap: epipolar constraint – Stereo image rectification – Stereo solutions • Computing correspondences • Non-geometric stereo constraints – Calibration – Example stereo applications Source: Alyosha Efros Correspondence problem Correspondence problem • Beyond the hard constraint of epipolar geometry, there are “soft” constraints to help Multiple match identify corresponding points hypotheses – Similarity satisfy epipolar – Uniqueness constraint, but – Ordering which is correct? – Disparity gradient • To find matches in the image pair, we will assume – Most scene points visible from both views – Image regions for the matches are similar in appearance Figure from Gee & Cipolla 1999 CS 376 Lecture 17 Stereo 3

3/29/2011 Dense correspondence search Correspondence problem For each epipolar line Parallel camera example: epipolar lines are For each pixel / window in the left image corresponding image scanlines • compare with every pixel / window on same epipolar line in right image • pick position with minimum match cost (e.g., SSD, correlation) Adapted from Li Zhang Source: Andrew Zisserman Correspondence problem Correspondence problem Neighborhoods of corresponding points are Intensity profiles similar in intensity patterns. Source: Andrew Zisserman Source: Andrew Zisserman Correlation ‐ based window matching Textureless regions Textureless regions are non ‐ distinct; high ambiguity for matches. Source: Andrew Zisserman Source: Andrew Zisserman CS 376 Lecture 17 Stereo 4

3/29/2011 Effect of window size Effect of window size W = 3 W = 20 Want window large enough to have sufficient intensity variation, yet small enough to contain only pixels with about the same disparity. Source: Andrew Zisserman Figures from Li Zhang Foreshortening effects Occlusion Source: Andrew Zisserman Slide credit: David Kriegman Correspondence problem Sparse correspondence search • Beyond the hard constraint of epipolar geometry, there are “soft” constraints to help identify corresponding points – Similarity – Uniqueness – Disparity gradient – Ordering • Restrict search to sparse set of detected features (e.g., corners) • Rather than pixel values (or lists of pixel values) use feature descriptor and an associated feature distance • Still narrow search further by epipolar geometry Tradeoffs between dense and sparse search? CS 376 Lecture 17 Stereo 5

3/29/2011 Uniqueness constraint Disparity gradient constraint • Up to one match in right image for every point in left • Assume piecewise continuous surface, so want disparity image estimates to be locally smooth Figure from Gee & Figure from Gee & Cipolla 1999 Cipolla 1999 Ordering constraint Ordering constraint • Points on same surface (opaque object) will be in same • Won’t always hold, e.g. consider transparent object, or order in both views an occluding surface Figure from Gee & Cipolla 1999 Figures from Forsyth & Ponce Scanline stereo • Try to coherently match pixels on the entire scanline • Different scanlines are still optimized independently • Beyond individual correspondences to estimate disparities: Left image Right image • Optimize correspondence assignments jointly – Scanline at a time (DP) – Full 2D grid (graph cuts) intensity CS 376 Lecture 17 Stereo 6

3/29/2011 “Shortest paths” for scan-line stereo Coherent stereo on 2D grid I  • Scanline stereo generates streaking artifacts Left image I Right image S left Right occlusion q occlusion Left t S right • Can’t use dynamic programming to find spatially s p coherent disparities/ correspondences on a 2D grid Can be implemented with dynamic programming Ohta & Kanade ’85, Cox et al. ‘96 Slide credit: Y . Boykov Stereo matching as energy minimization Recap: stereo with calibrated cameras I 2 D I 1 • Given image pair, R, T W 1 ( i ) W 2 ( i+D ( i )) D ( i ) • Detect some features • Compute essential matrix E • Match features using the     E E ( I , I , D ) E ( D ) epipolar and other data 1 2 smooth constraints             2 E D ( i ) D ( j ) E W ( i ) W ( i D ( i )) • Triangulate for 3d structure smooth data 1 2 neighbors i , j i • Energy functions of this form can be minimized using graph cuts Y. Boykov, O. Veksler, and R. Zabih, Fast Approximate Energy Minimization via Graph Cuts, PAMI 2001 Source: Steve Seitz Today Error sources • Low-contrast ; textureless image regions – Recap: epipolar constraint • Occlusions – Stereo image rectification • Camera calibration errors – Stereo solutions • Violations of brightness constancy (e.g., • Computing correspondences specular reflections) • Non-geometric stereo constraints • Large motions – Calibration – Example stereo applications CS 376 Lecture 17 Stereo 7

3/29/2011 Stereo in machine vision systems • Example depth maps (pentagon) Left : The Stanford cart sports a single camera moving in discrete increments along a straight line and providing multiple snapshots of outdoor scenes Right : The INRIA mobile robot uses three cameras to map its environment Forsyth & Ponce Depth for segmentation Depth for segmentation Edges in disparity in conjunction with image edges enhances contours found Danijela Markovic and Margrit Gelautz, Interactive Media Systems Group, Vienna University of Technology Danijela Markovic and Margrit Gelautz, Interactive Media Systems Group, Vienna University of Technology Model-based body tracking, Virtual viewpoint video stereo input C. Zitnick et al, High-quality video view interpolation using a layered representation, David Demirdjian, MIT Vision Interface Group SIGGRAPH 2004. http://people.csail.mit.edu/demirdji/movie/artic-tracker/turn-around.m1v CS 376 Lecture 17 Stereo 8