Convex Relaxation Methods for Computer Vision Daniel Cremers - PowerPoint PPT Presentation

Convex Relaxation Methods for Computer Vision Daniel Cremers Computer Science Department TU Munich with Kalin Kolev, Evgeny Strekalovskiy, Thomas Pock, Bastian Goldlücke, Antonin Chambolle & Jan Lellmann

3D Reconstruction from Multiple Views Daniel Cremers Convex Relaxation Methods for Computer Vision 2

Optimization in Computer Vision Image segmentation: Geman, Geman ’84, Blake, Zisserman ‘87, Kass et al. ’88, Mumford, Shah ’89, Caselles et al. ‘95, Kichenassamy et al. ‘95, Paragios, Deriche ’99, Chan, Vese ‘01, Tsai et al. ‘01, … Multiview stereo reconstruction: Non-convex energies Faugeras, Keriven ’98, Duan et al. ‘04, Yezzi, Soatto ‘03, Seitz et al. ‘06, Hernandez et al. ‘07, Labatut et al. ’07, … Optical flow estimation: Horn, Schunck ‘81, Nagel, Enkelmann ‘86, Black, Anandan ‘93, Alvarez et al. ‘99, Brox et al. ‘04, Baker et al. ‘07, Zach et al. ‘07, Sun et al. ‘08, Wedel et al. ’09, … Daniel Cremers Convex Relaxation Methods for Computer Vision 3

Non-convex versus Convex Energies Non-convex energy Convex energy Some related work: Brakke ‘95, Alberti et al. ‘01, Chambolle ‘01, Attouch et al. ‘06, Nikolova et al. ‘06, Cremers et al. ‘06, Bresson et al. ‘07, Lellmann et al. ‘08, Zach et al. ‘08, Chambolle et al. ’08, Pock et al. ‘09, Zach et al. ’09, Brown et al. ’10, Bae et al. ‘10, Yuan et al. ‘10,… Daniel Cremers Convex Relaxation Methods for Computer Vision 4

Overview Multiview reconstruction Super-res.textures 4D reconstruction Stereo reconstruction Segmentation Manifold-valued functions Daniel Cremers Convex Relaxation Methods for Computer Vision 5

Overview Multiview reconstruction Super-res.textures 4D reconstruction Stereo reconstruction Segmentation Manifold-valued functions Daniel Cremers Convex Relaxation Methods for Computer Vision 6

Stereo-weighted Minimal Surfaces 3D Reconstruction: Faugeras, Keriven ’98, Duan et al. ’04 Segmentation: Kichenassamy et al. ’95, Caselles et al. ’95 Optimal solution is the empty set: Local optimization: Faugeras, Keriven TIP ’98 Resort: Generative object/background modeling: Yezzi, Soatto ’03,… Constrain search space: Vogiatsis, Torr, Cipolla CVPR ’05 Intelligent ballooning: Boykov, Lempitsky BMVC ’06 Daniel Cremers Convex Relaxation Methods for Computer Vision 7

Silhouette Consistent Reconstructions Kolev et al., IJCV 2009, Cremers, Kolev, PAMI 2011 Daniel Cremers Convex Relaxation Methods for Computer Vision 8

Silhouette Consistent Reconstructions Σ = Proposition: The set of silhouette-consistent solutions is convex. Kolev et al., IJCV 2009, Cremers, Kolev, PAMI 2011 Daniel Cremers Convex Relaxation Methods for Computer Vision 9

Reconstruction of Fine-scale Structures Image data courtesy of Yasutaka Furukawa. Daniel Cremers Convex Relaxation Methods for Computer Vision 10

Overview Multiview reconstruction Super-res.textures 4D reconstruction Stereo reconstruction Segmentation Manifold-valued functions Daniel Cremers Convex Relaxation Methods for Computer Vision 11

Surface Evolution to Optimum Daniel Cremers Convex Relaxation Methods for Computer Vision 12

Super-Resolution Texture Map Given all images determine the surface color blur & downsample back-projection * Best Paper Goldlücke, Cremers, ICCV ’09, DAGM ’09 * , IJCV ‘13 Award Daniel Cremers Convex Relaxation Methods for Computer Vision 13

Super-Resolution Texture Map * Best Paper Goldlücke, Cremers, ICCV ’09, DAGM ’09 * , IJCV ‘13 Award Daniel Cremers Convex Relaxation Methods for Computer Vision 14

Super-Resolution Texture Map Closeup of input image Super-resolution texture * Best Paper Goldlücke, Cremers, ICCV ’09, DAGM ’09 * , IJCV ‘13 Award Daniel Cremers Convex Relaxation Methods for Computer Vision 15

Reconstructing the Niobids Statues Kolev, Cremers, ECCV ’08, PAMI 2011 Daniel Cremers Convex Relaxation Methods for Computer Vision 16

Overview Multiview reconstruction Super-res.textures 4D reconstruction Stereo reconstruction Segmentation Manifold-valued functions Daniel Cremers Convex Relaxation Methods for Computer Vision 17

Action Reconstruction Oswald, Cremers, ICCV ‘13 4DMoD Workshop Daniel Cremers Convex Relaxation Methods for Computer Vision 18

Action Reconstruction Oswald, Cremers, ICCV ‘13 4DMoD Workshop Daniel Cremers Convex Relaxation Methods for Computer Vision 19

Action Reconstruction Daniel Cremers Convex Relaxation Methods for Computer Vision 20

Action Reconstruction Daniel Cremers Convex Relaxation Methods for Computer Vision 21

Overview Multiview reconstruction Super-res.textures 4D reconstruction Stereo reconstruction Segmentation Manifold-valued functions Daniel Cremers Convex Relaxation Methods for Computer Vision 22

From Binary to Multilabel Optimization Example: Stereo Daniel Cremers Convex Relaxation Methods for Computer Vision 23

Cartesian Currents and Relaxation nonconvex data term label regularity Pock , Schoenemann, Graber, Bischof, Cremers ECCV ’08 Daniel Cremers Convex Relaxation Methods for Computer Vision 24

Cartesian Currents and Relaxation Pock , Schoenemann, Graber, Bischof, Cremers ECCV ’08 Daniel Cremers Convex Relaxation Methods for Computer Vision 25

Cartesian Currents and Relaxation nonconvex functional Theorem: Minimizing is equivalent to minimizing convex functional Solve in relaxed space ( ) and threshold to obtain a globally optimal solution. Pock , Schoenemann, Graber, Bischof, Cremers ECCV ’08 Daniel Cremers Convex Relaxation Methods for Computer Vision 26

Global Optima for Convex Regularizers Let be continuous in and , and convex in Theorem: For any function we have: where is constrained to the convex set Pock, Cremers, Bischof, Chambolle, SIAM J. on Imaging Sciences ’10 Daniel Cremers Convex Relaxation Methods for Computer Vision 27

Global Optima for Convex Regularizers The functional can be minimized by solving the relaxed saddle point problem Theorem: The functional fulfills a generalized coarea formula: As a consequence, we have a thresholding theorem assuring that we can globally minimize the functional Pock, Cremers, Bischof, Chambolle, SIAM J. on Imaging Sciences ’10 Daniel Cremers Convex Relaxation Methods for Computer Vision 28

An Efficient Saddle Point Solver Given the saddle point problem with close convex sets and and linear operator of norm The iterative algorithm converges with rate to a saddle point for Pock, Cremers, Bischof, Chambolle, ICCV ‘09, Chambolle, Pock ‘10 Daniel Cremers Convex Relaxation Methods for Computer Vision 29

Evolution to Global Minimum Daniel Cremers Convex Relaxation Methods for Computer Vision 30

Reconstruction from Aerial Images One of two input images Depth reconstruction Courtesy of Microsoft Daniel Cremers Convex Relaxation Methods for Computer Vision 31

Reconstruction from Aerial Images Daniel Cremers Convex Relaxation Methods for Computer Vision 32

Overview Multiview reconstruction Super-res.textures 4D reconstruction Stereo reconstruction Segmentation Manifold-valued functions Daniel Cremers Convex Relaxation Methods for Computer Vision 33

The Minimal Partition Problem Potts ’52, Blake, Zisserman ’87, Mumford - Shah ’89, Vese, Chan ’02 Proposition: With , this is equivalent to where Chambolle, Cremers, Pock ’08, SIIMS ‘12, Pock et al. CVPR ’09 Daniel Cremers Convex Relaxation Methods for Computer Vision 34

Test Case: The Triple Junction Lellmann et al. ’08 Zach et al. ’08 Input image our approach Proposition: The proposed relaxation strictly dominates alternative relaxations. Chambolle, Cremers, Pock ’08, SIIMS ‘12, Pock et al. CVPR ’09 Daniel Cremers Convex Relaxation Methods for Computer Vision 35

Minimal Surfaces in 3D 3D min partition inpainting Photograph of a soap film Chambolle, Cremers, Pock ’08, SIIMS ‘12, Pock et al. CVPR ’09 Daniel Cremers Convex Relaxation Methods for Computer Vision 36

The Minimal Partition Problem Input color image 10 label segmentation Chambolle, Cremers, Pock ’08, SIIMS ‘12, Pock et al. CVPR ’09 Daniel Cremers Convex Relaxation Methods for Computer Vision 37

Segmentation with Proportion Priors Nieuwenhuis, Strekalovskiy, Cremers ICCV ’13 Daniel Cremers Convex Relaxation Methods for Computer Vision 38

Segmentation with Proportion Priors Idea: Impose a prior on the relative size of object parts Nieuwenhuis, Strekalovskiy, Cremers ICCV ’13 Daniel Cremers Convex Relaxation Methods for Computer Vision 39

Segmentation with Proportion Priors with length regularity with proportion prior Nieuwenhuis, Strekalovskiy, Cremers ICCV ’13 Daniel Cremers Convex Relaxation Methods for Computer Vision 40

Piecewise Smooth Approximation Mumford, Shah ’89 For can be written as with a convex set Alberti, Bouchitte, Dal Maso ’04 Daniel Cremers Convex Relaxation Methods for Computer Vision 41