Convex Relaxation Methods for Computer Vision Daniel Cremers - - PowerPoint PPT Presentation

Daniel Cremers Computer Science Department TU Munich

Convex Relaxation Methods for Computer Vision

with Kalin Kolev, Evgeny Strekalovskiy, Thomas Pock, Bastian Goldlücke, Antonin Chambolle & Jan Lellmann

2

Daniel Cremers Convex Relaxation Methods for Computer Vision

3D Reconstruction from Multiple Views

3

Daniel Cremers Convex Relaxation Methods for Computer Vision

Image segmentation:

Optimization in Computer Vision

Geman, Geman ’84, Blake, Zisserman ‘87, Kass et al. ’88, Mumford, Shah ’89, Caselles et al. ‘95, Kichenassamy et al. ‘95, Paragios, Deriche ’99, Chan, Vese ‘01, Tsai et al. ‘01, … Multiview stereo reconstruction: Faugeras, Keriven ’98, Duan et al. ‘04, Yezzi, Soatto ‘03, Seitz et al. ‘06, Hernandez et al. ‘07, Labatut et al. ’07, … Optical flow estimation: Horn, Schunck ‘81, Nagel, Enkelmann ‘86, Black, Anandan ‘93, Alvarez et al. ‘99, Brox et al. ‘04, Baker et al. ‘07, Zach et al. ‘07, Sun et al. ‘08, Wedel et al. ’09, …

Non-convex energies

4

Daniel Cremers Convex Relaxation Methods for Computer Vision

Non-convex energy Convex energy

Non-convex versus Convex Energies

Some related work: Brakke ‘95, Alberti et al. ‘01, Chambolle ‘01, Attouch et al. ‘06, Nikolova et al. ‘06, Cremers et al. ‘06, Bresson et al. ‘07, Lellmann et al. ‘08, Zach et al. ‘08, Chambolle et al. ’08, Pock et al. ‘09, Zach et al. ’09, Brown et al. ’10, Bae et al. ‘10, Yuan et al. ‘10,…

5

Daniel Cremers Convex Relaxation Methods for Computer Vision

Overview

Multiview reconstruction Stereo reconstruction Super-res.textures Manifold-valued functions Segmentation 4D reconstruction

6

Daniel Cremers Convex Relaxation Methods for Computer Vision

Overview

Multiview reconstruction Stereo reconstruction Super-res.textures Manifold-valued functions Segmentation 4D reconstruction

7

Daniel Cremers Convex Relaxation Methods for Computer Vision

Optimal solution is the empty set: Resort: Local optimization: Faugeras, Keriven TIP ’98 Generative object/background modeling: Yezzi, Soatto ’03,… Constrain search space: Vogiatsis, Torr, Cipolla CVPR ’05 Intelligent ballooning: Boykov, Lempitsky BMVC ’06 Segmentation: Kichenassamy et al. ’95, Caselles et al. ’95 3D Reconstruction: Faugeras, Keriven ’98, Duan et al. ’04

Stereo-weighted Minimal Surfaces

8

Daniel Cremers Convex Relaxation Methods for Computer Vision

Silhouette Consistent Reconstructions

Kolev et al., IJCV 2009, Cremers, Kolev, PAMI 2011

9

Daniel Cremers Convex Relaxation Methods for Computer Vision

Proposition: The set of silhouette-consistent solutions is convex.

Σ=

Silhouette Consistent Reconstructions

Kolev et al., IJCV 2009, Cremers, Kolev, PAMI 2011

10

Daniel Cremers Convex Relaxation Methods for Computer Vision

Image data courtesy of Yasutaka Furukawa.

Reconstruction of Fine-scale Structures

11

Daniel Cremers Convex Relaxation Methods for Computer Vision

Overview

Multiview reconstruction Stereo reconstruction Super-res.textures Manifold-valued functions Segmentation 4D reconstruction

12

Daniel Cremers Convex Relaxation Methods for Computer Vision

Surface Evolution to Optimum

13

Daniel Cremers Convex Relaxation Methods for Computer Vision

Super-Resolution Texture Map

Given all images determine the surface color back-projection blur & downsample * Best Paper Award Goldlücke, Cremers, ICCV ’09, DAGM ’09*, IJCV ‘13

14

Daniel Cremers Convex Relaxation Methods for Computer Vision

Goldlücke, Cremers, ICCV ’09, DAGM ’09*, IJCV ‘13

Super-Resolution Texture Map

* Best Paper Award

15

Daniel Cremers Convex Relaxation Methods for Computer Vision

Closeup of input image Super-resolution texture

Super-Resolution Texture Map

Goldlücke, Cremers, ICCV ’09, DAGM ’09*, IJCV ‘13 * Best Paper Award

16

Daniel Cremers Convex Relaxation Methods for Computer Vision

Kolev, Cremers, ECCV ’08, PAMI 2011

Reconstructing the Niobids Statues

17

Daniel Cremers Convex Relaxation Methods for Computer Vision

Overview

Multiview reconstruction Stereo reconstruction Super-res.textures Manifold-valued functions Segmentation 4D reconstruction

18

Daniel Cremers Convex Relaxation Methods for Computer Vision

Action Reconstruction

Oswald, Cremers, ICCV ‘13 4DMoD Workshop

19

Daniel Cremers Convex Relaxation Methods for Computer Vision

Action Reconstruction

Oswald, Cremers, ICCV ‘13 4DMoD Workshop

20

Daniel Cremers Convex Relaxation Methods for Computer Vision

Action Reconstruction

21

Daniel Cremers Convex Relaxation Methods for Computer Vision

Action Reconstruction

22

Daniel Cremers Convex Relaxation Methods for Computer Vision

Overview

Multiview reconstruction Stereo reconstruction Super-res.textures Manifold-valued functions Segmentation 4D reconstruction

23

Daniel Cremers Convex Relaxation Methods for Computer Vision

Example: Stereo

From Binary to Multilabel Optimization

24

Daniel Cremers Convex Relaxation Methods for Computer Vision

Cartesian Currents and Relaxation

nonconvex data term label regularity

Pock , Schoenemann, Graber, Bischof, Cremers ECCV ’08

25

Daniel Cremers Convex Relaxation Methods for Computer Vision

Cartesian Currents and Relaxation

Pock , Schoenemann, Graber, Bischof, Cremers ECCV ’08

26

Daniel Cremers Convex Relaxation Methods for Computer Vision

convex functional Solve in relaxed space ( ) and threshold to obtain a globally optimal solution. Theorem: Minimizing is equivalent to minimizing

Cartesian Currents and Relaxation

nonconvex functional Pock , Schoenemann, Graber, Bischof, Cremers ECCV ’08

27

Daniel Cremers Convex Relaxation Methods for Computer Vision

Pock, Cremers, Bischof, Chambolle, SIAM J. on Imaging Sciences ’10 Let be continuous in and , and convex in Theorem: For any function we have: where is constrained to the convex set

Global Optima for Convex Regularizers

28

Daniel Cremers Convex Relaxation Methods for Computer Vision

Pock, Cremers, Bischof, Chambolle, SIAM J. on Imaging Sciences ’10 The functional can be minimized by solving the relaxed saddle point problem Theorem: The functional fulfills a generalized coarea formula: As a consequence, we have a thresholding theorem assuring that we can globally minimize the functional

Global Optima for Convex Regularizers

29

Daniel Cremers Convex Relaxation Methods for Computer Vision

Pock, Cremers, Bischof, Chambolle, ICCV ‘09, Chambolle, Pock ‘10 Given the saddle point problem with close convex sets and and linear operator of norm

An Efficient Saddle Point Solver

converges with rate to a saddle point for

The iterative algorithm

30

Daniel Cremers Convex Relaxation Methods for Computer Vision

Evolution to Global Minimum

31

Daniel Cremers Convex Relaxation Methods for Computer Vision

One of two input images Depth reconstruction Courtesy of Microsoft

Reconstruction from Aerial Images

32

Daniel Cremers Convex Relaxation Methods for Computer Vision

Reconstruction from Aerial Images

33

Daniel Cremers Convex Relaxation Methods for Computer Vision

Overview

Multiview reconstruction Stereo reconstruction Super-res.textures Manifold-valued functions Segmentation 4D reconstruction

34

Daniel Cremers Convex Relaxation Methods for Computer Vision

Potts ’52, Blake, Zisserman ’87, Mumford-Shah ’89, Vese, Chan ’02 Proposition: With , this is equivalent to

The Minimal Partition Problem

Chambolle, Cremers, Pock ’08, SIIMS ‘12, Pock et al. CVPR ’09 where

35

Daniel Cremers Convex Relaxation Methods for Computer Vision

Input image Lellmann et al. ’08 Zach et al. ’08

• ur approach

Proposition: The proposed relaxation strictly dominates alternative relaxations.

Test Case: The Triple Junction

Chambolle, Cremers, Pock ’08, SIIMS ‘12, Pock et al. CVPR ’09

36

Daniel Cremers Convex Relaxation Methods for Computer Vision

3D min partition inpainting Photograph of a soap film

Minimal Surfaces in 3D

Chambolle, Cremers, Pock ’08, SIIMS ‘12, Pock et al. CVPR ’09

37

Daniel Cremers Convex Relaxation Methods for Computer Vision

Input color image 10 label segmentation

The Minimal Partition Problem

Chambolle, Cremers, Pock ’08, SIIMS ‘12, Pock et al. CVPR ’09

38

Daniel Cremers Convex Relaxation Methods for Computer Vision

Segmentation with Proportion Priors

Nieuwenhuis, Strekalovskiy, Cremers ICCV ’13

39

Daniel Cremers Convex Relaxation Methods for Computer Vision

Segmentation with Proportion Priors

Nieuwenhuis, Strekalovskiy, Cremers ICCV ’13

Idea: Impose a prior on the relative size of object parts

40

Daniel Cremers Convex Relaxation Methods for Computer Vision

Segmentation with Proportion Priors

Nieuwenhuis, Strekalovskiy, Cremers ICCV ’13

with length regularity with proportion prior

41

Daniel Cremers Convex Relaxation Methods for Computer Vision

For can be written as with a convex set

Mumford, Shah ’89

Piecewise Smooth Approximation

Alberti, Bouchitte, Dal Maso ’04

42

Daniel Cremers Convex Relaxation Methods for Computer Vision

piecewise constant piecewise smooth Input image

Piecewise Smooth Approximation

Pock, Cremers, Bischof, Chambolle ICCV ’09

43

Daniel Cremers Convex Relaxation Methods for Computer Vision

inpainted crack tip surface plot fixed boundary values

The Crack Tip & Open Boundaries

Pock, Cremers, Bischof, Chambolle ICCV ’09

44

Daniel Cremers Convex Relaxation Methods for Computer Vision

Strekalovskiy, Chambolle, Cremers, CVPR ‘12

The Vectorial Mumford-Shah Problem

with the convex set: For , we consider the functional Proposition: For , we have:

45

Daniel Cremers Convex Relaxation Methods for Computer Vision

TV denoised Vectorial Mumford-Shah Input image

The Vectorial Mumford-Shah Problem

Strekalovskiy, Chambolle, Cremers, CVPR ‘12

46

Daniel Cremers Convex Relaxation Methods for Computer Vision

Channelwise MS Vectorial MS Input image

Channelwise versus Vectorial

Jump set Jump set

47

Daniel Cremers Convex Relaxation Methods for Computer Vision

Channelwise MS Vectorial MS Input image

Channelwise versus Vectorial

Strekalovskiy, Chambolle, Cremers, CVPR ‘12

48

Daniel Cremers Convex Relaxation Methods for Computer Vision

Overview

Multiview reconstruction Stereo reconstruction Super-res.textures Manifold-valued functions Segmentation 4D reconstruction

49

Daniel Cremers Convex Relaxation Methods for Computer Vision

color image processing

Functions with Values in a Manifold

• ptical flow estimation

normal field inpainting Cremers, Strekalovskiy, Siims ‘12 Lellmann, Strekalovskiy, Kötter, Cremers, ICCV ‘13

50

Daniel Cremers Convex Relaxation Methods for Computer Vision

Vectorial Total Variation ( )

Separate directions, uncoupled (Blomgren, Chan, TIP '98): Separate directions, coupled (Sapiro, Ringach, TIP '96): Shared direction, coupled (Goldlücke et al., SIIMS '12):

51

Daniel Cremers Convex Relaxation Methods for Computer Vision

with a Riemannian manifold . Cremers, Strekalovskiy, Siims ‘12 Lellmann, Strekalovskiy, Kötter, Cremers, ICCV ‘13

Total Variation for Functions with Values in a Manifold

Consider the problem geodesic distance

• n the manifold

52

Daniel Cremers Convex Relaxation Methods for Computer Vision

Lellmann, Strekalovskiy, Kötter, Cremers, ICCV ‘13

Total Variation for Functions with Values in a Manifold

Continuous labeling problem with all points of : Proposition: The pairwise constraints are equivalent to with spectral norm linear number of constraints, respects manifold structure

53

Daniel Cremers Convex Relaxation Methods for Computer Vision

Total Variation for Functions with Values in a Manifold

no orientation bias sub-label accuracy / no grid bias Lellmann, Strekalovskiy, Kötter, Cremers, ICCV ‘13

54

Daniel Cremers Convex Relaxation Methods for Computer Vision

Total Variation for Functions with Values in a Manifold

flow with finite labeling flow with continuous labeling Lellmann, Strekalovskiy, Kötter, Cremers, ICCV ‘13

55

Daniel Cremers Convex Relaxation Methods for Computer Vision

Total Variation for Functions with Values in a Manifold

noisy normal field

• denoised

Lellmann, Strekalovskiy, Kötter, Cremers, ICCV ‘13

56

Daniel Cremers Convex Relaxation Methods for Computer Vision

Total Variation for Functions with Values in a Manifold

shading with noisy normal field shading with denoised normals Lellmann, Strekalovskiy, Kötter, Cremers, ICCV ‘13

57

Daniel Cremers Convex Relaxation Methods for Computer Vision

Total Variation for Functions with Values in a Manifold

normals on the boundary

• inpainted normal field

Lellmann, Strekalovskiy, Kötter, Cremers, ICCV ‘13

58

Daniel Cremers Convex Relaxation Methods for Computer Vision

Conclusion

We can express image analysis problems in terms

• f convex functionals.

We can minimize these functionals using provably convergent primal-dual algorithms. We can define relaxations for functions with values in a manifold using continuous labeling. Solutions are independent of initialization and either optimal or within a bound of the optimum.