Regularizing with BregmanMoreau envelopes Heinz H. Bauschke Minh N. - PowerPoint PPT Presentation

Moreau Envelopes Bregman Distance Bregman Envelopes Conclusion Regularizing with Bregman–Moreau envelopes Heinz H. Bauschke Minh N. Dao Scott B Lindstrom COCANA CARMA CARMA University of British University of Newcastle University of Newcastle Columbia daonminh@gmail.com scott.lindstrom@uon.edu.au heinz.bauschke@ubc.ca AMSI Optimise 2017 Revised July 4, 2017 1 / 39

Moreau Envelopes Bregman Distance Bregman Envelopes Conclusion Outline I 1 Moreau Envelopes Introduction 3 Bregman Envelopes Epigraph Intuition Introduction Classical Results Results Prox Operators Envelopes 2 Bregman Distance Prox Operators Definition 4 Conclusion Our assumptions on f Summary Three demo functions References What Changes? 2 / 39

Moreau Envelopes Introduction Bregman Distance Epigraph Intuition Bregman Envelopes Classical Results Conclusion Prox Operators Definition 1 (Moreau envelope) Moreau envelope with parameter γ ∈ R ++ y ∈ X θ ( y ) + 1 env γ 2 γ � x − y � 2 . θ : x �→ inf (1) A special case of infimal convolution: θ � f : R n → [ −∞ , ∞ ] : x �→ inf y ∈ R n ( θ ( y ) + f ( x − y )) ( “exact” if θ � f ( x ) = min y ∈ R n ( θ ( y ) + f ( x − y )) ∀ x ∈ dom θ � f Moreau only considered γ = 1 Systematic study involving γ originated with Attouch [2][3] 3 / 39

Moreau Envelopes Introduction Bregman Distance Epigraph Intuition Bregman Envelopes Classical Results Conclusion Prox Operators Epigraph Intuition Think: smoothing through epigraph addition epi( θ � f ) = epi θ + epi f 1 is always true when θ � f is exact. 1Minkowski sum 4 / 39

Moreau Envelopes Introduction Bregman Distance Epigraph Intuition Bregman Envelopes Classical Results Conclusion Prox Operators Limiting case As γ → 0 we recover θ 5 / 39

Moreau Envelopes Introduction Bregman Distance Epigraph Intuition Bregman Envelopes Classical Results Conclusion Prox Operators Limiting Case As γ → ∞ we recover min θ 6 / 39

Moreau Envelopes Introduction Bregman Distance Epigraph Intuition Bregman Envelopes Classical Results Conclusion Prox Operators Varying the Parameter Varying the parameter 7 / 39

Moreau Envelopes Introduction Bregman Distance Epigraph Intuition Bregman Envelopes Classical Results Conclusion Prox Operators Prox Operators Throughout: θ is a lower semicontinuous convex function of Legendre type Definition 2 (Prox Operator) Prox γθ ( x ) is the unique point satisfying � θ ( y ) + 1 � 2 γ � x − y � 2 env γθ ( x ) = min y ∈ R n = θ (Prox γθ ( x )) + 1 2 γ � x − Prox γθ ( x ) � 2 8 / 39

Moreau Envelopes Introduction Bregman Distance Epigraph Intuition Bregman Envelopes Classical Results Conclusion Prox Operators Prox Operators: Geometric Intuition Figure: Where θ = | y + x − 1 | and Figure: The net Prox γθ (1 , 2) where γ = 1 / 2 γ ∈ ]0 , ∞ [ 9 / 39

Moreau Envelopes Introduction Bregman Distance Epigraph Intuition Bregman Envelopes Classical Results Conclusion Prox Operators Prox Operators: Limiting Cases Figure: lim γ →∞ Prox γθ ( x ) = P argmin θ ( x ) 10 / 39

Moreau Envelopes Introduction Bregman Distance Epigraph Intuition Bregman Envelopes Classical Results Conclusion Prox Operators Prox Operators: Limiting Cases Figure: lim γ → 0 Prox γθ ( x ) = x 11 / 39

Moreau Envelopes Introduction Bregman Distance Epigraph Intuition Bregman Envelopes Classical Results Conclusion Prox Operators Special case: projection Remark 1 (Special case: projection) When θ = ι C is the indicator of C Prox γθ ( x ) = P C ( x ) is the projection operator. 12 / 39

Moreau Envelopes Definition Bregman Distance Our assumptions on f Bregman Envelopes Three demo functions Conclusion What Changes? Bregman Distance Definition 3 (Bregman Distance) The Bregman Distance of a function f between two points x , y is D f ( x , y ) = f ( x ) − f ( y ) − �∇ f ( y ) , x − y � y 0 x f ( x ) D f ( x , y ) �∇ f ( y ) , x − y � f ( y ) Figure: Bregman distance where the function f is the Boltzmann-Shannon Entropy x �→ x log( x ) − x 13 / 39

Moreau Envelopes Definition Bregman Distance Our assumptions on f Bregman Envelopes Three demo functions Conclusion What Changes? Our assumptions on f We assume: 1 f is a lower semicontinuous convex function of Legendre type and U := int dom f . 2 ∇ 2 f exists and is continuous on U ; 3 D f is jointly convex , i.e., convex on X × X ; 4 ( ∀ x ∈ U ) D f ( x , · ) is strictly convex on U ; 5 ( ∀ x ∈ U ) D f ( x , · ) is coercive, i.e., D f ( x , y ) → + ∞ as � y � → + ∞ . 14 / 39

Moreau Envelopes Definition Bregman Distance Our assumptions on f Bregman Envelopes Three demo functions Conclusion What Changes? Our demo functions Where x , y ∈ R J : Energy: If f : x �→ 1 2 � x � 2 , then 1 D f ( x , y ) = 1 2 � x − y � 2 J Boltzmann–Shannon 2 entropy: If f : x �→ � x j ln( x j ) − x j , then one obtains the 2 j =1 Kullback–Leibler divergence �� J j =1 x j ln( x j / y j ) − x j + y j , if x ≥ 0 and y > 0; D f ( x , y ) = + ∞ , otherwise. J � Fermi–Dirac entropy: If f : x �→ x j ln( x j ) + (1 − x j ) ln(1 − x j ), then 3 j =1 �� J j =1 x j ln( x j / y j ) + (1 − x j ) log � (1 − x j ) / (1 − y j ) � , if 0 ≤ x ≤ 1 and 0 < y < 1; D f ( x , y ) = + ∞ , otherwise. 2With Boltzmann–Shannon entropy and Fermi–Dirac entropy, we use convention 0 · ln(0) := 0. 15 / 39

Moreau Envelopes Definition Bregman Distance Our assumptions on f Bregman Envelopes Three demo functions Conclusion What Changes? What Changes? We lose triangle Inequality. y 0 z x f ( x ) D f ( x , z ) D f ( x , y ) f ( z ) D f ( z , y ) f ( y ) Figure: Where f is the Boltzmann-Shannon Entropy, D f ( x , y ) > D f ( z , y ) + D f ( x , z ). 16 / 39

Moreau Envelopes Definition Bregman Distance Our assumptions on f Bregman Envelopes Three demo functions Conclusion What Changes? What Changes? We also lose symmetry... and translation invariance. 17 / 39

Moreau Envelopes Definition Bregman Distance Our assumptions on f Bregman Envelopes Three demo functions Conclusion What Changes? What Changes? Except when using the energy, of course. 18 / 39

Moreau Envelopes Introduction Bregman Distance Results Bregman Envelopes Envelopes Conclusion Prox Operators Bregman Envelopes Definition 4 For a given θ, f where int dom f ∩ dom θ � = ∅ : The left Bregman envelope is x ∈ X θ ( x ) + 1 ← − env γ θ : X → [ −∞ , + ∞ ] : y �→ inf γ D f ( x , y ) (2) The right Bregman envelope is y ∈ X θ ( y ) + 1 − → env γ θ : X → [ −∞ , + ∞ ] : x �→ inf γ D f ( x , y ) , (3) If f = 1 2 � · � 2 , then D f : ( x , y ) �→ 1 2 � x − y � 2 , and ← − θ = − → env γ env γ θ = θ � ( 1 2 γ � · � 2 ) is the classical Moreau envelope of θ of parameter γ . 19 / 39

Moreau Envelopes Introduction Bregman Distance Results Bregman Envelopes Envelopes Conclusion Prox Operators Bregman Envelopes Consider the following properties: (a) U ∩ dom θ is bounded. (b) inf θ ( U ) > −∞ . (c) f is supercoercive, i.e., f ( x ) / � x � → + ∞ as � x � → + ∞ . (d) ( ∀ x ∈ U ) D f ( x , · ) is supercoercive. Then the following hold (and we suppose them moving forward): If any of (a), (b), or (c) holds, then θ ( · ) + 1 ( ∀ y ∈ U ) γ D f ( · , y ) is coercive If any of (a), (b), or (d) holds, then θ ( · ) + 1 ( ∀ x ∈ U ) γ D f ( x , · ) is coercive. 20 / 39

Moreau Envelopes Introduction Bregman Distance Results Bregman Envelopes Envelopes Conclusion Prox Operators Bregman Proximity Operators Definition 5 (Bregman Proximity Operators) For a given θ, f where int dom f ∩ dom θ � = ∅ : 1 The left prox operator is ← − θ ( x ) + 1 P θ : int dom f → int dom f : y �→ argmin γ D f ( x , y ) . x ∈ X 2 The right prox operator is − → θ ( y ) + 1 P θ : int dom f → int dom f : x �→ argmin γ D f ( x , y ) . y ∈ X 21 / 39

Moreau Envelopes Introduction Bregman Distance Results Bregman Envelopes Envelopes Conclusion Prox Operators Epigraphs Shown: left envelope with Boltzman- Shannon entropy Still regularizes Addition changes 22 / 39

Moreau Envelopes Introduction Bregman Distance Results Bregman Envelopes Envelopes Conclusion Prox Operators Epigraphs Shown: right envelope with Boltzman- Shannon entropy Think: what about limiting cases? 23 / 39

Moreau Envelopes Introduction Bregman Distance Results Bregman Envelopes Envelopes Conclusion Prox Operators Results With x , y ∈ int dom f the following hold: As γ ↓ 0: θ ( y ) ↑ θ ( y ), θ ( ← − Left case: ← − env γ P γθ ( y )) ↑ θ ( y ), 1 γ D f ( ← − P γθ ( y ) , y ) → 0, and ← − 1 P γθ ( y ) → y . θ ( x ) ↑ θ ( x ), θ ( − → − → env γ Right case: P γθ ( x )) ↑ θ ( x ), 2 γ D f ( x , − → P γθ ( x )) → 0, and − → 1 P γθ ( x ) → x . As γ ↑ ∞ : Left case: ← − env γ θ ( y ) ↓ inf θ ( X ) and if argmin θ ⊆ int dom f , 1 then ← − P γθ ( y ) → ← − P argmin θ y as γ ↑ + ∞ . Right case: − → env γ θ ( x ) ↓ inf θ ( X ) and if argmin θ ⊆ int dom f , 2 then − → P γθ ( x ) → − → P argmin θ x as γ ↑ + ∞ . 24 / 39