Convex Functions (I) Lijun Zhang zlj@nju.edu.cn http://cs.nju.edu.cn/zlj
Outline Basic Properties Definition First-order Conditions, Second-order Conditions Jensen’s inequality and extensions Epigraph Operations That Preserve Convexity Nonnegative Weighted Sums Composition with an affine mapping Pointwise maximum and supremum Composition Minimization Perspective of a function Summary
Outline Basic Properties Definition First-order Conditions, Second-order Conditions Jensen’s inequality and extensions Epigraph Operations That Preserve Convexity Nonnegative Weighted Sums Composition with an affine mapping Pointwise maximum and supremum Composition Minimization Perspective of a function Summary
is convex if Convex Function is convex �
Convex Function � is convex if is convex � is strictly convex if
Convex Function � is convex if is convex is concave if is convex is convex Affine functions are both convex and concave, and vice versa.
Extended-value Extensions The extended-value extension of is � Example � � � � � � � �
Extended-value Extensions The extended-value extension of is � Example Indicator Function of a Set �
Zeroth-order Condition Definition High-dimensional space A function is convex if and only if it is convex when restricted to any line that intersects its domain. � , is convex is convex One-dimensional space
First-order Conditions is differentiable. Then is convex if and only if is convex For all 𝑔 𝑧 � 𝑔 𝑦 � 𝛼𝑔 𝑦 � �𝑧 � 𝑦� First-order Taylor approximation
First-order Conditions is differentiable. Then is convex if and only if is convex For all 𝑔 𝑧 � 𝑔 𝑦 � 𝛼𝑔 𝑦 � �𝑧 � 𝑦� Local Information Global Information is strictly convex if and only if 𝑔 𝑧 � 𝑔 𝑦 � 𝛼𝑔 𝑦 � �𝑧 � 𝑦�
Proof is convex � Necessary condition: 𝑔 𝑦 � 𝑢 𝑧 � 𝑦 � 1 � 𝑢 𝑔 𝑦 � 𝑢𝑔 𝑧 , 0 � 𝑢 � 1 � ��� ��� �� � ⇒ 𝑔 𝑧 � 𝑔 𝑦 � � �→� 𝑔 𝑧 � 𝑔 𝑦 � 𝑔 � 𝑦 𝑧 � 𝑦 Sufficient condition: 𝑨 � 𝜄𝑦 � 1 � 𝜄 𝑧 � ⇒ 𝑔 𝑦 � 𝑔 𝑨 � �1 � 𝜄�𝑔 � 𝑨 𝑦 � 𝑧 𝑔 𝑦 � 𝑔 𝑨 � 𝑔 � 𝑨 𝑦 � 𝑨 � 𝑔 𝑧 � 𝑔 𝑨 � 𝜄𝑔 � 𝑨 𝑦 � 𝑧 𝑔 𝑧 � 𝑔 𝑨 � 𝑔 � 𝑨 𝑧 � 𝑨 ⇒ 𝜄𝑔 𝑦 � 1 � 𝜄 𝑔 𝑧 � 𝑔 𝑨 ⇒ 𝑔 𝜄𝑦 � 1 � 𝜄 𝑧 � 𝜄𝑔 𝑦 � 1 � 𝜄 𝑔 𝑧
Proof is convex � � is convex � 𝑢 � 𝑔 𝑢𝑧 � 1 � 𝑢 𝑦 , ′ 𝑢 � 𝛼𝑔 𝑢𝑧 � 1 � 𝑢 𝑦 � 𝑧 � 𝑦 𝑔 is convex ⇒ is convex ⇒ 1 � 0 � � 0 ⇒ 𝑔 𝑧 � 𝑔 𝑦 � 𝛼𝑔 𝑦 � 𝑧 � 𝑦 𝑔 is is First-order First-order condition of condition of 𝑔 convex convex
Proof is convex � � is convex � ′ 𝑢 � 𝛼𝑔 𝑢𝑧 � 1 � 𝑢 𝑦 � 𝑧 � 𝑦 𝑢 � 𝑔 𝑢𝑧 � 1 � 𝑢 𝑦 , 𝑔 𝑢𝑧 � 1 � 𝑢 𝑦 � 𝑔 𝑢̃𝑧 � 1 � 𝑢̃ 𝑦 �𝛼𝑔 𝑢̃𝑧 � 1 � 𝑢̃ 𝑦 � 𝑧 � 𝑦 𝑢 � 𝑢̃ ⇒ 𝑢 � 𝑢̃ � � 𝑢̃ 𝑢 � 𝑢̃ ⇒ 𝑢 is convex ⇒ 𝑔 is convex 𝑔 is is First-order First-order condition of condition of 𝑔 convex convex
Second-order Conditions is twice differentiable. Then is convex if and only if is convex � For all , Attention � is strictly convex � is strict convex � is strict convex but � is convex is necessary,
Examples Functions on �� is convex on , � is convex on �� when or , and concave for � , for , is convex on is concave on �� Negative entropy is convex on ��
Examples � Functions on � is convex Every norm on � � � � Quadratic-over-linear: � dom 𝑔� �𝑦, 𝑧� ∈ 𝐒 � 𝑧 � 0� � � � � � � � � � �/� is concave on � � �� ��� � is concave on ��
Examples � Functions on � is convex Every norm on 𝑔�𝑦� is a norm on 𝐒 � 𝑔 𝜄𝑦 � 1 � 𝜄 𝑧 � 𝑔 𝜄𝑦 � 𝑔 1 � 𝜄 𝑧 � 𝜄𝑔 𝑦 � 1 � 𝜄 𝑔�𝑧� � � � � 𝑔 𝜄𝑦 � 1 � 𝜄 𝑧 � max � � 𝜄𝑦 � � 1 � 𝜄 𝑧 � � � 𝜄max � �𝑦 � � � 1 � 𝜄 max � �𝑧 � �
Examples � Functions on � � � � 𝑧 � �𝑦𝑧 𝑧 𝑧 � 𝛼 � 𝑔 𝑦, 𝑧 � � � � ≽ 0 �𝑦 �𝑦 𝑦 � � � � � �𝑦𝑧
� � � Functions on � � Examples
Examples � Functions on � � � � � 𝛼 � 𝑔 𝑦 � 𝟐 � 𝑨 diag 𝑨 � 𝑨𝑨 � 𝟐 � � � 𝑨 � 𝑓 � � , … 𝑓 � � � � � � 𝑨 � 𝑤 � 𝛼 � 𝑔 𝑦 𝑤 � 𝟐 � � � � ∑ ∑ 𝑨 � 𝑤 � � ��� ��� � � � � 0 ∑ 𝑤 � 𝑨 � ��� Cauchy-Schwarz inequality: �𝑏 � 𝑏��𝑐 � 𝑐� � 𝑏 � 𝑐 �
Examples � Functions on � is concave on �� 𝑢 � 𝑔 𝑎 � 𝑢𝑊 , 𝑎 � 𝑢𝑊 ≻ 0, 𝑎 ≻ 0 𝑢 � log det�𝑎 � 𝑢𝑊� � � 𝐽 � 𝑢𝑎 � � � 𝑊𝑎 � � � � 𝑎 � log det 𝑎 � � � ∑ log�1 � 𝑢𝜇 � � � log det 𝑎 ��� 𝜇 � , … 𝜇 � are the eigenvalues of 𝑎 � � � 𝑊𝑎 � � � � � 𝑢 � ∑ , �� 𝑢 � � ∑ � � � � � � ��� ��� ���� � � ���� � det 𝐵𝐶 � det 𝐵 det�𝐶� https: / / en.wikipedia.org/ wiki/ Determinant
Sublevel Sets -sublevel set 𝐷 � � 𝑦 ∈ dom 𝑔 𝑔�𝑦� � 𝛽� is convex � is convex � is convex is convex � -superlevel set 𝐷 � � 𝑦 ∈ dom 𝑔 𝑔 𝑦 � 𝛽� is concave convex � � � � � � � � ��� ��� � � is convex �
Epigraph � Graph of function � Epigraph of function
Epigraph � Epigraph of function Hypograph Conditions is convex is convex is concave is convex
Example Matrix Fractional Function 𝑔 𝑦, 𝑍 � 𝑦 � 𝑍 �� 𝑦, dom 𝑔 � 𝐒 � � 𝐓 �� � � Quadratic-over-linear: � �� � Schur complement condition is convex Recall Example 2.10 in the book
Application of Epigraph First order Condition 𝑔 𝑧 � 𝑔 𝑦 � 𝛼𝑔 𝑦 � �𝑧 � 𝑦� 𝑧, 𝑢 ∈ epi 𝑔 ⇒ 𝑢 � 𝑔 𝑧 � 𝑔 𝑦 � 𝛼𝑔 𝑦 � �𝑧 � 𝑦�
Application of Epigraph First order Condition 𝑔 𝑧 � 𝑔 𝑦 � 𝛼𝑔 𝑦 � �𝑧 � 𝑦� 𝑧, 𝑢 ∈ epi 𝑔 ⇒ 𝑢 � 𝑔 𝑦 � 𝛼𝑔 𝑦 � �𝑧 � 𝑦� � 𝑦 𝑧 𝑧, 𝑢 ∈ epi 𝑔 ⇒ 𝛼𝑔 𝑦 𝑢 � � 0 𝑔 𝑦 �1
Jensen’s Inequality Basic inequality points � � � � � � � � � � �
Jensen’s Inequality Infinite points � � � 𝑔 𝑦 � 𝐅𝑔�𝑦 � 𝑨�, 𝑨 is a zero-mean noisy Hölder’s inequality � � � � � � � � � � � � � � � � � ��� ��� ���
Outline Basic Properties Definition First-order Conditions, Second-order Conditions Jensen’s inequality and extensions Epigraph Operations That Preserve Convexity Nonnegative Weighted Sums Composition with an affine mapping Pointwise maximum and supremum Composition Minimization Perspective of a function Summary
Nonnegative Weighted Sums Finite sums 𝑥 � � 0, 𝑔 � is convex 𝑔 � 𝑥 � 𝑔 � � ⋯ � 𝑥 � 𝑔 � is convex Infinite sums 𝑔�𝑦, 𝑧� is convex in 𝑦, ∀𝑧 ∈ , 𝑥 𝑧 � 0 𝑦 � � 𝑔 𝑦, 𝑧 𝑥�𝑧� 𝑒𝑧 is convex Epigraph interpretation 𝐟𝐪𝐣 𝑥𝑔 � ��𝑦, 𝑢�|𝑥𝑔�𝑦� � 𝑢� 𝐽 0 𝑥 𝐟𝐪𝐣 𝑔 � ��𝑦, 𝑥𝑢�|𝑔 𝑦 � 𝑢� 0 𝐟𝐪𝐣 𝑥𝑔 � 𝐽 0 𝑥 𝐟𝐪𝐣 𝑔 0
Composition with an affine mapping � ��� � Affine Mapping 𝑦 � 𝑔�𝐵𝑦 � 𝑐� If is convex, so is If is concave, so is
Recommend
More recommend
