Lecture 3: Convex Function CK Cheng Dept. of Computer Science and - - PowerPoint PPT Presentation

1

CSE203B Convex Optimization: Lecture 3: Convex Function

CK Cheng

• Dept. of Computer Science and Engineering

University of California, San Diego

Outlines

• 1. Definitions: Convexity, Examples & Views
• 2. Conditions of Optimality
• 1. First Order Condition
• 2. Second Order Condition
• 3. Operations that Preserve the Convexity
• 1. Pointwise Maximum
• 2. Partial Minimization
• 4. Conjugate Function
• 5. Log-Concave, Log-Convex Functions

2

Outlines

• 1. Definitions
• 1. Convex Function vs Convex Set
• 2. Examples

1. Norm 2. Entropy 3. Affine 4. Determinant 5. Maximum

• 3. Views of Functions and Related Hyperplanes

3

• 1. Definitions: Convex Function vs Convex Set

4

Theorem: Given 𝑇 = 𝑦 𝑔 𝑦 ≤ 𝑐 If function 𝑔 𝑦 is convex, then 𝑇 is a convex set. Proof: We prove by the definition of convex set. For every 𝑣, 𝑤 ∈ 𝑇, i. e. 𝑔 𝑣 ≤ 𝑐, 𝑔 𝑤 ≤ 𝑐, We want to show that α𝑣 + 𝛾𝑤 ∈ 𝑇, ∀α + 𝛾 = 1, 𝛽, 𝛾 ≥ 0. We have 𝑔 𝛽𝑣 + 𝛾𝑤 ≤ 𝛽𝑔 𝑣 + 𝛾𝑔 𝑤 (𝑔 𝑗𝑡 𝑑𝑝𝑜𝑤𝑓𝑦) ≤ 𝛽𝑐 + 𝛾𝑐 (𝛽, 𝛾 ≥ 0) = 𝛽 + 𝛾 ∙ 𝑐 = 𝑐 (𝛽 + 𝛾 = 1) Thus α𝑣 + 𝛾𝑤 ∈ 𝑇 Remark: Convex function => Convex Set 𝑔(𝑦) ≤ 𝑐 => Convex Set 𝑔(𝑦) ≥ 𝑐 => ?

• 1. Convex Function Definitions: Examples

5

𝑔: 𝑆𝑜 → 𝑆 is convex if 𝑒𝑝𝑛 𝑔 is a convex set and 𝑔 𝜄𝑦 + 1 − 𝜄 𝑧 ≤ 𝜄𝑔 𝑦 + 1 − 𝜄 𝑔(𝑧) ∀𝑦, 𝑧 ∈ 𝑒𝑝𝑛 𝑔, 0 ≤ 𝜄 ≤ 1 Example on R: Convex Functions Affine: 𝑏𝑦 + 𝑐 𝑝𝑜 𝑆 for any 𝑏, 𝑐 ∈ 𝑆 Exponential: 𝑓𝑏𝑦 for any 𝑏 ∈ 𝑆 Power: 𝑦𝛽 𝑝𝑜 𝑆++ for 𝛽 ≥ 1 or 𝛽 ≤ 0 𝑦 𝑞 𝑝𝑜 𝑆 for 𝑞 ≥ 1 Concave Functions Affine: 𝑏𝑦 + 𝑐 𝑝𝑜 𝑆 for any 𝑏, 𝑐 ∈ 𝑆 Power: 𝑦𝛽 𝑝𝑜 𝑆++ for 0 ≤ 𝛽 ≤ 1 Logarithm: 𝑚𝑝𝑕𝑦 𝑝𝑜 𝑆++

6

Example on 𝑆𝑜: Affine: 𝑔 𝑦 = 𝑏𝑈𝑦 + 𝑐 Norms: 𝑦 𝑞 = σ𝑗=1

𝑜

𝑦 𝑞

ൗ

1 𝑞 𝑔𝑝𝑠 𝑞 ≥ 1;

𝑦 ∞ = max

𝑙

|𝑦𝑙| Example on 𝑆𝑛×𝑜: Affine: 𝑔 𝑌 = 𝑢𝑠 𝐵𝑈𝑌 = σ𝑗=1

𝑛 σ𝑘=1 𝑜

𝐵𝑗𝑘𝑦𝑗𝑘 Spectral (max singular value): 𝑔 𝑌 = 𝑌 2 = 𝜏𝑛𝑏𝑦 𝑌 = (𝜇𝑛𝑏𝑦 𝑌𝑈𝑌 ) Τ

1 2

• 1. Convex Function Definitions: Examples

7

Concave Functions: Log Determinant: 𝑔 𝑌 = log det 𝑌 , 𝑒𝑝𝑛 𝑔 = S++

𝑜

Proof: Let 𝑕 𝑢 = 𝑔 𝑌 + 𝑢𝑊 𝑊 ∈ 𝑇𝑜 𝑕 𝑢 = 𝑚𝑝𝑕 𝑒𝑓𝑢 (𝑌 + 𝑢𝑊) = 𝑚𝑝𝑕 𝑒𝑓𝑢 𝑌 + 𝑚𝑝𝑕𝑒𝑓𝑢(𝐽 + 𝑢𝑌−1

2𝑊𝑌−1 2)

= 𝑚𝑝𝑕 𝑒𝑓𝑢 𝑌 + σ𝑗=1

𝑜

𝑚𝑝𝑕(1 + 𝑢𝜇𝑗) 𝜇𝑗: 𝑓𝑗𝑕𝑓𝑜𝑤𝑏𝑚𝑣𝑓 𝑝𝑔 𝑌−1

2𝑊𝑌−1 2

𝑕 is concave in 𝑢 ⇒ 𝑔 is concave

• 1. Convex Function Definitions: Examples

8

Convex function examples: norm, max, expectation norm: If 𝑔: 𝑆𝑜 → 𝑆 is a norm and 0 ≤ 𝜄 ≤ 1 𝑔 𝜄𝑦 + 1 − 𝜄 𝑧 ≤ 𝑔 𝜄𝑦 + 𝑔 1 − 𝜄 𝑧 = 𝜄𝑔(𝑦) + (1 − 𝜄)𝑔(𝑧) Max function: 𝑔 𝑦 = max

𝑗

𝑦𝑗 , 𝑦 = 𝑦1, 𝑦2, … , 𝑦𝑜 𝑈 𝑔 𝜄𝑦 + 1 − 𝜄 𝑧 = max

𝑗

𝜄𝑦𝑗 + 1 − 𝜄 𝑧𝑗 ≤ 𝜄 max

𝑗

𝑦𝑗 + 1 − 𝜄 max

𝑗

𝑧𝑗 = 𝜄𝑔 𝑦 + 1 − 𝜄 𝑔 𝑧 for 0 ≤ 𝜄 ≤ 1 Probability: (Expectation) If 𝑔 𝑦 is convex with 𝑞 𝑦 a probability at 𝑦,

• i. e. 𝑞 𝑦 ≥ 0, ∀𝑦 and ׬ 𝑞(𝑦) 𝑒𝑦 = 1

Then 𝑔 𝐹𝑦 ≤ 𝐹𝑔 𝑦 , where 𝐹𝑦 = ׬𝑦 𝑞 𝑦 𝑒𝑦 𝐹𝑔(𝑦) = ׬ 𝑔(𝑦) 𝑞 𝑦 𝑒𝑦 triangle inequality scalability

1.3 Views of Functions and Related Hyperplanes

9

Given 𝑔 𝑦 , 𝑦 ∈ 𝑆𝑜, we plot the function in 𝑆𝑜 and 𝑆𝑜+1 spaces.

• 1. Draw function in 𝑆𝑜 space

Equipotential surface: tangent plane 𝛼𝑔 ෤ 𝑦 𝑈 𝑦 − ෤ 𝑦 = 0 at ෤ 𝑦

• 2. Draw function in 𝑆𝑜+1 space

2.1 Graph of function: {(𝑦, ℎ)|𝑦 ∈ 𝑒𝑝𝑛 𝑔, ℎ = 𝑔 𝑦 } 𝐢𝐳𝐪𝐟𝐬𝐪𝐦𝐛𝐨𝐟 (h = 𝛼𝑔 ෤ 𝑦 𝑈 𝑦 − ෤ 𝑦 + 𝑔(෤ 𝑦)) 𝛼𝑔 ෤ 𝑦 𝑈 − 1 𝑦 ℎ − ෤ 𝑦 𝑔 ෤ 𝑦 = 0 Example: 𝑔 𝑦 = 𝑦2. We show the hyperplane with 𝛼𝑔 𝑦 2.2. Epigraph: epi 𝑔: {(x, 𝑢)|𝑦 ∈ 𝑒𝑝𝑛 𝑔, 𝑔 𝑦 ≤ 𝑢} A function is convex iff its epigraph is a convex set. Example: 𝑔 𝑦 = max 𝑔

𝑗 𝑦 | 𝑗 = 1 … 𝑠 , 𝑔 𝑗 𝑦 𝑏𝑠𝑓 𝑑𝑝𝑜𝑤𝑓𝑦.

Since epi 𝑔 is the intersect of epi 𝑔

𝑗, epi 𝑔 is convex.

Thus, function 𝑔 is convex.

• 2. Conditions of Optimality: First Order Condition

10

Defintion: 𝑔 is differentiable if 𝑒𝑝𝑛𝑔 is open and 𝛼𝑔(𝑦) ≡ (

𝜖𝑔 𝑦 𝜖𝑦1 , 𝜖𝑔 𝑦 𝜖𝑦2 , … , 𝜖𝑔 𝑦 𝜖𝑦𝑜 ) exists at each 𝑦 ∈ 𝑒𝑝𝑛𝑔

Theorem: Differentiable 𝑔 with convex domain is convex iff 𝑔 𝑧 ≥ 𝑔 𝑦 + 𝛼𝑔 𝑦 T 𝑧 − 𝑦 , ∀𝑦, 𝑧 ∈ 𝑒𝑝𝑛𝑔 Proof => If 𝑔 is convex 𝑈ℎ𝑓𝑜 1 − 𝑢 𝑔 𝑦 + 𝑢𝑔 𝑧 ≥ 𝑔 1 − 𝑢 𝑦 + 𝑢𝑧 , ∀0 ≤ 𝑢 ≤ 1 𝑢 𝑔 𝑧 − 𝑔 𝑦 ≥ 𝑔 𝑦 + 𝑢 𝑧 − 𝑦 − 𝑔(𝑦) 𝑔 𝑧 − 𝑔 𝑦 ≥

1 𝑢 (𝑔 𝑦 + 𝑢 𝑧 − 𝑦

− 𝑔 𝑦 ) = 𝛼𝑔 𝑦 𝑧 − 𝑦 𝑥ℎ𝑓𝑜 𝑢 → 0 <= 𝐻𝑗𝑤𝑓𝑜 𝑔 𝑧 ≥ 𝑔 𝑦 + 𝛼𝑔 𝑦 T 𝑧 − 𝑦 , ∀𝑦, 𝑧 ∈ 𝑒𝑝𝑛𝑔 𝑀𝑓𝑢 𝑨 = 1 − 𝑢 𝑦 + 𝑢𝑧 where ൝𝑔 𝑦 ≥ 𝑔 𝑨 + 𝛼𝑔 𝑨 T 𝑦 − 𝑨 𝑔 𝑧 ≥ 𝑔 𝑨 + 𝛼𝑔 𝑨 T 𝑧 − 𝑨 Thus 1 − 𝑢 𝑔 𝑦 + 𝑢𝑔 𝑧 ≥ 𝑔(𝑨)

• 2. Conditions: Second Order Condition

11

Definition: 𝑔 is twice differentiable if 𝑒𝑝𝑛𝑔 is open and the Hessian 𝛼2𝑔 𝑦 ∈ 𝑇𝑜 𝛼2𝑔 𝑦 𝑗𝑘 ≡

𝜖2𝑔 𝑦 𝜖𝑦𝑗𝜖𝑦𝑘 , 𝑗, 𝑘 = 1, … , 𝑜 exists at each 𝑦 ∈ 𝑒𝑝𝑛𝑔

Theorem: Twice Differentiable 𝑔 with convex domain is convex iff 𝛼2𝑔 𝑦 ≽ 0, ∀𝑦 ∈ 𝑒𝑝𝑛𝑔 Proof: Using Lagrange remainder, we can find a z

𝑔 𝑦 + 𝑢(𝑧 − 𝑦) = 𝑔 𝑦 + 𝛼𝑔 𝑦 𝑈𝑢 𝑧 − 𝑦 + 1 2 𝑢2 𝑧 − 𝑦 𝑈𝛼2𝑔 𝑨 𝑧 − 𝑦 ,

∀0 ≤ 𝑢 ≤ 1, 𝑨 is between 𝑦 and 𝑦 + 𝑢(𝑧 − 𝑦) Since the last term is always positive by assumption, the first order condition is satisfied.

• 2. Conditions: Second Order Condition

12

Example: Negative Entropy: 𝑔 𝑦 = 𝑦 log 𝑦 , 𝑦 ∈ 𝑆++ 𝑔′ 𝑦 = 𝑦 𝑦 + log 𝑦 = 1 + log 𝑦 , 𝑔′′ 𝑦 = 1 𝑦 Since 𝑦 ∈ 𝑆++, 𝑔′′ 𝑦 > 0 ⇒ 𝑔 𝑦 is convex Show the plot of 𝑦 log 𝑦 Remark:

• 1st order condition can be used to design and prove the

property of opt. algorithms.

• 2nd order condition implies the 1st order condition
• 2nd order condition can be used to prove the convexity of

the functions.

• 2. Conditions: Examples

13

• Quadratic Function: 𝑔 𝑦 =

1 2 𝑦𝑈𝑄𝑦 + 𝑟𝑈𝑦 + 𝑠, 𝑄 ∈ 𝑇𝑜

𝛼𝑔 𝑦 = 𝑄𝑦 + 𝑟, 𝛼2𝑔 𝑦 = 𝑄

• Least Square: 𝑔 𝑦 =

𝐵𝑦 − 𝑐 2

2

𝛼𝑔 𝑦 = 2𝐵𝑈 𝐵𝑦 − 𝑐 , 𝛼2𝑔 𝑦 = 𝐵𝑈𝐵

• Quadratic over linear: 𝑔 𝑦, 𝑧 =

𝑦2 𝑧 , 𝑧 > 0

𝛼𝑔 𝑦, 𝑧 = 2𝑦 𝑧 , − 𝑦2 𝑧2

𝑈

, , 𝛼2𝑔 𝑦 = 2 𝑧 − 2𝑦 𝑧2 − 2𝑦 𝑧2 2𝑦2 𝑧3 = 2 𝑧3 𝑧 −𝑦 𝑧 −𝑦

• 2. Conditions: Examples

14

• Log-sum-exp: 𝑔 𝑦 = log σ𝐿=1

𝑜

𝑓𝑦𝑙 (Smooth max of softmax function) 𝛼2𝑔 𝑦 =

1 1𝑈𝑨 𝑒𝑗𝑏𝑕 𝑨 − 1 1𝑈𝑨 𝑨𝑨𝑈, 𝑨𝑙 = 𝑓𝑦𝑙

𝑤𝑈𝛼2𝑔 𝑦 𝑤 =

1 1𝑈𝑨 2 [ σ𝑗=1 𝑜

𝑨𝑗 σ𝑗=1

𝑜

𝑤𝑗

2𝑨𝑗 − σ𝑗=1 𝑜

𝑤𝑗𝑨𝑗 2] ≥ 0, for all 𝑤 ∈ 𝑆𝑜 (Cauchy-Schwarz inequality) Thus, 𝑔(𝑦) is a convex function Cauchy-Schwarz inequality: 𝑏𝑈𝑏

𝑐𝑈𝑐 ≥ 𝑏𝑈𝑐 2, 𝑏𝑗 = 𝑨𝑗, 𝑐𝑗 = 𝑤𝑗 𝑨𝑗

Proof 1: Let 𝑨 = 𝑏 −

𝑏𝑈 𝑐 𝑐𝑈 𝑐 𝑐, or 𝑏 = 𝑨 + 𝑏𝑈𝑐 𝑐𝑈𝑐 𝑐

We have a𝑈a = zTz + 𝑏𝑈𝑐 2 𝑐𝑈𝑐 2 𝑐𝑈𝑐 ≥ 𝑏𝑈𝑐 2 𝑐𝑈𝑐 2 𝑐𝑈𝑐 = 𝑏𝑈𝑐 2 𝑐𝑈𝑐 Proof 2: By induction

• 3. Operations that preserve convexity

15

• Nonnegative multiple: 𝛽𝑔, where 𝛽 ≥ 0, 𝑔 is convex
• Sum: 𝑔

1 + 𝑔 2, where 𝑔 1, 𝑏𝑜𝑒 𝑔 2 are convex

• Composition with affine function: 𝑔 𝐵𝑦 + 𝑐 , where 𝑔 is

convex Proof: 𝛼

𝑦 2𝑔 𝐵𝑦 + 𝑐 = 𝐵𝑈𝛼 𝑧 2𝑔 𝑧|𝑧 = 𝐵𝑦 + 𝑐 𝐵

E.g. 𝑔 𝑦 = − σ𝑗=1

𝑛 log 𝑐𝑗 − 𝑏𝑗 𝑈𝑦𝑗 ,

𝑒𝑝𝑛 𝑔 = {𝑦|𝑏𝑗

𝑈𝑦 < 𝑐𝑗, 𝑗 = 1, … , 𝑛}

𝑔 𝑦 = 𝐵𝑦 + 𝑐 (if 𝑔 is twice differentiable)

• 3. Operations that preserve convexity

16

• Pointwise maximum: 𝑔 𝑦 = max{𝑔

1 𝑦 , … , 𝑔 𝑠(𝑦)} , 𝑔 𝑗 are

convex

• Pointwise supremum:

𝑕 𝑦 = sup

𝑧∈𝐷

𝑔(𝑦, 𝑧) , where 𝑔 𝑦, 𝑧 is convex in 𝑦 and 𝐷 is an arbitrary set Examples

• 𝑇𝑑 𝑦 = sup

𝑧∈𝐷

𝑧𝑈𝑦, for an arbitrary set 𝐷

• 𝑔 𝑦 = sup

𝑧∈𝐷

𝑦 − 𝑧 , for an arbitrary set 𝐷

• 𝜇𝑛𝑏𝑦 𝑌 =

sup

𝑧 2=1

𝑧𝑈𝑌𝑧, 𝑌 ∈ 𝑇𝑜

• 3. Operations that preserve convexity: Dual norm

17

Example: 𝑔 𝑦 = max

𝑧 2≤1 𝑧𝑈𝑦

𝑔 𝑦 = max

𝑧 1≤1 𝑧𝑈𝑦

𝑔 𝑦 = max

𝑧 𝑞≤1 𝑧𝑈𝑦

18

Theorem: Pointwise maximum of convex functions is convex Given 𝑔 𝑦 = max 𝑔

1 𝑦 , 𝑔 2 𝑦

, where 𝑔

1 and 𝑔 2 are convex

and 𝑒𝑝𝑛 𝑔 = 𝑒𝑝𝑛 𝑔

1

∩ 𝑒𝑝𝑛 𝑔

2 is convex, then 𝑔 𝑦 is

convex. Proof: For 0 ≤ 𝜄 ≤ 1, 𝑦, 𝑧 ∈ 𝑒𝑝𝑛 𝑔 𝑔 𝜄𝑦 + 1 − 𝜄 𝑧 = max{𝑔

1 𝜄𝑦 + 1 − 𝜄 𝑧 , 𝑔 2 𝜄𝑦 + 1 − 𝜄 𝑧 }

≤ max{𝜄𝑔

1 𝑦) + 1 − 𝜄 𝑔 1(𝑧 , 𝜄𝑔 2 𝑦) + 1 − 𝜄 𝑔 2(𝑧 }

≤ 𝜄 max{𝑔

1 𝑦), 𝑔 2(𝑦)} + 1 − 𝜄 max {𝑔 1(𝑧 , 𝑔 2(𝑧)}

= 𝜄𝑔 𝑦 + 1 − 𝜄 𝑔 𝑧 i.e. 𝑔 𝜄𝑦 + 1 − 𝜄 𝑧 ≤ 𝜄𝑔 𝑦 + 1 − 𝜄 𝑔 𝑧 Thus, function 𝑔 𝑦 is convex.

• 3. Operations that preserve convexity: max function

19

Theorem: Partial minimization If 𝑕 𝑦, 𝑧 is convex in 𝑦 and 𝑧, and a set 𝐷 is convex Then 𝑔 𝑦 = min

𝑧∈𝐷 𝑕 𝑦, 𝑧 is convex.

Proof: Let 𝑧1 ∈ {𝑧| min

𝑧∈𝐷 𝑕(𝑦1, 𝑧)} and 𝑧2 ∈ {𝑧| min 𝑧∈𝐷 (𝑕(𝑦2, 𝑧)},

we can write 𝜄𝑔 𝑦1 + 1 − 𝜄 𝑔 𝑦2 = 𝜄𝑕 𝑦1, 𝑧1 + 1 − 𝜄 𝑕(𝑦2, 𝑧2) ≥ 𝑕(𝜄𝑦1 + 1 − 𝜄 𝑦2, 𝜄𝑧1 + 1 − 𝜄 𝑧2) 𝑕 𝑗𝑡 𝑑𝑝𝑜𝑤𝑓𝑦 ≥ min

𝑧∈𝐷 𝑕 𝜄𝑦1 + 1 − 𝜄 𝑦2, 𝑧

𝐷 𝑗𝑡 𝑑𝑝𝑜𝑤𝑓𝑦 = 𝑔(𝜄𝑦1 + 1 − 𝜄 𝑦2) i.e. we have 𝜄𝑔 𝑦1 + 1 − 𝜄 𝑔 𝑦2 ≥ 𝑔 𝜄𝑦1 + 1 − 𝜄 𝑦2 Therefore, 𝑔 𝑦 = min

𝑧∈𝐷 𝑕 𝑦, 𝑧 is convex.

• 3. Operations that preserve convexity: minimization

Examples for Partial Minimization Given 𝑔 𝑦, 𝑧 = 𝑦𝑈 𝑧𝑈 𝐵 𝐶 𝐶𝑈 𝐷 𝑦 𝑧 𝑦 ∈ 𝑆𝑜, 𝑧 ∈ 𝑆𝑛, 𝐵 ∈ 𝑇+

𝑜, 𝐷 ∈ 𝑇+ 𝑛,

𝐵 𝐶 𝐶𝑈 𝐷 ∈ 𝑇+

𝑜+𝑛

Let 𝑕 𝑦 = min

𝑧 𝑔(𝑦, 𝑧) = 𝑦𝑈 𝐵 − 𝐶𝐷+𝐶𝑈 𝑦,

𝐷+: 𝐪𝐭𝐟𝐯𝐞𝐩 𝐣𝐨𝐰𝐟𝐬𝐭𝐟 of matrix 𝐷. (Drazin inverse, or generalized inverse) We can claim that function 𝑕 𝑦 is convex. Proof: (1) 𝑔 𝑦, 𝑧 is convex (2) 𝑧 ∈ 𝑆𝑛 where 𝑆𝑛 is a convex non-empty set (3) Therefore, 𝑕(𝑦) is convex, i.e. 𝐵 − 𝐶𝐷+𝐶𝑈 ≽ 0

• 3. Operations that preserve convexity

20

Composition: Given 𝑕: 𝑆𝑜 → 𝑆 𝑏𝑜𝑒 ℎ: 𝑆 → 𝑆, we set 𝑔 𝑦 = ℎ(𝑕 𝑦 ) f is convex if g convex, h convex, ෨ ℎ nondecreasing g concave, h convex, ෨ ℎ nonincreasing f is concave if g convex, h concave, ෨ ℎ nonincreasing g concave, h concave, ෨ ℎ nondecreasing Proof : for 𝑜=1 𝑔′′ 𝑦 = ℎ′′ 𝑕 𝑦 𝑕′ 𝑦 2 + ℎ′ 𝑕 𝑦 𝑕′′ 𝑦 Ex1: exp 𝑕(𝑦) is convex if g is convex Ex2: 1/𝑕 𝑦 is convex if g is concave and positive Note that we set ෨ ℎ 𝑦 = ∞ if 𝑦 ∉ 𝑒𝑝𝑛 ℎ, h is convex ෨ ℎ 𝑦 = −∞ if 𝑦 ∉ 𝑒𝑝𝑛 ℎ, h is concave

• 3. Operations that preserve convexity

21

Show that ℎ 𝑕 𝜄𝑦 + 1 − 𝜄 𝑧 ≤ 𝜄ℎ 𝑕 𝑦 + 1 − 𝜄 ℎ(𝑕 𝑧 ) for the case that g, h are convex, and ෨ ℎ is nondecreasing (1) g is convex 𝑕 𝜄𝑦 + 1 − 𝜄 𝑧 ≤ 𝜄𝑕 𝑦 + 1 − 𝜄 𝑕(𝑧) (2) h is nondecreasing: From (1), we have ℎ 𝑕 𝜄𝑦 + 1 − 𝜄 𝑧 ≤ ℎ 𝜄𝑕 𝑦 + 1 − 𝜄 𝑕 𝑧 (3) h is convex ℎ 𝜄𝑕 𝑦 + 1 − 𝜄 𝑕 𝑧 ≤ 𝜄ℎ 𝑕 𝑦 + 1 − 𝜄 ℎ 𝑕 𝑧 (4) From (2) & (3) ℎ 𝑕 𝜄𝑦 + 1 − 𝜄 𝑧 ≤ 𝜄ℎ 𝑕 𝑦 + 1 − 𝜄 ℎ 𝑕 𝑧

• 3. Operations that preserve convexity

22

The setting of conjugate functions starts from the following problem (which may not be convex) 𝑛𝑗𝑜 𝑔(𝑦) subject to 𝑦 ≤ 0 We convert to a function of 𝑧 𝑗𝑜𝑔

𝑦

𝑔 𝑦 − 𝑧𝑈𝑦 The conjugate function is 𝑔∗ 𝑧 = sup

𝑦

𝑧𝑈𝑦 − 𝑔(𝑦) In the class, we interchange min and inf; max and sup to simplify the notation.

• 4. Conjugate Functions

23

Given 𝑔: 𝑆𝑜 → 𝑆, we have 𝑔∗: 𝑆𝑜 → 𝑆 𝑔∗ 𝑧 = sup

𝑦∈𝑒𝑝𝑛 𝑔

𝑧𝑈𝑦 − 𝑔 𝑦 ; (−𝑔∗ 𝑧 = min

𝑦∈𝑒𝑝𝑛 𝑔 − 𝑧𝑈𝑦 + 𝑔(𝑦))

Constraint: 𝑧 ∈ 𝑆𝑜 for which the supremum is finite (bounded) 𝑔∗ 𝑧 is called the conjugate of function f Theorem : 𝑔∗(𝑧) is convex (pointwise maximum) Proof : 𝑔∗ 𝜄𝑧1 + 1 − 𝜄 𝑧2 = sup

𝑦

𝜄𝑧1 + 1 − 𝜄 𝑧2 𝑈𝑦 − 𝑔(𝑦) ≤ sup

𝑦

𝜄𝑧1

𝑈𝑦 + 𝜄𝑔 𝑦

+ sup

𝑦

( 1 − 𝜄 𝑧2

𝑈 𝑦 − 1 − 𝜄 𝑔(𝑦))

= 𝜄𝑔∗ 𝑧1 + 1 − 𝜄 𝑔∗ 𝑧2 Remark: 𝑔∗(𝑧) is convex even if 𝑔(𝑦) is not convex

• 4. Conjugate Functions

24

Suppose we have a pair ҧ 𝑦, ത 𝑧, such that 𝑔∗ ത 𝑧 = ത 𝑧𝑈 ҧ 𝑦 − 𝑔 ҧ 𝑦 , we can show that ത 𝑧 = ∇𝑦𝑔 ҧ 𝑦 (exercise 3.40) And the supporting hyperplane : ത 𝑧𝑈𝑦 − ℎ = 𝑔∗ ത 𝑧 ത 𝑧𝑈 −1 𝑦 ℎ = 𝑔∗(ത 𝑧)

• Ex. 𝑔 𝑦 = 𝑦2 − 2𝑦, 𝑦 ∈ 𝑆

𝑔∗ 𝑧 = sup

𝑦

𝑧𝑦 − 𝑦2 + 2𝑦, 𝑧 ∈ 𝑆

• 4. Conjugate Functions

25

One way to view conjugate function 𝑔∗ 𝑧 = sup

𝑦∈𝑒𝑝𝑛 𝑔

𝑧𝑈𝑦 − 𝑔(𝑦) x : negative slack y : shadow price (loss) to accommodate the slack f*(y) : balance between price slack product (𝑧𝑈𝑦) and objective function 𝑔 𝑦 . Remark: When 𝑔∗ 𝑧 is unbounded, the shadow price 𝑧 is not reasonable.

• 4. Conjugate Functions

26

Ex: 𝑔 𝑦 = 𝑏𝑦 + 𝑐, 𝑦 ∈ 𝑆 𝑔∗ 𝑧 = sup

𝑦

(𝑧𝑦 − 𝑏𝑦 − 𝑐) (1) If 𝑧 ≠ 𝑏, 𝑔∗ 𝑧 = ∞ (2) If 𝑧 = 𝑏, 𝑔∗ 𝑧 = −𝑐 → 𝑒𝑝𝑛 𝑔∗ = 𝑏, 𝑔∗ 𝑧 = −𝑐

• 4. Conjugate Functions: Examples (single variable)

27

Ex: 𝑔 𝑦 = −𝑚𝑝𝑕𝑦, 𝑦 ∈ R++ 𝑔∗ 𝑧 = sup

𝑦∈𝑆++

𝑧𝑦 + 𝑚𝑝𝑕𝑦 (1) If 𝑧 ≥ 0, 𝑔∗ 𝑧 = ∞ (2) If 𝑧 < 0, 𝑔∗ 𝑧 = max

𝑦∈𝑆++𝑦𝑧 + 𝑚𝑝𝑕𝑦

Let 𝑕 𝑦 = 𝑦𝑧 + 𝑚𝑝𝑕𝑦, 𝑕′ 𝑦 = 𝑧 +

1 𝑦

If 𝑕′ 𝑦 = 0, 𝑦 = −

1 𝑧

Thus, 𝑔∗ 𝑧 = −1 + log −

1 𝑧 = −1 − log −𝑧

→ 𝑒𝑝𝑛 𝑔∗ = −𝑆++, 𝑔∗ 𝑧 = −1 − log(−𝑧)

• 4. Conjugate Functions: Examples (single variable)

28

Ex: 𝑔 𝑦 = 𝑓𝑦, 𝑦 ∈ 𝑆 𝑔∗ 𝑧 = 𝑡𝑣𝑞

𝑦

𝑦𝑧 − 𝑓𝑦 (1) 𝑧 < 0 ∶ 𝑔∗ 𝑧 = ∞ (2) 𝑧 > 0 ∶ Let 𝑕 𝑦 = 𝑦𝑧 − 𝑓𝑦→ 𝑕′ 𝑦 = 𝑧 − 𝑓𝑦 If 𝑕′ 𝑦 = 0, then 𝑦 = 𝑚𝑝𝑕𝑧 Thus 𝑔∗ 𝑧 = 𝑧𝑚𝑝𝑕𝑧 − 𝑧 (3) 𝑧 = 0 ∶ 𝑔∗ 𝑧 = 0 → 𝑒𝑝𝑛 𝑔∗ = 𝑆+, 𝑔∗ 𝑧 = 𝑧𝑚𝑝𝑕𝑧 − 𝑧 Therefore, we have 𝑔∗ 𝑧 = 𝑧𝑚𝑝𝑕𝑧 − 𝑧, where 𝑧 ≥ 0.

• 4. Conjugate Functions

29

Ex: 𝑔 𝑦 = 𝑦𝑚𝑝𝑕𝑦, 𝑦 ∈ 𝑆+, 𝑔 0 = 0 𝑔∗ 𝑧 = 𝑡𝑣𝑞

𝑦

𝑦𝑧 − 𝑦𝑚𝑝𝑕𝑦 Let 𝑕 𝑦 = 𝑦𝑧 − 𝑦𝑚𝑝𝑕𝑦 → 𝑕′ 𝑦 = 𝑧 − 𝑚𝑝𝑕𝑦 − 1 Suppose 𝑕′ 𝑦 = 0, we have 𝑧 = 1 + 𝑚𝑝𝑕𝑦 or 𝑦 = 𝑓𝑧−1 Thus 𝑔∗ 𝑧 = 𝑧𝑓𝑧−1 − 𝑓𝑧−1(𝑧 − 1) = 𝑓𝑧−1 where 𝑧 ∈ 𝑆

• 4. Conjugate Functions

30

Ex: 𝑔 𝑦 =

1 2 𝑦𝑈𝑅𝑦, 𝑦 ∈ 𝑆𝑜, 𝑅 ∈ 𝑇++ 𝑜

𝑔∗ 𝑧 = 𝑡𝑣𝑞

𝑦

𝑦𝑈𝑧 −

1 2 𝑦𝑈𝑅𝑦

Let 𝑕 𝑦 = 𝑦𝑈𝑧 −

1 2 𝑦𝑈𝑅𝑦 → 𝛼𝑕 𝑦 = 𝑧 − 𝑅𝑦

If 𝛼𝑕 𝑦 = 0, we have 𝑦 = 𝑅−1𝑧 Thus, 𝑔∗ 𝑧 =

1 2 𝑧𝑈𝑅−1𝑧

Remark: Suppose that 𝑔∗ ത 𝑧 = ത 𝑧𝑈 ҧ 𝑦 − 𝑔 ҧ 𝑦 and ∇2𝑔 ҧ 𝑦 ≻ 0 We have ∇𝑔∗ ത 𝑧 = ҧ 𝑦 and ∇2𝑔∗ ത 𝑧 = ∇2𝑔 ҧ 𝑦

−1 (exercise 3.40)

• 4. Conjugate Functions

31

Basic Properties (1) 𝑔 𝑦 + 𝑔∗ 𝑧 ≥ 𝑦𝑈𝑧 Fenchel’s inequality. Thus, in the above example 𝑦𝑈𝑧 ≤

1 2 𝑦𝑈𝑅𝑦 + 1 2 𝑧𝑈𝑅−1𝑧, ∀𝑦, 𝑧 ∈ 𝑆𝑜, 𝑅 ∈ 𝑇++ 𝑜

(2) 𝑔∗∗ = 𝑔, if f is convex & f is closed (i.e. epi f is a closed set) (3) If f is convex & differentiable, 𝑒𝑝𝑛 𝑔 = 𝑆𝑜 For max 𝑧𝑈𝑦 − 𝑔(𝑦), we have 𝑧 = 𝛼𝑔(𝑦∗) Thus, 𝑔∗ 𝑧 = 𝑦∗𝑈𝛼𝑔 𝑦∗ − 𝑔 𝑦∗ , 𝑧 = 𝛼𝑔(𝑦∗)

• 4. Conjugate Functions

32

Ex : 𝑔 𝑦 = 𝑚𝑝𝑕 σ𝑗=1

𝑜

𝑓𝑦𝑗 ↔ 𝑔∗ 𝑧 = σ𝑗=1

𝑜

𝑧𝑗𝑚𝑝𝑕𝑧𝑗 𝑔∗ 𝑧 = 𝑡𝑣𝑞

𝑦

𝑧𝑈𝑦 − 𝑔 𝑦 = 𝑡𝑣𝑞

𝑦

𝑧𝑈𝑦 − 𝑚𝑝𝑕 σ𝑗=1

𝑜

𝑓𝑦𝑗 Let 𝑕 𝑦 = 𝑧𝑈𝑦 − 𝑚𝑝𝑕 σ𝑗=1

𝑜

𝑓𝑦𝑗

𝜖𝑕 𝑦 𝜖𝑦𝑗 = 𝑧𝑗 − 𝑓𝑦𝑗 σ𝑗=1

𝑜

𝑓𝑦𝑗 = 0

Thus, 𝑧𝑗 =

𝑓𝑦𝑗 σ𝑗=1

𝑜

𝑓𝑦𝑗, i.e. 1𝑈𝑧 = 1

(1) 1𝑈𝑧 ≠ 1 → unbounded (2) 𝑧𝑗 < 0 → unbounded (3) 𝑔∗ 𝑧 = σ𝑗=1

𝑜

𝑧𝑗𝑚𝑝𝑕𝑧𝑗 , 𝑧 ≥ 0, 1𝑈𝑧 = 1

• 4. Conjugate Functions

33

Log function : 𝑚𝑝𝑕𝑔 𝑦 , 𝑔: 𝑆𝑜 → 𝑆, 𝑔 𝑦 > 0, ∀𝑦 ∈ 𝑒𝑝𝑛 𝑔 Suppose f is twice differentiable, 𝑒𝑝𝑛 𝑔 is convex. 𝛼2𝑚𝑝𝑕𝑔 𝑦 =

1 𝑔 𝑦 𝛼2𝑔 𝑦 − 1 𝑔 𝑦 2 𝛼𝑔 𝑦 𝛼𝑔 𝑦 𝑈

Then f is log-convex iff ∀𝑦 ∈ 𝑒𝑝𝑛 𝑔 𝑔 𝑦 𝛼2𝑔 𝑦 ≥ 𝛼𝑔 𝑦 𝛼𝑔 𝑦 𝑈 f is log-concave iff ∀𝑦 ∈ 𝑒𝑝𝑛 𝑔 𝑔 𝑦 𝛼2𝑔 𝑦 ≤ 𝛼𝑔 𝑦 𝛼𝑔 𝑦 𝑈

• 5. Log-Concave, Log-Convex Functions

34

𝑔 ∶ 𝑆𝑜 → 𝑆, 𝑔 𝑦 > 0, ∀𝑦 ∈ 𝑒𝑝𝑛 𝑔 Definition: If log 𝑔 is concave, f is log-concave. Definition: If log 𝑔 is convex, f is log-convex. Ex : 𝑔 𝑦 = 𝑏𝑈𝑦 + 𝑐, 𝑒𝑝𝑛 𝑔 = 𝑦 𝑏𝑈𝑦 + 𝑐 : log-concave 𝑔 𝑦 = 𝑦𝑏, 𝑦 ∈ 𝑆++, 𝑏 ≤ 0 : log-convex 𝑏 > 0 ∶ log-concave 𝑔 𝑦 = 𝑓𝛽𝑦 : log convex & log-concave 𝑔 𝑦 =

1 2𝜌 ׬ −∞ 𝑦 𝑓−𝑣2

2 𝑒𝑣 : cumulative distribution function of

Gaussian density log-concave 𝑔 𝑦 =

1 2𝜌 𝑜𝑒𝑓𝑢σ 𝑓−1

2 𝑦− ҧ

𝑦 𝑈 σ−1 𝑦− ҧ 𝑦

: log-concave

• 5. Log-Concave, Log-Convex Functions

35

Properties 𝛼2𝑚𝑝𝑕𝑔 𝑦 = 1 𝑔 𝑦 𝛼2𝑔 𝑦 − 1 𝑔 𝑦 2 𝛼𝑔 𝑦 𝛼f x T 𝑔 𝑦 𝛼2𝑔 𝑦 ≥ 𝛼𝑔 𝑦 𝛼𝑔 𝑦 𝑈, ∀𝑦 ∈ 𝑒𝑝𝑛 𝑔 : log-convex 𝑔 𝑦 𝛼2𝑔 𝑦 ≤ 𝛼𝑔 𝑦 𝛼𝑔 𝑦 𝑈, ∀𝑦 ∈ 𝑒𝑝𝑛 𝑔 : log-concave

• 5. Log-Concave, Log-Convex Functions

36

Outlines

• 1. Definitions: Convexity, Examples & Views
• 2. Conditions of Optimality
• 1. First Order Condition
• 2. Second Order Condition
• 3. Operations that Preserve the Convexity
• 1. Pointwise Maximum
• 2. Partial Minimization
• 4. Conjugate Function
• 5. Log-Concave, Log-Convex Functions

37