Convex Functions (I) Lijun Zhang zlj@nju.edu.cn - - PowerPoint PPT Presentation

▶
convex functions i
SMART_READER_LITE
LIVE PREVIEW

Convex Functions (I) Lijun Zhang zlj@nju.edu.cn - - PowerPoint PPT Presentation

Feb 12, 2024 105 likes •683 views

Convex Functions (I) Lijun Zhang zlj@nju.edu.cn http://cs.nju.edu.cn/zlj Outline Basic Properties Definition First-order Conditions, Second-order Conditions Jensens inequality and extensions Epigraph Operations

slide-1
SLIDE 1

Convex Functions (I)

Lijun Zhang zlj@nju.edu.cn http://cs.nju.edu.cn/zlj

slide-2
SLIDE 2

Outline

 Basic Properties

 Definition  First-order Conditions, Second-order Conditions  Jensen’s inequality and extensions  Epigraph

 Operations That Preserve Convexity

 Nonnegative Weighted Sums  Composition with an affine mapping  Pointwise maximum and supremum  Composition  Minimization  Perspective of a function

 Summary

slide-3
SLIDE 3

Outline

 Basic Properties

 Definition  First-order Conditions, Second-order Conditions  Jensen’s inequality and extensions  Epigraph

 Operations That Preserve Convexity

 Nonnegative Weighted Sums  Composition with an affine mapping  Pointwise maximum and supremum  Composition  Minimization  Perspective of a function

 Summary

slide-4
SLIDE 4

Convex Function

  • is convex if

 is convex 

slide-5
SLIDE 5

Convex Function

  • is convex if

 is convex 

  • is strictly convex if

slide-6
SLIDE 6

Convex Function

  • is convex if

 is convex 

 is concave if is convex

 is convex

 Affine functions are both convex and concave, and vice versa.

slide-7
SLIDE 7

Extended-value Extensions

 The extended-value extension of is

 

 Example

slide-8
SLIDE 8

Extended-value Extensions

 The extended-value extension of is

 

  •  Example

 Indicator Function of a Set

slide-9
SLIDE 9

Zeroth-order Condition

 Definition

 High-dimensional space

 A function is convex if and only if it is convex when restricted to any line that intersects its domain.

,

 is convex is convex  One-dimensional space

slide-10
SLIDE 10

First-order Conditions

 is differentiable. Then is convex if and only if

 is convex  For all

𝑔 𝑧 𝑔 𝑦 𝛼𝑔 𝑦 𝑧 𝑦

First-order Taylor approximation

slide-11
SLIDE 11

First-order Conditions

 is differentiable. Then is convex if and only if

 is convex  For all  Local Information Global Information 

 is strictly convex if and only if

𝑔 𝑧 𝑔 𝑦 𝛼𝑔 𝑦 𝑧 𝑦 𝑔 𝑧 𝑔 𝑦 𝛼𝑔 𝑦 𝑧 𝑦

slide-12
SLIDE 12

Proof

 is convex

  •  Necessary condition:

𝑔 𝑦 𝑢 𝑧 𝑦 1 𝑢 𝑔 𝑦 𝑢𝑔 𝑧 , 0 𝑢 1 ⇒ 𝑔 𝑧 𝑔 𝑦

  • → 𝑔 𝑧 𝑔 𝑦 𝑔 𝑦

𝑧 𝑦

 Sufficient condition:

𝑨 𝜄𝑦 1 𝜄 𝑧 𝑔 𝑦 𝑔 𝑨 𝑔 𝑨 𝑦 𝑨 𝑔 𝑧 𝑔 𝑨 𝑔 𝑨 𝑧 𝑨 ⇒ 𝑔 𝑦 𝑔 𝑨 1 𝜄𝑔 𝑨 𝑦 𝑧 𝑔 𝑧 𝑔 𝑨 𝜄𝑔 𝑨 𝑦 𝑧

  • ⇒ 𝜄𝑔 𝑦 1 𝜄 𝑔 𝑧 𝑔 𝑨 ⇒ 𝑔 𝜄𝑦 1 𝜄 𝑧 𝜄𝑔 𝑦 1 𝜄 𝑔 𝑧
slide-13
SLIDE 13

Proof

 is convex

is convex

  •  𝑔 is convex ⇒

is convex ⇒ 𝑕 1 𝑕 0 𝑕 0 ⇒ 𝑔 𝑧 𝑔 𝑦 𝛼𝑔 𝑦 𝑧 𝑦 𝑕 𝑢 𝑔 𝑢𝑧 1 𝑢 𝑦 , 𝑕′ 𝑢 𝛼𝑔 𝑢𝑧 1 𝑢 𝑦 𝑧 𝑦

𝑔 is convex 𝑕 is convex First-order condition of 𝑕 First-order condition of 𝑔

slide-14
SLIDE 14

Proof

 is convex

is convex

⇒ 𝑕 𝑢 𝑕 𝑢̃ 𝑕 𝑢̃ 𝑢 𝑢̃ ⇒ 𝑕 𝑢 is convex ⇒ 𝑔 is convex 𝑕 𝑢 𝑔 𝑢𝑧 1 𝑢 𝑦 , 𝑔 𝑢𝑧 1 𝑢 𝑦 𝑔 𝑢̃𝑧 1 𝑢̃ 𝑦 𝛼𝑔 𝑢̃𝑧 1 𝑢̃ 𝑦 𝑧 𝑦 𝑢 𝑢̃ 𝑕′ 𝑢 𝛼𝑔 𝑢𝑧 1 𝑢 𝑦 𝑧 𝑦

𝑔 is convex 𝑕 is convex First-order condition of 𝑕 First-order condition of 𝑔

slide-15
SLIDE 15

Second-order Conditions

 is twice differentiable. Then is convex if and only if

 is convex  For all ,

  •  Attention

  • is strictly convex

 is strict convex

  • is strict convex but

 is convex is necessary,

slide-16
SLIDE 16

Examples

 Functions on

is convex on ,

is convex on when

  • r

, and concave for 

, for

, is convex on  is concave on

  •  Negative entropy

is convex on

slide-17
SLIDE 17

Examples

 Functions on

  •  Every norm on

is convex

  •  Quadratic-over-linear:
  •  dom 𝑔 𝑦, 𝑧 ∈ 𝐒 𝑧 0

  • / is concave on

is concave on

slide-18
SLIDE 18

Examples

 Functions on

  •  Every norm on

is convex

 𝑔𝑦 is a norm on 𝐒  𝑔 𝜄𝑦 1 𝜄 𝑧 𝑔 𝜄𝑦 𝑔 1 𝜄 𝑧 𝜄𝑔 𝑦 1 𝜄 𝑔𝑧

  •  𝑔 𝜄𝑦 1 𝜄 𝑧 max

𝜄𝑦 1 𝜄 𝑧

𝜄max

𝑦 1 𝜄 max 𝑧

slide-19
SLIDE 19

Examples

 Functions on

  •  𝛼𝑔 𝑦, 𝑧
  • 𝑧

𝑦𝑧 𝑦𝑧 𝑦

  • 𝑧

𝑦 𝑧 𝑦

  • ≽ 0
slide-20
SLIDE 20

Examples

 Functions on

slide-21
SLIDE 21

Examples

 Functions on

  •  𝛼𝑔 𝑦
  • 𝟐

𝟐𝑨 diag 𝑨 𝑨𝑨  𝑨 𝑓, … 𝑓  𝑤𝛼𝑔 𝑦 𝑤

  • 𝟐 ∑

𝑨

  • ∑

𝑤

𝑨

  • ∑

𝑤𝑨

  •  Cauchy-Schwarz inequality: 𝑏𝑏𝑐𝑐

𝑏𝑐

slide-22
SLIDE 22

Examples

 Functions on

is concave on

  •  𝑕 𝑢 𝑔 𝑎 𝑢𝑊 , 𝑎 𝑢𝑊 ≻ 0, 𝑎 ≻ 0

 𝑕 𝑢 log det𝑎 𝑢𝑊 log det 𝑎

  • 𝐽 𝑢𝑎

𝑊𝑎 𝑎

  • ∑

log1 𝑢𝜇

  • log det 𝑎

 𝜇, … 𝜇 are the eigenvalues of 𝑎

𝑊𝑎

  •  𝑕 𝑢 ∑
  • , 𝑕 𝑢 ∑
  • det 𝐵𝐶 det 𝐵 det𝐶 https: / / en.wikipedia.org/ wiki/ Determinant
slide-23
SLIDE 23

Sublevel Sets

  • sublevel set

 is convex

is convex

is convex

is convex

  • superlevel set

 is concave

  • convex

  • is convex

𝐷 𝑦 ∈ dom 𝑔 𝑔𝑦 𝛽 𝐷 𝑦 ∈ dom 𝑔 𝑔 𝑦 𝛽

slide-24
SLIDE 24

Epigraph

 Graph of function

 Epigraph of function

slide-25
SLIDE 25

Epigraph

 Epigraph of function

 Hypograph

 Conditions

 is convex is convex  is concave is convex

slide-26
SLIDE 26

Example

 Matrix Fractional Function

 Quadratic-over-linear:

  •  Schur complement condition

 is convex

 Recall Example 2.10 in the book

𝑔 𝑦, 𝑍 𝑦𝑍𝑦, dom 𝑔 𝐒 𝐓

slide-27
SLIDE 27

Application of Epigraph

 First order Condition

 𝑔 𝑧 𝑔 𝑦 𝛼𝑔 𝑦 𝑧 𝑦  𝑧, 𝑢 ∈ epi 𝑔 ⇒ 𝑢 𝑔 𝑧 𝑔 𝑦 𝛼𝑔 𝑦 𝑧 𝑦

slide-28
SLIDE 28

Application of Epigraph

 First order Condition

 𝑔 𝑧 𝑔 𝑦 𝛼𝑔 𝑦 𝑧 𝑦  𝑧, 𝑢 ∈ epi 𝑔 ⇒ 𝑢 𝑔 𝑦 𝛼𝑔 𝑦 𝑧 𝑦  𝑧, 𝑢 ∈ epi 𝑔 ⇒ 𝛼𝑔 𝑦 1

  • 𝑧

𝑢 𝑦 𝑔 𝑦

slide-29
SLIDE 29

Jensen’s Inequality

 Basic inequality

 

 points

slide-30
SLIDE 30

Jensen’s Inequality

 Infinite points

 𝑔 𝑦 𝐅𝑔𝑦 𝑨, 𝑨 is a zero-mean noisy

 Hölder’s inequality

slide-31
SLIDE 31

Outline

 Basic Properties

 Definition  First-order Conditions, Second-order Conditions  Jensen’s inequality and extensions  Epigraph

 Operations That Preserve Convexity

 Nonnegative Weighted Sums  Composition with an affine mapping  Pointwise maximum and supremum  Composition  Minimization  Perspective of a function

 Summary

slide-32
SLIDE 32

Nonnegative Weighted Sums

 Finite sums  𝑥 0, 𝑔

is convex

 𝑔 𝑥𝑔

⋯ 𝑥𝑔 is convex

 Infinite sums  𝑔𝑦, 𝑧 is convex in 𝑦, ∀𝑧 ∈ 𝒝, 𝑥 𝑧 0  𝑕 𝑦 𝑔 𝑦, 𝑧 𝑥𝑧

𝒝

𝑒𝑧 is convex  Epigraph interpretation  𝐟𝐪𝐣 𝑥𝑔 𝑦, 𝑢|𝑥𝑔𝑦 𝑢  𝐽 𝑥 𝐟𝐪𝐣 𝑔 𝑦, 𝑥𝑢|𝑔 𝑦 𝑢  𝐟𝐪𝐣 𝑥𝑔 𝐽 𝑥 𝐟𝐪𝐣 𝑔

slide-33
SLIDE 33

Composition with an affine mapping

  •  Affine Mapping

 If is convex, so is  If is concave, so is

𝑕 𝑦 𝑔𝐵𝑦 𝑐

slide-34
SLIDE 34

Pointwise Maximum

  • is convex

is convex with

max𝑔

𝜄𝑦 1 𝜄 𝑧 , 𝑔 𝜄𝑦 1 𝜄 𝑧

max𝜄𝑔

𝑦 1 𝜄 𝑔 𝑧 , 𝜄𝑔 𝑦 1

𝜄𝑔

𝑧

𝜄 max 𝑔

𝑦 , 𝑔 𝑦

1 𝜄 max 𝑔

𝑧 , 𝑔 𝑧

𝜄𝑔 𝑦 1 𝜄 𝑔 𝑧

 𝑔

, … 𝑔 is convex ⇒ 𝑔 𝑦 max𝑔 𝑦 , … 𝑔 𝑦

slide-35
SLIDE 35

Examples

 Piecewise-linear functions

  •  Sum of

largest components

  • is convex
  •  Pointwise maximum of

! ! ! linear

functions

slide-36
SLIDE 36

Pointwise Supremum

 is convex in is convex with

∈𝒝

 Epigraph interpretation

∈𝒝

 Intersection of convex sets is convex

 Pointwise infimum of a set of concave functions is concave

∈𝒝

slide-37
SLIDE 37

Examples

 Support function of a set

  • ∈
  •  Distance to farthest point of a set

∈

slide-38
SLIDE 38

Examples

 Maximum eigenvalue of a symmetric matrix

  •  Norm of a matrix

is maximum singular value

  • f

slide-39
SLIDE 39

Representation

 Almost every convex function can be expressed as the pointwise supremum

  • f a family of affine functions.

⟹ 𝑔 𝑦 sup𝑕𝑦|𝑕 affine, 𝑕 z 𝑔 𝑨 ∀𝑨 𝑔: 𝐒 → 𝐒 is convex and dom 𝑔 𝐒

slide-40
SLIDE 40

Compositions

 Definition

 Chain Rule

  • 𝑔 𝑦 ℎ 𝑕 𝑦

𝛼𝑔𝑦 ℎ𝑕𝑦𝛼𝑕𝑦 ℎ 𝑕 𝑦 𝛼𝑕 𝑦 𝛼𝑕𝑦

slide-41
SLIDE 41

Scalar Composition

 and are twice differentiable   is convex, if

  •  ℎ is convex and nondecreasing 𝑕 is convex

  •  ℎ is convex and nonincreasing, 𝑕 is concave

𝑔 𝑦 ℎ 𝑕 𝑦 𝑕′ 𝑦 ℎ 𝑕 𝑦 𝑕′′𝑦

slide-42
SLIDE 42

Scalar Composition

 and are twice differentiable   is concave, if

  •  ℎ is concave and nondecreasing 𝑕 is concave

  •  ℎ is concave and nonincreasing, 𝑕 is convex

𝑔 𝑦 ℎ 𝑕 𝑦 𝑕′ 𝑦 ℎ 𝑕 𝑦 𝑕′′𝑦

slide-43
SLIDE 43

  •  Without differentiability assumption

 Without domain condition  ℎ 𝑦 0 with dom ℎ 1,2, which is convex and non-decreasing  𝑕 𝑦 𝑦 with dom 𝑕 𝐒, which is convex  dom 𝑔 2, 1 ∪ 1, 2 𝑔 𝑦 ℎ 𝑕 𝑦

Scalar Composition

slide-44
SLIDE 44

  •  Without differentiability assumption

 Without domain condition  ℎ is convex, ℎ is nondecreasing, and 𝑕 is convex ⇒ 𝑔 is convex  ℎ is convex, ℎ is nonincreasing, and 𝑕 is concave ⇒ 𝑔 is convex  The conditions for concave are similar

Scalar Composition

slide-45
SLIDE 45

Extended-value Extensions

slide-46
SLIDE 46

Examples

 is convex is convex  is concave and positive is concave  is concave and positive is convex  is convex and nonnegative and

is convex

 is convex is convex on

slide-47
SLIDE 47

Vector Composition

  •  ℎ and 𝑕 are twice differentiable

 dom 𝑕 𝐒, dom ℎ 𝐒 𝑔 ℎ ∘ 𝑕 ℎ𝑕 𝑦 , … , 𝑕 𝑦 𝑔 𝑦 𝑕 𝑦 𝛼ℎ 𝑕 𝑦 𝑕′𝑦 𝛼ℎ 𝑕 𝑦

𝑕′′𝑦

𝑔′ 𝑦 𝛼ℎ 𝑕 𝑦

𝑕′𝑦

slide-48
SLIDE 48

Vector Composition

  •  ℎ and 𝑕 are twice differentiable

 dom 𝑕 𝐒, dom ℎ 𝐒  𝑔 is convex, if 𝑔 𝑦 0

 ℎ is convex, ℎ is nondecreasing in each argument, and 𝑕 are convex  ℎ is convex, ℎ is nonincreasing in each argument, and 𝑕 are concave

𝑔 ℎ ∘ 𝑕 ℎ𝑕 𝑦 , … , 𝑕 𝑦 𝑔 𝑦 𝑕 𝑦 𝛼ℎ 𝑕 𝑦 𝑕′𝑦 𝛼ℎ 𝑕 𝑦

𝑕′′𝑦

slide-49
SLIDE 49

Vector Composition

  •  ℎ and 𝑕 are twice differentiable

 dom 𝑕 𝐒, dom ℎ 𝐒  𝑔 is concave, if 𝑔 𝑦 0

 ℎ is concave, ℎ is nondecreasing in each argument, and 𝑕 are concave

 The general case is similar

𝑔 ℎ ∘ 𝑕 ℎ𝑕 𝑦 , … , 𝑕 𝑦 𝑔 𝑦 𝑕 𝑦 𝛼ℎ 𝑕 𝑦 𝑕′𝑦 𝛼ℎ 𝑕 𝑦

𝑕′′𝑦

slide-50
SLIDE 50

Examples

 ℎ 𝑨 𝑨 ⋯ 𝑨 , 𝑨 ∈ 𝐒, 𝑕, … , 𝑕 is convex ⇒ ℎ ∘ 𝑕 is convex  ℎ 𝑨 log∑ 𝑓

  • , 𝑕, … , 𝑕 is convex ⇒ ℎ ∘

𝑕 is convex  ℎ 𝑨 ∑ 𝑨

  • / on 𝐒

is concave for 0 𝑞

1, and its extension is nondecreasing. If 𝑕 is concave and nonnegative ⇒ ℎ ∘ 𝑕 is concave

slide-51
SLIDE 51

Minimization

 is convex in is convex

 𝑕 𝑦 inf

∈ 𝑔𝑦, 𝑧 is convex if 𝑕 𝑦

∞, ∀ 𝑦 ∈ dom 𝑕  dom 𝑕 𝑦 𝑦, 𝑧 ∈ dom 𝑔 for some 𝑧 ∈ 𝐷

 Proof by Epigraph

 epi 𝑕 𝑦, 𝑢| 𝑦, 𝑧, 𝑢 ∈ epi 𝑔 for some 𝑧 ∈ 𝐷  The projection of a convex set is convex.

slide-52
SLIDE 52

Examples

 Schur complement

 𝑔 𝑦, 𝑧 𝑦𝐵𝑦 2𝑦𝐶𝑧 𝑧𝐷𝑧  𝐵 𝐶 𝐶 𝐷 ≽ 0, 𝐵, 𝐷 is symmetric ⇒ 𝑔𝑦, 𝑧 is convex  𝑕 𝑦 inf

𝑔 𝑦, 𝑧 𝑦 𝐵 𝐶𝐷𝐶 𝑦 is convex

⇒ 𝐵 𝐶𝐷𝐶 ≽ 0, 𝐷 is the pseudo-inverse of 𝐷

 Distance to a set

 𝑇 is a convex nonempty set,𝑔 𝑦, 𝑧 ‖𝑦 𝑧‖ is convex in 𝑦, 𝑧  𝑕 𝑦 dist 𝑦, 𝑇 inf

∈ ‖𝑦 𝑧‖

slide-53
SLIDE 53

Examples

 Affine domain

 ℎ𝑧 is convex  𝑕 𝑦 inf ℎ𝑧|𝐵𝑧 𝑦 is convex

 Proof

 𝑔 𝑦, 𝑧 ℎ 𝑧 if 𝐵𝑧 𝑦 ∞ otherwise  𝑔𝑦, 𝑧 is convex in 𝑦, 𝑧  𝑕 is the minimum of 𝑔 over 𝑧

slide-54
SLIDE 54

Perspective of a function

  • defined as

is the perspective of

 dom 𝑕 𝑦, 𝑢|𝑦/𝑢 ∈ dom 𝑔, 𝑢 0

 𝑔 is convex ⇒ 𝑕 is convex

 Proof

 Perspective mapping preserve convexity 𝑦, 𝑢, 𝑡 ∈ epi 𝑕 ⇔ 𝑢𝑔 𝑦 𝑢 𝑡 ⇔ 𝑔 𝑦 𝑢 𝑡 𝑢 ⇔ 𝑦/𝑢, 𝑡/𝑢 ∈ epi 𝑔

slide-55
SLIDE 55

Example

 Euclidean norm squared

  •  Composition with an Affine function

  • is convex

  • is convex
slide-56
SLIDE 56

Outline

 Basic Properties

 Definition  First-order Conditions, Second-order Conditions  Jensen’s inequality and extensions  Epigraph

 Operations That Preserve Convexity

 Nonnegative Weighted Sums  Composition with an affine mapping  Pointwise maximum and supremum  Composition  Minimization  Perspective of a function

 Summary

slide-57
SLIDE 57

Summary

 Basic Properties

 Definition  First-order Conditions, Second-order Conditions  Jensen’s inequality and extensions  Epigraph

 Operations That Preserve Convexity

 Nonnegative Weighted Sums  Composition with an affine mapping  Pointwise maximum and supremum  Composition  Minimization  Perspective of a function