Convex Optimization 3. Convex Functions Prof. Ying Cui Department - - PowerPoint PPT Presentation

Convex Optimization

• 3. Convex Functions
• Prof. Ying Cui

Department of Electrical Engineering Shanghai Jiao Tong University

2018

SJTU Ying Cui 1 / 42

Outline

Basic properties and examples Operations that preserve convexity The conjugate function Quasiconvex functions Log-concave and log-convex functions Convexity with respect to generalized inequalities

SJTU Ying Cui 2 / 42

Definition

◮ convex: f : Rn → R is convex if domf is a convex set and if f (θx + (1 − θ)y) ≤ θf (x) + (1 − θ)f (y) for all x, y ∈ domf , and θ with 0 ≤ θ ≤ 1

◮ geometric interpretation: line segment between (x, f (x)) and (y, f (y)) (i.e., chord from x to y) lies above graph of f

(x, f(x)) (y, f(y)) Figure 3.1 Graph of a convex function. The chord (i.e., line segment) be- tween any two points on the graph lies above the graph.

◮ concave: f is concave if −f is convex

SJTU Ying Cui 3 / 42

Definition

◮ strictly convex: f : Rn → R is strictly convex if domf is a convex set and if f (θx + (1 − θ)y) < θf (x) + (1 − θ)f (y) for all x, y ∈ domf , x = y, and θ with 0 < θ < 1 ◮ strictly concave: f is strictly concave if −f is strictly convex ◮ affine functions are both convex and concave ◮ any function that is convex and concave is affine

SJTU Ying Cui 4 / 42

Examples on R

convex: ◮ affine: ax+b on R, for any a, b ∈ R ◮ exponential: eax on R, for any a ∈ R ◮ powers: xα on R++, for α ≥ 1 or α ≤ 0 ◮ powers of absolute value: |x|p on R, for p ≥ 1 ◮ negative entropy: x log x on R++ concave: ◮ affine: ax+b on R, for any a, b ∈ R ◮ powers: xα on R++, for 0≤α≤ 1 ◮ logarithm: log x on R++

SJTU Ying Cui 5 / 42

Examples on Rn and Rm×n

Examples on Rn: ◮ affine function f (x)=aTx+b is both convex and concave ◮ every norm is convex

◮ due to triangle inequality and homogeneity ◮ lp-norms: ||x||p=(n

i=1 | xi |p)1/p for p ≥ 1 (||x||1=n i=1 |xi|,

||x||∞=maxk |xk|)

◮ max function f (x) = max{x1, · · · , xn} is convex ◮ log-sum-exp f (x) = log(ex1 + · · · + exn) is convex

◮ a differentiable approximation of the max function: log(ex1+· · ·+exn)−log n ≤ max{x1, · · · , xn} ≤ log(ex1+· · ·+exn)

Examples on Rm×n: ◮ affine function f (X) = tr(ATX) + b = m

i=1

n

j=1 AijXij + b

is both convex and concave ◮ spectral (maximum singular value) norm f (X) = X2 = σmax(X) = (λmax(X TX))1/2 on is convex

SJTU Ying Cui 6 / 42

Restriction of a convex function to a line

◮ a function f : Rn → R is convex iff it is convex when restricted to any line that intersects its domain, i.e.,

◮ g(t) = f (x + tv) is convex on {t|x + tv ∈ domf } for all x ∈ domf and all v ∈ Rn

◮ check convexity of a function of multiple variables by restricting it to a line and checking convexity of a function of

• ne variable

◮ example: f : Sn → R with f (X) = log det X, domf = Sn

++

Consider an arbitrary line X = Z + tV ∈ Sn

++ with Z, V ∈ Sn.

• w. l. o. g., assume t = 0 is in the interval, i.e., Z ∈ Sn

++.

g(t) = log det(Z + tV ) = log det(Z 1/2(I + tZ −1/2VZ −1/2)Z 1/2) = log det Z + log det(I + tZ −1/2VZ −1/2) = log det Z +

n

• i=1

log(1 + tλi) λi: eigenvalues of Z −1/2VZ −1/2 g is concave in t. Thus, f is concave.

SJTU Ying Cui 7 / 42

Extended-value extension

◮ extended-value extension ˜ f of a convex function f is ˜ f (x) =

• f (x),

x ∈ domf ∞, x / ∈ domf

◮ ˜ f is defined on all Rn, and takes values in R ∪ {∞} ◮ recover domain of f from ˜ f as domf = {x|˜ f (x) < ∞}

◮ extension can simplify notation, as no need to explicitly describe the domain, or add the qualifier ‘for all x ∈ domf ’ ◮ basic defining inequality for convexity can be expressed as: for 0 < θ < 1, ˜ f (θx + (1 − θ)y) ≤ θ˜ f (x) + (1 − θ)˜ f (y) for any x and y

◮ the inequality always holds for θ = 0, 1 ◮ no need to mention the two conditions: domf is convex (can be shown by contradiction) and x, y ∈ domf (x, y ∈ Rn is used instead, which can be omitted)

SJTU Ying Cui 8 / 42

First-order conditions

Suppose f is differentiable, i.e., domf is open and gradient ∇f (x) =

• ∂f (x)

∂x1 , · · · , ∂f (x) ∂xn

• exists at any x ∈ domf

◮ f is convex iff domf is convex and f (y) ≥ f (x) + ∇f (x)T (y − x) for all x, y ∈ domf

◮ first-order Taylor approx. of a convex function is a global underestimator of it; if first-order Taylor approx. of a function is always a global underestimator of it, then it is convex ◮ local information about a convex function (value and derivative at a point) implies global information (a global underestimator) ◮ if f is convex and ∇f (x) = 0, then x is a global minimizer of f

◮ f is strictly convex iff domf is convex and f (y) > f (x) + ∇f (x)T (y − x) for all x, y ∈ domf and x = y

(x, f(x)) f(y) f(x) + ∇f(x)T (y − x) Figure 3.2 If f is convex and differentiable, then f(x)+∇f(x)T (y−x) ≤ f(y) for all x, y ∈ dom f.

SJTU Ying Cui 9 / 42

Second-order conditions

Suppose f is twice differentiable, i.e., domf is open and Hessian ∇2f (x) ∈ Sn exists at any x ∈ domf , where ∇2f (x)ij = ∂2f (x)

∂xi∂xj ,

i, j = 1, · · · , n ◮ f is convex iff domf is convex and ∇2f (x) 0 for all x ∈ domf

◮ for a function on R, this reduces to domf is an interval and f ′′(x) ≥ 0 for all x in the interval ◮ ∇2f (x) 0 means the graph of f has positive (upward) curvature at x

◮ if domf is convex and ∇2f (x) ≻ 0 for all x ∈ domf , then f is strictly convex

◮ the converse is not true, e.g., f (x) = x4 is strictly convex but f ′′(0) = 0

SJTU Ying Cui 10 / 42

Second-order conditions

Examples ◮ quadratic function: f (x) = (1/2)xT Px + qTx + r (P ∈ Sn) ∇f (x) = Px + q, ∇2f (x) = P convex iff P ∈ Sn

+

◮ least-squares objective: f (x) = Ax − b2

2 = xTATAx − 2xT ATb + bTb

∇f (x) = 2AT(Ax − b), ∇2f (x) = 2ATA convex for all A ∈ Rm×n (as ATA 0 for all A ∈ Rm×n) ◮ quadratic-over-linear function: f (x, y) = x2/y ∇2f (x, y) = 2 y 3 y −x y −x T convex for all x ∈ R and y ∈ R++ (as zzT 0 for all z ∈ Rn)

SJTU Ying Cui 11 / 42

Second-order conditions

Examples ◮ log-sum-exp: f (x) =logn

k=1 exp xk is convex

◮ proof: ∇2f (x) = 1 1Tz diag(z) − 1 (1Tz)2 zzT (zk = exp xk) to show ∇2f (x) 0, we must verify that v T∇2f (x)v ≥ 0 for all v: v T ∇2f (x)v = (

k zkv 2 k )( k zk) − ( k vkzk)2

(

k zk)2

≥ 0 since (

k vkzk)2 ≤ ( k zkv 2 k )( k zk) (from Cauchy-Schwarz

inequality (aT a)(bTb) ≥ (aTb)2 by treating ai = vi√zi and bi = √zi)

◮ geometric mean: f (x) = (n

k=1 xk)1/n on Rn ++ is concave

(similar proof as for log-sum-exp)

SJTU Ying Cui 12 / 42

Sublevel set and superlevel set

Sublevel set ◮ α-sublevel set of f : Rn → R: {x ∈ domf |f (x) ≤ α} ◮ sublevel sets of a convex function are convex

◮ the converse is false (e.g., f (x) = − exp x is not convex (indeed, strictly concave) but all its sublevel sets are convex)

Superlevel set ◮ α-superlevel set of f : Rn → R: {x ∈ domf |f (x) ≥ α} ◮ superlevel sets of a concave function are convex To establish convexity of a set, express it as a sublevel set of a convex function, or as the superlevel set of a concave function.

SJTU Ying Cui 13 / 42

Epigraph and hypergraph

◮ graph of f : Rn → R: {(x, f (x))|x ∈ domf } ⊆ Rn+1 ◮ epigraph of f : Rn → R: epi f = {(x, t) ∈ Rn+1|x ∈ domf , f (x) ≤ t} ⊆ Rn+1

◮ f is convex iff epi f is a convex set

◮ hypograph of f : Rn → R: hypo f = {(x, t) ∈ Rn+1|x ∈ domf , f (x) ≥ t} ⊆ Rn+1

◮ f is concave iff hypo f is a convex set

epi f f Figure 3.5 Epigraph of a function f, shown shaded. The lower boundary, shown darker, is the graph of f.

SJTU Ying Cui 14 / 42

Jensen’s inequality and extensions

◮ basic inequality: if f is convex, x, y ∈ domf and 0 ≤ θ ≤ 1, then f (θx + (1 − θ)y) ≤ θf (x) + (1 − θ)f (y) ◮ extension to convex combinations of more than two points: if f is convex, x1, · · · , xk ∈ domf , and θ1, · · · , θk ≥ 0 with θ1 + · · · + θk = 1, then f (θ1x1 + · · · + θkxk) ≤ θ1f (x1) + · · · + θkf (xk) ◮ extensions to infinite sums and integrals (if p(x) ≥ 0 on S ⊆ domf ,

• S p(x)dx = 1, then

f

• S p(x)xdx
• ≤
• S f (x)p(x)dx, provided the integrals exist)

◮ extension to expected values: if f is convex and X is a random variable such that X ∈ domf w.p. 1, then f (EX) ≤ Ef (X), provided the expectations exist ◮ many famous inequalities (e.g., arithmetic-geometric mean inequality and H¨

• lder’s inequality) can be derived by applying

Jensen’s inequality to some convex function

SJTU Ying Cui 15 / 42

Operations that preserve convexity

practical methods for establishing convexity of a function ◮ verify definition (often simplified by restricting to a line) ◮ for twice differentiable functions, show ∇2f (x) 0 ◮ show that f is obtained from simple convex functions by

• perations that preserve convexity

◮ nonnegative weighted sum ◮ composition with affine function ◮ pointwise maximum and supremum ◮ composition ◮ minimization ◮ perspective

SJTU Ying Cui 16 / 42

Nonnegative weighted sums

◮ nonnegative multiple: if f is convex and α ≥ 0, then αf is convex ◮ sum: if f1 and f2 are convex, then f1 + f2 is convex ◮ nonnegative weighted sum: if f1, · · · , fm are convex and w1, · · · , wm ≥ 0, then w1f1 + · · · + wmfm is convex

◮ extension to infinite sums ◮ extension to integrals: if f (x, y) is convex in x for each y ∈ A, and w(y) ≥ 0 for each y ∈ A, then g(x) =

• A w(y)f (x, y)dy

is convex in x, provided the integral exists

SJTU Ying Cui 17 / 42

Composition with an affine function

Suppose f : Rn → R, A ∈ Rn×m, and b ∈ Rn. Define g : Rm → R by g(x) = f (Ax + b), domg = {x|Ax + b ∈ domf } Then if f is convex, so is g; if f is concave, so is g. examples ◮ log barrier for linear inequalities f (x) = −

m

• i=1

log(bi−aT

i x), domf = {x|aT i x < bi, i = 1, ..., m}

is convex ◮ (any) norm of affine function f (x) = ||Ax + b|| is convex

SJTU Ying Cui 18 / 42

Pointwise maximum

If f1, ..., fm are convex, then their pointwise maximum f , defined by f (x) = max{f1(x), ...., fm(x)}, domf = domf1 ∩ · · · ∩ domfm is convex. (over a finite set of convex functions) examples ◮ piecewise-linear function: f (x) = maxi=1,...,m(aT

i x + bi) is

convex ◮ sum of r largest components of x ∈ Rn: f (x) = r

i=1 x[i] is

convex, where x[i] is ith largest component of x

◮ proof: f (x) = max{xi1 + · · · + xir |1 ≤ i1 < i2 < · · · < ir ≤ n} is the pointwise maximum of n

r

• linear functions

◮ extension: if w1 ≥ w2 ≥ · · · ≥ wr ≥ 0, then f (x) = r

i=1 wix[i] is convex

◮ proof: f (x) = wr r

i=1 x[i]+(wr−1 −wr) r−1 i=1 x[i] +· · ·+(w2−w1)x[1]

is a nonnegtive weighted sum of (r + 1) convex functions

SJTU Ying Cui 19 / 42

Pointwise supremum

If f (x, y) is convex in x for each y ∈ A, then g(x) = sup

y∈A

f (x, y), domg = {x|(x, y) ∈ domf ∀y ∈ A, g(x) < ∞} is convex in x. (over an infinite set of convex functions) examples ◮ support function of set C ⊆ Rn, C = ∅: SC(x) = sup

y∈C

y Tx is convex ◮ distance to farthest point in set C ⊆ Rn: f (x) = sup

y∈C

||x − y|| is convex ◮ maximum eigenvalue of symmetric matrix: λmax(X) = sup

||y||2=1

y TXy with domf = Sn is convex ◮ spectral norm (maximum singular value) of a matrix: ||X||2 = sup

||u||2=1,||v||2=1

uTXv with domf = Rp×q is convex

SJTU Ying Cui 20 / 42

Scalar Composition

composition of g : Rn → R and h : R → R: f (x) = h(g(x)), domf = {x ∈ domg|g(x) ∈ domh} f is convex if

• h convex, ˜

h nondecreasing, g convex h convex, ˜ h nonincreasing, g concave f is concave if

• h concave, ˜

h nondecreasing, g concave h concave, ˜ h nonincreasing, g convex ◮ proof (n = 1, g, h twice differentiable, domg = domh = R) f ′′(x) = h′′(g(x))g′(x)2 + h′(g(x))g′′(x) ◮ note: monotonicity must hold for extended-value extension ˜ h

SJTU Ying Cui 21 / 42

Scalar Composition

examples ◮ if g is convex then exp g(x) is convex ◮ if g is concave and positive, then log g(x) is concave ◮ if g is concave and positive, then 1/g(x) is convex ◮ if g is convex and nonnegative and p ≥ 1, then g(x)p is convex ◮ if g is convex then − log(−g(x)) is convex on {x|g(x) < 0}

SJTU Ying Cui 22 / 42

Vector composition

composition of g : Rn → Rk and h : Rk → R: f (x) = h(g(x)) = h(g1(x), g2(x), ..., gk(x)) f is convex if

• h convex, ˜

h nondecreasing in each argument, gi convex h convex, ˜ h nonincreasing in each argument, gi concave f is concave if

• h concave, ˜

h nondecreasing in each argument, gi concave h concave, ˜ h nonincreasing in each argument, gi convex ◮ proof (n = 1, g, h twice differentiable, domg = R, domh = Rk): f ′′(x) = g′(x)T ∇2h(g(x))g′(x) + ∇h(g(x))T g′′(x)

SJTU Ying Cui 23 / 42

Vector composition

examples ◮ log(m

i=1 exp gi(x)) is convex if gi are convex

◮ m

i=1 log gi(x) is concave if gi are concave and positive

◮ (k

i=1 gi(x)p)1/p, 0 < p ≤ 1 is concave if gi are concave and

nonnegative ◮ (k

i=1 gi(x)p)1/p, p ≥ 1 is convex if gi are convex and

nonnegative ◮ (k

i=1 gi(x))1/k is concave if gi are concave and nonnegative

SJTU Ying Cui 24 / 42

Minimization

If f (x, y) is convex in (x, y) and C is a convex nonempty set, then g(x) = inf

y∈C f (x, y),

domg = {x|(x, y) ∈ domf for some y ∈ C} is convex examples ◮ Schur complement: f (x, y) = xT Ax + 2xT By + y TCy with X = A B BT C

• 0, C ≻ 0

Thus, g(x) = inf

y f (x, y) = xT(A − BC −1BT)x is convex,

implying Schur complement of C in matrix X: A − BC −1BT 0

◮ note: f (x, y) = x y T X x y

• is convex in (x, y) as X 0

◮ distance to a set: dist(x, S) = inf

y∈S ||x − y|| is convex if S is

convex

◮ note: ||x − y|| is convex in in (x, y)

SJTU Ying Cui 25 / 42

Perspective of a function

perspective of a function f : Rn → R is the function g : Rn × R → R g(x, t) = tf (x/t), domg = {(x, t)|x/t ∈ domf , t > 0} If f is convex, g is convex; if f is concave, g is concave examples ◮ Euclidean norm squared: perspective of convex function f (x) = xTx on Rn: g(x, t) = t(x/t)T(x/t) = xT x/t is convex in (x, t) for t > 0 ◮ negative logarithm: perspective of convex function f (x) = − log x on R++: g(x, t) = −t log(x/t) = t log(t/x) = t log t − t log x (called relative entropy) is convex on R2

++

◮ if f is convex, then g(x) = (cT x + d)f ((Ax + b)/(cT x + d)) is convex on domg = {x|cT x + d > 0, (Ax + b)/(cT x + d) ∈ domf }

SJTU Ying Cui 26 / 42

Conjugate function

conjugate f ∗ : Rn → R of a function f : Rn → R is f ∗(y) = sup

x∈domf

(y Tx − f (x)) domf ∗ consists of y ∈ Rn for which the supremum is finite, i.e., for which the difference y Tx − f (x) is bounded above on domf

f(x) (0, −f ∗(y)) xy x Figure 3.8 A function f : R → R, and a value y ∈ R. The conjugate function f ∗(y) is the maximum gap between the linear function yx and f(x), as shown by the dashed line in the figure. If f is differentiable, this

• ccurs at a point x where f ′(x) = y.

◮ f ∗ is convex (no matter whether f is or not)

◮ proof: it is the pointwise supremum of a family of convex (indeed, affine) functions of y

◮ will be useful in Chapter 5

SJTU Ying Cui 27 / 42

Examples

◮ affine function f (x) = ax + b domf ∗ = {a}, f ∗(y) = sup

x (yx−(ax+b)) =

• −b,

y = a ∞,

• therwise

◮ negative logarithm f (x) = − log x with domf = R++ domf ∗ = −R++, f ∗(y) = sup

x>0

(yx + log x) =

• −1 − log(−y),

y < 0 ∞,

• therwise

◮ strictly convex quadratic function f (x) = 1

2xTQx with

Q ∈ Sn

++

domf ∗ = Rn, f ∗(y) = sup

x (y Tx − 1

2xTQx) = 1 2y TQ−1y

SJTU Ying Cui 28 / 42

Basic properties

◮ Fenchel’s inequality: f (x) + f ∗(y) ≥ xTy for all x, y ◮ conjugate of the conjugate: if f is convex and closed (i.e., epif is a closed set), then f ∗∗ = f ◮ differentiable functions: if f is convex and differentiable with domf = Rn, then f ∗(y) = x∗T ∇f (x∗) − f (x∗), where x∗ satisfies y = ∇f (x∗) ◮ scaling and composition with affine transformation:

◮ for a > 0 and b ∈ R, the conjugate of g(x) = af (x) + b is g ∗(y) = af ∗(y/a) − b ◮ for nonsigular A ∈ Rn×n and b ∈ Rn, the conjugate of g(x) = f (Ax + b) is g ∗(y) = f ∗(A−T y) − bTA−Ty with domg ∗ = AT domf

◮ sum of independent functions: if f (u, v) = f1(u) + f2(v), where f1 and f2 are convex functions with conjugates f ∗

1 and

f ∗

2 , respectively, then f ∗(w, z) = f ∗ 1 (w) + f ∗ 2 (z)

SJTU Ying Cui 29 / 42

Quasiconvex functions

f : Rn → R is quasiconvex if domf is convex and all its sublevel sets Sα = {x ∈ domf | f (x) ≤ α}, a ∈ R are convex

α β a b c Figure 3.9 A quasiconvex function on R. For each α, the α-sublevel set Sα is convex, i.e., an interval. The sublevel set Sα is the interval [a, b]. The sublevel set Sβ is the interval (−∞, c].

◮ convex/quasiconvex functions have convex sublevel sets ◮ for a function on R, quasiconvexity requires that each sublevel set is an interval ◮ f is quasiconcave if −f is quasiconvex, i.e., all its super level sets {x ∈ domf | f (x) ≥ α}, a ∈ R are convex ◮ f is quasilinear if it is quasiconvex and quasiconcave, i.e., all its level sets {x ∈ domf | f (x) = α}, a ∈ R are convex

SJTU Ying Cui 30 / 42

Examples

◮ |x| is quasiconvex on R ◮ ceil(x) = inf{z ∈ Z | z ≥ x} is quasilinear ◮ log x is quasilinear on R++ ◮ f (x1, x2) = x1x2 is quasiconcave on R2

++

◮ linear-fractional function f (x) = aTx + b cTx + d , domf = {x | cTx + d > 0} is quasilinear ◮ distance ratio function f (x) = x − a2 x − b2 , domf = {x | x − a2 ≤ x − b2} is quasiconvex

SJTU Ying Cui 31 / 42

Basic properties

A function f is quasiconvex iff domf is convex and for any x, y ∈ domf and 0 ≤ θ ≤ 1, f (θx + (1 − θ)y) ≤ max{f (x), f (y)} which is called Jensen’s inequality for quasiconvex functions. ◮ the value of the function on a segment does not exceed the maximum of its values at the endpoints

(x, f(x)) (y, f(y)) max{f(x), f(y)}

Figure 3.10 A quasiconvex function on R. The value of f between x and y is no more than max{f(x), f(y)}.

SJTU Ying Cui 32 / 42

Basic properties

Examples ◮ cardinality (number of nonzero components) of a nonnegative vector: card(x) is quasiconcave on Rn

+ (but not Rn), as

card(x + y) ≥ min{card(x), card(y)} holds for x, y 0 ◮ rank of positive semidefinite matrix: rankX is quasiconcave

• n Sn

+, as rank(X + Y ) ≥ min{rankX, rankY } which holds

for X, Y ∈ Sn

+

SJTU Ying Cui 33 / 42

Basic properties

◮ f is quasiconvex iff its restriction to any line intersecting its domain is quasiconvex ◮ a continuous function f : R → R is quasiconvex iff at least

• ne of the following conditions holds:

◮ f is nondecreasing ◮ f is nonincreasing ◮ there is a point c ∈ domf such that for t ≤ c (and t ∈ domf ), f is nonincreasing, and for t ≥ c (and t ∈ domf ), f is nondecreasing, implying c is a global minimizer of f

c t

Figure 3.11 A quasiconvex function on R. The function is nonincreasing for t ≤ c and nondecreasing for t ≥ c.

SJTU Ying Cui 34 / 42

Differentiable quasiconvex functions

First-order conditions Suppose f : Rn → R is differentiable. ◮ f is quasiconvex iff domf is convex and for all x, y ∈ domf , f (y) ≤ f (x) ⇒ ∇f (x)T (y − x) ≤ 0

◮ geometric interpretation: ∇f (x) (when = 0) defines a supporting hyperplane to sublevel set {y|f (y) ≤ f (x)}, at x

x ∇f(x) Figure 3.12 Three level curves of a quasiconvex function f are shown. The vector ∇f(x) defines a supporting hyperplane to the sublevel set {z | f(z) ≤ f(x)} at x.

SJTU Ying Cui 35 / 42

Differentiable quasiconvex functions

Second-order conditions Suppose f : Rn → R is twice differentiable. ◮ if f is quasiconvex, then for all x ∈ domf and all y ∈ Rn, y T∇f (x) = 0 ⇒ y T∇2f (x)y ≥ 0

◮ when ∇f (x) = 0, ∇2f (x) 0 ◮ when ∇f (x) = 0, ∇2f (x) is positive semidefinite on the (n − 1)-dimensional subspace ∇f (x)⊥, implying that ∇2f (x) can have at most one negative eigenvalue

◮ if f is quasiconvex on R, then for all x ∈ domf , f ′(x) = 0 ⇒ f ′′(x) ≥ 0

◮ at any point with zero slope, the second derivative is nonnegative

◮ partial converse: if f satisfies y T∇f (x) = 0 ⇒ y T∇2f (x)y > 0 for all x ∈ domf and all y ∈ Rn, y = 0, then f is quasiconvex

SJTU Ying Cui 36 / 42

Operations that preserve quasiconvexity

Nonnegative weighted maximum ◮ if f1, · · · , fm are quasiconvex and w1, · · · , wm ≥ 0, then f = max{w1f1, · · · , wmfm} is quasiconvex ◮ if g(x, y) is quasiconvex in x and w(y) ≥ 0, for all y ∈ C, then f (x) = supy∈C(w(y)g(x, y)) is quasiconvex Composition ◮ if g : Rn → R is quasiconvex and h : R → R is nondecreasing, then h(g(x)) is quasiconvex ◮ composition of a quasiconvex function with an affine or linear-fractional transformation yields a quasiconvex function Minimization ◮ if f (x, y) is quasiconvex jointly in x and y and C is a convex set, then the function g(x) = infy∈C f (x, y) is quasiconvex

SJTU Ying Cui 37 / 42

Log-concave and log-convex functions

◮ a function f : Rn → R is log-concave if f (x) > 0 for all x ∈ domf and log f is concave ◮ a function f : Rn → R is log-convex if f (x) > 0 for all x ∈ domf and log f is convex ◮ f is log-convex iff 1/f is log-concave ◮ a function f : Rn → R with convex domain and f (x) > 0 for all x ∈ domf is log-concave iff for all x, y ∈ domf and 0 ≤ θ ≤ 1, f (θx + (1 − θ)y) ≥ f (x)θf (y)1−θ

◮ interpretation: the value at the average of two points is at least the geometric mean of the values at the two points

◮ a log-convex function is convex

◮ composition rule: if g is convex, then eg is convex (g = log f )

◮ a nonnegative concave function is log-concave

◮ composition rule: if g is concave, then log g is concave

◮ a log-convex (-concave) function is quasiconvex (-concave)

◮ any sublevel (superlevel) set of log f is convex and log is monotone increasing

SJTU Ying Cui 38 / 42

Examples

◮ affine function: f (x) = aTx + b is log-concave on {x|aT x + b > 0} ◮ powers: xa on R++ is log-convex for a ≤ 0, and log-concave for a ≥ 0 ◮ exponentials: f (x) = exp(ax) is log-convex and log-concave ◮ cumulative Gaussian distribution function: Φ(x) =

1 √ 2π

x

−∞ e−u2/2du is log-concave

◮ Gamma function: Γ(x) = ∞

0 ux−1e−u is log-convex for x ≥ 1

◮ determinant: detX is log-concave on Sn

++

◮ determinant over trace: detX/trX is log-concave on Sn

++

◮ many common probability densities are log-concave, e.g., multivariate normal distribution, exponential distribution on Rn

+, and uniform distribution over a convex set

SJTU Ying Cui 39 / 42

Properties

Twice differentiable log-convex/concave functions Suppose f is twice differentiable, with domf convex, so ∇2 log f (x) = 1 f (x)∇2f (x) − 1 f (x)2 ∇f (x)∇f (x)T ◮ f is log-concave iff for all x ∈ domf , f (x)∇2f (x) ∇f (x)∇f (x)T ◮ f is log-convex iff for all x ∈ domf , f (x)∇2f (x) ∇f (x)∇f (x)T Multiplication, addition, and integration ◮ product of log-concave (log-convex) functions is log-concave (log-convex) ◮ positive scaling of log-concave (log-convex) functions is log-concave (log-convex) ◮ sum (integral) of log-concave functions is not always log-concave, but sum (integral) of log-convex functions is always log-convex

SJTU Ying Cui 40 / 42

Consequences of integration property

Integration of log-concave functions If f : Rn × Rm → R is log-concave, then g(x) =

• f (x, y)dy is

log-concave in x on Rn (integration is over Rm) ◮ marginal distributions of log-concave probability densities are log-concave ◮ convolution f ∗ g of log-concave functions f , g, (f ∗ g)(x) =

• f (x − y)g(y)dy, is log-concave

◮ proof: f (x − y) is log-concave in (x, y) (as f is concave), and product of log-concave functions is log-concave

◮ if C ⊆ Rn is convex and y is a random variable in Rn with log-concave p.d.f. p(y), then f (x) = prob(x + y ∈ C) is log-concave

◮ proof: write f (x) as integral of product of log-concave functions (which is still log-concave) f (x) =

• g(x + y)p(y)dy,

g(u) =

• 1

u ∈ C u / ∈ C

SJTU Ying Cui 41 / 42

Convexity with respect to generalized inequalities

Suppose K ⊆ Rm is a proper cone with associated generalized inequality K ◮ f : Rn → Rm is K-convex if domf is convex and f (θx + (1 − θ)y) K θf (x) + (1 − θ)f (y) for x, y ∈ domf and 0 ≤ θ ≤ 1 ◮ f : Rn → Rm is strictly K-convex if domf is convex and f (θx + (1 − θ)y) ≺K θf (x) + (1 − θ)f (y) for x, y ∈ domf , x = y and 0 ≤ θ ≤ 1 ◮ these definitions reduce to ordinary convexity and strict convexity when m = 1 and K = R+. Many of the results for convex functions have extensions to K-convex functions.

SJTU Ying Cui 42 / 42