Convex Optimization 8. Geometric Problems Prof. Ying Cui - - PowerPoint PPT Presentation

Convex Optimization

• 8. Geometric Problems
• Prof. Ying Cui

Department of Electrical Engineering Shanghai Jiao Tong University

2018

SJTU Ying Cui 1 / 26

Outline

Extremal volume ellipsoids Centering Classification Placement and facility location

SJTU Ying Cui 2 / 26

Minimum volume ellipsoid covering a set

L¨

• wner-John ellipsoid of a set C ⊆ Rn (denoted by Elj):

minimum volume ellipsoid that contains set C ◮ parametrize a general ellipsoid E as E = {v | Av + b2 ≤ 1} (A ∈ Sn, b ∈ Rn)

◮ inverse image of Euclidean unit ball under an affine mapping ◮ w.l.o.g. assume A ∈ Sn

++

◮ volume of E is proportional to detA−1 ◮ find minimum volume ellipsoid containing C min

A,b

log det A−1 s.t. supv∈C Av + b2 ≤ 1

◮ convex problem (objective and constraint functions are both convex in A and b) ◮ tractable only in certain special cases (evaluating constraint for general C can be hard)

SJTU Ying Cui 3 / 26

Minimum volume ellipsoid covering a finite set

finite set: ◮ find minimum volume ellipsoid containing finite set {x1, . . . , xm} min

A,b

log det A−1 s.t. Axi + b2 ≤ 1, i = 1, . . . , m

◮ norm constraints can be replaced with squared versions which are convex quadratic inequalities

◮ equivalent to finding minimum volume ellipsoid containing polyhedron conv{x1, . . . , xm}

◮ an ellipsoid covers C iff it covers its convex hull

SJTU Ying Cui 4 / 26

Maximum volume inscribed ellipsoid

◮ parametrize a general ellipsoid E as E = {Bu + d | u2 ≤ 1} (B ∈ Sn, d ∈ Rn)

◮ image of Euclidean unit ball under an affine mapping ◮ w.l.o.g. assume B ∈ Sn

++

◮ volume of E is proportional to det B ◮ find maximum volume ellipsoid inside a convex set C ⊆ Rn max

B,d

log det B s.t. supu2≤1 IC(Bu + d) ≤ 0 where IC(x) = 0 for x ∈ C and IC(x) = ∞ for x / ∈ C

◮ convex problem (log det is concave and IC is convex, in B and d)

SJTU Ying Cui 5 / 26

Maximum volume ellipsoid in a polyhedron

polyhedron: ◮ find maximum volume ellipsoid inside polyhedron C = {x | aT

i x ≤ bi, i = 1, . . . , m}

max

B,d

log det B s.t. Bai2 + aT

i d ≤ bi, i = 1, . . . , m

◮ constraint follows from supu2≤1 aT

i (Bu + d) = Bai2 + aT i d

SJTU Ying Cui 6 / 26

Efficiency of ellipsoidal approximations

C ⊆ Rn convex, bounded, with nonempty interior ◮ L¨

• wner-John ellipsoid of C, shrunk by a factor n, lies inside C

◮ maximum volume inscribed ellipsoid of C, expanded by a factor n, covers C ◮ factor n can be improved to √n if C is symmetric example (for two polyhedra P = conv{x1, . . . , x6} in R2)

SJTU Ying Cui 7 / 26

Affine invariance of extremal volume ellipsoids

L¨

• wner-John ellipsoid ellipsoid and maximum volume inscribed

ellipsoid are both affinely invariant ◮ if E is L¨

• wner-John ellipsoid of C and T ∈ Rn×n is

nonsingular, then L¨

• wner-John ellipsoid of TC is TE

◮ if E is maximum volume inscribed ellipsoid of C and T ∈ Rn×n is nonsingular, then maximum volume inscribed ellipsoid of TC is TE

SJTU Ying Cui 8 / 26

Chebyshev center

Chebyshev center xcheb(C) of set C ⊆ Rn (bounded, with nonempty interior) is ◮ any point of maximum depth in C ◮ any point inside C that is farthest from exterior of C ◮ the center of the largest ball that lies inside C

SJTU Ying Cui 9 / 26

Chebyshev center of a convex set

◮ C is convex and defined by a set of convex inequalities: C = {x|f1(x) ≤ 0, · · · , fm(x) ≤ 0} ◮ find a Chebyshev center of convex set C: max

x,R

R s.t. gi(x, R) ≤ 0, i = 1, · · · , m where gi(x, R) = supu≤1 fi(x + Ru)

◮ convex problem (gi is the pointwise maximum of a family of convex functions of x and R, hence convex), xcheb(C) = x∗ ◮ tractable only in certain special cases (evaluate gi involving solving a convex problem which can be hard)

SJTU Ying Cui 10 / 26

Chebyshev center of a polyhedron

◮ C is a polyhedron defined by a set of linear inequalities: C = {x|aT

i x ≤ bi, i = 1, · · · , m}

◮ find a Chebyshev center of polyhedron C: max

x,R

R s.t. aT

i x + Rai∗ ≤ bi, i = 1, · · · , m

R ≥ 0 where each constraint follows from gi(x, R) = sup

u≤1

• aT

i (x + Ru) − bi

• = aT

i x + Rai∗ − bi

◮ LP

SJTU Ying Cui 11 / 26

Maximum volume ellipsoid center

◮ maximum volume ellipsoid center xmve(C) of set C is center

• f maximum volume ellipsoid in C

◮ find maximum volume ellipsoid center of set C max

B,d

log det B s.t. supu2≤1 IC(Bu + d) ≤ 0

◮ convex problem, xmve(C) = d∗

◮ maximum volume ellipsoid center is affine invariant, as maximum volume inscribed ellipsoid is affine invariant

SJTU Ying Cui 12 / 26

Analytic center of a set of inequalities

analytic center xac of a set of convex inequalities and linear equations fi(x) ≤ 0, i = 1, . . . , m, Fx = g is an optimal point of min

x

− m

i=1 log(−fi(x))

s.t. Fx = g ◮ objective called logarithmic barrier associated with inequalities ◮ interpretation of fi(x) at a feasible point x: margin or slack in i-th inequality ◮ xac is point that maximizes product (or geometric mean) of these margins or slacks subject to linear equality and implicit convex inequality constraints

◮ xac is not a function of the set: two sets of inequalities can describe the same set, but have different analytic centers ◮ xac is independent of affine changes of coordinates ◮ xac is invariant under (positive) scalings of inequality functions, and any reparametrization of equality constraints

SJTU Ying Cui 13 / 26

Analytic center of a set of linear inequalities

analytic center of a set of linear inequalities aT

i x ≤ bi, i = 1, . . . , m

is an optimal point of unconstrained minimization problem min

x

φ(x) = −

m

• i=1

log(bi − aT

i x)

◮ xac is the point that maximizes product of distances to hyperplanes Hi = {x|aT

i x = bi}, assuming ai2 = 1 w.l.o.g.

◮ xac is independent of positive scaling of constraint functions

SJTU Ying Cui 14 / 26

Inner and outer ellipsoids from analytic center of linear inequalities

analytic center of a set of linear inequalities P implicitly defines an inscribed and a covering ellipsoid, i.e., Einner ⊆ P ⊆ Eouter, where

P ={x|aT

i x ≤ bi, i = 1, . . . , m}

Einner ={x | (x − xac)T∇2φ(xac)(x − xac) ≤ 1} Eouter ={x | (x − xac)T∇2φ(xac)(x − xac) ≤ m(m − 1)} ◮ Einner and Eouter are related by scale factor (m(m − 1))1/2, which is always at least n

SJTU Ying Cui 15 / 26

Classification

given two sets of points in Rn, {x1, · · · , xN} and {y1, · · · , yN}, and find a function f : Rn → R (within a given family of functions) such that f (xi) > 0, i = 1, . . . , N, f (yi) < 0, i = 1, . . . , M 0-level set of f {z|f (z) = 0} separates, classifies, or discriminates two sets of points ◮ linear discrimination: f is linear (or affine) ◮ nonlinear discrimination: otherwise

SJTU Ying Cui 16 / 26

Linear discrimination

◮ find an affine function f (z) = aTz − b (i.e., a hyperplane) aTxi + b > 0, i = 1, . . . , N, aTyi + b < 0, i = 1, . . . , M

◮ strict inequalities homogeneous in a and b, hence equivalent to aTxi + b ≥ 1, i = 1, . . . , N, aTyi + b ≤ −1, i = 1, . . . , M

◮ problem: find a, b s.t. aTxi − b ≥ 1, i = 1, · · · , N aTyi − b ≤ −1, i = 1, · · · , M

SJTU Ying Cui 17 / 26

Robust linear discrimination

◮ existence of an affine classifying function f (z) = aTz − b is equivalent to a set of linear inequalities in a and b that define f and choose one that optimizes some measure of robustness ◮ seek function giving maximum possible gap between (positive) values at points xi and (negative) values at points yi

◮ normalize a and b, since otherwise scaling a and b by a positive constant can make gap arbitrarily large

SJTU Ying Cui 18 / 26

Robust linear discrimination

◮ primal problem: max

a,b,t

t s.t. aTxi − b ≥ t, i = 1, · · · , N aTyi − b ≤ −t, i = 1, · · · , M a2 ≤ 1

◮ convex problem ◮ t∗ is positive iff two sets of points can be linearly discriminated and in this case a∗2 = 1

◮ dual problem: max

u,v

− N

i=1 uixi −

M

i=1 viyi2

s.t. u 0, 1T u = 1/2 v 0, 1T v = 1/2

◮ find (half) distance between convex hulls of two sets

SJTU Ying Cui 19 / 26

Approximate linear separation of non-separable sets

◮ when two sets of points cannot be linearly separated, seek an affine function that approximately classifies them ◮ a heuristic for minimizing number of misclassified points min

a,b,u,v

1Tu + 1Tv s.t. aTxi + b ≥ 1 − ui, i = 1, . . . , N aTyi + b ≤ −1 + vi, i = 1, . . . , M u 0, v 0

◮ LP ◮ ui (vi): a measure of how much aTxi + b ≥ 1 (aTyi + b ≤ −1) is violated ◮ objective: a convex relaxation of number of points xi violating aTxi − b ≥ 1 plus number of points yi violating aTyi − b ≤ −1 ◮ u∗

i = max{0, 1 − a∗Txi − b∗}, v ∗ i = max{0, 1 + a∗Ty ∗ i + b}

SJTU Ying Cui 20 / 26

Support vector classifier

◮ more generally, consider trade-off between number of misclassified points and width of slab {z| − 1 ≤ aTz − b ≤ 1}, 2/a2 min

a,b,u,v

a2 + γ(1T u + 1Tv) s.t. aTxi + b ≥ 1 − ui, i = 1, · · · , N aTyi + b ≤ −1 + vi, i = 1, · · · , M u 0, v 0

SJTU Ying Cui 21 / 26

Nonlinear discrimination

separate two sets of points by a nonlinear function: f (xi) > 0, i = 1, . . . , N, f (yi) < 0, i = 1, . . . , M ◮ choose a linearly parametrized family of functions f (z) = θTF(z) F = (F1, . . . , Fk) : Rn → Rk are basis functions ◮ solve a set of linear inequalities in θ: θTF(xi) ≥ 1, i = 1, . . . , N, θTF(yi) ≤ −1, i = 1, . . . , M

SJTU Ying Cui 22 / 26

Nonlinear discrimination

quadratic discrimination: find f (z) = zTPz + qTz + r where P ∈ Sn, q ∈ Rn, r ∈ R satisfy xT

i Pxi+qTxi+r < 0, i = 1, · · · , N, y T i Pyi+qTyi+r > 0, i = 1, · · · , M

◮ strict inequalities homogeneous in P, q, r, and hence equivalent to xT

i Pxi + qTxi + r ≥ 1, y T i Pyi + qTyi + r ≤ −1

◮ separating surface {z|zTPz + qTz + r = 0} is quadratic ◮ can impose conditions on shape of separating surface

◮ add additional constraints, e.g., P −0 (equivalent to P −I due to homogeneity in P, q, r) to separate by an ellipsoid

◮ problem: SDP feasibility problem find P, q, r s.t. xT

i Pxi + qTxi + r ≥ 1,

i = 1, · · · , N y T

i Pyi + qTyi + r ≤ −1,

i = 1, · · · , M P −I

SJTU Ying Cui 23 / 26

Nonlinear discrimination

polynomial discrimination: consider set of polynomials with degree ≤ d, f (z) =

i1+···+in≤d ai1···inzi1 1 · · · zin n where ai1···in satisfy

• i1+···+in≤d

ai1···inxi1

1 · · · xin n < 0,

i = 1, · · · , N,

• i1+···+in≤d

ai1···iny i1

1 · · · y in n > 0,

i = 1, · · · , M ◮ problem: LP feasibility problem example:

SJTU Ying Cui 24 / 26

Placement and facility location

◮ setup:

◮ N points with coordinates xi ∈ R2 (or R3) and some pairs of points connected by links in A ◮ some positions xi are given, while the other xi’s are variables ◮ a cost fij(xi, xj) associated with each pair (xi, xj)

◮ placement problem: optimize positions of free points min

x

• (i,j)∈A fij(xi, xj)

◮ convex problem, assuming fij convex ◮ linear facility location problems: fij(xi, xj) = wijxi − xj, wij ≥ 0 ◮ nonlinear facility location problems: fij(xi, xj) = wijh(xi − xj), h increasing and convex and wij ≥ 0

◮ interpretations:

◮ points represent plants or warehouses, and fij is transportation cost between facilities i and j ◮ points represent cells on an IC, and fij represents wirelength

SJTU Ying Cui 25 / 26

Example

◮

(i,j)∈A h(xi − xj2), with 6 free points and 27 links

◮ histograms of connection lengths xi − xj2 of optimal placement for h(z) = z, h(z) = z2, h(z) = z4

SJTU Ying Cui 26 / 26