Interior-point methods 10-725 Optimization Geoff Gordon Ryan - - PowerPoint PPT Presentation

▶

Sep 07, 2022 256 likes •466 views

Interior-point methods 10-725 Optimization Geoff Gordon Ryan Tibshirani Review SVM duality min v T v/2 + 1 T s s.t. Av yd + s 1 ! 0 s ! 0 max 1 T T K /2 s.t. y T = 0 0 " " 1 Gram

SLIDE 1

Interior-point methods

10-725 Optimization Geoff Gordon Ryan Tibshirani

SLIDE 2

Geoff Gordon—10-725 Optimization—Fall 2012

Review

SVM duality
min vTv/2 + 1Ts s.t. Av – yd + s – 1 ! 0 s ! 0
max 1Tα – αTKα/2 s.t. yTα = 0 0 " α " 1
Gram matrix K
Interpretation
support vectors

& complementarity

reconstruct primal

solution from dual

SLIDE 3

Geoff Gordon—10-725 Optimization—Fall 2012

Review

Kernel trick
high-dim feature spaces, fast
positive definite function
Examples
polynomial
homogeneous polynomial
linear
Gaussian RBF

2 1 1 2 2 1 1 2

SLIDE 4

Geoff Gordon—10-725 Optimization—Fall 2012

Review: LF problem Ax + b ! 0

Ball center
bad summary of LF problem
Max-volume ellipsoid / ellipsoid center
good summary (1/n of volume), but expensive
Analytic center of LF problem
maximize product of distances to constraints
min –# ln(aiTx + bi)
Dikin ellipsoid @ analytic center: not quite as

good (just 1/m < 1/n), but much cheaper

SLIDE 5

Geoff Gordon—10-725 Optimization—Fall 2012

Force-field interpretation

f analytic center
Pretend constraints

are repelling a particle

normal force for each

constraint

force ! 1/distance
Analytic center =

equilibrium = where forces balance

SLIDE 6

Geoff Gordon—10-725 Optimization—Fall 2012

Newton for analytic center

f(x) = –# ln(aiTx + bi)
df/dx =
d2f/df2 =

SLIDE 7

Geoff Gordon—10-725 Optimization—Fall 2012

Dikin ellipsoid

E(x0) = { x | (x–x0)TH(x–x0) " 1 }
H = Hessian of log barrier at x0
unit ball of Hessian norm at x0
E(x0) ⊆ X for any strictly feasible x0
affine constraints can be just feasible
E(x0): as above, but intersected w/ affine constraints
vol(E(xac)) ! vol(X)/m
weaker than ellipsoid center, but still very useful

SLIDE 8

Geoff Gordon—10-725 Optimization—Fall 2012

E(x0) ⊆ X

E(x0) = { x | (x–x0)TH(x–x0) " 1 }
H = ATS-2A
S = diag(s) = diag(Ax0 + b)

SLIDE 9

Geoff Gordon—10-725 Optimization—Fall 2012

mE(x0) ⊇ X

Feasible point x: Ax + b ! 0
Analytic center xac: ATy = 0 y = 1./(Axac+b)
Let

Y = diag(yac), H = ATY2A; show:

(x–xac)TH(x–xac) " m2 [+ m]

SLIDE 10

Geoff Gordon—10-725 Optimization—Fall 2012

Combinatorics v. analysis

Two ways to find a feasible point of Ax+b ! 0
find analytic center—minimize a smooth function
find a feasible basis—combinatorial search

SLIDE 11

Geoff Gordon—10-725 Optimization—Fall 2012

Bad conditioning? No problem.

Analytic center & Dikin

ellipsoids invariant to affine xforms w = Mx+q

W = { w | AM-1(w–q) + b ! 0 }
Can always xform so that a

ball takes up ! vol(Y)/m

Dikin ellipsoid @ac → sphere

SLIDE 12

Geoff Gordon—10-725 Optimization—Fall 2012

LF→LP: the central path

Analytic center was for: find x st Ax + b ! 0
Now: min cTx st Ax + b ! 0
Same trick:
min ft(x) = cTx – (1/t) # ln(aiTx + bi)
parameter t > 0
central path =
t → 0: t → !:

SLIDE 13

Geoff Gordon—10-725 Optimization—Fall 2012

Force-field interpretation

f central path
Force along objective; normal forces for each

constraint

−c −3c

t=1 t=3

SLIDE 14

Geoff Gordon—10-725 Optimization—Fall 2012

Newton for central path

min ft(x) = cTx – (1/t) # ln(aiTx + bi)
df/dx =
d2f/dx2 =

SLIDE 15

Geoff Gordon—10-725 Optimization—Fall 2012

Central path example

bjective

t→0 t→∞

SLIDE 16

Geoff Gordon—10-725 Optimization—Fall 2012

New LP algorithm?

Set t=1012. Find corresponding point on central

path by Newton’s method.

worked for example on previous slide!
but has convergence problems in general
Alternatives?

SLIDE 17

Geoff Gordon—10-725 Optimization—Fall 2012

Constraint form of central path

min –# ln si st Ax + b ! 0 cTx " λ
∃ a 1-1 mapping λ(t) w/ x(λ(t)) = x(t) ∀t>0
but this form is slightly less convenient since we

don’t know minimal feasible value of λ or maximal nontrivial value of λ

SLIDE 18

Geoff Gordon—10-725 Optimization—Fall 2012

Dual of central path

min cTx – (1/t) # ln si st Ax + b = s ! 0
minx,s maxy L(x,s,y) = cTx – (1/t) # ln si + yT(s–Ax–b)

SLIDE 19

Geoff Gordon—10-725 Optimization—Fall 2012

Primal-dual correspondence

Primal and dual for central path:
min cTx – (1/t) # ln si st Ax + b = s ! 0
max (m ln t)/t + m/t + (1/t) # ln yi – yTb st

ATy = c y ! 0

L(x,s,y) = cTx – (1/t) # ln si + yT(s–Ax–b)
grad wrt s:
to get x:

SLIDE 20

Geoff Gordon—10-725 Optimization—Fall 2012

Duality gap

At optimum:
primal value cTx – (1/t) # ln si =

dual value (m ln t)/t + m/t + (1/t) # ln yi – yTb

s y = te