Convex Optimization 11. Interior-point methods Prof. Ying Cui - - PowerPoint PPT Presentation

Convex Optimization

• 11. Interior-point methods
• Prof. Ying Cui

Department of Electrical Engineering Shanghai Jiao Tong University

2018

SJTU Ying Cui 1 / 42

Outline

Inequality constrained minimization problems Logarithmic barrier function and central path Barrier method Feasibility and phase I methods Complexity analysis via self-concordance Problems with generalized inequalities Primal-dual interior-point methods

SJTU Ying Cui 2 / 42

Inequality constrained minimization

min

x

f0(x) s.t. fi(x) ≤ 0, i = 1, ..., m Ax = b assumptions: ◮ fi : Rn → R are convex and twice continuously differentiable ◮ A ∈ Rp×n with rank A = p < n ◮ there exists an optimal point x∗, i.e., optimal value p∗ = inf{f(x)|Ax = b} is attained and finite ◮ problem is strictly feasible, i.e., there exists x ∈ D with fi(x) < 0, i = 1, ..., m, Ax = b

◮ means that Slater’s constraint qualification holds

hence, strong duality holds and there exist dual optimal λ∗ ∈ Rm and ν∗ ∈ Rp which together with x∗ satisfy KKT conditions

SJTU Ying Cui 3 / 42

Inequality constrained minimization

interior-point methods solve convex optimization problems with inequality constraints ◮ barrier method: solve the problem by applying Newton’s method to a sequence of equality constrained problems ◮ primal-dual interior-point method: solve the problem by applying Newton’s method to a sequence of modified versions

• f KKT conditions

SJTU Ying Cui 4 / 42

Examples

◮ LP, QP, QCQP, GP ◮ entropy maximization with linear inequality constraints (domf0 = Rn

++)

max

x n

• i=1

xi log xi s.t. Fx g Ax = b ◮ differentiability may require reformulation, e.g., piecewise-linear minimization or ℓ∞-norm approximation via LP ◮ SDPs and SOCPs are not in the required form, but can be handled by extensions of interior-point methods to problems with generalized inequalities

SJTU Ying Cui 5 / 42

Logarithmic barrier function and central path

goal is to approximately formulate the inequality constrained problem as an equality constrained problem to which Newton’s method can be applied

SJTU Ying Cui 6 / 42

Logarithmic barrier

reformulation via indicator function: min

x

f0(x) +

m

• i=1

I−(fi(x)) s.t. Ax = b where I− : R → R is indicator function for R−: I−(u) =

• 0,

u ≤ 0 ∞,

• therwise

◮ an equality constrained problem ◮ objective function is not (in general) differentiable, so Newton’s method cannot be applied

SJTU Ying Cui 7 / 42

Logarithmic barrier

approximation via logarithmic barrier: min

x

f0(x) − (1/t)

m

• i=1

log(−fi(x)) s.t. Ax = b where t > 0 is a parameter that sets the accuracy of approximation ◮ −(1/t) log(−u), t > 0 is a smooth approximation of I− and the approximation becomes more accurate as t increases ◮ an equality constrained problem which is an approximation of the original problem

u −3 −2 −1 1 −5 5 10 Figure 11.1 The dashed lines show the function I−(u), and the solid curves show I−(u) = −(1/t) log(−u), for t = 0.5, 1, 2. The curve for t = 2 gives the best approximation.

SJTU Ying Cui 8 / 42

Logarithmic barrier

logarithmic barrier function: φ(x) = −

m

• i=1

log(−fi(x)), dom φ = {x|f1(x) < 0, ..., fm(x) < 0} ◮ convex (follows from composition rules) ◮ twice continuously differentiable, with derivatives ∇φ(x) =

m

• i=1

1 −fi(x)∇fi(x) ∇2φ(x) =

m

• i=1

1 fi(x)2 ∇fi(x)∇fi(x)T +

m

• i=1

1 −fi(x)∇2fi(x)

SJTU Ying Cui 9 / 42

Central path

equivalent problem of approximation via logarithmic barrier: min

x

tf0(x) + φ(x) s.t. Ax = b assume existence and uniqueness of solution x∗(t) for each t > 0 ◮ central path: {x∗(t)|t > 0} ◮ central path conditions: x∗(t) satisfies Ax∗(t) = b, fi(x∗(t)) < 0, i = 1, · · · , m and there exists a w ∈ Rp such that 0 =t∇f0(x∗(t)) + ∇φ(x∗(t)) + AT w =t∇f0(x∗(t)) +

m

• i=1

1 −fi(x∗(t))∇fi(x∗(t)) + AT w

SJTU Ying Cui 10 / 42

Central path

example: central path for an LP min

x

cT x s.t. aT

i x ≤ bi,

i = 1, ..., 6 hyperplane cT x = cT x∗(t) is tangent to level curve of φ through x∗(t)

c x⋆ x⋆(10)

Figure 11.2 Central path for an LP with n = 2 and m = 6. The dashed curves show three contour lines of the logarithmic barrier function φ. The central path converges to the optimal point x⋆ as t → ∞. Also shown is the point on the central path with t = 10. The optimality condition (11.9) at this point can be verified geometrically: The line cT x = cT x⋆(10) is tangent to the contour line of φ through x⋆(10). SJTU Ying Cui 11 / 42

Dual points from central path

property of central path: every central point yields a dual feasible point, and hence a lower bound on optimal value p∗ ◮ define λ∗

i (t) = 1/(−tfi(x∗(t)) > 0 and ν∗(t) = w/t

◮ express optimality condition as ∇f0(x∗(t)) + m

i=1 λ∗ i (t)∇fi(x∗(t)) + AT ν∗(t) = 0, implying

x∗(t) = arg min

x L(x, λ∗(t), ν∗(t))

◮ dual function is g(λ∗(t), ν∗(t)) = L(x∗(t), λ∗(t), ν∗(t)) = f0(x∗(t)) − m/t ◮ weak duality g(λ∗(t), ν∗(t)) ≤ p∗ = ⇒ f0(x∗(t)) − p∗ ≤ m/t x∗(t) is no more than m/t-suboptimal and converges to an optimal point as t → ∞

SJTU Ying Cui 12 / 42

Interpretation via KKT conditions

interpret central path conditions as a continuous deformation of KKT optimality conditions x = x∗(t), λ = λ∗(t), ν = ν∗(t) satisfy ◮ primal constraints: fi(x) ≤ 0, i = 1, ..., m, Ax = b ◮ dual constraints: λ 0 ◮ approximate complementary slackness: −λifi(x) = 1/t, i = 1, ..., m

◮ replaces complementary slackness λifi(x) = 0 in KKT conditions

◮ gradient of Lagrangian with respect to x vanishes: ∇f0(x) +

m

• i=1

λi∇fi(x) + AT ν = 0

SJTU Ying Cui 13 / 42

Force field interpretation

centering problem (for problem with no equality constraints) min

x

tf0(x) −

m

• i=1

log(−fi(x)) force field interpretation ◮ tf0(x) is potential of force field F0(x) = −t∇f0(x) ◮ − log(−fi(x)) is potential of force field Fi(x) = (1/fi(x))∇fi(x) the forces balance at x∗(t): F0(x∗(t)) +

m

• i=1

Fi(x∗(t)) = 0

SJTU Ying Cui 14 / 42

Force field interpretation

example min

x

cT x s.t. aT

i x ≤ bi,

i = 1, ..., m ◮ objective force field is constant: F0(x) = −tc ◮ constraint force field decays as inverse distance to constraint hyperplane: Fi(x) = −ai bi − aT

i x,

||Fi(x)||2 = 1 dist(x, Hi) where Hi = {x|aT

i x = bi}

−c −3c Figure 11.3 Force field interpretation of central path. The central path is shown as the dashed curve. The two points x⋆(1) and x⋆(3) are shown as dots in the left and right plots, respectively. The objective force, which is equal to −c and −3c, respectively, is shown as a heavy arrow. The other arrows represent the constraint forces, which are given by an inverse-distance

• law. As the strength of the objective force varies, the equilibrium position
• f the particle traces out the central path.

SJTU Ying Cui 15 / 42

Barrier method

Barrier method.

given strictly feasible x, t := t(0) > 0, µ > 1, tolerance ǫ > 0. repeat

• 1. Centering step. Compute x∗(t) by minimizing tf0 + φ, subject to Ax = b,

starting at x.

• 2. Update. x := x∗(t).
• 3. Stopping criterion. quit if m/t < ǫ
• 4. Increase t. t := µt.

◮ solve a sequence of unconstrained or linearly constrained minimization problems

◮ at each iteration, start from previously computed central point and perform Newton’s method

◮ terminate at ǫ-suboptimal solution (f0(x∗(t)) − p∗ ≤ m/t < ǫ) ◮ choice of µ involves a trade-off: large µ means fewer outer iterations, more inner (Newton) iterations

◮ typical values: µ = 10-20

◮ several heuristics for choice of t(0)

SJTU Ying Cui 16 / 42

Variation of barrier method

◮ initialize with a point x not necessarily satisfy Ax = b ◮ an infeasible start Newton method is used for first centering step ◮ a full Newton step is taken at some point during first centering step and thereafter the iterates are all primal feasible ◮ algorithm coincides with standard barrier method from second centering step

SJTU Ying Cui 17 / 42

Convergence analysis

number of outer (centering) iterations: exactly

• log(m/(ǫt(0)))

log µ

• plus initial centering step to achieve desired accuracy ǫ

◮ duality gap after initial centering step and k additional ones is m/(µkt(0)) convergence of Newton’s method for centering problem: min

x

tf0(x) + φ(x) s.t. Ax = b ◮ tf0 + φ must have closed sublevel sets for t ≥ t(0) ◮ classical analysis requires strong convexity of tf0 + φ and Lipschitz condition for Hessian of tf0 + φ ◮ analysis via self-concordance requires self-concordance of tf0 + φ

SJTU Ying Cui 18 / 42

Examples

inequality form LP (m = 100 inequalities, n = 50 variables) duality gap on a log scale versus cumulative total number of Newton iterations (natural measure of computational effort)

Newton iterations duality gap µ = 2 µ = 50 µ = 150 20 40 60 80 10−6 10−4 10−2 100 102 Figure 11.4 Progress of barrier method for a small LP, showing duality gap versus cumulative number of Newton steps. Three plots are shown, corresponding to three values of the parameter µ: 2, 50, and 150. In each case, we have approximately linear convergence of duality gap. µ Newton iterations 40 80 120 160 200 20 40 60 80 100 120 140 Figure 11.5 Trade-off in the choice of the parameter µ, for a small LP. The vertical axis shows the total number of Newton steps required to reduce the duality gap from 100 to 10−3, and the horizontal axis shows µ. The plot shows the barrier method works well for values of µ larger than around 3, but is otherwise not sensitive to the value of µ.

◮ starts with t(0) = 1 (duality gap m/t(0) = 100) ◮ terminates when t = 108 (duality gap m/t = 10−6) ◮ centering uses Newton’s method with backtracking ◮ each stair is associated with one outer iteration

◮ width of each stair tread: number of Newton iterations required for that outer iteration ◮ height of each stair riser: µ

SJTU Ying Cui 19 / 42

Examples

inequality form LP (m = 100 inequalities, n = 50 variables) plots illustrate several typical features of barrier method ◮ reason: approximately linear convergence of duality gap

◮ approximately constant number of Newton steps required to re-center

◮ trade-off in choice of µ

◮ for small µ, number of Newton steps required to re-center is small and total number of outer loops is large ◮ for large µ, number of Newton steps required to re-center is large and total number of outer loops is small

◮ total number of iterations not very sensitive for µ ≥ 10

◮ a good choice of µ is in range 10-100

SJTU Ying Cui 20 / 42

Examples

geometric program (m = 100 inequalities, n = 50 variables) min

x

log 5

• k=1

exp(aT

0kx + b0k)

• s.t.

log 5

• k=1

exp(aT

ikx + bik)

• ≤ 0,

i = 1, ..., m

Newton iterations duality gap µ = 2 µ = 50 µ = 150 20 40 60 80 100 120 10−6 10−4 10−2 100 102 Figure 11.6 Progress of barrier method for a small GP, showing duality gap versus cumulative number of Newton steps. Again we have approximately linear convergence of duality gap.

◮ approximately linear convergence of duality gap

SJTU Ying Cui 21 / 42

Examples

family of standard LPs (A ∈ Rm×2m, m = 10, · · · , 1000, 100 randomly generated LPs for each m) min

x

cT x s.t. Ax = b, x 0

m Newton iterations 101 102 103 15 20 25 30 35 Figure 11.8 Average number of Newton steps required to solve 100 randomly generated LPs of different dimensions, with n = 2m. Error bars show stan- dard deviation, around the average value, for each value of m. The growth in the number of Newton steps required, as the problem dimensions range

• ver a 100:1 ratio, is very small.

◮ number of iterations grows very slowly as m ranges over a 100 : 1 ratio

SJTU Ying Cui 22 / 42

Feasibility and phase I methods

feasibility problem: find x such that fi(x) ≤ 0, i = 1, ..., m, Ax = b phase I of barrier method: computes a strictly feasible starting point for barrier method phase II of barrier method: use the strictly feasible starting point for barrier method

SJTU Ying Cui 23 / 42

Basic phase I method

minimize maximum infeasibility p∗ = min

x,s

s s.t. fi(x) ≤ s, i = 1, ..., m Ax = b ◮ if p∗ < 0, then feasibility problem is strictly feasible

◮ if (x, s) is feasible for feasibility problem with s < 0, no need to solve with high accuracy and can terminate

◮ if p∗ > 0, then feasibility problem is infeasible

◮ when a dual feasible point is found with positive dual objective (which proves p∗ > 0), no need to solve with high accuracy and can terminate

◮ if p∗ = 0 and attained, then feasibility problem is feasible (but not strictly) ◮ if p∗ = 0 and not attained, then feasibility problem is infeasible

SJTU Ying Cui 24 / 42

Sum of infeasibilities phase I method

minimize sum of the infeasibilities p∗ = min

x,s

1T s s.t. s 0, fi(x) ≤ si, i = 1, ..., m Ax = b ◮ p∗ = 0 and attained, iff original set of equalities and inequalities is feasible ◮ if p∗ > 0, often violate only a small number and can identify a large subset of inequalities that is feasible

SJTU Ying Cui 25 / 42

Example

apply two phase I methods to an infeasible set of 100 linear inequalities aT

i x ≤ bi in 50 variables and show distributions of the

infeasibilities bi − aT

i x for two solutions xmax and xsum

bi − aT

i xmax

number −1 −0.5 0.5 1 1.5 10 20 30 40 50 60 number −1 −0.5 0.5 1 1.5 10 20 30 40 50 60 bi − aT

i xsum

Figure 11.9 Distributions of the infeasibilities bi − aT

i x for an infeasible set

• f 100 inequalities aT

i x ≤ bi, with 50 variables. The vector xmax used in

the left plot was obtained by the basic phase I algorithm. It satisfies 39

• f the 100 inequalities. In the right plot the vector xsum was obtained by

minimizing the sum of the infeasibilities. This vector satisfies 79 of the 100 inequalities.

◮ left: basic phase I solution xmax satisfies 39 inequalities ◮ right: sum of infeasibilities phase I solution xsum satisfies 79 inequalities

SJTU Ying Cui 26 / 42

Example

family of linear inequalities Ax b + γ∆b ◮ data chosen to be strictly feasible for γ > 0, infeasible for γ ≤ 0 ◮ use basic phase I method, terminate when s < 0 or dual

• bjective is positive

◮ number of iterations roughly proportional to log(1/|γ|)

γ Newton iterations Infeasible Feasible −1 −0.5 0.5 1 20 40 60 80 100 Figure 11.10 Number of Newton iterations required to detect feasibility or infeasibility of a set of linear inequalities Ax b + γ∆b parametrized by γ ∈ R. The inequalities are strictly feasible for γ > 0, and infeasible for γ < 0. For γ larger than around 0.2, about 30 steps are required to compute a strictly feasible point; for γ less than −0.5 or so, it takes around 35 steps to produce a certificate proving infeasibility. For values of γ in between, and especially near zero, more Newton steps are required to determine feasibility.

γ Newton iterations −100 −10−2 −10−4 −10−6 20 40 60 80 100 γ Newton iterations 10−6 10−4 10−2 100 20 40 60 80 100 Figure 11.11 Left. Number of Newton iterations required to find a proof of infeasibility versus γ, for γ small and negative. Right. Number of Newton iterations required to find a strictly feasible point versus γ, for γ small and positive.

SJTU Ying Cui 27 / 42

Complexity analysis via self-concordance

same assumptions as on page 3, plus: ◮ sublevel sets (of f0, on the feasible set) are bounded ◮ tf0 + φ is self-concordant with closed sublevel sets

◮ holds for LP, QP, QCQP ◮ may require reformulating the problem, e.g., min

x n

• i=1

xi log xi = ⇒ min

x n

• i=1

xi log xi s.t. Fx g s.t. Fx g, x 0 ◮ needed for complexity analysis, but barrier method works even when self-concordance assumption does not apply

convergence analysis of Newton’s method for self-concordant functions ◮ gives a rigorous bound on total number of Newton steps ◮ justifies that centering problems do not become more difficult as t increases

SJTU Ying Cui 28 / 42

Newton iterations per centering step

number of Newton iterations per centering step ≤ m(µ − 1 − log µ) γ + c γ, c are constants (depend on Newton algorithm parameters) proof: From self-concordance theory, number of Newton iterations for computing x+ = x∗(µt) starting at x = x∗(t) is given by µtf0(x) + φ(x) − µtf0(x+) − φ(x+) γ + c Let λ = λ∗(t) and ν = ν∗(t).

µtf0(x) + φ(x) − µtf0(x+) − φ(x+) =µtf0(x) − µtf0(x+) +

m

• i=1

log(−µtλifi(x+)) − m log µ ≤µtf0(x) − µtf0(x+) − µt

m

• i=1

λifi(x+) − m − m log µ ≤µtf0(x) − µtg(λ, ν) − m − m log µ = m(µ − 1 − log µ)

SJTU Ying Cui 29 / 42

Total number of Newton iterations

total number of Newton iterations (excluding first centering step) ≤N =

• log(m/(t(0)ǫ))

log µ m(µ − 1 − log µ) γ + c

• ◮ if µ and m are fixed , N is proportional to log(m/(t(0)ǫ)) (the

log of the ratio of the initial duality gap m/t(0) to the final duality gap ǫ) ◮ if µ and m/(t(0)ǫ) are fixed, N grows linearly with m and is independent of the other problem dimensions n and p ◮ as µ approaches one, N grows large; as µ becomes large, N grows approximately as µ/ log µ

SJTU Ying Cui 30 / 42

Total number of Newton iterations

µ N 1 1.1 1.2 1 104 2 104 3 104 4 104 5 104 Figure 11.14 The upper bound N on the total number of Newton iterations, given by equation (11.27), for c = 6, γ = 1/375, m = 100, and a duality gap reduction factor m/(t(0)ǫ) = 105, versus the barrier algorithm parameter µ.

figure shows N for typical values of γ, c, m = 100,

m t(0)ǫ = 105

◮ confirms trade-off in choice of µ ◮ in practice, N is in the tens and not very sensitive for µ ≥ 10

SJTU Ying Cui 31 / 42

Total number of Newton iterations

polynomial-time complexity of barrier method ◮ for µ = 1 + 1/√m: N = O

• √m log
• m/t(0)

ǫ

• ◮ number of Newton iterations for fixed gap reduction is O(√m)

◮ multiply with cost of one Newton iteration (a polynomial function of problem dimensions), to get bound on number of flops this choice of µ optimizes worst-case complexity; in practice we choose µ fixed (µ = 10, · · · , 20)

SJTU Ying Cui 32 / 42

Generalized inequalities

minimize f0(x) subject to fi(x) Ki 0, i = 1, · · · , m Ax = b ◮ f0 convex, fi : Rn → Rki, i = 1, · · · , m, convex with respect to proper cones Ki ∈ Rki ◮ fi twice continuously differentiable ◮ A ∈ Rp×n with rank A = p ◮ we assume p∗ is finite and attained ◮ we assume problem is strictly feasible; hence strong duality holds and dual optimum is attained examples of greatest interest: SOCP, SDP

SJTU Ying Cui 33 / 42

Generalized logarithm for proper cone

ψ : Rq → R is generalized logarithm for proper cone K ⊆ Rq if: ◮ domψ = intK and ∇2ψ(y) ≺ 0 for y ≻K 0 ◮ ψ(sy) = ψ(y) + θ log s for y ≻K 0, s > 0 (θ is the degree of ψ) examples ◮ nonnegative orthant K = Rn

+ : ψ(y) = n i=1 log yi,with

degree θ = n ◮ positive semidefinite cone K = Sn

+

ψ(Y ) = log det Y (θ = n) ◮ second-order cone K = {y ∈ Rn+1|(y2

1 + · · · + y2 n)1/2 ≤ yn+1} :

ψ(y) = log(y2

n+1 − y2 1 − · · · − y2 n)

(θ = 2)

SJTU Ying Cui 34 / 42

properties(without proof): for y ≻K 0 ∇ψ(y) K∗ 0, yT ∇ψ(y) = θ ◮ nonnegative orthant Rn

+ : ψ(y) = n i=1 log yi

∇ψ(y) = (1/y1, · · · , 1/yn), yT ∇ψ(y) = n ◮ positive semidefinite cone Sn

+ : ψ(Y ) = log det Y

∇ψ(Y ) = Y −1, tr(Y ∇ψ(Y )) = n ◮ second-order cone K = {y ∈ Rn+1|(y2

1 + · · · + y2 n)1/2 ≤ yn+1} :

∇ψ(y) = 2 y2

n+1 − y2 1 − · · · − y2 n

     −y1 . . . −yn yn+1      , yT ∇ψ(y) = 2

SJTU Ying Cui 35 / 42

Logarithmic barrier and central path

logarithmic barrier for f1(x) K1 0, · · · , fm(x) Km 0: φ(x) = −

m

• i=1

ψi(−fi(x)), domφ = {x|fi(x) ≺Ki 0, i = 1, · · · , m} ◮ ψi is generalized logarithm for Ki, with degree θi ◮ φ is convex, twice continuously differentiable central path: {x∗(t)|t > 0} where x∗(t) solves minimize tf0(x) + φ(x) subject to Ax = b

SJTU Ying Cui 36 / 42

Dual points on central path

x = x∗(t) if there exists w ∈ Rp t∇f0(x) +

m

• i=0

Dfi(x)T ∇ψi(−fi(x)) + AT w = 0 (Dfi(x) ∈ Rki×n is derivative matrix of fi) ◮ therefore, x∗(t) minimizes Lagrangian L(x, λ∗(t), ν∗(t)), where λ∗

i (t) = 1

t ∇ψi(−fi(x∗(t))), ν∗(t) = w t ◮ from properties of ψi: λ∗

i (t) ≻K∗

i 0, with duality gap

f0(x∗(t)) − g(λ∗(t), ν∗(t)) = (1/t)

m

• i=1

θi

SJTU Ying Cui 37 / 42

Example

semidefinite programming (with Fi ∈ Sp): minimize cT x subject to F(x) =

n

• i=1

xiFi + G 0 ◮ logarithmic barrier: φ(x) = log det(−F(x)−1) ◮ central path: x∗(t) minimizes tcT x − log det(−F(x)); hence tci − tr(FiF(x∗(t))−1) = 0, i = 1, · · · , n ◮ dual point on central path: Z∗(t) = −(1/t)F(x∗(t))−1 is feasible for maximize tr(GZ) subject to tr(FiZ) + ci = 0, i = 1, · · · , n Z 0 ◮ duality gap on central path: cT x∗(t) − tr(GZ∗(t)) = p/t

SJTU Ying Cui 38 / 42

Barrier method

given strictly feasible x, t := t(0) > 0, µ > 1,tolerance ǫ > 0 repeat

• 1. Centering step. Compute x∗(t) by minimizing tf0 + φ, subject to Ax = b.
• 2. Update. x := x∗(t)
• 3. Stopping criterion. quit if (

i θi)/t < ǫ

• 4. Increase t. t := µt

◮ only difference is duality gap m/t on central path is replaced by

i θi/t

◮ number of outer iterations: log((

i θi)/(ǫt(0)))

log µ

• ◮ complexity analysis via self-concordance applies to SDP, SOCP

SJTU Ying Cui 39 / 42

Examples

second-order cone program (50 variables, 50 SOC constraints in R6)

Newton iterations duality gap µ = 2 µ = 50 µ = 200 20 40 60 80 10−6 10−4 10−2 100 102 Figure 11.15 Progress of barrier method for an SOCP, showing duality gap versus cumulative number of Newton steps. µ Newton iterations 40 80 120 160 200 20 40 60 80 100 120 140 Figure 11.16 Trade-off in the choice of the parameter µ, for a small SOCP. The vertical axis shows the total number of Newton steps required to reduce the duality gap from 100 to 10−3, and the horizontal axis shows µ.

semidefinite program (100 variables, LMI constraint in S100)

Newton iterations duality gap µ = 2 µ = 50 µ = 150 20 40 60 80 100 10−6 10−4 10−2 100 102 Figure 11.17 Progress of barrier method for a small SDP, showing duality gap versus cumulative number of Newton steps. Three plots are shown, corresponding to three values of the parameter µ: 2, 50, and 150. µ Newton iterations 20 40 60 80 100 120 20 40 60 80 100 120 140 Figure 11.18 Trade-off in the choice of the parameter µ, for a small SDP. The vertical axis shows the total number of Newton steps required to reduce the duality gap from 100 to 10−3, and the horizontal axis shows µ.

SJTU Ying Cui 40 / 42

family of SDPs (A ∈ Sn, x ∈ Rn) minimize 1T x subject to A + diag(x) 0 n = 10, · · · , 1000, for each n solve 100 randomly generated instances

n Newton iterations 101 102 103 15 20 25 30 35

Figure 11.20 Average number of Newton steps required to solve 100 ran- domly generated SDPs (11.47) for each of 20 values of n, the problem size. Error bars show standard deviation, around the average value, for each value

• f n. The growth in the average number of Newton steps required, as the

problem dimensions range over a 100:1 ratio, is very small.

SJTU Ying Cui 41 / 42

Primal-dual interior-point methods

• ften more efficient than barrier method when high accuracy is

needed (exhibit superlinear asymptotic convergence) ◮ only one loop

◮ no distinction between inner and outer iterations ◮ at each iteration, both the primal and dual variables are update ◮ cost per iteration same as barrier method

◮ search directions can be interpreted as Newton directions for modified KKT condition (optimality conditions for logarithmic barrier centering problem) ◮ primal and dual iterates are not necessarily feasible

SJTU Ying Cui 42 / 42