[PPT] - Polynomial optimization and sums of squares Sums-of-squares PowerPoint Presentation

SLIDE 1

POLYNOMIAL OPTIMIZATION WITH SUMS-OF-SQUARES INTERPOLANTS

Sercan Yıldız

syildiz@samsi.info in collaboration with D´ avid Papp (NCSU)

OPT Transition Workshop May 02, 2017

SLIDE 2

OUTLINE

Polynomial optimization and sums of squares

– Sums-of-squares hierarchy – Semidefinite representation of sums-of-squares constraints

Interior-point methods and conic optimization
Sums-of-squares optimization with interior-point methods

– Computational complexity – Preliminary results

SLIDE 3

POLYNOMIAL OPTIMIZATION

Polynomial optimization problem:

min

z∈Rn

f(z) s.t. gi(z) ≥ 0 for i = 1, . . . , m where f, g1, . . . , gm are n-variate polynomials Example: min

z∈R2

z3

1 + 3z2 1z2 − 6z1z2 2 + 2z3 2

s.t. z2

1 + z2 2 ≤ 1

Some applications:

– Shape-constrained

estimation

– Design of experiments – Control theory – Combinatorial optimization – Computational geometry – Optimal power flow

Source: https://upload.wikimedia.

rg/wikipedia/commons/thumb/c/

cf/Max- cut.svg/200px- Max- cut. svg.png Source: http://www.cds.caltech.edu/ ˜murray/amwiki/images/thumb/1/ 19/Doscpp.png/270px- Doscpp.png Source: https://upload.wikimedia.

rg/wikipedia/commons/thumb/d/

d2/Kissing- 2d.svg/ 190px- Kissing- 2d.svg.png

SLIDE 4

SUMS-OF-SQUARES RELAXATIONS

Let f be an n-variate degree-2d polynomial.
Unconstrained polynomial optimization:

min

z∈Rn

f(z) NP-hard already for d = 2!

Equivalent “dual” formulation:

max

y∈R

y s.t. f(z) − y ∈ P where P = {f : f(z) ≥ 0 ∀z ∈ Rn}

Sums-of-squares cone:

SOS = {f : f = N

j=1f 2 j for some degree-d polynomials fj}

f ∈ SOS ⇒ f ∈ P

SOS relaxation:

max

y∈R

y s.t. f(z) − y ∈ SOS

SLIDE 5

SUMS-OF-SQUARES RELAXATIONS

Let f, g1, . . . , gm be n-variate polynomials.
Constrained polynomial optimization:

min

z∈Rn

f(z) s.t. gi(z) ≥ 0 for i = 1, . . . , m

Feasible set: G = {z ∈ Rn : gi(z) ≥ 0 for i = 1, . . . , m}.
Dual formulation:

max

y∈R

y s.t. f(z) − y ∈ PG where PG = {f : f(z) ≥ 0 ∀z ∈ G}

“Weighted” SOS cone of order r:

SOSG,r = {f : f = m

i=0gisi, si ∈ SOS, deg(gisi) ≤ r} where g0 ≡ 1

f ∈ SOSG,r ⇒ f ∈ PG

SOS relaxation of order r:

max

y∈R

y s.t. f(z) − y ∈ SOSG,r

SLIDE 6

WHY DO WE LIKE SOS?

Polynomial optimization is NP-hard.
Increasing r produces a hierarchy of SOS relaxations for polynomial
ptimization problems.
Under mild assumptions:

– The lower bounds from SOS relaxations converge to the true optimal

value as r ↑ ∞ (follows from Putinar’s Positivstellensatz).

SOS relaxations can be represented as semidefinite programs of size

O(mL2) where L = n+r/2

n

(follows from the results of Shor, Nesterov,

Parrilo, Lasserre).

There are efficient and stable numerical methods for solving SDPs.

SLIDE 7

SOS CONSTRAINTS ARE SEMIDEFINITE REPRESENTABLE

Consider the cone of degree-2d SOS polynomials:

SOS2d = {f ∈ R[z]2d : f = N

j=1f 2 j for some fj ∈ R[z]d}

Theorem (Nesterov, 2000) The univariate polynomial f(z) = 2d

u=0 ¯

fuzu is SOS iff there exists a (d + 1) × (d + 1) PSD matrix S such that ¯ fu =

k+ℓ=uSkℓ

∀u = 0, . . . , 2d.

SLIDE 8

SOS CONSTRAINTS ARE SEMIDEFINITE REPRESENTABLE

More generally, in the n-variate case:

– Let L := dim(R[z]d) =

n+d

n

and U := dim(R[z]2d) =

n+2d

n

.

– Fix bases {pℓ}L ℓ=1 and {qu}U u=1 for the linear spaces R[z]d and R[z]2d.

Theorem (Nesterov, 2000) The polynomial f(z) = U

u=1 ¯

fuqu(z) is SOS iff there exists a L × L PSD matrix S such that ¯ f = Λ∗(S) where Λ : RU → SL is the linear map satisfying Λ([qu]U

u=1) = [pℓ]L ℓ=1[pℓ]L ℓ=1 ⊤.

If univariate polynomials are represented in the monomial basis:

Λ(x) =        x0 x1 x2 . . . xd x1 x2 x2 . . . x2d−1 xd x2d−1 x2d        Λ∗(S) =

k+ℓ=uSkℓ

2d

u=0 .

These results easily extend to the weighted case.

SLIDE 9

SOS CONSTRAINTS ARE SEMIDEFINITE REPRESENTABLE

To keep things simple, we focus on optimization over a single SOS cone.
SOS problem:

max

y∈Rk ,S∈SL

b⊤y s.t. A⊤y + Λ∗(S) = c S 0 A⊤y + s = c s ∈ SOS := {s ∈ RU : s = Λ∗(S), S 0}

Moment problem:

min

x∈RU

c⊤x s.t. Ax = b Λ(x) 0 x ∈ SOS∗ := {x ∈ RU : Λ(x) 0} Disadvantages of existing approaches

Problem: SDP representation roughly squares the number of variables.

Solution: We solve the SOS and moment problems in their original space.

Problem: Standard basis choices lead to ill-conditioning.

Solution: We use orthogonal polynomials and interpolation bases.

SLIDE 10

INTERIOR-POINT METHODS FOR CONIC PROGRAMMING

Primal-dual pair of conic programs:

min c⊤x s.t. Ax = b x ∈ K max b⊤y s.t. A⊤y + s = c s ∈ K ∗

K: closed, convex, pointed cone with nonempty interior

Examples: K = Rn

+, Sn +, SOS, P

Interior-point methods make use of self-concordant barriers (SCBs).

Examples:

– K = Rn +: F(x) = − log x – K = Sn +: F(X) = − log det X

SLIDE 11

INTERIOR-POINT METHODS FOR CONIC PROGRAMMING

Given SCBs F and F ∗, IPMs converge to the optimal solution by solving a

sequence of equality-constrained barrier problems for µ ↓ 0: min c⊤x + µF(x) s.t. Ax = b max b⊤y − µF ∗(s) s.t. A⊤y + s = c

Primal IPMs:

– solve only the primal barrier problem, – are not considered to be practical.

Primal-dual IPMs:

– solve both the primal and dual barrier problems simultaneously, – are preferred in practice.

SLIDE 12

INTERIOR-POINT METHODS FOR CONIC PROGRAMMING

In principle, any closed convex cone admits a SCB (Nesterov and

Nemirovski, 1994).

However, the success of the IPM approach depends on the availability of a

SCB whose gradient and Hessian can be computed efficiently.

– For the cone P, there are complexity-based reasons for suspecting that

there are no computationally tractable SCBs.

– For the cone SOS, there is evidence suggesting that there may not exist

any tractable SCBs.

– On the other hand, the cone SOS∗ inherits a tractable SCB from the PSD

cone.

SLIDE 13

INTERIOR-POINT METHOD OF SKAJAA AND YE

Until recently:

– The practical success of primal-dual IPMs had been limited to

ptimization over symmetric cones: LP

, SOCP , SDP .

– Existing primal-dual IPMs for non-symmetric conic programs required

both the primal and dual SCBs (e.g., Nesterov, Todd, and Ye, 1999; Nesterov, 2012).

Skajaa and Ye (2015) proposed a primal-dual IPM for non-symmetric conic

programs which

– requires a SCB for only the primal cone, – achieves the best-known iteration complexity.

SLIDE 14

SOLVING SOS PROGRAMS WITH INTERIOR-POINT METHODS

Using Skajaa and Ye’s IPM with the SCB for SOS∗, the SOS and moment

problems can be solved without recourse to SDP .

For any ǫ ∈ (0, 1), the algorithm finds a primal-dual solution that has ǫ

times the duality gap of an initial solution in O( √ L log(1/ǫ)) iterations where L = dim(R[x]d).

Each iteration of the IPM requires

– the computation of the Hessian of the SCB for SOS∗, – the solution of a Newton system.

Solving the Newton system requires O(U3) operations where

U = dim(R[z]2d).

SLIDE 15

SOLVING SOS PROGRAMS WITH INTERIOR-POINT METHODS

The choice of bases {pℓ}L

ℓ=1 and {qu}U u=1 for R[z]d and R[z]2d has a

significant effect on how efficiently the Newton system can be compiled.

– In general, computing the Hessian requires O(L2U2) operations. – If both bases are chosen to be monomial bases, the Hessian can be

computed faster but requires specialized methods such as FFT and the “inversion” of Hankel-like matrices.

– Following L¨

fberg and Parrilo (2004), we choose
{qu}U

u=1: Lagrange interpolating polynomials,

{pℓ}L

ℓ=1: orthogonal polynomials. – With this choice, the Hessian can be computed in O(LU2) operations.

SLIDE 16

SOLVING SOS PROGRAMS WITH INTERIOR-POINT METHODS

Putting everything together:

– The algorithm runs in O(

√ L log(1/ǫ)) iterations.

– At each iteration:

Computing the Hessian requires O(LU2) operations,
Solving the Newton system requires O(U3) operations.
Overall complexity: O(U3√

L log(1/ǫ)) operations.

This matches the best-known complexity bounds for LP!
In contrast:

– Solving the SDP formulation with a primal-dual IPM requires

O(L6.5 log(1/ǫ)) operations.

– For fixed n:

L2 U = Θ(dn).

SLIDE 17

SOLVING SOS PROGRAMS WITH INTERIOR-POINT METHODS

The conditioning of the moment problem is directly related to the

conditioning of the interpolation problem with {pkpℓ}L

k,ℓ=1.

Good interpolation nodes are understood well only in few low-dimensional

domains:

– Chebyshev points in [−1, 1], – Padua points in [−1, 1]2 (Caliari et al., 2005).

For problems in higher dimensions, we follow a heuristic approach to

choose interpolation nodes.

SLIDE 18

A TOY EXAMPLE

Minimizing the Rosenbrock function on the square:

min

z∈R2

(z1 − 1)2 + 100(z2 − z2

1)2

s.t. z ∈ [−1, 1]2

We can obtain a lower bound on the optimal value of this problem from the

moment relaxation of order r.

For r = 60:

– The moment relaxation has 1891 variables. – The SDP representation requires one 496 × 496 and two 494 × 494

matrix inequalities.

– Sedumi quits with numerical errors after 948 seconds. Primal

infeasibility: 1.2 × 10−4.

– Our implementation solves the moment relaxation in 482 seconds.

SLIDE 19

FINAL REMARKS

We have shown that SOS programs can be solved using a primal-dual IPM

in O(U3√ L log(1/ǫ)) operations where L = n+d

n

and U =

n+2d

n

.
This improves upon the standard SDP-based approach which requires

O(L6.5 log(1/ǫ)) operations.

In progress: Numerical stability.

– For multivariate problems, the conditioning of the problem formulation

becomes important.

– How do we choose interpolation nodes for better-conditioned problems?

Thank you!

SLIDE 20

REFERENCES

M. Caliari, S. De Marchi, and M. Vianello. Bivariate polynomial interpolation
n the square at new nodal sets. Applied Mathematics and Computation,

165:261-274, 2005.

J. L¨
fberg and P

. A. Parrilo. From coefficients to samples: a new approach to sos optimization. In 43rd IEEE Conference on Decision and Control, volume 3, pages 3154-3159. IEEE, 2004.

Yu. Nesterov. Squared functional systems and optimization problems. In H.

Frenk, K. Roos, T. Terlaky, and S. Zhang, editors, High Performance Optimization, volume 33 of Applied Optimization, chapter 17, pages 405-440. Springer, Boston, MA, 2000.

Yu. Nesterov. Towards non-symmetric conic optimization. Optimization

Methods & Software, 27(4-5):893-917, 2012.

SLIDE 21

REFERENCES

Yu. Nesterov and A. Nemirovski. Interior-Point Polynomial Algorithms in

Convex Programming, volume 13 of SIAM Studies in Applied Mathematics. Society for Industrial and Applied Mathematics (SIAM), Philadelphia, PA, 1994.

Yu. Nesterov, M. J. Todd, and Y. Ye. Infeasible-start primal-dual methods

and infeasibility detectors for nonlinear programming problems. Mathematical Programming, 84:227-267, 1999.

A. Skajaa and Y. Ye. A homogeneous interior-point algorithm for