New Applications of Moment-SOS Hierarchies Victor Magron , RA - - PowerPoint PPT Presentation

New Applications of Moment-SOS Hierarchies

Victor Magron, RA Imperial College

17 October 2014

Imagination Technologies Seminar London

b y b → sin( √ b) par−

b1

par−

b2

par−

b3

par+

b1

par+

b2

par+

b3

1 b1 b2 b3 = 500

• V. Magron

New Applications of Moment-SOS Hierarchies 1 / 68

Personal Background

2008 − 2010: Master at Tokyo University HIERARCHICAL DOMAIN DECOMPOSITION METHODS (S. Yoshimura) 2010 − 2013: PhD at Inria Saclay LIX/CMAP FORMAL PROOFS FOR NONLINEAR OPTIMIZATION (S. Gaubert and B. Werner) 2014 Jan-Sept: Postdoc at LAAS-CNRS MOMENT-SOS APPLICATIONS (D. Henrion and J.B. Lasserre)

• V. Magron

New Applications of Moment-SOS Hierarchies 2 / 68

Errors and Proofs

Mathematicians want to eliminate all the uncertainties on their results. Why?

• M. Lecat, Erreurs des Mathématiciens des origines à

nos jours, 1935. 130 pages of errors! (Euler, Fermat, Sylvester, . . . )

• V. Magron

New Applications of Moment-SOS Hierarchies 3 / 68

Errors and Proofs

Possible workaround: proof assistants COQ (Coquand, Huet 1984) HOL-LIGHT (Harrison, Gordon 1980) Built in top of OCAML

• V. Magron

New Applications of Moment-SOS Hierarchies 3 / 68

Computer Science and Mathematics

Tool: Formal Bounds for Global Optimization Collaboration with: Benjamin Werner (LIX Polytechnique) Stéphane Gaubert (Maxplus Team CMAP/INRIA Polytechnique) Xavier Allamigeon (Maxplus Team)

• V. Magron

New Applications of Moment-SOS Hierarchies 4 / 68

Complex Proofs

Complex mathematical proofs / mandatory computation

• K. Appel and W. Haken , Every Planar Map is

Four-Colorable, 1989.

• T. Hales, A Proof of the Kepler Conjecture, 1994.
• V. Magron

New Applications of Moment-SOS Hierarchies 5 / 68

From Oranges Stack...

Kepler Conjecture (1611): The maximal density of sphere packings in 3D-space is

π √ 18

Face-centered cubic Packing Hexagonal Compact Packing

• V. Magron

New Applications of Moment-SOS Hierarchies 6 / 68

...to Flyspeck Nonlinear Inequalities

The proof of T. Hales (1998) contains mathematical and computational parts Computation: check thousands of nonlinear inequalities Robert MacPherson, editor of The Annals of Mathematics: “[...] the mathematical community will have to get used to this state of affairs.” Flyspeck [Hales 06]: Formal Proof of Kepler Conjecture

• V. Magron

New Applications of Moment-SOS Hierarchies 7 / 68

...to Flyspeck Nonlinear Inequalities

The proof of T. Hales (1998) contains mathematical and computational parts Computation: check thousands of nonlinear inequalities Robert MacPherson, editor of The Annals of Mathematics: “[...] the mathematical community will have to get used to this state of affairs.” Flyspeck [Hales 06]: Formal Proof of Kepler Conjecture Project Completion on 10 August by the Flyspeck team!!

• V. Magron

New Applications of Moment-SOS Hierarchies 7 / 68

...to Flyspeck Nonlinear Inequalities

Nonlinear inequalities: quantified reasoning with “∀” ∀x ∈ K, f(x) 0 NP-hard optimization problem

• V. Magron

New Applications of Moment-SOS Hierarchies 8 / 68

A “Simple” Example

In the computational part: Multivariate Polynomials:

∆x := x1x4(−x1 + x2 + x3 − x4 + x5 + x6) + x2x5(x1 − x2 + x3 + x4 − x5 + x6) + x3x6(x1 + x2 − x3 + x4 + x5 − x6) − x2(x3x4 + x1x6) − x5(x1x3 + x4x6)

• V. Magron

New Applications of Moment-SOS Hierarchies 9 / 68

A “Simple” Example

In the computational part: Semialgebraic functions: composition of polynomials with | · |, √, +, −, ×, /, sup, inf, . . . p(x) := ∂4∆x q(x) := 4x1∆x r(x) := p(x)/

• q(x)

l(x) := −π 2 + 1.6294 − 0.2213 (√x2 + √x3 + √x5 + √x6 − 8.0) + 0.913 (√x4 − 2.52) + 0.728 (√x1 − 2.0)

• V. Magron

New Applications of Moment-SOS Hierarchies 9 / 68

A “Simple” Example

In the computational part: Transcendental functions T : composition of semialgebraic functions with arctan, exp, sin, +, −, ×, . . .

• V. Magron

New Applications of Moment-SOS Hierarchies 9 / 68

A “Simple” Example

In the computational part: Feasible set K := [4, 6.3504]3 × [6.3504, 8] × [4, 6.3504]2 Lemma9922699028 from Flyspeck: ∀x ∈ K, arctan p(x)

• q(x)
• + l(x) 0
• V. Magron

New Applications of Moment-SOS Hierarchies 9 / 68

Existing Formal Frameworks

Formal proofs for Global Optimization: Bernstein polynomial methods [Zumkeller’s PhD 08] SMT methods [Gao et al. 12] Interval analysis and Sums of squares

• V. Magron

New Applications of Moment-SOS Hierarchies 10 / 68

Existing Formal Frameworks

Interval analysis Certified interval arithmetic in COQ [Melquiond 12] Taylor methods in HOL Light [Solovyev thesis 13]

Formal verification of floating-point operations

robust but subject to the Curse of Dimensionality

• V. Magron

New Applications of Moment-SOS Hierarchies 10 / 68

Existing Formal Frameworks

Lemma9922699028 from Flyspeck: ∀x ∈ K, arctan

• ∂4∆x

√4x1∆x

• + l(x) 0

Dependency issue using Interval Calculus:

One can bound ∂4∆x/√4x1∆x and l(x) separately Too coarse lower bound: −0.87 Subdivide K to prove the inequality

K = ⇒ K0 K1 K2 K3 K4

• V. Magron

New Applications of Moment-SOS Hierarchies 10 / 68

Existing Formal Frameworks

Sums of squares techniques Formalized in HOL-LIGHT [Harrison 07] COQ [Besson 07] Precise methods but scalibility and robustness issues (numerical) powerful: global optimality certificates without branching but not so robust: handles moderate size problems Restricted to polynomials

• V. Magron

New Applications of Moment-SOS Hierarchies 10 / 68

Existing Formal Frameworks

Approximation theory: Chebyshev/Taylor models mandatory for non-polynomial problems hard to combine with SOS techniques (degree of approximation)

• V. Magron

New Applications of Moment-SOS Hierarchies 10 / 68

Existing Formal Frameworks

Can we develop a new approach with both keeping the respective strength of interval and precision of SOS? Proving Flyspeck Inequalities is challenging: medium-size and tight

• V. Magron

New Applications of Moment-SOS Hierarchies 10 / 68

New Framework (in my PhD thesis)

Certificates for lower bounds of Nonlinear optimization using:

Moment-SOS hierarchies Maxplus approximation (Optimal Control)

Verification of these certificates inside COQ

• V. Magron

New Applications of Moment-SOS Hierarchies 11 / 68

New Framework (in my PhD thesis)

Software Implementation NLCertify: https://forge.ocamlcore.org/projects/nl-certify/ 15 000 lines of OCAML code 4000 lines of COQ code

• V. Magron

New Applications of Moment-SOS Hierarchies 11 / 68

Introduction Moment-SOS relaxations Semialgebraic Maxplus Optimization Formal Nonlinear Optimization Approximating Pareto curves Polynomial image of semialgebraic sets Program Analysis with Polynomial Templates Conclusion

Polynomial Optimization Problems

Semialgebraic set K := {x ∈ Rn : g1(x) 0, . . . , gm(x) 0} p∗ := min

x∈K p(x): NP hard

Sums of squares Σ[x] e.g. x2

1 − 2x1x2 + x2 2 = (x1 − x2)2

Q(K) :=

• σ0(x) + ∑m

j=1 σj(x)gj(x), with σj ∈ Σ[x]

• V. Magron

New Applications of Moment-SOS Hierarchies 12 / 68

Polynomial Optimization Problems

Archimedean module The set K is compact and the polynomial N − x2

2 belongs to

Q(K) for some N > 0. Assume that K is a box: product of closed intervals Normalize the feasibility set to get K′ := [−1, 1]n K′ := {x ∈ Rn : g1 := 1 − x2

1 0, · · · , gn := 1 − x2 n 0}

n − x2

2 belongs to Q(K′)

• V. Magron

New Applications of Moment-SOS Hierarchies 13 / 68

Convexification and the K Moment Problem

Borel σ-algebra B (generated by the open sets of Rn) M+(K): set of probability measures supported on K. If µ ∈ M+(K) then

1 µ : B → [0, 1], µ(∅) = 0, µ(Rn) < ∞ 2 µ( i Bi) = ∑i µ(Bi), for any countable (Bi) ⊂ B 3 K µ(dx) = 1

supp(µ) is the smallest set K such that µ(Rn\K) = 0

• V. Magron

New Applications of Moment-SOS Hierarchies 14 / 68

Convexification and the K Moment Problem

p∗ = inf

x∈K p(x) =

inf

µ∈M+(K)

• K p dµ
• V. Magron

New Applications of Moment-SOS Hierarchies 14 / 68

Convexification and the K Moment Problem

Let (xα)α∈Nn be the monomial basis Definition A sequence y has a representing measure on K if there exists a finite measure µ supported on K such that yα =

• K xαµ(dx) ,

∀ α ∈ Nn .

• V. Magron

New Applications of Moment-SOS Hierarchies 14 / 68

Convexification and the K Moment Problem

Ly(q) : q ∈ R[x] → ∑α qαyα Theorem [Putinar 93] Let K be compact and Q(K) be Archimedean. Then y has a representing measure on K iff Ly(σ) 0 , Ly(gj σ) 0 , ∀σ ∈ Σ[x] .

• V. Magron

New Applications of Moment-SOS Hierarchies 14 / 68

Lasserre’s Hierarchy of SDP relaxations

Moment matrix M(y)u,v := Ly(u · v), u, v monomials Localizing matrix M(gj y) associated with gj M(gj y)u,v := Ly(u · v · gj), u, v monomials

• V. Magron

New Applications of Moment-SOS Hierarchies 15 / 68

Lasserre’s Hierarchy of SDP relaxations

Mk(y) contains (n+2k

n ) variables, has size (n+k n )

Truncated matrix of order k = 2 with variables x1, x2: M2(y) =               1 | x1 x2 | x2

1

x1x2 x2

2

1 1 | y1,0 y0,1 | y2,0 y1,1 y0,2 − − − − − − − − x1 y1,0 | y2,0 y1,1 | y3,0 y2,1 y1,2 x2 y0,1 | y1,1 y0,2 | y2,1 y1,2 y0,3 − − − − − − − − − x2

1

y2,0 | y3,0 y2,1 | y4,0 y3,1 y2,2 x1x2 y1,1 | y2,1 y1,2 | y3,1 y2,2 y1,3 x2

2

y0,2 | y1,2 y0,3 | y2,2 y1,3 y0,4              

• V. Magron

New Applications of Moment-SOS Hierarchies 15 / 68

Lasserre’s Hierarchy of SDP relaxations

Consider g1(x) := 2 − x2

1 − x2

• 2. Then v1 = ⌈deg g1/2⌉ = 1.

M1(g1 y) =   1 x1 x2 1 2 − y2,0 − y0,2 2y1,0 − y3,0 − y1,2 2y0,1 − y2,1 − y0,3 x1 2y1,0 − y3,0 − y1,2 2y2,0 − y4,0 − y2,2 2y1,1 − y3,1 − y1,3 x2 2y0,1 − y2,1 − y0,3 2y1,1 − y3,1 − y1,3 2y0,2 − y2,2 − y0,4  

M1(g1 y)(3, 3) = L(g1(x) · x2 · x2) = L(2x2

2 − x2 1x2 2 − x4 2)

= 2y0,2 − y2,2 − y0,4

• V. Magron

New Applications of Moment-SOS Hierarchies 15 / 68

Lasserre’s Hierarchy of SDP relaxations

Truncation with moments of order at most 2k vj := ⌈deg gj/2⌉ Hierarchy of semidefinite relaxations:          infy Ly(p) = ∑α

• K pα xα µ(dx) = ∑α pα yα

Mk(y)

• 0 ,

Mk−vj(gj y)

• 0 ,

1 j m, y1 = 1 .

• V. Magron

New Applications of Moment-SOS Hierarchies 15 / 68

Semidefinite Optimization

F0, Fα symmetric real matrices, cost vector c Primal-dual pair of semidefinite programs: (SDP)              P : infy ∑α cαyα s.t. ∑α Fα yα − F0 0 D : supY Trace (F0 Y) s.t. Trace (Fα Y) = cα , Y 0 . Freely available SDP solvers (CSDP, SDPA, SEDUMI)

• V. Magron

New Applications of Moment-SOS Hierarchies 16 / 68

Primal-dual Moment-SOS

M+(K): space of probability measures supported on K Polynomial Optimization Problems (POP) (Primal) (Dual) inf

• K p dµ

= sup λ s.t. µ ∈ M+(K) s.t. λ ∈ R , p − λ ∈ Q(K)

• V. Magron

New Applications of Moment-SOS Hierarchies 17 / 68

Primal-dual Moment-SOS

Truncated quadratic module Qk(K) := Q(K) ∩ R2k[x] For large enough k, zero duality gap [Lasserre 01]: Polynomial Optimization Problems (POP) (Moment) (SOS) inf

∑

α

pα yα = sup λ s.t. Mk−vj(gj y) 0 , 0 j m, s.t. λ ∈ R , y1 = 1 p − λ ∈ Qk(K)

• V. Magron

New Applications of Moment-SOS Hierarchies 17 / 68

Practical Computation

Hierarchy of SOS relaxations: λk := sup

λ

• λ : p − λ ∈ Qk(K)
• Convergence guarantees λk ↑ p∗ [Lasserre 01]

If p − p∗ ∈ Qk(K) for some k then: y∗ := (1, x∗

1, x∗ 2, (x∗ 1)2, x∗ 1x∗ 2, . . . , (x∗ 1)2k, . . . , (x∗ n)2k)

is a global minimizer of the primal SDP [Lasserre 01].

• V. Magron

New Applications of Moment-SOS Hierarchies 18 / 68

Practical Computation

Caprasse Problem ∀x ∈ [−0.5, 0.5]4, −x1x3

3 + 4x2x2 3x4 + 4x1x3x2 4 + 2x2x3 4 +

4x1x3 + 4x2

3 − 10x2x4 − 10x2 4 + 5.1801 0.

scale_pol = true: scaled on [0, 1]4 relax_order = 2: SOS of degree at most 4 bound_squares_variables = true: redundant constraints x2

1 1, . . . , x2 4 1

• V. Magron

New Applications of Moment-SOS Hierarchies 18 / 68

The “No Free Lunch” Rule

Exponential dependency in

1 Relaxation order k (SOS degree) 2 number of variables n

Computing λk involves (n+2k

n ) variables

At fixed k, O(n2k) variables

• V. Magron

New Applications of Moment-SOS Hierarchies 19 / 68

Semialgebraic Extension [Lasserre-Putinar 10]

Example from Flyspeck

K := [4, 6.3504]6 ∆(x) = x1x4(−x1 + x2 + x3 − x4 + x5 + x6) + x2x5(x1 − x2 + x3 + x4 − x5 + x6) + x3x6(x1 + x2 − x3 + x4 + x5 − x6) − x2(x3x4 + x1x6) − x5(x1x3 + x4x6) ∂4∆x = x1(−x1 + x2 + x3 − x4 + x5 + x6) + x2x5 + x3x6 − x2x3 − x5x6 p(x) = ∂4∆x , q(x) := 4x1∆x f ∗

sa := inf x∈K

p(x)

• q(x)
• V. Magron

New Applications of Moment-SOS Hierarchies 20 / 68

Semialgebraic Extension [Lasserre-Putinar 10]

Example from Flyspeck:

z1 :=

• q(x) ,

m1 inf

x∈K z1(x) ,

M1 sup

x∈K

z1(x) , ˆ K := {(x, z) ∈ R8 : x ∈ K, h1(x, z) 0, · · · , h6(x, z) 0} h1 := z1 − m1 h4 := −z2

1 + q(x)

h2 := M1 − z1 h5 := z2z1 − p(x) h3 := z2

1 − q(x)

h6 := −z2z1 + p(x) f ∗

sa := inf(x,z)∈ ˆ K z2 and SOS yields λ2 = −0.618 < λ3 = −0.445.

• V. Magron

New Applications of Moment-SOS Hierarchies 20 / 68

Introduction Moment-SOS relaxations Semialgebraic Maxplus Optimization Formal Nonlinear Optimization Approximating Pareto curves Polynomial image of semialgebraic sets Program Analysis with Polynomial Templates Conclusion

The General “Informal Framework”

Given K a compact set and f a transcendental function, bound f ∗ = inf

x∈K f(x) and prove f ∗ 0

f is underestimated by a semialgebraic function fsa Reduce the problem f ∗

sa := infx∈K fsa(x) to a polynomial

• ptimization problem (POP)
• V. Magron

New Applications of Moment-SOS Hierarchies 21 / 68

Maxplus Approximation

Initially introduced to solve Optimal Control Problems [Fleming-McEneaney 00] Curse of dimensionality reduction [McEaneney Kluberg, Gaubert-McEneaney-Qu 11, Qu 13]. Allowed to solve instances of dim up to 15 (inaccessible by grid methods) In our context: approximate transcendental functions

• V. Magron

New Applications of Moment-SOS Hierarchies 22 / 68

Maxplus Approximation

Definition Let γ 0. A function φ : Rn → R is said to be γ-semiconvex if the function x → φ(x) + γ

2 x2 2 is convex.

a y par+

a1

par+

a2

par−

a2

par−

a1

a2 a1 arctan m M

• V. Magron

New Applications of Moment-SOS Hierarchies 22 / 68

Nonlinear Function Representation

Exact parsimonious maxplus representations

a y

• V. Magron

New Applications of Moment-SOS Hierarchies 23 / 68

Nonlinear Function Representation

Exact parsimonious maxplus representations

a y

• V. Magron

New Applications of Moment-SOS Hierarchies 23 / 68

Nonlinear Function Representation

Abstract syntax tree representations of multivariate transcendental functions: leaves are semialgebraic functions of A nodes are univariate functions of D or binary operations

• V. Magron

New Applications of Moment-SOS Hierarchies 23 / 68

Nonlinear Function Representation

For the “Simple” Example from Flyspeck:

+ l(x) arctan r(x)

• V. Magron

New Applications of Moment-SOS Hierarchies 23 / 68

Maxplus Optimization Algorithm

First iteration:

+ l(x) arctan r(x) a y par−

a1

arctan m M a1 1 control point {a1}: m1 = −4.7 × 10−3 < 0

• V. Magron

New Applications of Moment-SOS Hierarchies 24 / 68

Maxplus Optimization Algorithm

Second iteration:

+ l(x) arctan r(x) a y par−

a1

par−

a2

arctan m M a1 a2 2 control points {a1, a2}: m2 = −6.1 × 10−5 < 0

• V. Magron

New Applications of Moment-SOS Hierarchies 24 / 68

Maxplus Optimization Algorithm

Third iteration:

+ l(x) arctan r(x) a y par−

a1

par−

a2

par−

a3

arctan m M a1 a2 a3 3 control points {a1, a2, a3}: m3 = 4.1 × 10−6 > 0

OK!

• V. Magron

New Applications of Moment-SOS Hierarchies 24 / 68

Maxplus Optimization Algorithm

Input: tree t, box K, SOS relaxation order k, precision p Output: bounds m and M, approximations t−

2 and t+ 2

1: if t ∈ A then t− := t, t+ := t 2: else if u := root(t) ∈ D with child c then 3:

mc, Mc, c−, c+ := samp_approx(c, K, k, p)

4:

I := [mc, Mc]

5:

u−, u+ := unary_approx(u, I, c, p)

6:

t−, t+ := compose_approx(u, u−, u+, I, c−, c+)

7: else if bop := root(t) with children c1 and c2 then 8:

mi, Mi, c−

i , c+ i := samp_approx(ci, K, k, p) for i ∈ {1, 2}

9:

t−, t+ := compose_bop(c−

1 , c+ 1 , c− 2 , c+ 2 , bop, [m2, M2])

10: end 11: return min_sa(t−, K, k), max_sa(t+, K, k), t−, t+

• V. Magron

New Applications of Moment-SOS Hierarchies 25 / 68

Minimax Approximation / For Comparison

The precision is an integer d The best-uniform degree-d polynomial approximation of u: min

h∈Rd[x] u − h∞ = min h∈Rd[x](sup x∈I

|u(x) − h(x)|) Implementation in Sollya [Chevillard-Joldes-Lauter 10] Interface of NLCertify with Sollya

• V. Magron

New Applications of Moment-SOS Hierarchies 26 / 68

High-degree Polynomial Approximation + SOS

SWF: minx∈[1,500]n f(x) = − ∑n

i=1 xi sin(√xi)

replace sin(√·) by a degree-d Chebyshev polynomial Hard to combine with SOS

• V. Magron

New Applications of Moment-SOS Hierarchies 27 / 68

High-degree Polynomial Approximation + SOS

Indeed: Small d: lack of accuracy = ⇒ expensive Branch and Bound Large d: “No free lunch” rule with (n+d

n ) SDP variables

• V. Magron

New Applications of Moment-SOS Hierarchies 27 / 68

High-degree Polynomial Approximation + SOS

SWF with n = 10, d = 4: 38 min to compute a lower bound of −430n

• V. Magron

New Applications of Moment-SOS Hierarchies 27 / 68

Comparison on Global Optimization Problems

min

x∈[1,500]n f(x) = − ∑n i=1 xi sin(√xi)

f ∗ −418.9n Interval Arithmetic for sin + SOS n lower bound nlifting #boxes time 10 −430n 3830 129 s 10 −430n 2n 16 40 s

• V. Magron

New Applications of Moment-SOS Hierarchies 28 / 68

Comparison on Global Optimization Problems

min

x∈[1,500]n f(x) = − n

∑

i=1

xi sin(√xi) f ∗ −418.9n

a y par−

a1

par−

a2

par−

a3

par+

a1

par+

a2

par+

a3

sin 1 a1 a2 a3 = √ 500

n lower bound nlifting #boxes time 10 −430n 3830 129 s 10 −430n 2n 16 40 s

• V. Magron

New Applications of Moment-SOS Hierarchies 29 / 68

Comparison on Global Optimization Problems

min

x∈[1,500]n f(x) = − ∑n i=1 xi sin(√xi)

f ∗ −418.9n Interval Arithmetic for sin + SOS n lower bound nlifting #boxes time 100 −440n > 10000 > 10 h 100 −440n 2n 274 1.9 h

• V. Magron

New Applications of Moment-SOS Hierarchies 30 / 68

Comparison on Global Optimization Problems

min

x∈[1,500]n f(x) = − n

∑

i=1

xi sin(√xi) f ∗ −418.9n

a y par−

a1

par−

a2

par−

a3

par+

a1

par+

a2

par+

a3

sin 1 a1 a2 a3 = √ 500

n lower bound nlifting #boxes time 100 −440n > 10000 > 10 h 100 −440n 2n 274 1.9 h

• V. Magron

New Applications of Moment-SOS Hierarchies 31 / 68

Comparison on Global Optimization Problems

min

x∈[1,500]n f(x) = − n−1

∑

i=1

(xi + xi+1) sin(√xi)

a y par−

a1

par−

a2

par−

a3

par+

a1

par+

a2

par+

a3

sin 1 a1 a2 a3 = √ 500

n lower bound nlifting #boxes time 1000 −967n 2n 1 543 s 1000 −968n n 1 272 s

• V. Magron

New Applications of Moment-SOS Hierarchies 32 / 68

Comparison on Global Optimization Problems

min

x∈[1,500]n f(x) = − n−1

∑

i=1

(xi + xi+1) sin(√xi)

b y b → sin( √ b) par−

b1

par−

b2

par−

b3

par+

b1

par+

b2

par+

b3

1 b1 b2 b3 = 500

n lower bound nlifting #boxes time 1000 −967n 2n 1 543 s 1000 −968n n 1 272 s

• V. Magron

New Applications of Moment-SOS Hierarchies 33 / 68

Convergence of the Optimization Algorithm

Let f be a multivariate transcendental function Let t−

p be the underestimator of f, obtained at precision p

Let xp

• pt be a minimizer of t−

p over K

Theorem [X. Allamigeon S. Gaubert VM B. Werner 13] Every accumulation point of the sequence (xp

• pt) is a global min-

imizer of f on K. Ingredients of the proof: Convergence of Lasserre SOS hierarchy Uniform approximation schemes (Maxplus/Minimax)

• V. Magron

New Applications of Moment-SOS Hierarchies 34 / 68

Polynomial Approximations for Semialgebraic Functions

Inspired from [Lasserre - Thanh 13] Let fsa ∈ A defined on a box K ⊂ Rn Let µn be the standard Lebesgue measure on Rn Best polynomial underestimator h ∈ Rd[x] of fsa for the L1 norm: (Psa)    min

h∈Rd[x]

• K(fsa − h)dµn

s.t. fsa − h 0 on K . Lemma Problem (Psa) has a degree-d polynomial minimizer hd.

• V. Magron

New Applications of Moment-SOS Hierarchies 35 / 68

Polynomial Approximations for Semialgebraic Functions

b.s.a.l. ˆ K := {(x, z) ∈ Rn+p : g1(x, z) 0, . . . , gm(x, z) 0} The quadratic module M( ˆ K) is Archimedean The optimal solution hd of (Psa) is a maximizer of: (Pd)    max

h∈Rd[x]

• [0,1]n h dµn

s.t. (zp − h) ∈ M( ˆ K) .

• V. Magron

New Applications of Moment-SOS Hierarchies 36 / 68

Polynomial Approximations for Semialgebraic Functions

Let md be the optimal value of Problem (Psa) Let hdk be a maximizer of the SOS relaxation of (Pd) Convergence of the SOS Hierarchy The sequence (fsa − hdk1)kk0 is non-increasing and converges to md. Each accumulation point of the sequence (hdk)kk0 is an

• ptimal solution of Problem (Psa).
• V. Magron

New Applications of Moment-SOS Hierarchies 37 / 68

Polynomial Approximations for Semialgebraic Functions

fsa(x) := ∂4∆x √4x1∆x d k Upper bound of fsa − hdk1 Bound 2 2 0.8024

• 1.171

3 0.3709

• 0.4479

4 2 1.617

• 1.056

3 0.1766

• 0.4493
• V. Magron

New Applications of Moment-SOS Hierarchies 37 / 68

Polynomial Approximations for Semialgebraic Functions

rad2 : (x1, x2) → −64x2

1+128x1x2+1024x1−64x2 2+1024x2−4096

−8x2

1+8x1x2+128x1−8x2 2+128x2−512

Linear and quadratic underestimators for rad2 (k = 3):

• V. Magron

New Applications of Moment-SOS Hierarchies 38 / 68

Polynomial Approximations for Semialgebraic Functions

rad2 : (x1, x2) → −64x2

1+128x1x2+1024x1−64x2 2+1024x2−4096

−8x2

1+8x1x2+128x1−8x2 2+128x2−512

Linear and quadratic underestimators for rad2 (k = 3):

0.2 0.4 0.6 0.8 1 0 0.5 1 0.11 0.11 0.12 0.12 d = 1 d = 2

• V. Magron

New Applications of Moment-SOS Hierarchies 38 / 68

Contributions

Published:

• X. Allamigeon, S. Gaubert, V. Magron, and B. Werner.

Certification of inequalities involving transcendental functions: combining sdp and max-plus approximation, ECC Conference 2013.

• X. Allamigeon, S. Gaubert, V. Magron, and B. Werner.

Certification of bounds of non-linear functions: the templates method, CICM Conference, 2013.

In revision:

• X. Allamigeon, S. Gaubert, V. Magron, and B. Werner.

Certification of Real Inequalities – Templates and Sums of Squares, arxiv:1403.5899, 2014.

• V. Magron

New Applications of Moment-SOS Hierarchies 39 / 68

Introduction Moment-SOS relaxations Semialgebraic Maxplus Optimization Formal Nonlinear Optimization Approximating Pareto curves Polynomial image of semialgebraic sets Program Analysis with Polynomial Templates Conclusion

The General “Formal Framework”

We check the correctness of SOS certificates for POP We build certificates to prove interval bounds for semialgebraic functions We bound formally transcendental functions with semialgebraic approximations

• V. Magron

New Applications of Moment-SOS Hierarchies 40 / 68

Formal SOS bounds

When q ∈ Q(K), σ0, . . . , σm is a positivity certificate for q Check symbolic polynomial equalities q = q′ in COQ Existing tactic ring [Grégoire-Mahboubi 05] Polynomials coefficients: arbitrary-size rationals bigQ [Grégoire-Théry 06] Much simpler to verify certificates using sceptical approach Extends also to semialgebraic functions

• V. Magron

New Applications of Moment-SOS Hierarchies 41 / 68

Checking Polynomial Equalities: ring tactic

Sparse Horner normal form

Inductive PolC: Type := | Pc : bigQ → PolC | Pinj: positive → PolC → PolC | PX : PolC → positive → PolC → PolC. (Pc c) for constant polynomials (Pinj i p) shifts the index of i in the variables of p (PX p j q) evaluates to pxj

1 + q(x2, . . . , xn)

Encoding SOS certificates with Sparse Horner polynomials

• V. Magron

New Applications of Moment-SOS Hierarchies 42 / 68

Bounding the Polynomial Remainder

Normalized POP (x ∈ [0, 1]n) ǫpop(x) := p(x) − λk − ∑m

j=0 σj(x)gj(x)

∀x ∈ [0, 1]n, ǫpop(x) ǫ∗

pop := ∑ ǫα0

ǫα

• V. Magron

New Applications of Moment-SOS Hierarchies 43 / 68

Formal SOS Results

POP1: ∀x ∈ K, ∂4∆x −41. POP2: ∀x ∈ K, ∆x 0. Problem n NLCertify micromega [Besson 07] POP1 6 0.08 s 9.00 s POP2 2 0.09 s 0.36 s 3 0.39 s − 6 13.2 s − Sparse SOS relaxations = ⇒ Speedup

• V. Magron

New Applications of Moment-SOS Hierarchies 44 / 68

Benchmarks for Flyspeck Inequalities

Inequality #boxes Time Time 9922699028 39 190 s 2218 s 3318775219 338 1560 s 19136 s Comparable with Taylor interval methods in HOL-LIGHT [Hales-Solovyev 13] No free lunch: SDP informal bottleneck 22 times slower than SDP: q = q′ formal bottleneck

• V. Magron

New Applications of Moment-SOS Hierarchies 45 / 68

Contribution

For more details on the formal side:

• X. Allamigeon, S. Gaubert, V. Magron and B. Werner. Formal

Proofs for Nonlinear Optimization. In Revision, arxiv:1404.7282

• V. Magron

New Applications of Moment-SOS Hierarchies 46 / 68

Introduction Moment-SOS relaxations Semialgebraic Maxplus Optimization Formal Nonlinear Optimization Approximating Pareto curves Polynomial image of semialgebraic sets Program Analysis with Polynomial Templates Conclusion

Bicriteria Optimization Problems

Let f1, f2 ∈ Rd[x] two conflicting criteria Let S := {x ∈ Rn : g1(x) 0, . . . , gm(x) 0} a semialgebraic set (P)

• min

x∈S (f1(x) f2(x))⊤

• Assumption

The image space R2 is partially ordered in a natural way (R2

+ is

the ordering cone).

• V. Magron

New Applications of Moment-SOS Hierarchies 47 / 68

Bicriteria Optimization Problems

g1 := −(x1 − 2)3/2 − x2 + 2.5 , g2 := −x1 − x2 + 8(−x1 + x2 + 0.65)2 + 3.85 , S := {x ∈ R2 : g1(x) 0, g2(x) 0} . f1 := (x1 + x2 − 7.5)2/4 + (−x1 + x2 + 3)2 , f2 := (x1 − 1)2/4 + (x2 − 4)2/4 .

• V. Magron

New Applications of Moment-SOS Hierarchies 47 / 68

Parametric sublevel set approximation

Inspired by previous research on multiobjective linear

• ptimization [Gorissen-den Hertog 12]

Workaround: reduce P to a parametric POP (Pλ) : f ∗(λ) := min

x∈S { f2(x) : f1(x) λ } ,

• V. Magron

New Applications of Moment-SOS Hierarchies 48 / 68

A Hierarchy of Polynomial underestimators

Moment-SOS approach [Lasserre 10]: (Dd)      max

q∈R2d[λ] 2d

∑

k=0

qk/(1 + k) s.t. f2(x) − q(λ) ∈ Q2d(K) . The hierarchy (Dd) provides a sequence (qd) of polynomial underestimators of f ∗(λ). limd→∞ 1

0 (f ∗(λ) − qd(λ))dλ = 0

• V. Magron

New Applications of Moment-SOS Hierarchies 49 / 68

A Hierarchy of Polynomial underestimators

Degree 4

• V. Magron

New Applications of Moment-SOS Hierarchies 50 / 68

A Hierarchy of Polynomial underestimators

Degree 6

• V. Magron

New Applications of Moment-SOS Hierarchies 50 / 68

A Hierarchy of Polynomial underestimators

Degree 8

• V. Magron

New Applications of Moment-SOS Hierarchies 50 / 68

Contributions

Numerical schemes that avoid computing finitely many points. Pareto curve approximation with polynomials, convergence guarantees in L1-norm

• V. Magron, D. Henrion, J.B. Lasserre. Approximating Pareto

Curves using Semidefinite Relaxations. Operations Research

• Letters. arxiv:1404.4772, April 2014.
• V. Magron

New Applications of Moment-SOS Hierarchies 51 / 68

Introduction Moment-SOS relaxations Semialgebraic Maxplus Optimization Formal Nonlinear Optimization Approximating Pareto curves Polynomial image of semialgebraic sets Program Analysis with Polynomial Templates Conclusion

Approximation of sets defined with “∃”

Let B ⊂ Rm be the unit ball and assume that F = f(S) ⊆ B. Another point of view: F = {y ∈ B : ∃x ∈ S s.t. h(x, y) 0} , with hf (x, y) := −y − f(x)2

2 .

Approximate F as closely as desired by a sequence of sets

• f the form :

Fk := {y ∈ B : qk(y) 0} , for some polynomials qk ∈ R2k[y].

• V. Magron

New Applications of Moment-SOS Hierarchies 52 / 68

A hierarchy of outer approximations of f(S)

Let K = S × B, g0 := 1 and Qk(K) be the k-truncated quadratic module generated by g0, . . . , gm: Qk(K) = m

∑

l=0

σl(x, y)gl(x), with σl ∈ Σk−vl[x, y]

• Define h(y) := supx∈S h(x, y)

Hierarchy of Semidefinite programs: ρk := min

q∈R2k[y],σl

• B(q − h)dy : q − hf ∈ Qk(K)
• .

Yet another SOS program with an optimal solution qk ∈ R2k[y]!

• V. Magron

New Applications of Moment-SOS Hierarchies 53 / 68

A hierarchy of outer approximations of f(S)

From the definition of qk, the sublevel sets Fk := {y ∈ B : qk(y) 0} ⊇ F , provide a sequence of certified outer approximations of F. It comes from the following: ∀(x, y) ∈ K, qk(y) hf (x, y) ⇐ ⇒ ∀y, qk(y) h(y) .

• V. Magron

New Applications of Moment-SOS Hierarchies 54 / 68

Strong convergence property

Theorem

1 The sequence of underestimators (qk)kk0 converges to h

w.r.t the L1(B)-norm: lim

k→∞

• B |qk − h|dy = 0 .
• V. Magron

New Applications of Moment-SOS Hierarchies 55 / 68

Strong convergence property

Theorem

1 The sequence of underestimators (qk)kk0 converges to h

w.r.t the L1(B)-norm: lim

k→∞

• B |qk − h|dy = 0 .

2

lim

k→∞ vol(Fk\F) = 0 .

• V. Magron

New Applications of Moment-SOS Hierarchies 55 / 68

Approximation for polynomial image of semialgebraic sets

Image of the unit ball S := {x ∈ R2 : x2

2 1} by

f(x) := (x1 + x1x2, x2 − x3

1)/2

F1

• V. Magron

New Applications of Moment-SOS Hierarchies 56 / 68

Approximation for polynomial image of semialgebraic sets

Image of the unit ball S := {x ∈ R2 : x2

2 1} by

f(x) := (x1 + x1x2, x2 − x3

1)/2

F2

• V. Magron

New Applications of Moment-SOS Hierarchies 56 / 68

Approximation for polynomial image of semialgebraic sets

Image of the unit ball S := {x ∈ R2 : x2

2 1} by

f(x) := (x1 + x1x2, x2 − x3

1)/2

F3

• V. Magron

New Applications of Moment-SOS Hierarchies 56 / 68

Approximation for polynomial image of semialgebraic sets

Image of the unit ball S := {x ∈ R2 : x2

2 1} by

f(x) := (x1 + x1x2, x2 − x3

1)/2

F4

• V. Magron

New Applications of Moment-SOS Hierarchies 56 / 68

Semialgebraic set projections

f(x) = (x1, x2): projection on R2 of the semialgebraic set S := {x ∈ R3 :x2

2 1, 1/4 − (x1 + 1/2)2 − x2 2 0,

1/9 − (x1 − 1/2)4 − x4

2 0}

F2

• V. Magron

New Applications of Moment-SOS Hierarchies 57 / 68

Semialgebraic set projections

f(x) = (x1, x2): projection on R2 of the semialgebraic set S := {x ∈ R3 :x2

2 1, 1/4 − (x1 + 1/2)2 − x2 2 0,

1/9 − (x1 − 1/2)4 − x4

2 0}

F3

• V. Magron

New Applications of Moment-SOS Hierarchies 57 / 68

Semialgebraic set projections

f(x) = (x1, x2): projection on R2 of the semialgebraic set S := {x ∈ R3 :x2

2 1, 1/4 − (x1 + 1/2)2 − x2 2 0,

1/9 − (x1 − 1/2)4 − x4

2 0}

F4

• V. Magron

New Applications of Moment-SOS Hierarchies 57 / 68

Support function of a closed convex set

Assume that F is strictly convex Let q := supx∈F{b⊤x} be the support function of F q is degree-1 positive homogeneous, subadditive One can show that f = ∇q For a convex homogeneous ˜ q, let q(x) := x˜ q( x

x)

One can show that ∇q(x) = ∇˜ q(x), for each x ∈ Sn−1

• V. Magron

New Applications of Moment-SOS Hierarchies 58 / 68

Support function of a closed convex set

˜ q(x) := x4

1 + x4 2 + 2x2 1x2 2 + 7/2(x2 1 + x2 2) − (x1x2 + x1 + x2)

F2

• V. Magron

New Applications of Moment-SOS Hierarchies 58 / 68

Support function of a closed convex set

˜ q(x) := x4

1 + x4 2 + 2x2 1x2 2 + 7/2(x2 1 + x2 2) − (x1x2 + x1 + x2)

F4

• V. Magron

New Applications of Moment-SOS Hierarchies 58 / 68

Approximating Pareto curves

Back our previous nonconvex example: F1

• V. Magron

New Applications of Moment-SOS Hierarchies 59 / 68

Approximating Pareto curves

Back our previous nonconvex example: F2

• V. Magron

New Applications of Moment-SOS Hierarchies 59 / 68

Approximating Pareto curves

Back our previous nonconvex example: F4

• V. Magron

New Applications of Moment-SOS Hierarchies 59 / 68

Approximating Pareto curves

“Zoom” on the region which is hard to approximate: F4

• V. Magron

New Applications of Moment-SOS Hierarchies 60 / 68

Approximating Pareto curves

“Zoom” on the region which is hard to approximate: F5

• V. Magron

New Applications of Moment-SOS Hierarchies 60 / 68

Semialgebraic image of semialgebraic sets

Image of the unit ball S := {x ∈ R2 : x2

2 1} by

f(x) := (min(x1 + x1x2, x2

1), x2 − x3 1)/3

F1

• V. Magron

New Applications of Moment-SOS Hierarchies 61 / 68

Semialgebraic image of semialgebraic sets

Image of the unit ball S := {x ∈ R2 : x2

2 1} by

f(x) := (min(x1 + x1x2, x2

1), x2 − x3 1)/3

F2

• V. Magron

New Applications of Moment-SOS Hierarchies 61 / 68

Semialgebraic image of semialgebraic sets

Image of the unit ball S := {x ∈ R2 : x2

2 1} by

f(x) := (min(x1 + x1x2, x2

1), x2 − x3 1)/3

F3

• V. Magron

New Applications of Moment-SOS Hierarchies 61 / 68

Semialgebraic image of semialgebraic sets

Image of the unit ball S := {x ∈ R2 : x2

2 1} by

f(x) := (min(x1 + x1x2, x2

1), x2 − x3 1)/3

F4

• V. Magron

New Applications of Moment-SOS Hierarchies 61 / 68

Introduction Moment-SOS relaxations Semialgebraic Maxplus Optimization Formal Nonlinear Optimization Approximating Pareto curves Polynomial image of semialgebraic sets Program Analysis with Polynomial Templates Conclusion

One-loop with Conditional Branching

r, s, Ti, Te ∈ R[x] x0 ∈ X0, with X0 semialgebraic set x = x0; while (r(x) 0){ if (s(x) 0){ x = Ti(x); } else{ x = Te(x); } }

• V. Magron

New Applications of Moment-SOS Hierarchies 62 / 68

Well-representative Templates w.r.t. Properties

Sufficient condition to get inductive invariant: α := min

q∈R[x]

sup

x∈X0

q(x) s.t. q − q ◦ Ti 0 , q − q ◦ Te 0 , q − κ 0 .

• k∈N

Xk ⊆ {x ∈ Rn : q(x) α} ⊆ {x ∈ Rn : κ(x) α}

• V. Magron

New Applications of Moment-SOS Hierarchies 63 / 68

Bounding Template using SOS

Sufficient condition to get bounding inductive invariant: α := min

q∈R[x]

sup

x∈X0

q(x) s.t. q − q ◦ Ti 0 , q − q ◦ Te 0 , q − · 2

2 0 .

• k∈N

Xk ⊆ {x ∈ Rn : q(x) α} ⊆ {x ∈ Rn : x2 α}

• V. Magron

New Applications of Moment-SOS Hierarchies 64 / 68

Bounds for

k∈N Xk

X0 := [0.9, 1.1] × [0, 0.2] r(x) := 1 s(x) := 1 − x2 Ti(x) := (x2

1 + x3 2, x3 1 + x2 2)

Te(x) := (1 2x2

1 + 2

5x3

2, −3

5x3

1 + 3

10x2

2)

κ(x) = x2 Degree 6

• V. Magron

New Applications of Moment-SOS Hierarchies 65 / 68

Bounds for

k∈N Xk

X0 := [0.9, 1.1] × [0, 0.2] r(x) := 1 s(x) := 1 − x2 Ti(x) := (x2

1 + x3 2, x3 1 + x2 2)

Te(x) := (1 2x2

1 + 2

5x3

2, −3

5x3

1 + 3

10x2

2)

κ(x) = x2 Degree 8

• V. Magron

New Applications of Moment-SOS Hierarchies 65 / 68

Bounds for

k∈N Xk

X0 := [0.9, 1.1] × [0, 0.2] r(x) := 1 s(x) := 1 − x2 Ti(x) := (x2

1 + x3 2, x3 1 + x2 2)

Te(x) := (1 2x2

1 + 2

5x3

2, −3

5x3

1 + 3

10x2

2)

κ(x) = x2 Degree 10

• V. Magron

New Applications of Moment-SOS Hierarchies 65 / 68

Does

k∈N Xk avoid unsafe region?

X0 := [0.5, 0.7]2 r(x) := 1 s(x) := x2 Ti(x) := (x2

1 + x3 2, x3 1 + x2 2)

Te(x) := (1 2x2

1 + 2

5x3

2, −3

5x3

1 + 3

10x2

2)

κ(x) = 1 4 − (x1 + 1 2)2 − (x2 + 1 2)2 Degree 6

• V. Magron

New Applications of Moment-SOS Hierarchies 66 / 68

Does

k∈N Xk avoid unsafe region?

X0 := [0.5, 0.7]2 r(x) := 1 s(x) := x2 Ti(x) := (x2

1 + x3 2, x3 1 + x2 2)

Te(x) := (1 2x2

1 + 2

5x3

2, −3

5x3

1 + 3

10x2

2)

κ(x) = 1 4 − (x1 + 1 2)2 − (x2 + 1 2)2 Degree 8

• V. Magron

New Applications of Moment-SOS Hierarchies 66 / 68

Does

k∈N Xk avoid unsafe region?

X0 := [0.5, 0.7]2 r(x) := 1 s(x) := x2 Ti(x) := (x2

1 + x3 2, x3 1 + x2 2)

Te(x) := (1 2x2

1 + 2

5x3

2, −3

5x3

1 + 3

10x2

2)

κ(x) = 1 4 − (x1 + 1 2)2 − (x2 + 1 2)2 Degree 10

• V. Magron

New Applications of Moment-SOS Hierarchies 66 / 68

Contributions

• A. Adjé, V. Magron. Polynomial template generation using

sum-of-squares programming. Submitted. arxiv:1409.3941, October 2014.

• V. Magron

New Applications of Moment-SOS Hierarchies 67 / 68

Introduction Moment-SOS relaxations Semialgebraic Maxplus Optimization Formal Nonlinear Optimization Approximating Pareto curves Polynomial image of semialgebraic sets Program Analysis with Polynomial Templates Conclusion

Conclusion

Formal nonlinear optimization: NLCertify Safe solutions for challenging problems, e.g. Flyspeck Approximation of Pareto Curves, images and projections

• f semialgebraic sets

Program Analysis with polynomial templates

• V. Magron

New Applications of Moment-SOS Hierarchies 68 / 68

Conclusion

Further research: OCAML API Alternative Polynomials bounds using geometric programming (T. de Wolff, S. Iliman) COQ tactic Improve formal polynomial checker Semialgebraic/transcendental program analysis

• V. Magron

New Applications of Moment-SOS Hierarchies 68 / 68

Conclusion

Further research: Generalized problem of moments (Moment) (SOS) inf

• K p0 dµ
• sup

λ0 + ∑

i

λi bi s.t.

• K pi dµ bi

s.t. λ0, λi 0 , µ ∈ M+(K) p0 − λ0 − ∑

i

λi pi ∈ Qk(K) Formal bounds using SDP and ring

• V. Magron

New Applications of Moment-SOS Hierarchies 68 / 68

Conclusion

Further research at IC:

1 Tuning FPGA hardware by performing program analysis

Krivine-Handelman representation of positive polynomials

• D. Boland, G. Constantinides. Automated Precision

Analysis: A Polynomial Algebraic Approach. Extension using Putinar representations Mixed LP/SOS certificates

• V. Magron

New Applications of Moment-SOS Hierarchies 68 / 68

Conclusion

Further research at IC:

2 Adapting existing framework in GPU Verification

Verification of race/divergence freedom

• A. Betts, N. Chong, A.F. Donaldson, S. Qadeer and P.
• Thomson. GPUVerify: A Verifier for GPU Kernels.

Built in top of Boogie, interface with Z3 Recent features to handle nonlinearity

• V. Magron

New Applications of Moment-SOS Hierarchies 68 / 68

End

Thank you for your attention! cas.ee.ic.ac.uk/people/vmagron