MINLP: Undecidability and Hardness A tutorial Leo Liberti, CNRS LIX - - PowerPoint PPT Presentation

MINLP: Undecidability and Hardness

A tutorial Leo Liberti, CNRS LIX Ecole Polytechnique liberti@lix.polytechnique.fr The Aussois COW, 2017

1 / 49

Section 1 Introduction

2 / 49

Mixed-Integer Nonlinear Programming

MINLP

◮ Formal declarative language

sentences describe optimization problems

◮ Can encode pure feasibility problems

by minimizing a constant function e.g. min{0 | g(x) ≤ 0}

◮ Includes most other MP classes

e.g. LP, MILP, NLP

◮ Interpreter = solver

shifts focus from algorithmics to modelling

◮ Only consider single-objective MP

3 / 49

Syntax

Given functions f, g1, . . . , gm : Qn → Q and Z ⊆ {1, . . . , n} min f(x) ∀i ≤ m gi(x) ≤ ∀j ∈ Z xj ∈ Z   

◮ f, gi represented by expression DAGs

E.g. min{x1 + 2x2 − log(x1x2) | x1x2

2 ≥ 1 ∧ x1 ≥ 0 ∧ x2 ∈ N}

4 / 49

Semantics

P ≡ min{x1 + 2x2 − log(x1x2) | x1x2

2 ≥ 1 ∧ x1 ≥ 0 ∧ x2 ∈ N}

P = (opt(P), val(P))

• pt(P) = (1, 1)

val(P) = 3

5 / 49

“Solving an MP”

◮ Given an MP P, there are three possibilities:

• 1. P exists
• 2. P is unbounded
• 3. P is infeasible

◮ P has a feasible solution iff P exists or is unbounded

• therwise it is infeasible

◮ P has an optimum iff P exists

• therwise it is infeasible or unbounded

◮ Asymmetry between optimization and feasibility

P ≡ min{0 | g(x) ≤ 0 ∧ x ∈ X}

• YES

∃P ∨ unbnd(P ) NO infeas(P ) Q ≡ min{f(x) | g(x) ≤ 0 ∧ x ∈ X}

• YES

∃Q NO unbnd(Q) ∨ infeas(Q) 6 / 49

Section 2 Undecidability

7 / 49

Formal systems (FS)

◮ Formal System F

◮ alphabet and formal grammar

well-formed formulæ and sentences

◮ Axioms A

(recursive1 consistent set of sentences)

◮ Inference rules R

derive new sentences from old ones

◮ Language L

set of all sentences of F

◮ Theory T

sentences obtained by iterated application of R to A

1M recursive if ∃ alg. solving “given a, is a ∈ M or not?”

8 / 49

Some FSs

◮ Peano Arithmetic (PA): →↔, ∧, ∨, ¬, ∀, ∃, +, ×, = and

variable names; 1st order sentences about N; A=PL+induction; modus ponens and generalization

◮ T : provable sentences about N

◮ Real-closed Fields (RLF): like above and >; polynomials

• ver R; field axioms for R, “basic operations on polynomials”

◮ T : polynomial systems over R with solution in R

◮ Diophantine Equations (DE):

existentially quantified subset of PA

◮ T : polynomial systems over Z with solution in N ◮ {[∃x ∈ Nn p(x) = 0] | p ∈ Z[x]} ≡ {[f ∧ x ∈ Nn] | f ∈ L(RLF)} ◮ “between PA and RLF” 9 / 49

What is decidability?

FS F is decidable if ∃ algorithm A : L → {0, 1} ∀f ∈ L A(f) = 1 if f ∈ T

• therwise

◮ PA: does a sentence have a proof in PA or not? ◮ RLF: does a polynomial over R have a solution in R or not? ◮ DE: does a polynomial over Z have a solution in N or not?

Only YES/NO answer required (rather than explicit proof)

10 / 49

Subsection 1 Polynomial systems in integer variables

11 / 49

DE: Relevance to MINLP

◮ DE ⊆ [MINLP feasibility]

• bvious

◮ DE ⊆ [MINLP optimality]

∀i ≤ m gi(x) = x ∈ Nn

• is feasible

⇔      u∗ = min u (1 − u)

i≤m

(gi(x))2 = (x, u) ∈ Nn+1      = 0

◮ if u∗ > 1 get −1 = 0 (contradiction) ◮ if u∗ = 1 get g(x) = 0 (infeasible) ◮ if u∗ = 0 get g(x) = 0 (feasible) Suppose u∗ = 1 and g(x) = 0 feas., then u = 0 would also satisfy constr. and contradict minimality of u∗ 12 / 49

Goal: MINLP is undecidable

◮ MINLP contains DE ◮ show DE is undecidable

◮ any r.e.2 subset of N can be encoded by a DE ◮ {a ∈ N | a ∈ Halting} is r.e. ◮ so Halting can be represented by a DE ◮ if every DE were decidable, we could solve Halting

2M ⊆ N is r.e. if ∃ alg. terminating on input a iff a ∈ M (nonterm. for a ∈ M)

13 / 49

DE and set membership

◮ DE sentence [p(y) = 0 ∧ y ∈ Nn+1]

y = (a, x) where a ∈ N “encodes the instance”

◮ DEs define subsets M ⊆ N:

a ∈ M ↔ ∃x ∈ Nn p(a, x) = 0 (†)

◮ Conversely, given M ⊆ N,

is there p(a, x) ∈ Z[a, x] s.t. (†)?

◮ Focus on r.e. sets M ⊆ N ◮ Matyiasevich-Davis-Putnam-Robinson thm. (MDPR, 1970):

For each r.e. set M there is a DE p(a, x) s.t. (†)

14 / 49

MDPR implies undecidability

◮ Suppose DE is decidable

◮ ⇒ given DE, can decide feasibility/infeas. ◮ ⇒ given r.e. set M, can decide a ∈ M and a ∈ M

◮ Halting: given TM T and input ι, will T(ι) terminate?

undecidable by [Turing 1936] (diagonal argument)

◮ Let H = {(T, ι) | T(ι) ↓}

H is r.e.: simulate T with input ι, terminates iff T(ι) ↓ MH = encoding of H in N ⇒ MH is r.e.

◮ ⇒ can decide MH and solve Halting, contradiction ◮ Hence DE undecidable ◮ ∃ Universal Diophantine Equations (UDE)

encoding the dynamics of a UTM

∃ UDE deg d and n vars where (d, n) ∈ {(4, 58), (1.6 × 1045, 9)}

15 / 49

Structure of the MDPR theorem

◮ Proof of Gödel’s 1st incompleteness thm.

r.e. sets ≡ DE with < ∞ ∃ and bounded ∀ quantifiers

◮ Davis’ normal form

• ne bounded quantifier suffices: ∃x0∀a ≤ x0∃x p(a, x) = 0

◮ (2 bnd qnt ≡ 1 bnd qnt on pairs) and induction

◮ Robinson’s idea

get rid of universal quantifier by using exponent vars

◮ idea: [∃x0∀a ≤ x0∃x p(a, x) = 0] “ → ”

• ∃x
• a≤x0

p(a, x)

• ◮ precise encoding needs variables in exponents

◮ Matyiasevic’s contribution

express c = ba using polynomials

◮ use Pell’s equation x2 − dy2 = 1 ◮ solutions (xn, yn) satisfy xn + yn

√ d = (x1 + y1 √ d)n

◮ xn, yn grow exponentially with n 16 / 49

Subsection 2 Polynomial systems in continuous variables

17 / 49

RLF: Relevance to MINLP

◮ RLF ⊇ [poly NLP feasibility]

• bvious

◮ RLF ⊆ [poly NLP optimality]

∀i ≤ m gi(x) = 0

is feasible

⇔

• α∗ = min u2

(1 − u2)

i≤m

(gi(x))2 =

• = 0

◮ if α∗ > 1 get −1 = 0 (contradiction) ◮ if α∗ > 0 get g(x) = 0 (infeasible) ◮ if α∗ = 0 get g(x) = 0 (feasible) 18 / 49

Goal 1: [poly NLP feas.] is decidable

◮ RLF contains [poly NLP feasibility] ◮ RLF decidable ◮ ⇒ [poly NLP feasibility] is decidable

19 / 49

Goal 2: [poly NLP] is decidable

◮ [poly NLP]:

we need to tell optimality and unboundedness apart

◮ RLF also includes universal quantifiers ◮ P ≡ min{f(x) g(x) ≤ 0} unbounded:

∀y f(x) = y ∧ g(x) ≤ 0

◮ P exists:

∃y f(x) = y ∧ g(x) ≤ 0 ∧ ¬unbounded(P)

◮ P infeasible:

¬∃y f(x) = y ∧ g(x) ≤ 0

20 / 49

Decidability of RLF

◮ RLF sentence [p(x) R 0]: p ∈ R[x], x ∈ Rn, R ∈ {=, <}

p(x) < 0 ← → p(x) − y = 0 ∧ y < 0 p(x) ≤ 0 ← → p(x) − y2 = 0 ∀i ≤ m pi(x) = 0 ← →

• i≤m

(pi(x))2 = 0

• pen constraints y < 0 invalid in MP, need not bother

◮ ∃? alg. for deciding if any p(x) = 0 solves R or not? ◮ RLF is decidable by quantifier elimination [Tarski 1948] ◮ Quantifier elimination:

◮ constructs solution sets (YES) or derives contradictions (NO) ◮ ⇒ RLF is complete, too ◮ think of Fourier-Motzkin elimination for linear RLF 21 / 49

Example: quantifier elimination in R1

◮ DLO (Dense Linear Order):

0, 1, ¬, ∨, ∧, ∃, ∀, <, =, vars; quantifiers over R

◮ Reduce to form ∃x i≤m

qi

where all qi’s have form x = v, x < v, or x > v (v var/const)

◮ x = x can be removed from conjunction ◮ x < x : sentence is false (and there’s a proof!) ◮ if v x differ, rewrite ∃x x = v ∧ r(x, v) ↔ ∃x r(x, x)

back to previous case

◮ remaining case: qi is

• i

(ui < x) ∧

• i

(x < vi) rewrite as

i ui < vi

◮ get [∃x¯

q where ¯ q does not involve x] or contradiction

◮ repeat until only constants in R left

get proof of YES and proof of NO

DLO is decidable and complete

22 / 49

Rationals

◮ [Robinson 1949]:

RT (1st ord. theory over Q) is undecidable

◮ [Pheidas 2000]: existential theory of Q (ERT) is open

can we decide wether p(x) = 0 has solutions in Q? Boh!

◮ [Matyiasevich 1993]:

◮ equivalence between DEH and ERT ◮ DEH = [DE restricted to homogeneous polynomials] ◮ but we don’t know whether DEH is decidable

Note that Diophantus solved DE in positive rationals

23 / 49

Subsection 3 Digressions

24 / 49

Proof complexity bounds with UDEs

The following surprising bound is due to [Jones 1982] For any axiomatizable theory T in PA1 and any sentence p ∈ T , if p has a proof in T , then it has a proof consisting of 100 additions and multiplications of integers

◮ Gödel numbering: T −

→ r.e. subset of N

◮ Search for proofs ←

→ search for DE solutions solution encodes whole proof

◮ ∃ UDE the evaluation of which takes 100 +, × operations ◮ Any solution of the UDE can be verified in at most 100

• perations

25 / 49

Common misconception 1

“Since N is contained in R, how is it possible that RLF is decidable but DE (= RLF ∩ N) is not?”

After all, if a problem contains a hard subproblem, it’s hard by inclusion, right?

◮ Can you express DE p(x) = 0 ∧ x ∈ N in RLF?

◮ p(x) = 0 belongs to both DE and RLF, OK ◮ “x ∈ N” in RLF?

⇐ find poly q(x) s.t. ∃x q(x) = 0 iff x ∈ Nn

◮ q(x) = x(x − 1) · · · (x − a) only good for {0, 1, . . . , a}

q(x) =

i∈ω

(x − i) is ∞ly long, invalid

◮ IMPOSSIBLE!

if it were possible, DE would be decidable, contradiction

26 / 49

Common misconception 2

“Decidability implies completeness”

For any sentence f we can decide whether ∃ proof or not, so either f or ¬f must be provable, right?

◮ Algebraically Closed Fields (ACF):

field axioms + “every polynomial splits” schema

◮ ACF is decidable by quantifier elimination ◮ Cp ≡ [ j≤p

1 = 0] (for any prime p): independent of ACF

◮ ∃ fields of every prime characteristic p ◮ each different field satisfies Cp and negates Cq for q = p

◮ ACF is incomplete: neither Cp nor ¬Cp is provable in ACF ◮ Decision algorithm for ACF returns NO for both

⇒ theories can be decidable and incomplete

27 / 49

Subsection 4 Application to MINLP

28 / 49

MIQCP is undecidable

◮ [Jeroslow 1973]: MIQCP:

min c⊤x ∀i ≤ m x⊤Qix + ai⊤x + bi ≥ x ∈ Zn    (†) is undecidable

Proof:

◮ Let U(a, x) = 0 be an UDE ◮ P(a) ≡ min{u ∈ N | (1 − u)U(a, x) = 0 ∧ x ∈ Zn}

P(a) describes an undecidable problem

◮ Linearize every product xixj by yij and add yij = xixj

until only degree 1 and 2 left

◮ Obtain MIQCP (†) 29 / 49

Some MIQCQPs are decidable

◮ If each Qi is diagonal PSD, decidable [Witzgall 1963] ◮ If x are bounded in [xL, xU] ∩ Zn, decidable

can express x ∈ {xL, xL + 1, . . . , xU} by polynomial ∀i ≤ m

• xL

i ≤i≤xU i

(x − i) = 0 turn into poly system in R (in RLF, decidable)

◮ ⇒ Bounded (vars) easier than unbounded (for Z) ◮ [MIQP decision vers.] is decidable x⊤Qx + c⊤x ≤ γ Ax ≥ b ∀j ∈ Z xj ∈ Z    (in NP [Del Pia et al. 2014])

30 / 49

NLP is undecidable

We can’t represent unbounded subsets of N by polynomials But we can if we allow some transcendental functions x ∈ Z ← → sin(πx) = 0

◮ Constrained NLP is undecidable:

min{0 | U(a, x) = 0 ∧ ∀j ≤ n sin(πxj) = 0}

◮ Even with just one nonlinear constraint:

min{0, | (U(a, x))2 +

• j≤n

(sin(πxj))2 = 0}

◮ Unconstrained NLP is undecidable:

min(U(a, x))2 +

• j≤n

(sin(πxj))2

◮ Box-constrained NLP is undecidable (boundedness doesn’t help):

min{(U(a, tan x1, . . . , tan xn))2+

• j≤n

(sin(π tan xj))2 | −π 2 ≤ x ≤ π 2 }

31 / 49

Some NLPs are decidable

◮ All polynomial NLPs are decidable

by decidability of RLF

◮ Quadratic Programming (QP) is decidable over Q

min x⊤Qx + c⊤x Ax ≥ b

• (P)

◮ Bricks of the proof

◮ if Q is PSD, P ∈ Q

• 1. replace components using active constraints at opt
• 2. work out KKT conditions, they are linear in rational coefficients
• 3. ⇒ solution is rational

◮ ∃ polytime IPM for solving P [Renegar&Shub 1992] ◮ unbounded case treated in [Vavasis 1990]

◮ ⇒ [QP decision version] is in NP

⇒ QP is decidable over Q

32 / 49

Section 3 Hardness

33 / 49

MILP is NP-hard

◮ sat is NP-hard by Cook’s theorem

Reduce from sat in CNF

• i≤m
• j∈Ci

ℓj where ℓj is either xj or ¯ xj ≡ ¬xj

◮ Polynomial reduction ρ

sat xj ¯ xj ∨ ∧ MILP xj 1 − xj + ≥ 1

◮ E.g. ρ maps (x1 ∨ x2) ∧ (¯

x2 ∨ x3) to MILP min{0 | x1 + x2 ≥ 1 ∧ x3 − x2 ≥ 0 ∧ x ∈ {0, 1}3}

◮ sat is YES iff MILP is feasible trivially shows MINLP is NP-hard

34 / 49

Focus on continuous problems

◮ Hardness of MINLP due to integer variables

question hardness without integrality constrs.

◮ Interesting issue: hardness in fixed dimension

◮ [ax2 + by − c = 0 ∧ (x, y) ∈ N2] is NP-complete

[Manders& Adleman 1978], from 3sat → knapsack → ·

◮ ⇒ [x2 ≡ a (mod b)] is NP-complete ◮ ⇒ min{(ax2 + by − c)2 | x, y ∈ N} is NP-hard 35 / 49

Subsection 1 Quadratic Programming

36 / 49

QP is NP-hard

◮ By reduction from SAT, let σ be an instance ◮ ˆ

ρ(σ, x) ≥ 1: linear constraints from sat → MILP reduction

◮ Consider QP

min f(x) =

j≤n

xj(1 − xj) ˆ ρ(σ, x) ≥ 1 0 ≤ x ≤ 1      (†)

◮ Claim: σ is YES iff val(†) = 0 ◮ Proof:

◮ assume σ YES with soln. x∗, then x∗ ∈ {0, 1}n, hence

f(x∗) = 0, since f(x) ≥ 0 for all x, val(†) = 0

◮ assume σ NO, suppose val(†) = 0, then (†) feasible

with soln. x′, since f(x′) = 0 then x′ ∈ {0, 1}, feasible in sat hence σ is YES, contradiction

37 / 49

Box-constrained QP is NP-hard

◮ Add surplus vars v to sat→MILP constraints:

ˆ ρ(σ, x) − 1 − v = 0 (denote by ∀i ≤ m (a⊤

i x − bi − vi = 0))

◮ Now sum them on the objective

min

• j≤n

xj(1 − xj) +

i≤m

(a⊤

i x − bi − vi)2

0 ≤ x ≤ 1, v ≥ 0

• ◮ Issue: v not bounded above

◮ Reduce from 3sat, get ≤ 3 literals per clause

⇒ can consider 0 ≤ v ≤ 2

38 / 49

cQKP is NP-hard

◮ continuous Quadratic Knapsack Problem (cQKP)

min f(x) = x⊤Qx + c⊤x

• j≤n

ajxj = γ x ∈ [0, 1]n,     

◮ Reduction from subset-sum given list a ∈ Qn and γ, is there J ⊆ {1, . . . , n} s.t.

j∈J

aj = γ? reduce to f(x) =

j xj(1 − xj)

◮ σ is a YES instance of subset-sum

◮ let x∗ j = 1 iff j ∈ J, x∗ j = 0 otherwise ◮ feasible by construction ◮ f is non-negative on [0, 1]n and f(x∗) = 0: optimum

◮ σ is a NO instance of subset-sum

◮ suppose opt(cQKP) = x∗ s.t. f(x∗) = 0 ◮ then x∗ ∈ {0, 1}n because f(x∗) = 0 ◮ feasibility of x∗ → supp(x∗) solves σ, contradiction, hence f(x∗) > 0 39 / 49

One negative eigenvalue: hard

◮ Convex QP is in P

need some negative eigenvalue for hardness: how many?

◮ QP with 1 Negative Eigenvalue (QP1NE) is NP-hard

reduction from k-clique instance (G = (V, E), k)

◮ Reduce to the following QP: min z − w2 (1)

• j∈V

4jxj = w (2)

• j∈V

42jxj + 2

• i<j

4i+jyij = z (3) ∀i < j ∈ V max(0, xi + xj − 1) ≤ yij (4) ∀{i, j} ∈ E xi + xj ≤ 1 (5)

• j∈V

xj = k (6) 0 ≤ x ≤ 1. (7) ◮ z = w2 iff yij = xixj, integrality nontrivial, Eq. (5)-(7) encode k-clique

40 / 49

QP on a simplex is NP-hard

min f(x) = x⊤Qx + c⊤x

• j≤n

xj = 1 ∀j ≤ n xj ≥     

◮ Reduce max clique to subclass f(x) = −

{i,j}∈E

xixj Motzkin-Straus formulation (MSF)

◮ Theorem [Motzkin& Straus 1964] Let C be the maximum clique of the instance G = (V, E) of max clique

∃x∗ ∈ opt (MSF) f ∗ = f(x∗) = 1

2

• 1 −

1 ω(G)

• ∀j ∈ V

x∗

j =

• 1

ω(G)

if j ∈ C

• therwise

41 / 49

Proof of the Motzkin-Straus theorem

x∗ = opt( max

x∈[0,1]n

• j xj =1
• ij∈E

xixj) s.t. |C = {j ∈ V |; x∗

j > 0}| smallest (‡)

• 1. C is a clique

◮ Suppose 1, 2 ∈ C but {1, 2} ∈ E[C], then x∗

1, x∗ 2 > 0, can perturb by small

ǫ ∈ [−x∗

1, x∗ 2], get xǫ = (x∗ 1 + ǫ, x∗ 2 − ǫ, . . .), feasible w.r.t. simplex and bounds

◮ {1, 2} ∈ E ⇒ x1x2 does not appear in f(x) ⇒ f(xǫ) depends linearly on ǫ; by

• ptimality of x∗, f achieves max for ǫ = 0, in interior of its range ⇒ f(ǫ)

constant ◮ set ǫ = −x∗

1 or = x∗ 2 yields global optima with more zero components than x∗,

against assumption (‡), hence {1, 2} ∈ E[C], by relabeling C is a clique

42 / 49

Proof of the Motzkin-Straus theorem

x∗ = opt( max

x∈[0,1]n

• j xj =1
• ij∈E

xixj) s.t. |C = {j ∈ V |; x∗

j > 0}| smallest (‡)

• 2. |C| = ω(G)

◮ square simplex constraint

j xj = 1, get

• j∈V

x2

j + 2

• i<j∈V

xixj = 1 ◮ by construction x∗

j = 0 for j ∈ C ⇒

ψ(x∗) =

• j∈C

(x∗

j )2 + 2

• i<j∈C

x∗

j x∗ j =

• j∈C

(x∗

j )2 + 2f(x∗) = 1

◮ ψ(x) = 1 for all feasible x, so f(x) achieves maximum when

j∈C(x∗ j )2 is

minimum, i.e. x∗

j = 1 |C| for all j ∈ C

◮ again by simplex constraint f(x∗) = 1 −

• j∈C

(x∗

j )2 = 1 − |C|

1 |C|2 ≤ 1 − 1 ω(G) so f(x∗) attains maximum 1 − 1/ω(G) when |C| = ω(G)

43 / 49

Bilinear Programming

◮ Separable BPP is NP-hard

min

• j≤n

xjyj Ax ≥ b By ≥ c      (†)

◮ Reduction from 2ls [Bennett & Mangasarian 1993]

geometric piecewise-linear point partition problem

◮ (apparently still) Open research problem: Settled in [Matsui 1996]

min (c⊤x + γ)(d⊤x + δ) Ax ≥ b

• (‡)

Is (‡) NP-hard? [Vavasis 1995]

◮ Thanks to S. Iwata for updating me on the status of this problem!

44 / 49

Subsection 2 Tractable cases of QP

45 / 49

Unconstrained QP

min x⊤Qx

◮ Check constant Hessian

◮ If PSD, attains minimum ◮ If ∃ 1 negative eigenvalue, unbounded direction

◮ Always feasible

46 / 49

Trust Region Subproblems

◮ ℓ2-TRS

min{x⊤Qx + c⊤x | x2 ≤ 1} can be solved in polynomial time

◮ Many variants [Bienstock & Michalka, 2013]

◮ +1 linear constraint a⊤x ≤ b ◮ +1 ℓ2 norm constr. x − x02 ≤ r0 ◮ +2 linear constraints a⊤

i x ≤ bi (i ≤ 2)

s.t. (a⊤

1 x − b1)(a⊤ 2 x − b2) = 0

◮ +2 linear affinely parallel constraints ◮ +m linear constraints

(no two simultaneously binding in feas. reg.)

◮ . . .

◮ Unfortunately, not much practical use

Applications mostly use ℓ∞ norm, over which QP is hard

...but see [Buchheim et al. 2013 & 2015] 47 / 49

Piggy-backing on [HKLW 2010]

◮ Decidability of cvx constr. minimization in Zn

⇐ turns out it’s bounded

◮ Fixed Param. Tractable (FPT) cases of hard MINLP ◮ Integer convex maximization

◮ fixed number of objective fun. arguments ◮ some fixed data (e.g. constraint matrix) ◮ fixed number of variables

◮ Integer quasicvx constrained minimization

fixed number of variables

◮ ...and more ◮ QP over Z2 is in P [Del Pia & Weismantel 2014]

48 / 49

The end

49 / 49