and sum of squares polynomials Georgina Hall INSEAD, Decision - - PowerPoint PPT Presentation

Shape-constrained regression and sum of squares polynomials

Georgina Hall INSEAD, Decision Sciences Joint work with Mihaela Curmei (Berkeley, EECS)

1

Shape-constrained regression (1/2)

Data: 𝑌𝑗, 𝑍

𝑗 𝑗=1,…,𝑛 where 𝑌𝑗 ∈ 𝐶 ⊂ ℝ𝑜 (𝐶 is a box) and 𝑍 𝑗 ∈ ℝ

Goal: Fit a polynomial ො 𝑕𝑛,𝑒 of degree 𝑒 to the data that minimizes σ𝑗=1…𝑛 𝑍

𝑗 − 𝑕(𝑌𝑗) 2

and that has certain constraints on its shape.

2

Shape-constrained regression (2/2)

3

Convexity over B Monotonicity over B Lipschitz with constant 𝑳 For a full-dimensional box B and a twice-continuously differentiable function 𝑔: 𝑔 is convex over 𝐶 ⇔ ∇2𝑔 𝑦 ≽ 0, ∀𝑦 ∈ 𝐶 Example:

• Price of a car as a function
• f age

For a continuously differentiable function 𝑔: 𝑔 is increasing (resp. decreasing) in component 𝑦𝑗 ⇔

𝜖𝑔 𝑦 𝜖𝑦𝑗 ≥ 0 𝑠𝑓𝑡𝑞. ≤ 0 , ∀𝑦 ∈ 𝐶

Example:

• Demand as a function of

price For any function 𝑔 and a fixed scalar 𝐿 > 0: 𝑔 is Lipschitz with constant K ⇔ 𝑔 𝑦 − 𝑔 𝑧 ≤ 𝐿 𝑦 − 𝑧 , ∀𝑦, 𝑧 ∈ 𝐶 Use as a regularizer: stops 𝑔 from growing too steeply Focus on convex regression here.

Convex regression – possible candidate

4

A candidate for our regressor: ො 𝑕𝑛,𝑒 𝑦 ≔ arg min

𝑕 σ𝑗=1..𝑛 𝑍 𝑗 − 𝑕 𝑌𝑗 2

s.t. 𝑕 is a polynomial of degree 𝑒 ∇2𝑕 𝑦 ≽ 0, ∀𝑦 ∈ 𝐶 But… Theorem [Ahmadi, H.]: It is (strongly) NP-hard to test whether a polynomial 𝑞 of degree ≥ 3 is convex over a box 𝐶. (Reduction from problem of testing whether a matrix whose entries are affine polynomials in 𝑦 is positive semidefinite for all 𝑦 in 𝐶.)

A detour via sum of squares (1/5)

5

• If we can find a way of imposing that a polynomial be nonnegative,

then we are in business!

• Unfortunately, hard to test whether a polynomial 𝑞 is nonnegative

for degree of 𝑞 ≥ 4.

• What to do?

𝑕(𝑦) convex

• ver B

⇔ 𝑧𝑈∇2𝑕(𝑦)𝑧 ≥ 0, ∀𝑦 ∈ 𝐶, ∀𝑧 ∈ ℝ𝑜 ∇2𝑕(𝑦) ≽ 0, ∀𝑦 ∈ 𝐶 ⇔ Polynomial in 𝒚 and 𝒛

A detour via sum of squares (2/5)

6

Idea Find a property that implies nonnegativity but that is easy to test. = sum of squares (sos) Definition: A polynomial 𝑞 is sos if it can be written as 𝑞 𝑦 = σ𝑗 𝑟𝑗 𝑦 2.

Sos polynomials Nonnegative polynomials

① ② ① Yes! Even equal sometimes : 𝑜 = 1, 𝑒 = 2, 𝑜, 𝑒 = (3,4) [Hilbert] What about ②? Also yes! Let’s see why.

A detour via sum of squares (3/5)

A polynomial 𝑞(𝑦) of degree 2d is sos if and only if ∃𝑅 ≽ 0 such that where 𝑨 = 1, 𝑦1, … , 𝑦𝑜, 𝑦1𝑦2, … , 𝑦𝑜

𝑒 𝑈 is the vector of monomials of degree up to 𝑒.

𝑞 𝑦 = 𝑦1

4 − 6𝑦1 3𝑦2 + 2𝑦1 3𝑦3 + 6𝑦1 2𝑦3 2 + 9𝑦1 2𝑦2 2 − 6𝑦1 2𝑦2𝑦3 − 14𝑦1𝑦2𝑦3 2 + 4𝑦1𝑦3 3

+5𝑦3

4 − 7𝑦2 2𝑦3 2 + 16𝑦2 4

= 𝑦1

2 − 3𝑦1𝑦2 + 𝑦1𝑦3 + 2𝑦3 2 2 + 𝑦1𝑦3 − 𝑦2𝑦3 2 + 4𝑦2 2 − 𝑦3 2 2

Ex:

= 𝒚𝟐

𝟑

𝒚𝟐𝒚𝟑 𝒚𝟑

𝟑

𝒚𝟐𝒚𝟒 𝒚𝟑𝒚𝟒 𝒚𝟒

𝟑 T

𝟐 −𝟒 𝟏 𝟐 𝟏 𝟑 −𝟒 𝟘 𝟏 −𝟒 𝟏 −𝟕 𝟏 𝟏 𝟐𝟕 𝟏 𝟏 −𝟓 𝟐 −𝟒 𝟏 𝟑 −𝟐 𝟑 𝟏 𝟏 𝟏 −𝟐 𝟐 𝟏 𝟑 −𝟕 𝟓 𝟑 𝟏 𝟔 𝒚𝟐

𝟑

𝒚𝟐𝒚𝟑 𝒚𝟑

𝟑

𝒚𝟐𝒚𝟒 𝒚𝟑𝒚𝟒 𝒚𝟒

𝟑

𝑞 𝑦 = 𝒜 𝒚 𝑼𝑹𝒜(𝒚)

7

A detour via sum of squares (4/5)

• Testing if a polynomial is sos is a semidefinite program (SDP).
• In fact, even optimizing over the set of sos polynomials (of fixed

degree) is an SDP.

8

min

𝑅 0

s.t. 𝑞 𝑦 = 𝑨 𝑦 𝑈𝑅𝑨 𝑦 ∀𝑦 𝑅 ≽ 0

Linear equations involving coefficients of 𝑞 and entries of 𝑅

min

c1,c2 𝑑1 + 𝑑2

𝑡. 𝑢. 𝑑1 − 3𝑑2 = 4 𝑑1𝑦1

2 − 2𝑑2𝑦1𝑦2 + 5𝑦2 4 sos

min

c1,c2,𝑅 𝑑1 + 𝑑2

𝑡. 𝑢. 𝑑1 − 3𝑑2 = 4 𝑑1𝑦1

2 − 2𝑑2𝑦1𝑦2 + 5𝑦2 4 = 𝑨 𝑦 𝑈𝑅𝑨 𝑦

𝑅 ≽ 0

A detour via sum of squares (5/5)

• Slight subtlety here:
• How to impose nonnegativity over a set?

9

𝑕(𝑦) convex

• ver B

⇔ 𝑧𝑈∇2𝑕(𝑦)𝑧 ≥ 0, ∀𝒚 ∈ 𝑪, ∀𝑧 ∈ ℝ𝑜 ∇2𝑕(𝑦) ≽ 0, ∀𝑦 ∈ 𝐶 ⇔ Theorem [Putinar ’93]: For a box 𝐶 = 𝑦1, … , 𝑦𝑜 𝑚1 ≤ 𝑦1 ≤ 𝑣1, … , 𝑚𝑜 ≤ 𝑦𝑜 ≤ 𝑣𝑜}, we write instead: 𝑧𝑈∇2𝑕 𝑦 𝑧 = 𝜏0 𝑦, 𝑧 + 𝜏1 𝑦, 𝑧 𝑣1 − 𝑦1 𝑦1 − 𝑚1 + ⋯ + 𝜏𝑜 𝑦, 𝑧 𝑣𝑜 − 𝑦𝑜 (𝑦𝑜 − 𝑚𝑜) where 𝜏0 𝑦, 𝑧 , 𝜏1 𝑦, 𝑧 , … 𝜏𝑜(𝑦, 𝑧) are sos polynomials in 𝑦 and 𝑧

Convex regression – a new candidate

10

A new candidate for the regressor: ෤ 𝑕𝑛,𝑒,𝑠 𝑦 ≔ arg min

𝑕,𝜏0,…𝜏𝑜 σ𝑗=1..𝑛 𝑍 𝑗 − 𝑕 𝑌𝑗 2

s.t. 𝑕 is a polynomial of degree 𝑒 𝑧𝑈∇2𝑕 𝑦 𝑧 = 𝜏0 𝑦, 𝑧 + ⋯ + 𝜏𝑜 𝑦, 𝑧 𝑣𝑜 − 𝑦𝑜 𝑦𝑜 − 𝑚𝑜 𝜏0 𝑦, 𝑧 , 𝜏1 𝑦, 𝑧 , … , 𝜏𝑜(𝑦, 𝑧) are sos of degree 𝑠 in 𝑦 (and 2 in 𝑧)

• When 𝑠 is fixed, this is a semidefinite program to solve.
• As 𝑠 → ∞, we recover ො

𝑕𝑛,𝑒.

Comparison with existing methods

11

Our method Existing method

[Lim & Glynn, Seijo & Sen]

• Semidefinite program to obtain estimator
• Number of datapoints does not impact size
• f semidefinite program
• Size of the semidefinite program scales

polynomially in number of features.

• Obtaining a prediction: evaluation of our

polynomial estimator

• Smooth estimator
• Can be combined with monotonocity

constraints and Lipschitz constraints

• Quadratic program to obtain estimator
• Number of variables (resp. constraints) scales

linearly (resp. quadratically) with number of datapoints

• Obtaining a prediction: requires solving a linear

program

• Piecewise affine estimator (can be smoothed

see [Mazumder et al.])

• Can be combined with monotonicity constraints

(see [Lim & Glynn]) and Lipschitz constraints (see [Mazumder et al.])

Consistency of ෤ 𝑕𝑛,𝑒,𝑠 (1/4)

• Estimator of [Lim & Glynn, Seijo & Sen] is shown to be consistent.

What about ours?

12

Theorem [Curmei, H.] The regressor ෤ 𝑕𝑛,𝑒,𝑠 is a consistent estimator of 𝑔 over any compact 𝐷 ⊂ 𝐶, i.e., sup

𝑦∈𝐷

෤ 𝑕𝑛,𝑒,𝑠(𝑦) − 𝑔 𝑦 → 0 a.s., when 𝑒, 𝑛, 𝑠 → ∞

For 𝒀𝒋 𝑌𝑗 are iid, with support 𝐶, and 𝐹 𝑌𝑗

2 < ∞

For 𝒁𝒋 𝑍

𝑗 = 𝑔 𝑌𝑗 + 𝜗𝑗 for 𝑗 = 1, … , 𝑛

with 𝐹 𝜗𝑗 𝑌𝑗 = 0 a.s. and 𝐹 𝜗𝑗

2 < ∞

For 𝒈 𝑔 is twice continuously differentiable and convex over 𝐶

Assumptions on the data:

Consistency of ො 𝑕𝑛,𝑒 (2/4)

Proof ideas: inspired by [Lim and Glynn, OR’12]

• 1. Write

𝑔 𝑦 − ෤ 𝑕𝑛,𝑒,𝑠(𝑦) ≤ 𝑔 𝑦 − ො 𝑕𝑛,𝑒 𝑦 + ො 𝑕𝑛,𝑒 𝑦 − ෤ 𝑕𝑛,𝑒,𝑠 𝑦

• 2. Introduce a polynomial approximation of 𝑔: for any 𝜗 > 0, ∃ 𝑒 and

a convex polynomial 𝑕𝑒 of degree 𝑒 such that sup

𝑦∈𝐷

𝑔 𝑦 − 𝑕𝑒 𝑦 < 𝜗

13

Can show sup

𝑦∈𝐷

ො 𝑕𝑛,𝑒 𝑦 − ෤ 𝑕𝑛,𝑒,𝑠 𝑦 → 0 when 𝑠 → ∞

Consistency of ො 𝑕𝑛,𝑒 (3/4)

• 3. For 𝑦 ∈ 𝐷 and "𝑌𝑗 close to 𝑦":

𝑔 𝑦 − ො 𝑕𝑛,𝑒 𝑦 ≤ 𝑔 𝑦 − 𝑕𝑒 𝑦 + 𝑕𝑒 𝑦 − 𝑕𝑒 𝑌𝑗 + 𝑕𝑒 𝑌𝑗 − ො 𝑕𝑛,𝑒 𝑌𝑗 + | ො 𝑕𝑛,𝑒 𝑌𝑗 − ො 𝑕𝑛,𝑒 𝑦 | Remains to show that

1 𝑛 σ𝑗=1..𝑛 𝑕𝑒 𝑌𝑗 − ො

𝑕𝑛,𝑒 𝑌𝑗

2 → 0 a.s. when 𝑛 → ∞

14

Upper bound with 𝜗

Show that 𝑕𝑒 is Lipschitz (use convexity of 𝑕𝑒 over B) Show that ො 𝑕𝑛,𝑒 is Lipschitz (uniformly in 𝑛) (bound | ො 𝑕𝑛,𝑒| over 𝐷 unif. in 𝑛 and use convexity) Upper bound this (algebra) by

1 𝑛 σ𝑗=1..𝑛 𝑕𝑒 𝑌𝑗 − ො

𝑕𝑛,𝑒 𝑌𝑗

2

Consistency of ො 𝑕𝑛,𝑒 (4/4)

• 3. Show that

1 𝑛 σ𝑗=1..𝑛 𝑕𝑒 𝑌𝑗 − ො

𝑕𝑛,𝑒 𝑌𝑗

2 → 0 a.s. when 𝑛 → ∞

• Use the fact that ො

𝑕𝑛,𝑒 is a minimizer of σ𝑗 𝑍

𝑗 − 𝑕 𝑌𝑗 2 to obtain 1 𝑛 σ𝑗=1..𝑛 𝑕𝑒 𝑌𝑗 − ො

𝑕𝑛,𝑒 𝑌𝑗

2 ≤ 2 𝑛 σ𝑗 𝑍 𝑗 − 𝑕𝑒 𝑌𝑗

⋅ ( ො 𝑕𝑛,𝑒 𝑌𝑗 − 𝑕𝑒 𝑌𝑗 )

• Can’t use SLLN because ො

𝑕𝑛,𝑒 is a polynomial that depends on 𝑌𝑗 and 𝑍

𝑗

• Idea: approximate ෝ

𝒉𝒏,𝒆 by a deterministic function which is bounded over 𝐷.

• Show that ො

𝑕𝑛,𝑒 belongs (for large enough 𝑛) to a compact set whose elements are bounded over 𝐷

• Construct 𝜗-net of this set
• Replace ො

𝑕𝑛,𝑒 by an element of this set which is 𝜗-close and bounded over 𝐷

• Use SLLN now with 𝒁𝒋 − 𝒉𝒆 𝒀𝒋 ≈ 𝝑𝒋

15

Experiments: synthetic data (1/2)

16

𝑍

𝑗 = 𝑔 𝑌𝑗 + 𝝑 ⋅ 𝜏 ത

𝑍 ⋅ 𝜉𝑗

• 𝑔 is a convex and monotonous

function

• 𝜉𝑗 ∼ 𝑂 0,1
• 𝜏(ത

𝑍) is the (empirical) standard deviation of (𝑔 𝑌1 , … , 𝑔 𝑌𝑜 )

Experiments: synthetic data (2/2)

17

Experiments: production functions (1/3)

• In economics, production output (Out) is a function of labor (L),

capital (K) and intermediate goods (I).

• Out is assumed to be concave in L,K,I (diminishing returns)
• Out is assumed to be monotonous in L,K,I
• K,L,I and Out available in the KLEMS dataset for 65 different

industries

18

Experiments: production functions (2/3)

• How are production functions fitted to data in economics generally?

19

Cobb Douglas functions 𝑃𝑣𝑢 = 𝑏 ⋅ 𝐿𝑐𝑀𝑑𝐽𝑒 𝑏 > 0, b, c, d > 0 and b + c + 𝑒 ≤ 1 ⇒ concave +monotone Fit in log-space: linear regression

Experiments: production functions (3/3)

20

Outperforms Cobb-Douglas on 50/65 industries.

Main messages

• Shape-constrained regression: convexity, monotonicity, Lipschitz
• Discussed convex regression (results also hold for other shape

constraints)

• Proposed estimator: polynomial, obtained via SDP, consistent
• Numerical experiments: synthetic + production functions in economics

21

Thank you for listening

Questions? Want to know more? https://sites.google.com/view/georgina-hall

22