COMPUTATIONAL ASPECTS OF SELECTION OF EXPERIMENTS Yining Wang - - PowerPoint PPT Presentation

COMPUTATIONAL ASPECTS OF SELECTION OF EXPERIMENTS

Yining Wang

Machine Learning Department, Carnegie Mellon University Georgia Institute of Technology, Atlanta GA, USA

Arxiv:1711.05174

Joint work with Zeyuan Allen-Zhu, Yuanzhi Li and Aarti Singh

MOTIVATING APPLICATION

Worst-case structural analysis

• Maximum stress resulting from worst-case external forces
• Example application: lightweight structural design in automated fiber

process

MOTIVATING APPLICATION

Worst-case structural analysis

• Challenges: Finite Element Analysis (FEA) for every external force

locations would be computationally too expensive

Justification for single, normal, compressive load can be found in Ulu et al.’17, based on Rockafellar’s Theorem

MOTIVATING APPLICATION

Worst-case structural analysis

• Idea: Sample a few “representative” force locations and build a predictive

model for the rest locations

• Challenge: How to determine the “best” representative locations

~4000 nodes 200 nodes

PROBLEM FORMULATION

Linear regression model: Experiment selection:

yi = hxi, θ0i + εi

• max. stress response

top e-vec of surface Laplacian unknown regression model modeling error

x1 x2

dimension p Force location 1 Force location 2 Force location n

xn

X ∈ Rn×p

dimension p selected location 1 selected location 2 selected location k

XS ∈ Rk×p

~4000 nodes 200 nodes y1 y2 yk

PROBLEM FORMULATION

Linear regression model: Ordinary Least Squares:

• By CLT:

yi = hxi, θ0i + εi

b θ = (P

i2S xix> i )1(P i2S yixi)

√n(b θ − θ0)

d

→ N(0, (P

i2S xix> i )1)

(scaled) sample covariance, Fisher’s Information

Optimal experimental design Find subset , so as to minimize S ⊆ [n] |S| ≤ k

f ⇣P

j2S xjx> j

⌘

“optimality criteria”

PROBLEM FORMULATION

Predictive model:

yi = hxi, θ0i + εi

Optimal experimental design Find subset , so as to minimize S ⊆ [n] |S| ≤ k

f ⇣P

j2S xjx> j

⌘

Example: A-optimality D-optimality E-optimality V-optimality ….

fA(Σ) = tr(Σ−1)/p

fD(Σ) = det(Σ)−1/p fE(Σ) = 1/kΣ−1kop

MSE Ekˆ

θ θ0k2

2

“scale invariant” “optimality criteria”

PROBLEM FORMULATION

Predictive model:

yi = hxi, θ0i + εi

Optimal experimental design Find subset , so as to minimize S ⊆ [n] |S| ≤ k

f ⇣P

j2S xjx> j

⌘ Objective: efficient approximation algorithms f ⇣P

j2b S xjx> j

⌘ ≤ C(n, p) · min|S|k f ⇣P

j2S xjx> j

⌘

“approximation ratio”

EXISTING RESULTS

Existing positive results

• O(1) approximation for D-optimality
• O(n/k) approximation for A-optimality

Existing negative results

• NP-Hard for exact optimization of D/E-optimality
• NP-Hard for (1+𝜁) approximation for D-optimality when k=p

(Nikolov & Singh, STOC’15) (Avron & Boutsidis, SIMAX’13) (Summa et al., SODA’15) (Cerny & Hladik, Comput. Optim. Appl.’12)

Applicable to only one or two criteria f

REGULAR CRITERIA

Optimal experimental design Find subset , so as to minimize S ⊆ [n] |S| ≤ k

f ⇣P

j2S xjx> j

⌘

“Regular” criteria:

(A1) Convexity: f (or its surrogate) is convex; (A2) Monotonicity: (A3) Reciprocal linearity:

A B = ) f(A) f(B)

f(tA) = t−1f(A)

All popular optimality criteria are “regular”, e.g., A/D/E/V/G-optimality

OUR RESULT

• Remark 1: Concurrent to or after our works, 1+𝜁 approx. for D/A-
• ptimality are obtained under condition
• Remark 2: The condition is tight for E-optimality and

continuous relaxation type methods.

• Theorem. For all regular criteria f, there exists a polynomial

time (1+𝜁) approximation algorithm provided that

k = Ω(p/ε2)

k = Ω(p/ε2) k = Ω(p/ε + 1/ε2)

(Singh & Xie, SODA’18; Nikolov et al., arXiv’18) (Nikolov et al., arXiv’18) #. of design subsets #. of variables / dimension

ALGORITHMIC FRAMEWORK

Continuous relaxation of the discrete problem Whitening of candidate design points Regret minimization characterization of least eigenvalues Greedy swapping based on FTRL potential functions

ALGORITHMIC FRAMEWORK

Continuous relaxation of the discrete problem Whitening of candidate design points Regret minimization characterization of least eigenvalues Greedy swapping based on FTRL potential functions

CONTINUOUS RELAXATION

• Equivalent formulation:
• Convex! Can be solved using classical methods (e.g., projected gradient/

mirror descent)

Optimal experimental design Find subset , so as to minimize S ⊆ [n] |S| ≤ k

f ⇣P

j2S xjx> j

⌘

min

s1,··· ,sn f

n X

i=1

sixix>

i

! s.t.

n

X

i=1

si ≤ k, si ∈ {0, 1}

Relaxation: 0 ≤ si ≤ 1

CONTINUOUS RELAXATION

• Equivalent formulation:
• Question: Round {si} to integer values

Optimal experimental design Find subset , so as to minimize S ⊆ [n] |S| ≤ k

f ⇣P

j2S xjx> j

⌘

min

s1,··· ,sn f

n X

i=1

sixix>

i

! s.t.

n

X

i=1

si ≤ k, si ∈ {0, 1}

Relaxation: 0 ≤ si ≤ 1

ALGORITHMIC FRAMEWORK

Continuous relaxation of the discrete problem Whitening of candidate design points Regret minimization characterization of least eigenvalues Greedy swapping based on FTRL potential functions

WHITENING

• Whitening: where
• By monotonicity of f, the rounding problem is reduced to

Rounding problem. Given optimal continuous solution , round it to such that

π

b s ∈ {0, 1}n, P

i b

si ≤ k

f(P

i b

sixix>

i ) ≤ (1 + O(ε)) · f(P i πixix> i )

e xi = W −1/2xi W = P

i πixix> i

λmin(P

i b

sie xie x>

i ) ≥ 1 − O(ε)

ALGORITHMIC FRAMEWORK

Continuous relaxation of the discrete problem Whitening of candidate design points Regret minimization characterization of least eigenvalues Greedy swapping based on FTRL potential functions

REGRET MINIMIZATION

Matrix linear bandit/online learning:

• At each time t a player picks an action , observes a

reference and suffers loss

• Objective: minimize regret of the action sequences

At ∈ ∆p hAt, Fti

Action space ∆p = {A ⌫ 0, tr(A) = 1}

R(A) :=

T

X

t=1

hFt, Ati inf

U∈∆p T

X

t=1

hFt, ∆i

precisely λmin(

X Ft)

Ft

REGRET MINIMIZATION

Matrix linear bandit/online learning:

• At each time t a player picks an action , observes a

reference and suffers loss

• Objective: minimize regret of the action sequences R(A)
• Follow-The-Regularized-Leader policy:

At ∈ ∆p hAt, Fti

Ft

At = arg min

A∈∆p

( w(A) + α ·

t−1

X

τ=1

hFτ, Ai )

“regularizer”

Example regularizers:

• 1. MWU:
• 2. l1/2-regularization:

w(A) = tr(A>(log A − I))

At = exp ( cI − α

t−1

X

τ=1

Fτ )

w(A) = −2tr(A1/2)

At = cI − α

t−1

X

τ=1

Fτ !−2

REGRET MINIMIZATION

• Proved using classical analysis of regret of FTRL policies
• Ft: swapping of two design points from the pool.

Regret lemma. Suppose . Then

Ft = utu>

t − vtv> t

penalty parameter in FTRL FTRL solution at time t

inf

U2∆p k

X

t=0

hFt, Ui

k

X

t=1

u>

t Atut

1 + 2αu>

t A1/2 t

ut

• v>

t Atvt

1 2αv>

t A1/2 t

vt 2pp α

swapping of two design points

ALGORITHMIC FRAMEWORK

Continuous relaxation of the discrete problem Whitening of candidate design points Regret minimization characterization of least eigenvalues Greedy swapping based on FTRL potential functions

GREEDY SWAPPING

A “potential” function:

Regret lemma. Suppose . Then

Ft = utu>

t − vtv> t

inf

U2∆p k

X

t=0

hFt, Ui

k

X

t=1

u>

t Atut

1 + 2αu>

t A1/2 t

ut

• v>

t Atvt

1 2αv>

t A1/2 t

vt 2pp α

ψ(u, v; A) := u>Au 1 + 2αu>A1/2u − v>Av 1 − 2αv>A1/2v

GREEDY SWAPPING

The “greedy swapping” algorithm:

• Start with an arbitrary set of size k
• At each t, find that maximize
• Greedy swapping:

Regret lemma. Suppose . Then

Ft = utu>

t − vtv> t

inf

U2∆p k

X

t=0

hFt, Ui

k

X

t=1

u>

t Atut

1 + 2αu>

t A1/2 t

ut

• v>

t Atvt

1 2αv>

t A1/2 t

vt 2pp α

S0 ⊆ [n]

it ∈ St−1, jt / ∈ St−1

ψ(xjt, xit; At−1)

St ← St−1 ∪ {jt}\{it}

GREEDY SWAPPING

Proof framework:

• If then the “progress” of each swapping is lower

bounded by until

• Repeat the swapping for at most iterations until we’re done.

Regret lemma. Suppose . Then

Ft = utu>

t − vtv> t

inf

U2∆p k

X

t=0

hFt, Ui

k

X

t=1

u>

t Atut

1 + 2αu>

t A1/2 t

ut

• v>

t Atvt

1 2αv>

t A1/2 t

vt 2pp α

k ≥ 5p/ε2, α = √p/ε

ε/k

λmin ≥ 1 − O(ε)

O(k/ε)

SUMMARY

Summary of our result (re-cap):

• Objective: discrete optimization
• Regularity: f is “regular” if it is convex, monotonic and reciprocal linear
• Method: continuous relaxation + greedy swapping

mins f(P

i sixix> i ) s.t. si ∈ {0, 1}, P i si ≤ k

• Theorem. For all regular criteria f, there exists a polynomial

time (1+𝜁) approximation algorithm provided that

k = Ω(p/ε2)

APPLICATION

Worst-case structural analysis

• Sample a few “representative” force locations and build a predictive model

for the rest locations

~4000 nodes 200 nodes

APPLICATION

Worst-case structural analysis

• Sample a few “representative” force locations and build a predictive model

for the rest locations

• Predictive model: Laplacian (linear) smoothing
• Typical problem parameter range:

yi = hxi, θ0i + εi

• max. stress response

top e-vec of surface Laplacian unknown regression model modeling error

n = 4000 ∼ 6000, p = 10 ∼ 15, k = 25 ∼ 300

ALGORITHMIC FRAMEWORK

Input: a structure with fixed boundary conditions (blue) and contact regions (red).

RESULTS

Results for the “Fertility” model

Our algorithm

nF = 3914

RESULTS

Results for the “RockingChair” model

nF = 5348

RESULTS

Results for the “Shark” model

nF = 4281

SAMPLING ALGORITHM

Results: comparison with equi-distance sampling

• K=100 sampling points

“Sensitive” regions (e.g., arms, wingtips) more sampled “Easy” regions less sampled Equidistant (naive) Our solution

SAMPLING ALGORITHM

Results: comparison with equidistance sampling

• K=200 sampling points

“Sensitive” regions (e.g., arms, wingtips) more sampled “Easy” regions less sampled

EXTENSIONS

“Robust” experimental design

• Design points are subject to adversarial “perturbations”
• Example discrete optimization problem:

min

s1,··· ,sn max ξ1,··· ,ξn f

n X

i=1

(xi + ξi)(xi + ξi)> ! s.t. si 2 {0, 1}, P

i si  k, kξik2  δ

Adversarial perturbations

EXTENSIONS

“Random design” linear regression

• Random designs , but
• Worst-case optimal designs:

(xi, yi) ∼ D

E[yi|xi] 6= hxi, β0i min

S

sup

D∈D

ED,S h kb βS β0k2

2

i

S: selected design subset β0: best linear predictor w.r.t. D βS: OLS on XS Variance: f(P

i2S xix> i )

Bias: dependent on D

Thank you!