Online Learning, and Private Optimization Ellen Vitercik - - PowerPoint PPT Presentation

Dispersion for Data-Driven Algorithm Design, Online Learning, and Private Optimization

Ellen Vitercik Northwestern Quarterly Theory Workshop Joint work with Nina Balcan and Travis Dick

Many problems have fast, optimal algorithms

• E.g., sorting, shortest paths

Many problems have fast, optimal algorithms

• E.g., sorting, shortest paths

Many problems don’t

• E.g., integer programming, subset selection
• Many approximation and heuristic techniques
• Best method depends on the application
• Which to use?

Practitioners repeatedly solve problems Maintain same structure Differ on underlying data Should be algo that’s good across all instances Use information about prior instances to choose algorithm for future instances Use ML to automate algorithm design

Automated algorithm design

Use ML to automate algorithm design Large body of empirical work:

• Comp bio [DeBlasio and Kececioglu, ‘18]
• AI [Xu, Hutter, Hoos, and Leyton-Brown, ’08]

This work: formal guarantees for this approach Use ML to automate algorithm design

Simple example: knapsack

Problem instance:

• 𝑜 items; Item 𝑗 has value 𝑤𝑗 and size 𝑡𝑗
• Knapsack with capacity 𝐿

Goal: find most valuable items that fit Algorithm (parameterized by 𝜍 ≥ 0): Add items in decreasing order of

𝑤𝑗 𝑡𝑗

𝜍

[Gupta and Roughgarden, ‘17] How to set?

Application domain: stealing jewelry

Day 1

Online algorithm configuration

Knapsack algorithm parameter 𝜍 0.95

Day 2

Online algorithm configuration

Knapsack algorithm parameter 𝜍 0.45

Value of items in knapsack

Day 3

Online algorithm configuration

Knapsack algorithm parameter 𝜍 0.45 Parameter 𝜍

Day 3

Online algorithm configuration

Knapsack algorithm parameter 𝜍 0.45 Algorithm utility on 3rd instance Parameter 𝜍 𝑣3 𝜍

Day 4

Online algorithm configuration

Knapsack algorithm parameter 𝜍 0.75 Algorithm utility on 4th instance Parameter 𝜍 𝑣4 𝜍

Online algorithm configuration

Goal:Compete with best fixed parameters in hindsight. Minimize regret.

Optimizing piecewise Lipschitz functions

Worst-case impossible to optimize online! Configuration ⇔ optimizing sums of piecewise Lipschitz functions

Parameter 𝜍 Algorithm utility on 𝑢th instance

Our contributions

Structural property dispersion implies strong guarantees for:

• Online optimization of PWL functions
• Uniform convergence in statistical settings
• Differentially private optimization

Dispersion satisfied in real problems under very mild assumptions

Outline

• 1. Online learning setup
• 2. Dispersion
• 3. Regret bounds
• 4. Examples of dispersion
• 5. Other applications of dispersion
• 6. Conclusion

Online piecewise Lipschitz optimization

For each round 𝑢 ∈ 1, … , 𝑈 :

• 1. Learner chooses 𝝇𝑢 ∈ ℝ𝑒
• 2. Adversary chooses piecewise 𝑀-Lipschitz function 𝑣𝑢: ℝ𝑒 → ℝ
• 3. Learner gets reward 𝑣𝑢 𝝇𝑢
• 4. Full information: Learner observes function 𝑣𝑢

𝑣𝑢 𝜍 = Algorithm utility on 𝑢th instance 𝜍

Online piecewise Lipschitz optimization

For each round 𝑢 ∈ 1, … , 𝑈 :

• 1. Learner chooses 𝝇𝑢 ∈ ℝ𝑒
• 2. Adversary chooses piecewise 𝑀-Lipschitz function 𝑣𝑢: ℝ𝑒 → ℝ
• 3. Learner gets reward 𝑣𝑢 𝝇𝑢
• 4. Full information: Learner observes function 𝑣𝑢

Bandit feedback: Learner only observes 𝑣𝑢 𝝇𝑢

𝑣𝑢 𝜍 = Algorithm utility on 𝑢th instance 𝜍

Online piecewise Lipschitz optimization

For each round 𝑢 ∈ 1, … , 𝑈 :

• 1. Learner chooses 𝝇𝑢 ∈ ℝ𝑒
• 2. Adversary chooses piecewise 𝑀-Lipschitz function 𝑣𝑢: ℝ𝑒 → ℝ
• 3. Learner gets reward 𝑣𝑢 𝝇𝑢
• 4. Full information: Learner observes function 𝑣𝑢

Bandit feedback: Learner only observes 𝑣𝑢 𝝇𝑢 Goal: Minimize regret = max

𝝇∈ℝ𝑒 σ𝑢=1 𝑈

𝑣𝑢 𝝇 − σ𝑢=1

𝑈

𝑣𝑢 𝝇𝑢 Want regret sublinear in 𝑈

Avg regret 𝑈

Prior work on PWL online optimization

Gupta and Roughgarden [’17]: Max-Weight Independent Set algo configuration Cohen-Addad and Kanade [’17]: 1D piecewise constant functions

Mean adversary

Exists adversary choosing piecewise constant functions s.t.: Every full information online algorithm has linear regret. Round 1: Adversary chooses one or the other with equal prob.

Mean adversary

Exists adversary choosing piecewise constant functions s.t.: Every full information online algorithm has linear regret. Round 1: Round 2:

Mean adversary

Exists adversary choosing piecewise constant functions s.t.: Every full information online algorithm has linear regret. Round 1: Round 2: Repeatedly halves optimal region

Mean adversary

Exists adversary choosing piecewise constant functions s.t.: Every full information online algorithm has linear regret. Round 1: Round 2: Repeatedly halves optimal region

Mean adversary

Exists adversary choosing piecewise constant functions s.t.: Every full information online algorithm has linear regret. Round 1: Round 2: Repeatedly halves optimal region

Mean adversary

Exists adversary choosing piecewise constant functions s.t.: Every full information online algorithm has linear regret. Round 1: Round 2: Learner’s expected reward:

𝑈 2

Reward of best point in hindsight: 𝑈 Expected regret =

𝑈 2

Repeatedly halves optimal region

Outline

• 1. Online learning setup
• 2. Dispersion
• 3. Regret bounds
• 4. Examples of dispersion
• 5. Other applications of dispersion
• 6. Conclusion

Dispersion

Mean adversary concentrates discontinuities near maximizer 𝜍∗ Even points very close to 𝜍∗ have low utility! 𝑣1, … , 𝑣𝑈 are 𝒙, 𝒍 -dispersed at point 𝝇 if: ℓ2-ball 𝐶 𝝇, 𝑥 contains discontinuities for ≤ 𝑙 of 𝑣1, … , 𝑣𝑈

𝝇 𝑥

Ball of radius 𝑥 about 𝝇 contains 2 discontinuities. → (𝑥, 2)-dispersed at 𝝇.

Sums of piecewise dispersed functions

Given 𝑣1, … , 𝑣𝑈, plot of sum σ𝑢=1

𝑈

𝑣𝑢: Not dispersed Many discontinuities in interval Few discontinuities in interval Dispersed

𝜍 𝜍

Key property of dispersed functions

If 𝑣1, … , 𝑣𝑈: ℝ𝑒 → [0,1] are

• 1. Piecewise 𝑀-Lipschitz
• 2. (𝑥, 𝑙)-dispersed at maximizer 𝝇∗,

For every 𝝇 ∈ 𝐶 𝝇∗, 𝑥 : σ𝑢=1

𝑈

𝑣𝑢 𝝇 ≥ σ𝑢=1

𝑈

𝑣𝑢 𝝇∗ − 𝑈𝑀𝑥 − 𝑙. Proof idea : 𝑣1, … , 𝑣𝑈 Is 𝑣𝑢 𝑀-Lipschitz

• n 𝐶 𝝇∗, 𝑥 ?

𝝇∗

Key property of dispersed functions

If 𝑣1, … , 𝑣𝑈: ℝ𝑒 → [0,1] are

• 1. Piecewise 𝑀-Lipschitz
• 2. (𝑥, 𝑙)-dispersed at maximizer 𝝇∗,

For every 𝝇 ∈ 𝐶 𝝇∗, 𝑥 : σ𝑢=1

𝑈

𝑣𝑢 𝝇 ≥ σ𝑢=1

𝑈

𝑣𝑢 𝝇∗ − 𝑈𝑀𝑥 − 𝒍. Proof idea : 𝑣1, … , 𝑣𝑈 𝑣𝑢 𝝇 − 𝑣𝑢 𝝇∗ ≤ 1 No (≤ 𝑙 functions) Is 𝑣𝑢 𝑀-Lipschitz

• n 𝐶 𝝇∗, 𝑥 ?

𝝇∗

Key property of dispersed functions

If 𝑣1, … , 𝑣𝑈: ℝ𝑒 → [0,1] are

• 1. Piecewise 𝑀-Lipschitz
• 2. (𝑥, 𝑙)-dispersed at maximizer 𝝇∗,

For every 𝝇 ∈ 𝐶 𝝇∗, 𝑥 : σ𝑢=1

𝑈

𝑣𝑢 𝝇 ≥ σ𝑢=1

𝑈

𝑣𝑢 𝝇∗ − 𝑼𝑴𝒙 − 𝑙. Proof idea : 𝑣1, … , 𝑣𝑈 𝑣𝑢 𝝇 − 𝑣𝑢 𝝇∗ ≤ 1 𝑣𝑢 𝝇 − 𝑣𝑢 𝝇∗ ≤ 𝑀𝑥 Yes (≤ 𝑈 functions) No (≤ 𝑙 functions) Is 𝑣𝑢 𝑀-Lipschitz

• n 𝐶 𝝇∗, 𝑥 ?

𝝇∗

Outline

• 1. Online learning setup
• 2. Dispersion
• 3. Regret bounds

1. Full information 2. Bandit feedback

• 4. Examples of dispersion
• 5. Other applications of dispersion
• 6. Conclusion

Full information online learning

Exponentially Weighted Forecaster [Cesa-Bianchi & Lugosi ’06]: At round 𝑢, sample from dist. w/ PDF 𝑔

𝑢(𝝇) ∝ exp 𝜇 σ𝑡=1 𝑢−1 𝑣𝑡 𝝇

.

𝑣𝑢 𝜍 𝜍

Full information online learning

𝝇∗ Theorem: If 𝑣1, … , 𝑣𝑈: 𝐶𝑒(𝟏, 1) → 0,1 are:

• 1. Piecewise 𝑀-Lipschitz
• 2. (𝑥, 𝑙)-dispersed at 𝝇∗,

EWF has regret 𝑃 𝑼𝒆 𝐦𝐩𝐡

𝟐 𝒙 + 𝑼𝑴𝒙 + 𝒍 .

Intuition: Every 𝝇 ∈ 𝐶 𝝇∗, 𝑥 has utility ≥ 𝑃𝑄𝑈 − 𝑈𝑀𝑥 − 𝑙. EWF can compete with 𝐶 𝝇∗, 𝑥 up to 𝑃 𝑈𝑒 log

1 𝑥

factor. When is this a good bound? For 𝑥 =

1 𝑀 𝑈 and 𝑙 = ෨

𝑃 𝑈 , regret is ෨ 𝑃 𝑈𝑒

Full information online learning

𝝇∗ Theorem: If 𝑣1, … , 𝑣𝑈: 𝐶𝑒(𝟏, 1) → 0,1 are:

• 1. Piecewise 𝑀-Lipschitz
• 2. (𝑥, 𝑙)-dispersed at 𝝇∗,

EWF has regret 𝑃 𝑼𝒆 𝐦𝐩𝐡

𝟐 𝒙 + 𝑼𝑴𝒙 + 𝒍 .

Intuition: Every 𝝇 ∈ 𝐶 𝝇∗, 𝑥 has utility ≥ 𝑃𝑄𝑈 − 𝑼𝑴𝒙 − 𝒍. EWF can compete with 𝐶 𝝇∗, 𝑥 up to 𝑃 𝑈𝑒 log

1 𝑥

factor.

Full information online learning

𝝇∗ Theorem: If 𝑣1, … , 𝑣𝑈: 𝐶𝑒(𝟏, 1) → 0,1 are:

• 1. Piecewise 𝑀-Lipschitz
• 2. (𝑥, 𝑙)-dispersed at 𝝇∗,

EWF has regret 𝑃 𝑼𝒆 𝐦𝐩𝐡

𝟐 𝒙 + 𝑼𝑴𝒙 + 𝒍 .

Intuition: Every 𝝇 ∈ 𝐶 𝝇∗, 𝑥 has utility ≥ 𝑃𝑄𝑈 − 𝑼𝑴𝒙 − 𝒍. EWF can compete with 𝐶 𝝇∗, 𝑥 up to 𝑃 𝑼𝒆 𝐦𝐩𝐡

𝟐 𝒙

factor.

Matching lower bound

Theorem: For any algorithm, exist PW constant 𝑣1, … , 𝑣𝑈 s.t.: Algorithm’s regret is Ω inf

(𝑥,𝑙)

𝑈𝑒 log

1 𝑥 + 𝑙 .

Inf over all (𝑥, 𝑙)-dispersion parameters 𝑣1, … , 𝑣𝑈 satisfy at 𝝇∗. Upper bound = 𝑃 inf

(𝑥,𝑙)

𝑈𝑒 log

1 𝑥 + 𝑙 .

Outline

• 1. Online learning setup
• 2. Dispersion
• 3. Regret bounds

1. Full information 2. Bandit feedback

• 4. Examples of dispersion
• 5. Other applications of dispersion
• 6. Conclusion

Bandit feedback

Theorem: If 𝑣1, … , 𝑣𝑈: 𝐶𝑒(𝟏, 1) → 0,1 are:

• 1. Piecewise 𝑀-Lipschitz
• 2. (𝑥, 𝑙)-dispersed at 𝝇∗,

There is a bandit algorithm with regret ෨ 𝑃 𝑈𝑒

1 𝑥 𝑒

+ 𝑈𝑀𝑥 + 𝑙 .

𝑣𝑢 𝜍 𝜍

Bandit feedback

Theorem: Exists algorithm with regret ෨ 𝑃 𝑈𝑒

1 𝑥 𝑒

+ 𝑈𝑀𝑥 + 𝑙 . Proof idea.

• Let 𝝇1, … , 𝝇𝑂 be a 𝑥-net of 𝐶𝑒(𝟏, 1) (can take 𝑂 ≈ 1/𝑥𝑒).
• Reduce to 𝑂-armed bandit.
• Use EXP3 on discretization → regret 𝑃

𝑈𝑂 log 𝑂 .

• Ball of radius 𝑥 around 𝝇∗ must contain some 𝝇𝑗.
• Regret of 𝝇𝑗 compared to 𝝇∗ is ≤ 𝑈𝑀𝑥 + 𝑙.

When is this a good bound? If 𝑒 = 1, 𝑥 =

1

3 𝑈, and 𝑙 = ෨

𝑃 𝑈2/3 , regret is ෨ 𝑃 𝑀𝑈2/3 .

𝑣𝑢 𝜍 𝜍

Bandit feedback

Theorem: Exists algorithm with regret ෨ 𝑃 𝑈𝑒

1 𝑥 𝑒

+ 𝑈𝑀𝑥 + 𝑙 . Proof idea.

• Let 𝝇1, … , 𝝇𝑂 be a 𝑥-net of 𝐶𝑒(𝟏, 1) (can take 𝑂 ≈ 1/𝑥𝑒).
• Reduce to 𝑂-armed bandit.
• Use EXP3 on discretization → regret 𝑃

𝑈𝑂 log 𝑂 .

• Ball of radius 𝑥 around 𝝇∗ must contain some 𝝇𝑗.
• Regret of 𝝇𝑗 compared to 𝝇∗ is ≤ 𝑈𝑀𝑥 + 𝑙.

When is this a good bound? If 𝑥 = 𝑈

𝑒+1 𝑒+2−1, 𝑙 = ෨

𝑃 𝑈

𝑒+1 𝑒+2 , then regret is ෨

𝑃 𝑈

𝑒+1 𝑒+2

𝑒3𝑒 + 𝑀 .

Outline

• 1. Online learning setup
• 2. Dispersion
• 3. Regret bounds
• 4. Examples of dispersion
• 5. Other applications of dispersion
• 6. Conclusion

Smooth adversaries and dispersion

Adversary chooses thresholds 𝑣𝑢: 0,1 → 0,1 . 1 𝜐

Smooth adversaries and dispersion

Adversary chooses thresholds 𝑣𝑢: 0,1 → 0,1 . Discontinuity 𝜐 “smoothed” by adding 𝑎~𝑂(0, 𝜏2). Lemma: W.h.p., ∀𝑥, 𝑣1, … , 𝑣𝑈 are 𝑥, ෨ 𝑃

𝑈𝑥 𝜏 +

𝑈

• dispersed.

Corollary: 𝑥 =

𝜏 𝑈 ⇒ Full information regret = 𝑷

𝑼 𝐦𝐩𝐡

𝑼 𝝉 .

𝜐 + 𝑎 1 𝜐

Adversary chooses thresholds 𝑣𝑢: 0,1 → 0,1 . Discontinuity 𝜐 “smoothed” by adding 𝑎~𝑂(0, 𝜏2). Lemma: W.h.p., ∀𝑥, 𝑣1, … , 𝑣𝑈 are 𝑥, ෨ 𝑃

𝑈𝑥 𝜏 +

𝑈

• dispersed.

Proof idea: For any width-𝑥 interval, 𝔽[#discontinuities] = 𝑃

𝑈𝑥 𝜏

.

• VC-dim ⇒ w.h.p., every interval has ෨

𝑃

𝑈𝑥 𝜏 +

𝑈 discontinuities.

Smooth adversaries and dispersion

𝜐 + 𝑎 1 𝜐

Simple example: knapsack

Problem instance:

• 𝑜 items; Item 𝑗 has value 𝑤𝑗 and size 𝑡𝑗
• Knapsack with capacity 𝐿

Goal: find most valuable items that fit Algorithm (parameterized by 𝜍 ≥ 0): Add items in decreasing order of

𝑤𝑗 𝑡𝑗

𝜍

[Gupta and Roughgarden, ‘17]

Simple example: knapsack

Problem instance:

• 𝑜 items; Item 𝑗 has value 𝑤𝑗 and size 𝑡𝑗
• Knapsack with capacity 𝐿

Goal: find most valuable items that fit Algorithm (parameterized by 𝜍 ≥ 0): Add items in decreasing order of

𝑤𝑗 𝑡𝑗

𝜍

[Gupta and Roughgarden, ‘17]

𝜍 𝑣𝑢(𝜍) Algorithm utility on 𝑢th instance

Dispersion for knapsack

Theorem: If instances randomly distributed s.t. on each round:

• 1. Each 𝑤𝑗 independent from 𝑡𝑗
• 2. All 𝑤𝑗, 𝑤𝑘 have 𝜆-bounded joint density,

W.h.p., for any 𝛽 ≥

1 2, 𝑣1, … , 𝑣𝑈 are

෨ 𝑃

𝑈1−𝛽 𝜆

, ෨ 𝑃 (# items)2𝑈𝛽

• dispersed.

Corollary: Bandit feedback regret = ෨ 𝑃 𝑈

2 3

1 𝜆 + (# items)2

.

𝜍 𝑣𝑢(𝜍) Algorithm utility on 𝑢th instance

Corollary: Full information regret = ෨ 𝑃 (# items)2 𝑈 .

More Results for Algorithm Configuration

Prove dispersion under smoothness assumptions for:

• Maximum weight independent set

Under no assumptions, we show dispersion for:

• Integer quadratic programming approximation algos
• Based on semi-definite programming relaxations
• 𝑡-linear rounding [Feige & Langberg ‘06]
• Outward rotations [Zwick ‘99]
• Both generalizations of Goemans-Williamson max-cut algorithm [‘95].

Outline

• 1. Online learning setup
• 2. Dispersion
• 3. Regret bounds
• 4. Examples of dispersion
• 5. Other applications of dispersion
• 6. Conclusion

Uniform convergence for batch learning

Theorem: If 𝑣1, … , 𝑣𝑈: ℝ𝑒 → 0,1 are:

• 1. Independently drawn from a distribution 𝒠
• 2. Piecewise 𝑀-Lipschitz
• 3. Globally (𝑥, 𝑙)-dispersed,

W.h.p., for every 𝝇 ∈ ℝ𝑒, 1 𝑈 ෍

𝑢=1 𝑈

𝑣𝑢 𝝇 − 𝔽𝑣∼𝒠 𝑣 𝝇 = ෨ 𝑃 𝑒 𝑈 log 1 𝑥 + 𝑀𝑥 + 𝑙 𝑈 .

Differentially private optimization

Given 𝑣1, … , 𝑣𝑈: 𝐶𝑒(𝟏, 1) → 0,1 up front. Goal:

• Find (approximate) maximizer of

1 𝑈 σ𝑢=1 𝑈

𝑣𝑢.

• Preserve 𝜗-DP w.r.t. changing any one function.

Exponential mechanism [McSherry-Talwar ’07] has suboptimality ෨ 𝑃 𝑒 𝑈𝜗 log 1 𝑥 + 𝑀𝑥 + 𝑙 𝑈 . Matching lower bounds!

Outline

• 1. Online learning setup
• 2. Dispersion
• 3. Regret bounds
• 4. Examples of dispersion
• 5. Other applications of dispersion
• 6. Conclusion

Conclusions and open questions

• Introduced dispersion.
• Measures concentration of discontinuities of PWL functions.
• Implies regret bounds for online optimization of PWL functions.
• Batch learning and private optimization guarantees.
• Examples of dispersion in real problems.

Conclusions and open questions

• Introduced dispersion.
• Measures concentration of discontinuities of PWL functions.
• Implies regret bounds for online optimization of PWL functions.
• Batch learning and private optimization guarantees.
• Examples of dispersion in real problems.

Open Questions:

• Bad properties beyond discontinuities?
• Config. between full-info and bandit. Can we provide algos?