Optimistic Rates Nati Srebro Based on work with Karthik Sridharan - - PowerPoint PPT Presentation

“Optimistic” Rates

Nati Srebro Based on work with Karthik Sridharan and Ambuj Tewari

Examples based on work with Andy Cotter, Elad Hazan, Tomer Koren, Percy Liang, Shai Shalev-Shwartz, Ohad Shamir, Karthik Sridharan

Outline

• What?
• When?

(How?)

• Why?

Estimating the Bias of a Coin

Optimistic VC bound

(aka 𝑀∗-bound, multiplicative bound)

• For a hypothesis class with VC-dim D, w.p. 1-𝜀 over n samples:

Optimistic VC bound

(aka 𝑀∗-bound, multiplicative bound)

• For a hypothesis class with VC-dim D, w.p. 1-𝜀 over n samples:
• Sample complexity to get 𝑀 ℎ ≤ 𝑀∗ + 𝜗:
• Extends to bounded real-valued loss, D=VC subgraph dim

From Parametric to Scale Sensitive Classes

• Instead of VC-dim or VC-subgraph-dim (≈ #params), rely on

metric scale to control complexity, e.g.:

• Learning depends on:
• Metric complexity measures: fat shattering dimension, covering

numbers, Rademacher Complexity

• Scale sensitivity of loss 𝜚 (bound on derivatives or “margin”)
• For ℋwith Rademacher Complexity ℛ𝑜, and 𝜚′ ≤ 𝐻:

𝑆 =

Non-Parametric Optimistic Rate for Smooth Loss

• Theorem: for any ℋ with (worst case) Rademacher Complexity

ℛ𝑜(ℋ), and any smooth loss with 𝜚′′ ≤ 𝐼, 𝜚 ≤ 𝑐, w.p. 1 − 𝜀 over n samples:

[S Sridharan Tewari 2010]

• Sample complexity

Proof Ideas

• Smooth functions are self bounding: for any H-smooth non-

negative f: 𝑔′ 𝑢 ≤ 4𝐼𝑔 𝑢

• 2nd order version of Lipschitz composition Lemma, restricted to

predictors with low loss:

Rademacher  fat shattering  𝑀∞ covering  (compose with loss and use smoothness)  𝑀2 covering  Rademacher

• Local Rademacher analysis

Non-Parametric Optimistic Rate for Smooth Loss

• Theorem: for any ℋ with (worst case) Rademacher Complexity

ℛ𝑜(ℋ), and any smooth loss with 𝜚′′ ≤ 𝐼, 𝜚 ≤ 𝑐, w.p. 1 − 𝜀 over n samples:

[S Sridharan Tewari 2010]

• Sample complexity

Parametric vs Non-Parametric

Parametric dim(ℋ) ≤ 𝐄, 𝒊 ≤ 𝟐 Scale-Sensitive ℛ𝒐 ℋ ≤ 𝑺 𝒐 Lipschitz: 𝜚′ ≤ 𝐻 (e.g. hinge, ℓ1) 𝐻 𝐸 𝑜 + 𝑀∗𝐻𝐸 𝑜 𝐻2𝑆 𝑜 Smooth: 𝜚′′ ≤ 𝐼 (e.g. logistic, Huber, smoothed hinge) 𝐼 𝐸 𝑜 + 𝑀∗𝐼𝐸 𝑜 𝐼 𝑆 𝑜 + 𝑀∗𝐼𝑆 𝑜 Smooth & strongly convex: 𝜇 ≤ 𝜚′′ ≤ 𝐼 (e.g. square loss) 𝐼 𝜇 ⋅ 𝐼 𝐸 𝑜 𝐼 𝑆 𝑜 + 𝑀∗𝐼𝑆 𝑜

Min-max tight up to poly-log factors

Optimistic SVM-Type Bounds

𝜚01 𝜚hinge ≤

• Optimize
• Generalize

Optimistic SVM-Type Bounds

𝜚01 ≤ 𝜚smooth≤ 𝜚hinge

• Optimize
• Generalize

Optimistic Learning Guarantees

Parametric classes Scale-sensitive classes with smooth loss SVM-type bounds Margin Bounds Online Learning/Optimization with smooth loss Stability-based guarantees with smooth loss × Non-param (scale sensitive) classes with non-smooth loss × Online Learning/Optimization with non-smooth loss

Why Optimistic Guarantees?

• Optimistic regime typically relevant regime:
• Approximation error 𝑀∗ ≈ Estimation error 𝜗
• If 𝜗 ≪ 𝑀∗, better to spend energy on lowering approx. error

(use more complex class)

• Important in understanding statistical learning

Training Kernel SVMs

# Kernel evaluations to get excess error 𝜗: (𝑆 = 𝑥∗ 2)

• Using SGD:
• Using the Stochastic Batch Perceptron [Cotter et al 2012]:

(is this the best possible?)

Training Linear SVMs

Runtime (# feature evaluations): (𝑆 = 𝑥∗ 2)

• Using SGD:
• Using SIMBA [Hazan et al 2011]:

(is this the best possible?)

Mini-Batch SGD

• Stochastic optimization of smooth 𝑀 𝑥 using n training-points,

doing T=n/b iterations of SGD with mini-batches of size b

• Pessimistic Analysis (ignoring 𝑀∗):

 Can use minibatch of size 𝑐 ∝ 𝑜 , with 𝑈 ∝ 𝑜 iterations and get same error (up to constant factor) as sequential SGD

[Dekel et al 2010][Agarwal Duchi 2011]

• But taking into account 𝑀∗:

In Optimistic Regime: Can’t use b>1, no parallelization speedups!

• Use acceleration to get speedup in optimistic regime  [Cotter et al 2011]

Multiple Complexity Controls

[Liang Srebro 2010]

𝑀 𝑥 = 𝔽 𝑥, 𝑌 − 𝑍 2 , 𝑍 = 𝑥, 𝑌 + 𝒪(0, 𝜏2) 𝑥 ∈ ℝ𝐸 𝑥 2 ≤ 𝑆

𝑀∗/𝐸

𝑆/𝑜 𝑀∗𝑆/𝑜 𝑀∗𝐸/𝑜

𝔽[𝑍2] 𝑀∗ 𝑆/𝔽[𝑍2] 𝑆/𝑀∗ 𝑀∗𝐸2/𝑆

Be Optimistic

• For scale-sensitive non-parametric classes, with smooth

loss: [Srebro Sridharan Tewari 2010]

• Diff vs parametric: Not possible with non-smooth loss!
• Optimistic regime typically relevant regime:
• Approximation error 𝑀∗ ≈ Estimation error 𝜗
• If 𝜗 ≪ 𝑀∗, better to spend energy on lowering approx. error (use

more complex class)

• Important in understanding statistical learning