Pr Private Stochastic Convex Optimization wi with Optimal Ra Rate - - PowerPoint PPT Presentation

β–Ά
pr private stochastic convex optimization wi with optimal
SMART_READER_LITE
LIVE PREVIEW

Pr Private Stochastic Convex Optimization wi with Optimal Ra Rate - - PowerPoint PPT Presentation

Oct 29, 2022 40 likes β€’171 views

Pr Private Stochastic Convex Optimization wi with Optimal Ra Rate Raef Bassily Vitaly Feldman Kunal Talwar Abhradeep Guha Thakaurta UC Santa Cruz Ohio State University Google Brain Google Brain Google Brain This work

slide-1
SLIDE 1

Pr Private Stochastic Convex Optimization wi with Optimal Ra Rate

UC Santa Cruz Google Brain

Raef Bassily Vitaly Feldman Kunal Talwar Abhradeep Guha Thakaurta

Ohio State University Google Brain Google Brain

slide-2
SLIDE 2

This work

Differentially private (DP) algorithms for stochastic convex optimization with optimal excess population risk

slide-3
SLIDE 3

Stochastic Convex Optimization (SCO)

Unknown distribution (population) 𝒠 over data universe 𝒢 Dataset 𝑇 = 𝑨&, … , 𝑨) ∼ 𝒠) Convex loss function β„“: π’Ÿ Γ— 𝒢 β†’ ℝ Convex parameter space π’Ÿ βŠ‚ ℝ2

𝑀4/𝑀4 setting:

π’Ÿ and πœ–β„“ are bounded in 𝑀4-norm

slide-4
SLIDE 4

A SCO algorithm, given 𝑇, outputs 7 πœ„ ∈ π’Ÿ s.t. Excess Pop. Risk β‰œ 𝔽<βˆΌπ’  β„“ 7 πœ„, 𝑨 βˆ’ min

Aβˆˆπ’Ÿ 𝔽<βˆΌπ’  β„“ πœ„, 𝑨

is as small as possible

Stochastic Convex Optimization (SCO)

Well-studied problem: optimal rate β‰ˆ

& )

Unknown distribution (population) 𝒠 over data universe 𝒢 Dataset 𝑇 = 𝑨&, … , 𝑨) ∼ 𝒠) Convex loss function β„“: π’Ÿ Γ— 𝒢 β†’ ℝ Convex parameter space π’Ÿ βŠ‚ ℝ2

𝑀4/𝑀4 setting:

π’Ÿ and πœ–β„“ are bounded in 𝑀4-norm

slide-5
SLIDE 5

Private Stochastic Convex Optimization (PSCO)

Goal: πœ—, πœ€ -DP algorithm 𝒝FGHI that, given 𝑇, outputs 7

πœ„ ∈ π’Ÿ s.t. Excess Pop. Risk β‰œ 𝔽<βˆΌπ’  β„“ 7 πœ„, 𝑨 βˆ’ min

Aβˆˆπ’Ÿ 𝔽<βˆΌπ’  β„“ πœ„, 𝑨

is as small as possible Unknown distribution (population) 𝒠 over data universe 𝒢 Dataset 𝑇 = 𝑨&, … , 𝑨) ∼ 𝒠) Convex loss function β„“: π’Ÿ Γ— 𝒢 β†’ ℝ Convex parameter space π’Ÿ βŠ‚ ℝ2

𝑀4/𝑀4 setting:

π’Ÿ and πœ–β„“ are bounded in 𝑀4-norm

slide-6
SLIDE 6

Main Result

Optimal non-private population risk Optimal private empirical risk [BST14]

Optimal excess population risk for PSCO is β‰ˆ max

& ) , 2 L )

slide-7
SLIDE 7

Main Result

Optimal excess population risk for PSCO is β‰ˆ max

& ) , 2 L )

When 𝑒 = Θ π‘œ (common in modern ML)

  • Opt. risk for PSCO β‰ˆ

& ) = opt. risk for SCO

asymptotically no cost of privacy

slide-8
SLIDE 8

Algorithms

Two algorithms under mild smoothness assumption on β„“ : Ø A variant of mini-batch noisy SGD: Ø Objective Perturbation (entails rank assumption on βˆ‡4β„“ )

slide-9
SLIDE 9

Algorithms

Two algorithms under mild smoothness assumption on β„“ : Ø A variant of mini-batch noisy SGD: Ø Objective Perturbation (entails rank assumption on βˆ‡4β„“ )

  • The objective function in both algorithms is the empirical risk.
slide-10
SLIDE 10

Algorithms

Two algorithms under mild smoothness assumption on β„“ : Ø A variant of mini-batch noisy SGD: Ø Objective Perturbation (entails rank assumption on βˆ‡4β„“ )

  • The objective function in both algorithms is the empirical risk.
  • Generalization error is bounded via uniform stability:
  • For the first algorithm: uniform stability of SGD [HRS15, FV19].
  • For the second algorithm: uniform stability due to regularization.
slide-11
SLIDE 11

Algorithms

  • General non-smooth loss:

Ø A new, efficient, noisy stochastic proximal gradient algorithm:

  • Based on Moreau-Yosida smoothing
  • A gradient step w.r.t. the smoothed loss is equivalent to a

proximal step w.r.t. the original loss.

slide-12
SLIDE 12

Results vs. Prior Work on DP-ERM

This work

  • Optimal excess population risk for PSCO is β‰ˆ max

& ) , 2 L )

Previous work

  • Focused on the empirical version (DP-ERM): [CMS11, KST12, BST14, TTZ15, …]
  • Optimal empirical risk is previously known [BST14], but not optimal population

risk.

  • Best known population risk using DP-ERM algorithms β‰ˆ max

2Q/R ) , 2 L ) [BST14].

slide-13
SLIDE 13

Full version: arXiv:1908.09970

Poster #163