Poisson Subsampled Rnyi Differential Privacy Yuqing Zhu joint work - - PowerPoint PPT Presentation

Poisson Subsampled Rényi Differential Privacy

Yuqing Zhu

1

joint work with Yu-Xiang Wang

2

𝜗, 𝜀 -DP 𝑃 𝛿𝜗, 𝛿𝜀 -DP 𝜗, 𝛽 -Rényi DP

[KLNRS’08], [Li et al., 2011]

Sampling probability𝛅 What’s the optimal bound ?

Privacy Amplification by Sampling

Strong composition tool

Example: The Noisy SGD Algorithm

✓t+1 ✓t ⌘t 1 |I| X

i∈I

rfi(✓t) + Zt !

Song et al. 2013; Bassily et al. 2014

1.Randomly chosen minibatch (Poisson subsampling) 2.Then add Gaussian noise (Gaussian mechanism) RDP analysis for subsampled Gaussian mechanism (Abadi et al., 2016) Really what makes Deep Learning with Differential Privacy practical

3

Let M be any randomized algorithm that obeys (α,ϵ(α))−RDP γ be the subsampling probability and for integer α≥2

𝜗(∘𝒕𝒃𝒏𝒒𝒎𝒇 𝛽 ≤ Ο(𝛽𝛿4𝜗 2 )

Exact RDP of Subsampled Mechanism

Asymptotic rate

4

This asymptotic rate holds for any mechanism M !

Let M be any randomized algorithm that obeys (α,ϵ(α))−RDP γ be the subsampling probability and for integer α≥2

Exact RDP of Subsampled Mechanism

5

𝝑𝐍∘𝑻𝒃𝒏𝒒𝒎𝒇 𝜷 ≤ 𝟐 𝜷 𝐦𝐩𝐡{ 𝟐 − 𝜹 𝜷B𝟐 𝜷𝜹 − 𝜹 + 𝟐 + 𝜷 𝟑 𝜹𝟑 𝟐 − 𝜹 𝜷B𝟑𝒇𝝑 𝟑 +𝟒 F 𝜷 ℓ 𝟐 − 𝜹 𝜷Bℓ𝜹ℓ𝒇 ℓB𝟐 𝝑(ℓ)}

𝜷 ℓI𝟒

This bound is optimal, up to a factor of 3 on a low order term

𝝑𝐍∘𝑻𝒃𝒏𝒒𝒎𝒇 𝜷 ≤ 𝟐 𝜷 𝐦𝐩𝐡{ 𝟐 − 𝜹 𝜷B𝟐 𝜷𝜹 − 𝜹 + 𝟐 + 𝜷 𝟑 𝜹𝟑 𝟐 − 𝜹 𝜷B𝟑𝒇𝝑 𝟑 +𝟒 F 𝜷 ℓ 𝟐 − 𝜹 𝜷Bℓ𝜹ℓ𝒇 ℓB𝟐 𝝑(ℓ)}

𝜷 ℓI𝟒

Let M be any randomized algorithm that obeys (α,ϵ(α))−RDP γ be the subsampling probability and for integer α≥2

Exact Amplification Bound for RDP

6

Get rid of it

Matches the lower bound when M is Gaussian or Laplace mechanism

Subsampled Gaussian Mechanism

𝜏 = 5, 𝛿 = 1𝑓 − 3

7

𝑷(𝑳) 𝑷( 𝑳

• )

Overall (𝜗, 𝜀)-DP over composition

8

Low Privacy Regime

Subsampled Gaussian Mechanism

𝜏 = 1, 𝛿 = 1𝑓 − 3

Poster Number Pacific Ballroom #178

Thank you!

Code available:

https://github.com/yuxiangw/autodp

9

Or just use:

pip install autodp Get Paper