Locally private learning without interaction requires separation - - PowerPoint PPT Presentation

Locally private learning without interaction requires separation

Vitaly Feldman Research

with Amit Daniely

Hebrew University

[KLNRS ‘08] 𝜗-LDP if for every user 𝑗, message 𝑘 is sent using a local 𝜗𝑗,𝑘-DP randomizer 𝐵𝑗,𝑘 and 𝜗𝑗,𝑘

𝑘

≤ 𝜗

Local Differential Privacy (LDP)

𝑨2 𝑨1 𝑨3 𝑨𝑜

Server

Non-interactive LDP

𝑨2 𝑨1 𝑨3 𝑨𝑜

Server

PAC learning

4

PAC model [Valiant ‘84]: Let 𝐷 be a set of binary classifiers over 𝑌 𝐵 is a PAC learning algorithm for 𝐷 if ∀𝑔 ∈ 𝐷 and distribution 𝐸 over 𝑌, given i.i.d. examples 𝑦𝑗, 𝑔 𝑦𝑗 for 𝑦𝑗 ∼ 𝐸, 𝐵 outputs ℎ such that w.h.p. 𝐐𝐬

𝑦∼𝐸 ℎ 𝑦 ≠ 𝑔(𝑦) ≤ 𝛽

Distribution-specific learning: 𝐸 is fixed and known to 𝐵

Statistical query model [Kearns ‘93]

[KLNRS ‘08] Simulation with success prob. 1 − 𝛾 (𝜗 ≤ 1)

• 𝜗-LDP with 𝑛 messages ⇒ 𝑃 𝑛 queries with 𝜐 = Ω

𝛾 𝑛

• 𝑟 queries with tolerance 𝜐 ⇒ 𝜗-LDP with 𝑜 = 𝑃

𝑟 log 𝑟/𝛾 𝜐𝜗 2

samples/messages Non-interactive if and only if queries are non-adaptive

𝑤1 − 𝐅𝑨∼𝑄 𝜚1 𝑨 ≤ 𝜐

𝜐 is tolerance of the query; 𝜐 = 1/ 𝑜

𝜚𝑟

𝑤1

𝜚2

𝑤2 𝑤𝑟

𝜚1 SQ algorithm

𝜚1: 𝑎 → 0,1

𝑄

SQ oracle 𝑄 distribution over 𝑎 𝑎 = 𝑌 × {±1} 𝑄 is the distribution of (𝑦, 𝑔 𝑦 ) for 𝑦 ∼ 𝐸

Known results

Examples:

• Yes: halfspaces/linear classifiers [Dunagan,Vempala ‘04]
• No: parity functions [Kearns ‘93]
• Yes, non-adaptively: Boolean conjunctions

6

𝐷 is SQ-learnable efficiently (non-adaptively) if and only if learnable efficiently with 𝜗-LDP (non-interactively) [KLNRS 08] There exists 𝐷 that is 1. SQ/LDP-learnable efficiently over the uniform distribution on 0,1 𝑒 but 2. requires exponential num. of samples to learn non-interactively by an LDP algorithm

Masked parity

[KLNRS 08]:

Does separation hold for distribution-independent learning?

Margin Complexity

7

+ + + + + +

• +
• +

+ + +

• +

Margin complexity of 𝐷 over 𝑌 - 𝐍𝐃(𝐷): smallest 𝑁 such that exists an embedding Ψ: 𝑌 → 𝐂𝑒(1) under which every 𝑔 ∈ 𝐷 is linearly separable with margin 𝛿 ≥

1 𝑁

Positive examples Ψ 𝑦 𝑔 𝑦 = +1} Negative examples Ψ 𝑦 𝑔 𝑦 = −1}

Lower bound

8

Thm: Let 𝐷 be a negation-closed set of classifiers. Any non-interactive 1-LPD algorithm that learns 𝐷 with error 𝛽 < 1/2 and success probability Ω 1 needs 𝑜 = Ω 𝐍𝐃 𝐷 2/3

Corollaries:

• Decision lists over 0,1 𝑒: 𝑜 = 2Ω 𝑒1/3

[Buhrman,Vereshchagin,de Wolf ‘07] (Interactively) learnable with 𝑜 = poly

𝑒 𝛽𝜗 [Kearns ’93]

• Linear classifiers over 0,1 𝑒: 𝑜 = 2Ω 𝑒

[Goldmann,Hastad,Razborov ‘92; Sherstov ‘07] (Interactively) learnable with 𝑜 = poly

𝑒 𝛽𝜗 [Dunagan,Vempala ’04]

Upper bound

9

Thm: For any 𝐷 and distribution 𝐸 there exists a non-adaptive 𝜗-LPD algorithm that learns 𝐷 over 𝐸 with error 𝛽 and success probability 1 − 𝛾 using 𝑜 = poly 𝐍𝐃 𝐷 ⋅ log 1/𝛾 𝛽𝜗 Instead of fixed 𝐸

• access to public unlabeled samples from 𝐸
• (interactive) LDP access to unlabeled samples from 𝐸

Lower bound holds against the hybrid model

Lower bound technique

10

Thm: Let 𝐷 be a negation-closed set of classifiers. If exists a non-adaptive SQ algorithm that uses 𝑟 queries

• f tolerance 1/𝑟 to learn 𝐷 with error 𝛽 < 1/2 and success probability Ω 1 then

𝐍𝐃 𝐷 = 𝑃 𝑟3/2

Correlation dimension of 𝐷 - CSQdim(𝐷) [F. ’08] : smallest 𝑢 for which exist 𝑢 functions ℎ1, … , ℎ𝑢: 𝑌 → [−1,1] such that for every 𝑔 ∈ 𝐷 and distribution 𝐸 exists 𝑗 such that 𝐅

𝑦∼𝐸 𝑔 𝑦 ℎ𝑗 𝑦

≥ 1 𝑢

Thm: [F. ’08; Kallweit,Simon ‘11]: 𝐍𝐃 𝐷 ≤ CSQdim 𝐷 3/2

Proof

Let 𝜚1, … , 𝜚𝑟: 𝑌 × ±1 → 0,1 be the (non-adaptive) queries of 𝐵 Decompose 𝜚 𝑦, 𝑧 = 𝜚 𝑦, 1 + 𝜚 𝑦, −1 2 + 𝜚 𝑦, 1 − 𝜚 𝑦, −1 2 ⋅ 𝑧 𝐅

𝑦∼𝐸 𝜚𝑗(𝑦, 𝑔 𝑦 ) = 𝐅 𝑦∼𝐸 𝑕𝑗 𝑦

+ 𝐅

𝑦∼𝐸 𝑔 𝑦 ℎ𝑗 𝑦

If 𝐅

𝑦∼𝐸 𝑔 𝑦 ℎ𝑗 𝑦

≤ 1

𝑟 then 𝐅 𝑦∼𝐸 𝜚𝑗(𝑦, 𝑔 𝑦 ) ≈ 𝐅 𝑦∼𝐸 𝜚𝑗(𝑦, −𝑔 𝑦 )

If this holds for all 𝑗 ∈ [𝑟], then the algorithm cannot distinguish between 𝑔 and −𝑔 Cannot achieve error < 1/2

11

If exists a non-adaptive SQ algorithm 𝐵 that uses 𝑟 queries

• f tolerance 1/𝑟 to learn 𝐷 with error 𝛽 < 1/2 then

CSQdim 𝐷 ≤ 𝑟 ℎ 𝑕

Upper bound

12

Thm: For any 𝐷 and distribution 𝐸 there exists a non-adaptive 𝜗-LPD algorithm that learns 𝐷 over 𝐸 with error 𝛽 < 1/2 and success probability 1 − 𝛾 using 𝑜 = poly 𝐍𝐃 𝐷 ⋅ log 1/𝛾 𝛽𝜗

Margin complexity of 𝐷 over 𝑌 - 𝐍𝐃(𝐷): smallest 𝑁 such that exists an embedding Ψ: 𝑌 → 𝐂𝑒(1) under which every 𝑔 ∈ 𝐷 is linearly separable with margin 𝛿 ≥

1 𝑁

Thm [Arriaga,Vempala ’99; Ben-David,Eiron,Simon ‘02]: For every every 𝑔 ∈ 𝐷, random projection into 𝐂𝑒(1) for 𝑒 = 𝑃(𝐍𝐃 𝐷 2log(1/𝛾)) ensures that with prob. 1 − 𝛾, 1 − 𝛾 fraction of points are linearly separable with margin 𝛿 ≥

1 𝟑 𝐍𝐃 𝐷

Algorithm

Perceptron: if sign( 𝑥𝑢, 𝑦 ) ≠ 𝑧 then update 𝑥𝑢+1 ← 𝑥𝑢 + 𝑧𝑦 Expected update: 𝐅

(𝑦,𝑧)∼𝑄 𝑧𝑦 | sign( 𝑥𝑢, 𝑦 ) ≠ 𝑧

|| 𝐅

(𝑦,𝑧)∼𝑄 𝑧𝑦 ⋅ 𝟚(sign( 𝑥𝑢, 𝑦 ) ≠ 𝑧) /

𝐐𝐬

(𝑦,𝑧)∼𝑄 sign( 𝑥𝑢, 𝑦 ) ≠ 𝑧

|| 𝐅

(𝑦,𝑧)∼𝑄 𝑦 ⋅ 𝑧 − sign 𝑥𝑢, 𝑦

2 Estimate the mean vector with ℓ2 error

• LDP [Duchi,Jordan,Wainright ‘13]
• SQs [F.,Guzman,Vempala ‘15]

13

scalar ≥ 𝛽 independent of the label non-adaptive

= 𝐅

𝑦,𝑧 ∼𝑄 𝑦𝑧 +

𝐅

𝑦,𝑧 ∼𝑄 𝑦 sign 𝑥𝑢, 𝑦

2

Conclusions

• New approach to lower bounds for non-interactive LDP
• Reduction to margin-complexity lower bounds
• Lower bounds for classical learning problems
• Same results for communication constrained protocols
• Also equivalent to SQ
• Interaction is necessary for learning
• Open:
• Distribution-independent learning in poly 𝐍𝐃 𝐷
• Lower bounds against 2 + round protocols
• Stochastic convex optimization

https://arxiv.org/abs/1809.09165

14