Adversarial Classification Under Differential Privacy Jairo Giraldo - - PowerPoint PPT Presentation

Adversarial Classification Under Differential Privacy

Jairo Giraldo University of Utah Alvaro A. Cardenas UC Santa Cruz Murat Kantarcioglu UT Dallas Jonathan Katz GMU

Kevin Ashton (British entrepreneur) coined the term IoT in 1999.

2

• 20th Century: computers were

brains without senses—-they

• nly knew what we told them.
• More info in the world than what

people can type on keyboard

• 21st century: computers sense

things, e.g., GPS we take for granted in our phones

New Privacy Concerns

3

In Addition to Privacy, There is Another Problem: Data Trustworthiness

4

Privacy Security Utility This work Privacy vs. Utility

We Need to Provide 3 Properties

• 1. Classical Utility
• Usable Statistics
• Reason for data collection
• 2. Privacy
• Protect consumer data
• 3. Security
• Trustworthy data
• Detect data poisoning
• Different from classical utility because this

is an adversarial setting

5

DP

푑1 푑2 푑푛

Database

Query Response

DP DP

Sensor 1 Sensor 2 Sensor 3 Sensor n

New Adversary Model

• Consumer data

protected by Differential Privacy (DP)

• Classical adversary in

DP is curious

• Our adversary is

different: data poisoning by hiding their attacks in DP noise

• Global and local DP

6

Adversary Goals

• Intelligently poison the data in a way that is

hard to detect (hide attack in DP noise)

• Achieve maximum damage to the utility of the

system (deviate estimate as much as possible)

7

max

fa E[Y a]

s.t. DKL(fakf0)  γ fa 2 F

Classical DP Attack Goals: Multi-criteria Optimization ¯ Y ← M(D) ¯ Y ∼ f0 Attack Y a instead of ¯ Y

Functional Optimization Problem

• We have to find a probability distribution
• A probability density function
• Among all possible continuous functions as

long as

• What is the shape of ?

8

fa Z

r∈Ω

fa(r)dr = 1 fa

Solution: Variational Methods

• Variational methods are a useful tool to find

the shape of functions or the structure of matrices

• They replace the function or matrix
• ptimization problem with a parameterized

perturbation of the function or matrix

• We can then optimize with respect to the

parameter to find the “shape” of the function/ matrix

• The Lagrange multipliers give us the final

parameters of the function

9

Solution

10

Z

r∈Ω

rfa(r)dr Z

r∈Ω

fa(r) ln ✓fa(r) f0(r) ◆ dr ≤ γ. Z

r∈Ω

fa(r)dr = 1.

Maximize Subject to: q(r, α) = f ∗

a(r) + αp(r).

Auxiliary Function Lagrangian: Solution:

L(α) = Z

r∈Ω

rq(r, α)dr + κ1 @ Z

r∈Ω

q(r, α) ln q(r, α) f0(r) dr − γ 1 A + κ2 @ Z

r∈Ω

q(r, α)dr − 1 1 A

f ∗

a(y) =

f0(y)e

y κ1

R f0(r)e

r κ1 dr

,where κ1 is the solution to DKL(f ∗

akf0) = γ.

Least-Favorable Laplace Attack

11

User ID Data User 1 0.5 User 2 0.3 User 3 0.7 User 4 1

2.5

Diff. Privacy

2.3 2.2 2.7 2.4

2.4 2.9 2.8 2.6

Database Query response Possible private response Possible compromised response

Aggregation

f0(y) = 1 2be−|y−θ|/b f ∗

a(y) = κ2 1 − b2

2bκ2

1

e− |y−θ|

b

+ (y−θ)

κ1

2b2 κ2

1 − b2 + ln(1 − b2

κ2

1

) = γ

κ1 is the solution to

• 10
• 5

5 10 15 20 25 30 DP Aggregation 0.05 0.1 0.15 0.2 0.25 Probability

= 0 = 0.1 = 2

Example: Traffic Flow Estimation

12

• Vehicle count
• Occupancy

We use loop detection data from California

Classical Bad Data Detection in Traffic Flow Estimation

13 Sensor Readings

DP

BDD BDD

DP

Prediction

ˆ yi(k + 1) = ˆ yi(k) + T li ✓li−1 li F in

i (k)

−F out

i

(k)

• + Qi(yi(k) − ˆ

yi(k))

Cabinet

TMC

F out

i

(k) F in

i (k)

Li+1

Cell i − 1 Cell i Cell i + 1 λi−1 = 3

Loop detector

The Attack Can Hide in DP Noise and Cause a Larger Impact

14

Without DP the attack is limited With DP, the attacker can lie more without detection Can we do better?

Defense Against Adversarial (Adaptive) Distributions

• Player 1 designs classifier D ∈ S minimize Φ(D,A) (e.g.,

Pr[Miss Detection] Subject to fix false alarms)

• Player 1 makes the first move
• Player 2 (attacker) has multiple strategies A∈ F
• Makes the move after observing the move of the classifier
• Player 1 wants provable performance guarantees:
• Once it selects Do by minimizing Φ, it wants proof that no matter what

the attacker does, Φ<m, i.e.

• 15

Defense in Traffic Case

• With classical defense

16

Proposed new defense as game between attacker and defender:

• With our defense

Another Example: Sharing Electricity Consumption

17

10-2 10-1 100

Level of privacy ( )

20 40 60 80 100

Impact S (MW)

=0.03 and BDD =0.02 and BDD =0.01 and BDD =0.03 and DP-BDD =0.02 and DP-BDD =0.01 and DP-BDD

Conclusions

• Growing number of applications where we

need to provide utility, privacy, and security

• In particular, adversarial classification

under differential privacy

• Various possible extensions
• Different quantification of privacy loss

(e.g., Rényi DP)

• Adversary models (noiseless privacy), etc.
• Related work on DP and adversarial ML
• Certified robustness

18

Strategic Adversary + Defender

• Player 1 designs classifier D ∈ S minimizing

Φ(D,A) (e.g., Pr[Error])

– Defender makes the first move

• Player 2 (attacker) has multiple strategies

A∈ F

– Attacker makes the move after observing the move of the classifier

• Player 1 wants provable performance

guarantees:

– Once it selects Do by minimizing Φ, it wants proof that no matter what the attacker does, Φ<m, i.e.

19

Strategy: Solve maximin and Show Solution is equal to minimax

– For any finite, zero sum-game: – Minimax = Maximin = Nash Equilibrium (saddle point)

20

Sequential Hypothesis Testing

• Sequence of random variables X1,X2,...

– Honest sensors have X1,X2,...,Xi distributed as f0(X1,X2,...,Xi) (Defined by DP) – Tampered sensor has X1, X2,...,Xi distributed as f1(X1, X2,…,Xi) (note that f1 is unknown)

• Collect enough samples i until we have

enough information to make a decision!

– D=(N,dN) where N=stopping time, dN=decision

21

Sequential Probability Ratio Test (SPRT)

The solution of this problem is the SPRT:

22

time Undecided H1 H0 U L Sn

Sn = ln f1(x1, ..., xn) f0(x1, ...xn)