Data Poisoning Attack cks on Stoch chastic c Bandits Fang Liu - - PowerPoint PPT Presentation

Data Poisoning Attack cks on Stoch chastic c Bandits

Fang Liu and Ness Shroff

• Background

qWhat are bandits? qMotivations

• Data poisoning attacks on stochastic bandits

qOffline model qOnline model qSimulation results

• Conclusions and discussions

Outline

• Repeated game between an agent and an environment

What are bandits?

reward agent environment action Reinforcement learning

x

Online learning MAB

x

Stochastic

• Model

§ At each (discrete) time t, the agent plays action At from a set of K actions § The agent receives reward , drawn from unknown distribution At

What are bandits?

YAt,t

• Regret lower bounds

§ Problem-dependent: where is expected reward § Problem-independent:

• Performance measure

§ Regret(loss) § Minimize regret = maximize total reward

• Popular algorithms

§ Upper Confidence Bounds (UCB), Thompson Sampling, epsilon-greedy

R(T) = E " max

i∈[K] T

X

t=1

Yi,t −

T

X

t=1

YAt,t #

Ω X

i

µ∗ − µi KL(µa, µ∗) log T !

µi

Ω ⇣√ KT ⌘

• Adversarial learning is well studied in deep learning
• How robust are bandits?
• Many applications

§ Clinical trials § Recommendation systems § Ad placement § A/B test § A component of game-playing algorithms (MCTS), e.g. AlphaGo § Resource allocation

• If under stealthy attack, hard to detect (due to limited feedback)

Motivations

Offline model

• Distributed system
• Algorithm updates in batches

§ Yahoo! Front Page (daily)

• Attacker manipulates the

rewards by adding

• Target arm , sub-optimal
• Goal: bandit plays with

high prob. at T+1

• Cost:

bandit environment attacker bandit algorithm

at

data buffer

• f size T

(at, rt)

{~ ✏a}

rt

a∗

a∗

✏t

C(T)2 =

T

X

t=1

✏2

t =

X

a∈A

||~ ✏a||2

2.

1 − δ

Offline model: epsilon greedy algorithm

• Optimal para:
• Post-attack empirical mean:
• Attack error tolerance:
• Quadratic program with linear constraints

at = ( A, αt arg maxa∈A ˜ µa(t − 1), .

αt = Θ(1/t)

˜ µa(t)

δ = K − 1 K αT +1

P1 : min

~ ✏a:a∈A

X

a∈A

||~ ✏a||2

2

s.t. ˜ µa∗(T) ˜ µa(T) + ⇠, 8a 6= a∗

Offline model: UCB algorithm

• Attack error tolerance:
• Conditional “deterministic” algorithm
• Quadratic program with linear constraints

P2 : min

~ ✏a:a∈A

X

a∈A

||~ ✏a||2

2

s.t. ua∗(T + 1) ua(T + 1) + ⇠, 8a 6= a∗

at = arg max

a∈A ua(t) := ˜

µa(t − 1) + 3σ s log t Na(t − 1).

δ = 0

Offline model: Thompson Sampling

• Bayesian algorithm: prior-posterior, prob. matching
• Quadratic program with convex constraints

at = arg max

a∈A θa(t) ∼ N(˜

µa(t − 1)/σ2, σ2/Na(t − 1))

P3 : min

~ ✏a:a2A

X

a2A

||~ ✏a||2

2

s.t. X

a6=a∗

Φ ˜ µa(T) ˜ µa∗(T) 3p 1/ma + 1/ma∗ !  ˜ µa(T) ˜ µa∗(T)  0, 8a 6= a⇤

Online model

• Algorithm updates online
• Attacker manipulates the

rewards by adding

• Target arm , sub-optimal
• Goal: bandit plays in

with high prob.

• Cost:

rt

a∗

a∗

✏t

1 − δ

bandit environment attacker bandit algorithm

at

rt

r0

t

✏t

Θ(T)

C(T) =

T

X

t=1

|✏t|

Online model: Oracle attacks

• Attack against any bandit algorithm
• Not practical: unknown expectations

✏t = {at 6= a∗}[µat µa∗ + ⇠]+

O(log T) ξ > 0 E[Na∗(T)] = T − o(T) O(P

i6=a∗[µi − µa∗ + ξ]+ log T)

Adaptive attacks by constant Estimation (ACE)

• where is decreasing in n
• Pre-attack empirical mean:
• Attack against any bandit algorithm
• Adaptive and efficient: estimation
• How: concentration inequality + union bound

✏t = {at 6= a∗}[ˆ µat(t) ˆ µa∗(t) + (Nat(t)) + (Na∗(t))]+

β(n) = r 2σ2 n log π2Kn2 3δ

ˆ µa(t)

δ ∈ (0, 1) P(E) > 1 − δ E = {∀a ∈ A, ∀t : |ˆ µa(t) − µa| < β(Na(t))}.

Online model: ACE attacks

✏t = {at 6= a∗}[ˆ µat(t) ˆ µa∗(t) + (Nat(t)) + (Na∗(t))]+

∈ (0, 0.5) O(log T) 1 − 1 − 2 a⇤ Na∗(T) Na∗(T) = T − o(T)

T

X

t=1

|✏t| ≤ O @ X

a6=a∗

• [µa − µa∗]+ + 4(1)
• log T

1 A .

• Tight to oracle attack (with some additive constant)

Simulation results: offline model

• Gaussian distributions with random drawn expectations
• Parameters:
• Poisoning effort ratio:

K = 5, σ = 0.1, T = 1000, δ = 0.05

||~ ✏||2 ||~ r||2 = sP

a∈A ||~

✏a||2

2

P

a∈A ||~

ra||2

2

Poisoning effort ratio

0.02 0.04 0.06 0.08 0.1

Number of trials

20 40 60 80 100 120 140 160 180

Success rate =100%

Poisoning effort ratio

0.005 0.01 0.015 0.02 0.025

Number of trials

20 40 60 80 100 120

Success rate =100%

Poisoning effort ratio

0.015 0.02 0.025 0.03 0.035 0.04 0.045 0.05

Number of trials

20 40 60 80 100 120 140

Success rate =100%

Epsilon-greedy

UCB Thompson Sampling

Simulation results: online model

• Gaussian distributions with random drawn expectations
• Parameters:
• 3 cases:
• Jun’s attack is optimized if the bandit algo. is known (esp. deterministic).

Epsilon-greedy

UCB Thompson Sampling

Time

100 101 102 103 104 105

Cost

1 2 3 4 5 6 7 8 9 10

ACE,∆=1 Jun,∆=1 ACE,∆=0.5 Jun,∆=0.5 ACE,∆=0.2 Jun,∆=0.2

Time

100 101 102 103 104 105

Cost

50 100 150 200 250

ACE,∆=1 Jun,∆=1 ACE,∆=0.5 Jun,∆=0.5 ACE,∆=0.2 Jun,∆=0.2

Time

100 101 102 103 104 105

Cost

1 2 3 4 5 6

ACE,∆=1 ACE,∆=0.5 ACE,∆=0.2

K = 2, σ = 0.1, T = 105, δ = 0.05

µ1 = ∆, µ2 = 0

• Negative results: bandits are vulnerable!

§ Algorithm-specific attacks on 3 popular bandits in offline model § Adaptive attacks on any bandit in online model

• Any hope to build a robust world?
• Crack the model

§ Encrypt decision § Replicate reward records

• Detect by distribution outlier detection

Conclusions and discussions

Thanks!

End