Stronger and Faster Wasserstein Adversarial Attacks Kaiwen Wu - - PowerPoint PPT Presentation

Stronger and Faster Wasserstein Adversarial Attacks

Kaiwen Wu kaiwen.wu@uwaterloo.ca Joint work with Allen Wang and Yaoliang Yu

K.Wu, A.Wang and Y.Yu Wasserstein Adversarial Attacks July 29, 2020 1 / 18

Adversarial Examples

Adversarial examples:

(Goodfellow et al. 2015)

Generating adversarial examples: maximize

xadv

ℓ(f (xadv), y) subject to xadv ≈ x

K.Wu, A.Wang and Y.Yu Wasserstein Adversarial Attacks July 29, 2020 2 / 18

How “Similar” Is Similar?

How to quantify xadv ≈ x? x − xadvp ≤ ǫ (Szegedy et al. 2014) point-wise function (Laidlaw et al. 2019) geometric transformation (Engstrom et al. 2019) Wasserstein distance (Wong et al. 2019) ... Our contributions stronger and faster Wasserstein adversarial attacks higher robust accuracy using adversarial training

K.Wu, A.Wang and Y.Yu Wasserstein Adversarial Attacks July 29, 2020 3 / 18

What is Wasserstein Distance?

W(x, z) = min

Π≥0 Π, C s.t. Π1 = x, Π⊤1 = z

x ∈ Rn and z ∈ Rn: input images Π ∈ Rn×n: transportation matrix C ∈ Rn×n: transportation cost x z cost Πij × Cij

K.Wu, A.Wang and Y.Yu Wasserstein Adversarial Attacks July 29, 2020 4 / 18

Applications across Different Domains

(Arjovsky et al. 2017; Rabin et al. 2014; Solomon et al. 2015)

K.Wu, A.Wang and Y.Yu Wasserstein Adversarial Attacks July 29, 2020 5 / 18

Why Wasserstein Distance?

Captures geometry in image space, e.g. translation, rotation

ǫ = 0.05 ǫ = 0.10 ǫ = 0.20 ǫ = 0.40 ǫ = 0.50 ǫ = 1.00 ǫ = 2.00 ǫ = 4.00 ǫ = 0.05 ǫ = 0.10 ǫ = 0.20 ǫ = 0.40

ℓ∞ ℓ2

Wasserstein

predict: 4 predict: 9

Why Wasserstein Distance?

Captures geometry in image space, e.g. translation, rotation

ǫ = 0.05 ǫ = 0.10 ǫ = 0.20 ǫ = 0.40 ǫ = 0.50 ǫ = 1.00 ǫ = 2.00 ǫ = 4.00 ǫ = 0.05 ǫ = 0.10 ǫ = 0.20 ǫ = 0.40

ℓ∞ ℓ2

Wasserstein

predict: 4 predict: 9

K.Wu, A.Wang and Y.Yu Wasserstein Adversarial Attacks July 29, 2020 6 / 18

Computing Wasserstein Adversarial Examples

Search for adversarial examples: maximize

xadv

ℓ(xadv) subject to W (x, xadv) ≤ ǫ

K.Wu, A.Wang and Y.Yu Wasserstein Adversarial Attacks July 29, 2020 7 / 18

Computing Wasserstein Adversarial Examples

Search for adversarial examples: maximize

xadv

ℓ(xadv) subject to W (x, xadv) ≤ ǫ Alternatively, search for transportation matrix: maximize

Π≥0

ℓ(Π⊤1) subject to Π1 = x, Π, C ≤ ǫ Then, recover adversarial examples: xadv = Π⊤1

K.Wu, A.Wang and Y.Yu Wasserstein Adversarial Attacks July 29, 2020 7 / 18

Optimization in Transportation Matrix

K.Wu, A.Wang and Y.Yu Wasserstein Adversarial Attacks July 29, 2020 8 / 18

Optimization in Transportation Matrix

ǫ ∇Πℓ (Π)

(a) projected gradient

minimize

Π≥0 1 2Π − G2 F

subject to Π1 = x, Π, C ≤ ǫ

K.Wu, A.Wang and Y.Yu Wasserstein Adversarial Attacks July 29, 2020 8 / 18

Optimization in Transportation Matrix

ǫ ∇Πℓ (Π)

(a) projected gradient

minimize

Π≥0 1 2Π − G2 F

subject to Π1 = x, Π, C ≤ ǫ

(b) Frank-Wolfe (Jaggi 2011)

minimize

Π≥0

Π, H subject to Π1 = x, Π, C ≤ ǫ

K.Wu, A.Wang and Y.Yu Wasserstein Adversarial Attacks July 29, 2020 8 / 18

Optimization in Transportation Matrix

ǫ ∇Πℓ (Π)

(a) projected gradient

minimize

Π≥0 1 2Π − G2 F

subject to Π1 = x, Π, C ≤ ǫ

(b) Frank-Wolfe (Jaggi 2011)

minimize

Π≥0

Π, H subject to Π1 = x, Π, C ≤ ǫ For n dimensional images, Π has n2 variables...

K.Wu, A.Wang and Y.Yu Wasserstein Adversarial Attacks July 29, 2020 8 / 18

Solve Projection in PGD

minimize

Π≥0 1 2Π − G2 F

subject to Π1 = x, Π, C ≤ ǫ

K.Wu, A.Wang and Y.Yu Wasserstein Adversarial Attacks July 29, 2020 9 / 18

Solve Projection in PGD

minimize

Π≥0 1 2Π − G2 F

subject to Π1 = x, Π, C ≤ ǫ The Lagrange dual can be simplified as a univariate problem maximize

λ≥0

g(λ)

K.Wu, A.Wang and Y.Yu Wasserstein Adversarial Attacks July 29, 2020 9 / 18

Solve Projection in PGD

minimize

Π≥0 1 2Π − G2 F

subject to Π1 = x, Π, C ≤ ǫ The Lagrange dual can be simplified as a univariate problem maximize

λ≥0

g(λ) No closed-form expression...

K.Wu, A.Wang and Y.Yu Wasserstein Adversarial Attacks July 29, 2020 9 / 18

Solve Projection in PGD

minimize

Π≥0 1 2Π − G2 F

subject to Π1 = x, Π, C ≤ ǫ The Lagrange dual can be simplified as a univariate problem maximize

λ≥0

g(λ) No closed-form expression... But g′(λ) can be evaluated in O(n2 log n) time

Proposition

0 ≤ λ⋆ ≤ 2 vec(G)∞ + x∞ mini=j{Cij}

K.Wu, A.Wang and Y.Yu Wasserstein Adversarial Attacks July 29, 2020 9 / 18

Bisection on the Dual

maximize

λ≥0

g(λ) Converge to high precision ≤ 20 iterations in practice. λ g(λ)

Bisection on the Dual

maximize

λ≥0

g(λ) Converge to high precision ≤ 20 iterations in practice. λ g(λ) λ⋆

Bisection on the Dual

maximize

λ≥0

g(λ) Converge to high precision ≤ 20 iterations in practice. λ g(λ) λ⋆

2vec(G)∞+x∞ mini=j{Cij}

Bisection on the Dual

maximize

λ≥0

g(λ) Converge to high precision ≤ 20 iterations in practice. λ g(λ) λ⋆

2vec(G)∞+x∞ mini=j{Cij}

K.Wu, A.Wang and Y.Yu Wasserstein Adversarial Attacks July 29, 2020 10 / 18

Solve Linear Minimization in Frank-Wolfe

minimize

Π≥0

Π, H subject to Π1 = x, Π, C ≤ ǫ

K.Wu, A.Wang and Y.Yu Wasserstein Adversarial Attacks July 29, 2020 11 / 18

Solve Linear Minimization in Frank-Wolfe

minimize

Π≥0

Π, H subject to Π1 = x, Π, C ≤ ǫ The Lagrange dual can be simplified as a univariate problem maximize

λ≥0

g(λ)

K.Wu, A.Wang and Y.Yu Wasserstein Adversarial Attacks July 29, 2020 11 / 18

Solve Linear Minimization in Frank-Wolfe

minimize

Π≥0

Π, H subject to Π1 = x, Π, C ≤ ǫ The Lagrange dual can be simplified as a univariate problem maximize

λ≥0

g(λ) Bound on the optimum: 0 ≤ λ⋆ ≤ 2vec(H)∞

mini=j{Cij}

K.Wu, A.Wang and Y.Yu Wasserstein Adversarial Attacks July 29, 2020 11 / 18

Solve Linear Minimization in Frank-Wolfe

minimize

Π≥0

Π, H subject to Π1 = x, Π, C ≤ ǫ The Lagrange dual can be simplified as a univariate problem maximize

λ≥0

g(λ) Bound on the optimum: 0 ≤ λ⋆ ≤ 2vec(H)∞

mini=j{Cij}

Does not work...

◮ difficult to recover primal solution ◮ severe numerical instability K.Wu, A.Wang and Y.Yu Wasserstein Adversarial Attacks July 29, 2020 11 / 18

Entropic Regularization

minimize

Π≥0

Π, H + γ

n

• i=1

n

• j=1

Πij log Πij subject to Π1 = x, Π, C ≤ ǫ

K.Wu, A.Wang and Y.Yu Wasserstein Adversarial Attacks July 29, 2020 12 / 18

Entropic Regularization

minimize

Π≥0

Π, H + γ

n

• i=1

n

• j=1

Πij log Πij subject to Π1 = x, Π, C ≤ ǫ Closed-form expression to recover primal solution

K.Wu, A.Wang and Y.Yu Wasserstein Adversarial Attacks July 29, 2020 12 / 18

Entropic Regularization

minimize

Π≥0

Π, H + γ

n

• i=1

n

• j=1

Πij log Πij subject to Π1 = x, Π, C ≤ ǫ Closed-form expression to recover primal solution Entropic regularization introduces approximation error

K.Wu, A.Wang and Y.Yu Wasserstein Adversarial Attacks July 29, 2020 12 / 18

Entropic Regularization

minimize

Π≥0

Π, H + γ

n

• i=1

n

• j=1

Πij log Πij subject to Π1 = x, Π, C ≤ ǫ Closed-form expression to recover primal solution Entropic regularization introduces approximation error But the approximation error is guaranteed to be small

K.Wu, A.Wang and Y.Yu Wasserstein Adversarial Attacks July 29, 2020 12 / 18

Exploit Sparsity

Local transportation constraint (Wong et al. 2019) ⇒ structured sparsity in Π Per iteration cost is reduced to O(n) by exploiting sparsity

K.Wu, A.Wang and Y.Yu Wasserstein Adversarial Attacks July 29, 2020 13 / 18

Comparison

ǫ = 0.001 0.002 0.003 0.004 0.005 20 40 60 80 adversarial accuracy on CIFAR-10 (standard training) Wong et al. (2019) Dual Proj.(ours) Dual LMO(ours)

K.Wu, A.Wang and Y.Yu Wasserstein Adversarial Attacks July 29, 2020 14 / 18

Comparison

ǫ = 0.001 0.002 0.003 0.004 0.005 20 40 60 80 adversarial accuracy on CIFAR-10 (standard training) Wong et al. (2019) Dual Proj.(ours) Dual LMO(ours) 20 40 60 80 iterations 10 20 time per iteration in ms

K.Wu, A.Wang and Y.Yu Wasserstein Adversarial Attacks July 29, 2020 14 / 18

Entropic Regularization Reflects Shapes

Entropic Regularization Reflects Shapes

Entropic Regularization Reflects Shapes

K.Wu, A.Wang and Y.Yu Wasserstein Adversarial Attacks July 29, 2020 15 / 18

Scalable to High Dimensional Data

K.Wu, A.Wang and Y.Yu Wasserstein Adversarial Attacks July 29, 2020 16 / 18

Improved Adversarial Training

Stronger attacks improve adversarial training! ǫ = 0.001 0.002 0.003 0.004 0.005 20 40 60 80 adversarial accuracy of models on CIFAR-10 (adversarial training) Wong et al. (2019) FW + dual LMO (ours)

K.Wu, A.Wang and Y.Yu Wasserstein Adversarial Attacks July 29, 2020 17 / 18

Summary

PGD and Frank-Wolfe complement each other nicely PGD with dual projection is the strongest attack Frank-Wolfe with dual LMO is the fastest attack Improved adversarial training Applicable to any Wasserstein constrained optimization

K.Wu, A.Wang and Y.Yu Wasserstein Adversarial Attacks July 29, 2020 18 / 18