Wasserstein Adversarial Examples via Projected Sinkhorn Iterations - - PowerPoint PPT Presentation

Wasserstein Adversarial Examples via Projected Sinkhorn Iterations

ICML 19 Eric Wong 1 Frank R. Schmidt 2

• J. Zico Kolter 1

1Carnegie Mellon University 2Bosch Center for Artificial Intelligence

Presented by Kaiwen Wu August 2019

Kaiwen Wu Wasserstein Adversarial Examples August 2019 1 / 11

Contribution

Propose Wasserstein adversarial examples Develop an algorithm for Wasserstein projection

Kaiwen Wu Wasserstein Adversarial Examples August 2019 2 / 11

Wasserstein Distance for Images

Transportation between pixels: View an image x ∈ Rn as a histogram Euclidean distance between indices as the ground cost Normalize the image ⇒ balanced transport

Kaiwen Wu Wasserstein Adversarial Examples August 2019 3 / 11

Motivation

Wasserstein distance offers a different geometry

Wasserstein v.s. ℓ∞

Kaiwen Wu Wasserstein Adversarial Examples August 2019 4 / 11

Adversarial Example Generation

max

x′

ℓ(x′, y) s.t. x′ ∈ B(x, ǫ) Projected (normalized) gradient descent: xt+1 = ProjB(x,ǫ)

• xt + arg max

v≤α

v⊤∇ℓ(xt, y)

• ℓ2 ball: {z : z − x2 ≤ ǫ} ⇒ closed form

ℓ∞ ball: {z : z − x∞ ≤ ǫ} ⇒ closed form Wasserstein ball: {z : W(z, x) ≤ ǫ} ⇒ ???

Kaiwen Wu Wasserstein Adversarial Examples August 2019 5 / 11

Wasserstein Projection

Projecting w in to B(x, ǫ), min

z

w − z2

2

s.t. W(x, z) ≤ ǫ

Kaiwen Wu Wasserstein Adversarial Examples August 2019 6 / 11

Wasserstein Projection

Projecting w in to B(x, ǫ), min

z

w − z2

2

s.t. W(x, z) ≤ ǫ Equivalently, min

z,Π w − z2 2

s.t. Π1 = x, Π⊤1 = z Π, C ≤ ǫ

Kaiwen Wu Wasserstein Adversarial Examples August 2019 6 / 11

Acceleration: Entropic Regularization

Add an entropic regularization on Π min

z,Π

1 2w − z2

2 + 1

λ

• ij

Πij log Πij s.t. Π1 = x, Π⊤1 = z Π, C ≤ ǫ Dual form max

α∈Rn,β∈Rn,ψ∈R+ − 1

2λβ2

2 − ψǫ + α⊤x + β⊤w+

−

• ij

exp(αi)exp(−ψCij − 1)exp(βj)

Kaiwen Wu Wasserstein Adversarial Examples August 2019 7 / 11

Solving the Dual

max

α∈Rn,β∈Rn,ψ∈R+ − 1

2λβ2

2 − ψǫ + α⊤x + β⊤w+

−

• ij

exp(αi)exp(−ψCij − 1)exp(βj) Idea: block coordinate ascent on dual variables ⇒ a Sinkhorn-like algorithm Maximize α: ∂g

∂α = 0, closed form solution

Maximize β: ∂g

∂β = 0, closed form solution (using Lambert function)

Maximize ψ: Newton step ψ = ψ − t · 1 ∂2g/∂ψ2 · ∂g ∂ψ

Kaiwen Wu Wasserstein Adversarial Examples August 2019 8 / 11

Kaiwen Wu Wasserstein Adversarial Examples August 2019 9 / 11

Acceleration: Local Transport Plans

Kψ ∈ Rn×n, where (Kψ)ij = exp(−ψCij − 1) Multiple matrix-vector multiplications in each iteration, O(n2) Set Cij = ∞ ⇒ force Πij = 0 ⇒ Kψ is sparse max

α∈Rn,β∈Rn,ψ∈R+ − 1

2λβ2

2 − ψǫ + α⊤x + β⊤w+

−

• ❅

❅ ❅ ❅

n

• i=1

n

• j=1

exp(αi)exp(−ψCij − 1)exp(βj)

• Cij<∞

Only allow moving mass in k × k region Use convolutional filter to implement it

Kaiwen Wu Wasserstein Adversarial Examples August 2019 10 / 11

Experiment

Wasserstein adversarial training v.s. ℓ∞ robust training

Kaiwen Wu Wasserstein Adversarial Examples August 2019 11 / 11