Wasserstein Adversarial Examples via Projected Sinkhorn Iterations - PowerPoint PPT Presentation

Wasserstein Adversarial Examples via Projected Sinkhorn Iterations ICML 19 Eric Wong 1 Frank R. Schmidt 2 J. Zico Kolter 1 1 Carnegie Mellon University 2 Bosch Center for Artificial Intelligence Presented by Kaiwen Wu August 2019 Kaiwen Wu Wasserstein Adversarial Examples August 2019 1 / 11

Contribution Propose Wasserstein adversarial examples Develop an algorithm for Wasserstein projection Kaiwen Wu Wasserstein Adversarial Examples August 2019 2 / 11

Wasserstein Distance for Images Transportation between pixels: View an image x ∈ R n as a histogram Euclidean distance between indices as the ground cost Normalize the image ⇒ balanced transport Kaiwen Wu Wasserstein Adversarial Examples August 2019 3 / 11

Motivation Wasserstein distance offers a different geometry Wasserstein v.s. ℓ ∞ Kaiwen Wu Wasserstein Adversarial Examples August 2019 4 / 11

Adversarial Example Generation ℓ ( x ′ , y ) max x ′ s . t . x ′ ∈ B ( x , ǫ ) Projected (normalized) gradient descent: � � x t +1 = Proj B ( x ,ǫ ) x t + arg max v ⊤ ∇ ℓ ( x t , y ) � v �≤ α ℓ 2 ball: { z : � z − x � 2 ≤ ǫ } ⇒ closed form ℓ ∞ ball: { z : � z − x � ∞ ≤ ǫ } ⇒ closed form Wasserstein ball: { z : W ( z , x ) ≤ ǫ } ⇒ ??? Kaiwen Wu Wasserstein Adversarial Examples August 2019 5 / 11

Wasserstein Projection Projecting w in to B ( x , ǫ ), � w − z � 2 min 2 z s . t . W ( x , z ) ≤ ǫ Kaiwen Wu Wasserstein Adversarial Examples August 2019 6 / 11

Wasserstein Projection Projecting w in to B ( x , ǫ ), � w − z � 2 min 2 z s . t . W ( x , z ) ≤ ǫ Equivalently, z , Π � w − z � 2 min 2 s . t . Π1 = x , Π ⊤ 1 = z � Π , C � ≤ ǫ Kaiwen Wu Wasserstein Adversarial Examples August 2019 6 / 11

Acceleration: Entropic Regularization Add an entropic regularization on Π 1 2 + 1 2 � w − z � 2 � min Π ij log Π ij λ z , Π ij s . t . Π1 = x , Π ⊤ 1 = z � Π , C � ≤ ǫ Dual form α ∈ R n ,β ∈ R n ,ψ ∈ R + − 1 2 λ � β � 2 2 − ψǫ + α ⊤ x + β ⊤ w + max � − exp ( α i ) exp ( − ψ C ij − 1) exp ( β j ) ij Kaiwen Wu Wasserstein Adversarial Examples August 2019 7 / 11

Solving the Dual α ∈ R n ,β ∈ R n ,ψ ∈ R + − 1 2 λ � β � 2 2 − ψǫ + α ⊤ x + β ⊤ w + max � − exp ( α i ) exp ( − ψ C ij − 1) exp ( β j ) ij Idea: block coordinate ascent on dual variables ⇒ a Sinkhorn-like algorithm Maximize α : ∂ g ∂α = 0, closed form solution Maximize β : ∂ g ∂β = 0, closed form solution (using Lambert function) Maximize ψ : Newton step ∂ 2 g /∂ψ 2 · ∂ g 1 ψ = ψ − t · ∂ψ Kaiwen Wu Wasserstein Adversarial Examples August 2019 8 / 11

Kaiwen Wu Wasserstein Adversarial Examples August 2019 9 / 11

Acceleration: Local Transport Plans K ψ ∈ R n × n , where ( K ψ ) ij = exp( − ψ C ij − 1) Multiple matrix-vector multiplications in each iteration, O ( n 2 ) Set C ij = ∞ ⇒ force Π ij = 0 ⇒ K ψ is sparse α ∈ R n ,β ∈ R n ,ψ ∈ R + − 1 2 λ � β � 2 2 − ψǫ + α ⊤ x + β ⊤ w + max ❅ � n n � � � ❅ − exp ( α i ) exp ( − ψ C ij − 1) exp ( β j ) � ❅ � i =1 j =1 ❅ � C ij < ∞ Only allow moving mass in k × k region Use convolutional filter to implement it Kaiwen Wu Wasserstein Adversarial Examples August 2019 10 / 11

Experiment Wasserstein adversarial training v.s. ℓ ∞ robust training Kaiwen Wu Wasserstein Adversarial Examples August 2019 11 / 11