Robustness and geometry of deep neural networks Alhussein Fawzi - - PowerPoint PPT Presentation

Robustness and geometry of deep neural networks

Alhussein Fawzi DeepMind May 23rd 2019 The Mathematics of Deep Learning and Data Science University of Cambridge

1

Recent advances in machine learning

He et., al., “Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification” , 2015 Karpathy et., al, “Automated Image Captioning with ConvNets and Recurrent Nets” LeCun et. al.,, “Deep Learning” , 2015

Error rate (%)

DeepMind https://deepmind.com/research/alphago/

2

Robustness of classifiers to perturbations

In real world environments, images undergo perturbations.

3

Robustness of classifiers to perturbations

In real world environments, images undergo perturbations.

Lampshade

3

Robustness of classifiers to perturbations

In real world environments, images undergo perturbations.

Lampshade Perturbation

3

Robustness of classifiers to perturbations

In real world environments, images undergo perturbations.

Lampshade Perturbation

3

Robustness of classifiers to perturbations

In real world environments, images undergo perturbations.

Lampshade

f Perturbation

3

Robustness of classifiers to perturbations

In real world environments, images undergo perturbations.

Lampshade

f Lampshade? Perturbation

3

Robustness of classifiers to perturbations

In real world environments, images undergo perturbations.

Lampshade

f Lampshade? Perturbation

Broad range of perturbations

Adversarial perturbations

[Szegedy et. al. ICLR 2014], [Biggio et. al., PKDD 2013], ...

Random noise

[Fawzi et. al., NIPS 2016], [Franceschi et. al., AISTATS 2018]

Structured nuisances (geometric transformations

[Bruna et. al., TPAMI 2013], [Jaderberg et. al., NIPS 2015] , occlusions [Sharif et. al., CCS 2016] , etc...). 3

Robustness of classifiers to perturbations (Cont’d)

Safety of machine learning systems

?

4

Robustness of classifiers to perturbations (Cont’d)

Safety of machine learning systems

?

Better understanding of the geometry of state-of-the-art classifiers.

Class 1 Class 2

4

Talk outline

1 Fooling classifiers is easy: vulnerability to different perturbations. 2 Improving the robustness (i.e., “defending”) is difficult. 3 Geometric analysis of a successful defense: adversarial training. 5

Adversarial perturbations

State-of-the-art deep neural networks have been shown to be surprisingly unstable to adversarial perturbations.

6

Adversarial perturbations

State-of-the-art deep neural networks have been shown to be surprisingly unstable to adversarial perturbations.

School bus

6

Adversarial perturbations

State-of-the-art deep neural networks have been shown to be surprisingly unstable to adversarial perturbations.

School bus Ostrich

6

Adversarial perturbations

State-of-the-art deep neural networks have been shown to be surprisingly unstable to adversarial perturbations.

School bus Perturbation Ostrich

Figure from [Szegedy et. al., ICLR 2014]. 6

Adversarial perturbations

State-of-the-art deep neural networks have been shown to be surprisingly unstable to adversarial perturbations.

School bus Perturbation Ostrich

Figure from [Szegedy et. al., ICLR 2014].

Adversarial examples are found by seeking the minimal perturbation (in the ℓ2 sense) that switches the label of the classifier.

6

Adversarial perturbations

Robustness to adversarial noise

r∗( ) x

x

r∗(x) = min

r

r2 subject to f (x + r) = f (x).

7

Other types of adversarial perturbations

Universal perturbations [Moosavi-Dezfooli et. al., 2017]

Flagpole Joystick Chihuahua L a b r a d

• r

T e r r i e r Balloon

Geometric transformations [Fawzi et. al., 2015, Moosavi-Dezfooli et. al., 2018, Xiao et al., 2018]

8

Finding adversarial perturbations is easy...

http://robust.vision

9

... but designing defense mechanisms is hard!

Despite the huge number of proposed defenses, state-of-the-art classifiers are still vulnerable to small perturbations.

10

... but designing defense mechanisms is hard!

Despite the huge number of proposed defenses, state-of-the-art classifiers are still vulnerable to small perturbations.

10

... but designing defense mechanisms is hard!

Despite the huge number of proposed defenses, state-of-the-art classifiers are still vulnerable to small perturbations.

10

... but designing defense mechanisms is hard!

Despite the huge number of proposed defenses, state-of-the-art classifiers are still vulnerable to small perturbations.

10

Adversarial training

11

Adversarial training

Adversarial accuracy (CIFAR-10): 11

Adversarial training

Adversarial accuracy (CIFAR-10): [Madry et. al., 2017] 11

Adversarial training

Adversarial accuracy (CIFAR-10): [Madry et. al., 2017]

Adversarial training leads to state-of-the art robustness to adversarial perturbations.

11

Adversarial training

Adversarial accuracy (CIFAR-10): [Madry et. al., 2017]

Adversarial training leads to state-of-the art robustness to adversarial perturbations. But what does it actually do?

11

Decision boundaries

• Adv. direction

Random direction 12

Decision boundaries

Normal training

• Adv. direction

Random direction 12

Decision boundaries

Normal training Adversarial training

• Adv. direction

Random direction 12

Decision boundaries

Normal training Adversarial training

• Adv. direction

Random direction

After adversarial training, the decision boundaries are flatter and more regular.

12

Effect of adversarial training on loss landscape

Logit Label

13

Effect of adversarial training on loss landscape

Before adv. fine-tuning Logit Label

13

Effect of adversarial training on loss landscape

Before adv. fine-tuning After adv. fine-tuning Logit Label

13

Effect of adversarial training on loss landscape (Cont’d)

14

Quantitative analysis: curvature decrease with adversarial training

We compute the Hessian matrix at a test point x with respect to inputs. H = ∂2ℓ ∂xi∂xj

• The eigenvalues of H are the curvature of ℓ in the vicinity of x.

15

Quantitative analysis: curvature decrease with adversarial training

We compute the Hessian matrix at a test point x with respect to inputs. H = ∂2ℓ ∂xi∂xj

• The eigenvalues of H are the curvature of ℓ in the vicinity of x.
• 0.5

1.5 1.0 0.5 0.0 3000 2500 2000 1500 1000 500

Original Adversarial

Eigenvalue profile

Eigenvalue number Value 15

Quantitative analysis: curvature decrease with adversarial training

We compute the Hessian matrix at a test point x with respect to inputs. H = ∂2ℓ ∂xi∂xj

• The eigenvalues of H are the curvature of ℓ in the vicinity of x.
• 0.5

1.5 1.0 0.5 0.0 3000 2500 2000 1500 1000 500

Original Adversarial

Eigenvalue profile

Eigenvalue number Value 15

Relation between curvature and robustness

Locally quadratic approximation of the loss function

16

Relation between curvature and robustness

Locally quadratic approximation of the loss function We derive upper and lower bounds on the minimal perturbation required to fool a classifier.

16

Relation between curvature and robustness

Locally quadratic approximation of the loss function We derive upper and lower bounds on the minimal perturbation required to fool a classifier.

Threshold

16

Relation between curvature and robustness

Locally quadratic approximation of the loss function We derive upper and lower bounds on the minimal perturbation required to fool a classifier.

Threshold

16

Relation between curvature and robustness

Locally quadratic approximation of the loss function We derive upper and lower bounds on the minimal perturbation required to fool a classifier.

Threshold

16

Relation between curvature and robustness

Locally quadratic approximation of the loss function We derive upper and lower bounds on the minimal perturbation required to fool a classifier.

Threshold

16

Relation between curvature and robustness

Locally quadratic approximation of the loss function We derive upper and lower bounds on the minimal perturbation required to fool a classifier.

Threshold

Curvature

Upper bound Lower bound

Robustness

16

How important is the curvature decrease?

Is the curvature decrease the main effect of adversarial training leading to improved robustness?

17

How important is the curvature decrease?

Is the curvature decrease the main effect of adversarial training leading to improved robustness? → we regularize explicitly for the curvature.

17

How important is the curvature decrease?

Is the curvature decrease the main effect of adversarial training leading to improved robustness? → we regularize explicitly for the curvature. Idea: Regularize the norm of the Hessian of the loss wrt inputs.

17

How important is the curvature decrease?

Is the curvature decrease the main effect of adversarial training leading to improved robustness? → we regularize explicitly for the curvature. Idea: Regularize the norm of the Hessian of the loss wrt inputs. Use Hutichinson’s estimator HF =

• E

z∼N(0,I)

Hz2

2

17

How important is the curvature decrease?

Is the curvature decrease the main effect of adversarial training leading to improved robustness? → we regularize explicitly for the curvature. Idea: Regularize the norm of the Hessian of the loss wrt inputs. Use Hutichinson’s estimator HF =

• E

z∼N(0,I)

Hz2

2

In practice:

Compute Hessian-vector products with finite difference Selective sampling on directions corresponding to high curvature

17

How important is the curvature decrease?

Is the curvature decrease the main effect of adversarial training leading to improved robustness? → we regularize explicitly for the curvature. Idea: Regularize the norm of the Hessian of the loss wrt inputs. Use Hutichinson’s estimator HF =

• E

z∼N(0,I)

Hz2

2

In practice:

Compute Hessian-vector products with finite difference Selective sampling on directions corresponding to high curvature

CURE: Regularize using ℓr = ∇ℓ(x + hz) − ∇ℓ(x)

17

CUREing deep networks trained on CIFAR-10

Accuracy on clean samples: Adversarial accuracy: 18

CUREing deep networks trained on CIFAR-10

Accuracy on clean samples: Adversarial accuracy: 18

CUREing deep networks trained on CIFAR-10

Accuracy on clean samples: Adversarial accuracy: 18

CUREing deep networks trained on CIFAR-10

Accuracy on clean samples: Adversarial accuracy: 18

CUREing deep networks trained on CIFAR-10

Accuracy on clean samples: Adversarial accuracy:

[Moosavi-Dezfooli, et. al., Robustness via curvature regularization, and vice-versa, CVPR 2019]

18

Upper limits on adversarial robustness

Goal: examine the existence of upper bounds on the robustness to adversarial perturbations. Relate to quantities that we better understand/we can better measure: robustness to random noise. Comparison to the robustness to random noise quantifies the power

• f an adversary having access to the model vs. no clue about the

model.

19

From adversarial to random noise

Robustness to adversarial noise

r∗( ) x

x

min

r

r2 subject to f (x + r) = f (x).

20

From adversarial to random noise

Robustness to random noise

x

min

t

|t| subject to f (x + tv) = f (x) v uniformly sampled from SD−1.

20

Linear classifiers

Theorem (Fawzi et. al., NIPS ’16, Franceschi et. al., AISTATS ’18)

For affine classifiers, we have r∗2 = Θ 1 √ D r∗

rand2

• ,

with high probability (over the choice of random perturbation). In high dimensions, very large gap between both robustness measures. When data is bounded, we therefore typically get r∗2 = O

• 1

√ D

• →

Achieving robustness in high dimensions is difficult!

21

Intuition

Decision boundary

r∗(x) x

22

Intuition

Decision boundary

r∗(x) x

22

Intuition

Decision boundary

r∗(x) x

22

Intuition

Decision boundary

r∗(x) x

22

Intuition

Decision boundary

r∗(x) x √ d r∗(x) 2

22

Robustness and geometry of deep networks: SPM’17

23

Conclusions

Very simple strategies to fool classifiers: adversarial, geometric, universal perturbations, ... Difficulty of defending against adversarial perturbations; most “defenses” make the classifier robust against specific directions. Adversarial training is essentially reducing the curvature of the loss function, leading to an increased robustness.

24

References

Szegedy et. al., Intriguing properties of neural networks, ICLR 2014 Fawzi et. al, Manitest: Are classifiers really invariant?, BMVC 2015 Moosavi-Dezfooli et. al., Deepfool: a simple and accurate approach to fool deep neural networks, CVPR 2016 Fawzi et. al., Robustness of classifiers: from adversarial to random noise, NIPS 2016 Moosavi-Dezfooli et. al., Universal adversarial perturbations, CVPR 2017 Uesato et al., Adversarial risk and the dangers of evaluating against weak attacks, arXiv 2018 Fawzi et. al., Adversarial vulnerability of any classifier, NeurIPS 2018 Moosavi-Dezfooli et. al., Robustness via curvature regularization, and vice-versa, CVPR 2019 Survey: [Fawzi et. al., Robustness of deep networks: a geometric perspective, IEEE Signal Processing Magazine 2017]

25