[PPT] - Lessons Learned from Evaluating the Robustness of Defenses to PowerPoint Presentation

SLIDE 1

Lessons Learned from Evaluating the Robustness of Defenses to Adversarial Examples

Nicholas Carlini Google Research

SLIDE 2

Lessons Learned from Evaluating the Robustness of Defenses to Adversarial Examples

SLIDE 3

Lessons Learned from Evaluating the Robustness of Defenses to Adversarial Examples

SLIDE 4

SLIDE 5

SLIDE 6

SLIDE 7

SLIDE 8

Why should we care about adversarial examples?

Make ML robust Make ML better

SLIDE 9

SLIDE 10

SLIDE 11

How do we generate adversarial examples?

SLIDE 12 Truck Dog Random Direction Random Direction

SLIDE 13 Dog Truck Airplane Random Direction Adversarial Direction Adversarial Direction Random Direction

SLIDE 14

SLIDE 15 ( (

SLIDE 16

SLIDE 17

Lessons Learned from Evaluating the Robustness of Defenses to Adversarial Examples

SLIDE 18

A defense is a neural network that

1. Is accurate on the test data
2. Resists adversarial examples

SLIDE 19

For example: Adversarial Training

Claim: Neural networks don't generalize

Madry, A., Makelov, A., Schmidt, L., Tsipras, D., & Vladu, A. Towards deep learning models resistant to adversarial attacks. ICLR 2018

SLIDE 20

F

Normal Training

( , ) ( , ) 7 3

Training

SLIDE 21

7 ( , ) ( , )

Adversarial Training (1)

( , ) 7 3

Attack

3 ( , )

SLIDE 22

7

G

Adversarial Training (2)

3 ( , ) ( , ) 7 3

Training

( , ) ( , )

SLIDE 23

Or: Thermometer Encoding

Claim: Neural networks are "overly linear"

Buckman, J., Roy, A., Raffel, C., & Goodfellow, I. Thermometer encoding: One hot way to resist adversarial examples. ICLR 2018

SLIDE 24

T(0.13) = 1 1 0 0 0 0 0 0 0 0 T(0.66) = 1 1 1 1 1 1 0 0 0 0 T(0.97) = 1 1 1 1 1 1 1 1 1 1 Solution

SLIDE 25

Or: Input Transformations

Claim: Perturbations are brittle

Guo, C., Rana, M., Cisse, M., & Van Der Maaten, L. Countering adversarial images using input transformations. ICLR 2018

SLIDE 26

Solution

Random Transform

SLIDE 27

Solution

JPEG Compress

SLIDE 28

SLIDE 29

Lessons Learned from Evaluating the Robustness of Defenses to Adversarial Examples

SLIDE 30

What does it meant to evaluate the robustness of a defense?

SLIDE 31 model = train_model(x_train, y_train) acc, loss = model.evaluate(  x_test, y_test)  if acc > 0.96: print("State-of-the-art")  else: print("Keep Tuning Hyperparameters")

Standard ML Pipeline

SLIDE 32 model = train_model(x_train, y_train) acc, loss = model.evaluate(  x_test, y_test)  if acc > 0.96: print("State-of-the-art")  else: print("Keep Tuning Hyperparameters")

Standard ML Pipeline

SLIDE 33 model = train_model(x_train, y_train) acc, loss = model.evaluate(  x_test, y_test)  if acc > 0.96: print("State-of-the-art")  else: print("Keep Tuning Hyperparameters")

Standard ML Pipeline

SLIDE 34 model = train_model(x_train, y_train) acc, loss = model.evaluate(  x_test, y_test)  if acc > 0.96: print("State-of-the-art")  else: print("Keep Tuning Hyperparameters")

Standard ML Evaluations

SLIDE 35 model = train_model(x_train, y_train) acc, loss = model.evaluate(  x_test, y_test)  if acc > 0.96: print("State-of-the-art")  else: print("Keep Tuning Hyperparameters")

Standard ML Evaluations

SLIDE 36

What are robustness evaluations?

SLIDE 37 model = train_model(x_train, y_train) acc, loss = model.evaluate(  x_test, y_test)  if acc > 0.96: print("State-of-the-art")  else: print("Keep Tuning Hyperparameters")

Standard ML Evaluations

SLIDE 38 model = train_model(x_train, y_train) acc, loss = model.evaluate(  A(x_test, model), y_test)  if acc > 0.96: print("State-of-the-art")  else: print("Keep Tuning Hyperparameters")

Adversarial ML Evaluations

SLIDE 39

How complete are evaluations?

SLIDE 40

Case Study: ICLR 2018

SLIDE 41

Serious effort  to evaluate  By space, most papers are ½ evaluation

SLIDE 42

We re-evalauted these defenses ...

SLIDE 43

2 7 4

Out of scope Broken Defenses Correct Defenses

SLIDE 44

2 7 4

Out of scope Broken Defenses Correct Defenses

SLIDE 45

2 7 4

Out of scope Broken Defenses Correct Defenses

SLIDE 46

So what did defenses do?

SLIDE 47

SLIDE 48

SLIDE 49

SLIDE 50

SLIDE 51

SLIDE 52

Lessons Learned from Evaluating the Robustness of Defenses to Adversarial Examples

SLIDE 53

SLIDE 54

SLIDE 55

Lessons (1 of 3) what types of defenses are effective

SLIDE 56

First class of effective defenses:

SLIDE 57

First class of effective defenses: Adversarial Training

SLIDE 58

SLIDE 59

SLIDE 60

SLIDE 61

Second class of effective defenses:

SLIDE 62

Second class of effective defenses: _______________

SLIDE 63

SLIDE 64

Lessons (2 of 3) what we've learned from evaluations

SLIDE 65

SLIDE 66

SLIDE 67

SLIDE 68

SLIDE 69

SLIDE 70

So how to attack it?

SLIDE 71

SLIDE 72

"Fixing" Gradient Descent

[0.1,  0.3, 0.0,  0.2, 0.4]

SLIDE 73

SLIDE 74

Lessons (3 of 3) performing better evaluations

SLIDE 75

SLIDE 76

SLIDE 77

Everything the following papers do is standard practice Actionable advice requires specific, concrete examples

SLIDE 78

Perform an adaptive attack

SLIDE 79

SLIDE 80

SLIDE 81

A "hold out" set is not an adaptive attack

SLIDE 82

Stop using FGSM (exclusively)

SLIDE 83

Use more than 100 (or 1000?) iteration of gradient descent

SLIDE 84

Iterative attacks should always do better than single step attacks.

SLIDE 85

Unbounded optimization attacks should eventually reach in 0% accuracy

SLIDE 86

Unbounded optimization attacks should eventually reach in 0% accuracy

SLIDE 87

Unbounded optimization attacks should eventually reach in 0% accuracy

SLIDE 88

Model accuracy should be monotonically decreasing

SLIDE 89

Model accuracy should be monotonically decreasing

SLIDE 90

✓

SLIDE 91

Evaluate against the worst attack

SLIDE 92

Plot accuracy vs distortion

SLIDE 93

Verify enough iterations

f gradient descent

SLIDE 94

Try gradient-free attack algorithms

SLIDE 95

Try random noise

SLIDE 96

SLIDE 97

The Future

SLIDE 98

SLIDE 99

The Year is 1997

SLIDE 100

SLIDE 101

Back to (the future)

SLIDE 102

SLIDE 103

SLIDE 104

Are we crypto in the 90's?

SLIDE 105

Maybe not. Two reasons.

SLIDE 106

Reason 1.

SLIDE 107

SLIDE 108