Obfuscated Gradients Give a False Sense of Security: Circumventing - - PowerPoint PPT Presentation

▶

Dec 06, 2022 40 likes •544 views

Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples Anish Athalye* 1 , Nicholas Carlini * 2 , and David Wagner 3 1 Massachusetts Institute of Technology 2 University of California, Berkeley (now

SLIDE 1

Obfuscated Gradients

Give a False Sense of Security:

Circumventing Defenses to Adversarial Examples Anish Athalye*1, Nicholas Carlini*2 , and David Wagner3 1 Massachusetts Institute of Technology 2 University of California, Berkeley (now Google Brain) 3 University of California, Berkeley

SLIDE 2

Advice on performing adversarial example defense evaluations

Or,

SLIDE 3

SLIDE 4 Definition 1: Inputs specifically crafted to fool a neural network. Definition 2: Given an input x, find an input x' that is misclassified such that |x-x'| < 𝜁 Correct definition. Hard to formalize. Adversarial Examples Not complete. Easy to formalize.

SLIDE 5 Adversarial Examples Definition 1 Defn. 2

SLIDE 6

13 total defense papers at ICLR'18 9 are white-box, non-certified 6 of these are broken   (~0% accuracy) 1 of these is partially broken

SLIDE 7

~50% of our paper is our attacks

SLIDE 8

~50% of our paper is our attacks

This talk is about the other 50%.

SLIDE 9

SLIDE 10

This Talk:

How should we evaluate  adversarial example defenses?

SLIDE 11

SLIDE 12

1. A precise threat model
2. A clear defense proposal
3. A thorough evaluation

SLIDE 13

SLIDE 14

A threat model is a formal statement defining when a system is intended to be secure.

1. Threat Model

SLIDE 15

1. Threat Model

What dataset is considered? Adversarial example definition? What does the attacker know? (model architecture? parameters?  training data? randomness?) If black-box: are queries allowed?

SLIDE 16 All Possible Adversaries Threat Model

SLIDE 17 All Possible Adversaries Threat Model

SLIDE 18 All Possible Adversaries          Threat Model

SLIDE 19

Good Threat Model: "Robust when L2 distortion is less than 5, given the attacker has white-box knowledge" Claim: 90% accuracy on ImageNet

SLIDE 20

SLIDE 21

2. Defense Proposal

Precise proposal of one specific defense    

(with code and models available)

SLIDE 22

SLIDE 23 A defense evaluation has one purpose, to answer: 

"Is the defense secure under the threat model?"

3. Defense Evaluation

SLIDE 24 acc, loss = model.evaluate(  Xtest, Ytest)

Is no longer sufficient.

3. Defense Evaluation

SLIDE 25

This step is why security is hard

3. Defense Evaluation

SLIDE 26

Serious effort  to evaluate  By space, most papers are ½ evaluation

SLIDE 27

Going through the motions is

insufficient

to evaluate a defense to adversarial examples

SLIDE 28

The purpose of a defense evaluation is NOT to show the defense is RIGHT

SLIDE 29

The purpose of a defense evaluation is to FAIL to show the defense is WRONG

SLIDE 30

SLIDE 31

Everything the following papers do is standard practice Actionable advice requires specific, concrete examples

SLIDE 32

Perform an adaptive attack

SLIDE 33

A "hold out" set is not an adaptive attack

SLIDE 34

Stop using FGSM (exclusively)

SLIDE 35

Use more than 100 (or 1000?) iteration of gradient descent

SLIDE 36

Iterative attacks should always do better than single step attacks.

SLIDE 37

Unbounded optimization attacks should eventually reach in 0% accuracy

SLIDE 38

Unbounded optimization attacks should eventually reach in 0% accuracy

SLIDE 39

Unbounded optimization attacks should eventually reach in 0% accuracy

SLIDE 40

Model accuracy should be monotonically decreasing

SLIDE 41

Model accuracy should be monotonically decreasing

SLIDE 42

SLIDE 43

✓

SLIDE 44

Evaluate against the worst attack

SLIDE 45

Plot accuracy vs distortion

SLIDE 46

Verify enough iterations

f gradient descent

SLIDE 47

Try gradient-free attack algorithms

SLIDE 48

SLIDE 49

Conclusion

The hardest part of a  defense is the evaluation

SLIDE 50

Anish: aathalye@mit.edu Me: nicholas@carlini.com

Please do reach out to us if you have any evaluation questions

Thank You