Some thoughts on safety of machine learning Fabio Roli University - - PowerPoint PPT Presentation

Pattern Recognition and Applications Lab

University

• f Cagliari, Italy

Department of Electrical and Electronic Engineering

Some thoughts on safety

• f machine learning

Fabio Roli

HUML 2016, Venice, December 16th, 2016

http://pralab.diee.unica.it

The black cloud…

2

1915 - 2001

http://pralab.diee.unica.it

The black cloud

3

The novel tells the arrival of an enormous cloud of gas that appears to destroy the life on Earth by blocking the Sun's radiation. The motion of the cloud doesn’t follow physical laws, so one scientist argues that it might be a life-form with a degree of intelligence. Supposing that the cloud might be intelligent, the scientists try to communicate with it But human paradigms for information communication do not work !

“Could I put in this way? Said Kingsley. Between two absolutely identical individuals, no communication at all would be necessary because each individual would automatically know the experience of the other. …..Between two widely different individuals a vastly more complicated communication system is required”

• F. Hoyle, The black cloud, 1957 -

http://pralab.diee.unica.it

First issue

4

http://pralab.diee.unica.it 5

Is antropomorphism good ?

“But if cattle and horses and lions had hands

• r could paint with their hands and create works such as men do,

horses like horses and cattle like cattle also would depict the gods' shapes and make their bodies of such a sort as the form they themselves have” Xenophánes; 570 – 480 BC

Is antropomorphism good for the human use of machine learning ?

• Should we fabricate robots with human appearance ?
• Should we be disappointed if algorithms make errors that humans

wouldn’t have never done?

http://pralab.diee.unica.it

6

Vision of a humanoid robot

The iCub is the humanoid robot developed at the Italian Institute of Technology as part of the EU project RobotCub and subsequently adopted by more than 20 laboratories worldwide. It has 53 motors that move the head, arms and hands, waist, and legs. It can see and hear, it has the sense of proprioception (body configuration) and movement (using accelerometers and gyroscopes).

[http://www.icub.org]

The object recognition system of iCub uses visual features extracted with CNN models trained on the ImageNet dataset

[G. Pasquale et al. MLIS 2015]

http://pralab.diee.unica.it

7

[http://old.iit.it/projects/data-sets]

iCub object recognition: example images

http://pralab.diee.unica.it

8

The iCub object recognition pipeline

Credits: Lorenzo Natale, Visual Learning of Objects and Tools on the iCub Robot, 2015

http://pralab.diee.unica.it

9 9

Generation of adversarial noise against iCub

f(x) ≠ l

Plate Cup Adversarial Noise The adversarial image x + r is visually hard to distinguish from x

Disclaimer: unpublished work, work in progress at PRA Lab https://pralab.diee.unica.it

http://pralab.diee.unica.it

iCub is not a unique case…

10

http://pralab.diee.unica.it 11

[Szegedy et al., Intriguing properties of neural networks, 2014]

Black swans in ImageNet

http://pralab.diee.unica.it 12

Adversarial faces

• M. Sharif et al. developed a systematic method to automatically

generate attacks to face recognition systems, attacks which are realized through printing a pair of eyeglass frames. When worn by the attacker whose image is supplied to a state-of-the- art face-recognition algorithm, the eyeglasses allow her to evade being recognized or to impersonate another individual.

[M. Sharif et al., ACM CCS 2016]

http://pralab.diee.unica.it 13

Adversarial images in mission-critical apps…

[Patrick McDaniel et al., IEEE Security & Privacy, 2016]

http://pralab.diee.unica.it 14

Adversarial images in mission-critical apps…

[Patrick McDaniel et al., IEEE Security & Privacy, 2016]

To humans, adversarial images are indistinguishable from original images. Left an ordinary image of a stop sign Right an image manipulated with adversarial noise and classified as a yield sign

http://pralab.diee.unica.it 15

Big disappointment ?

http://pralab.diee.unica.it 16

Low probability blind spots

[M. Gori and F. Scarselli, PAMI 1998; B. Biggio et al., MCS 2015]

Adversarial inputs in low probability regions can evade easily the classifier

2−class classification −5 5 −5 5

Blue: legitimate class Red: illegal class Spam/Ham for example

http://pralab.diee.unica.it 17

Low probability blind spots

[M. Gori and F. Scarselli, PAMI 1998; B. Biggio et al., MCS 2015]

1.5C classification (MCS) −5 5 −5 5

Better enclosing legitimate data in feature space may improve classifier security

2−class classification −5 5 −5 5

http://pralab.diee.unica.it 18

Second issue

http://pralab.diee.unica.it 19

Can machine learning be safe?

• M. Barreno et al., Can machine learning be

secure? ASIACCS '06 Proceedings of the 2006 ACM Symposium on Information, computer and communications security

http://pralab.diee.unica.it 20

Android malware detection

About one billion of users

• f Android mobile
• perating system

Thousands of new Android malware samples every day

http://pralab.diee.unica.it 21

Benign Malware

w1 w2 w3 wn

• th

s s ≥ th s < th

...

DREBIN: Android malware detector

• D. Arp et al.

DREBIN: Effective and Explainable Detection of Android Malware in Your Pocket NDSS 2014 - The Network and Distributed System Security Symposium

http://pralab.diee.unica.it 22

Android.hardware.wifi

weights

SEND_SMS READ_SMS Android.hardware.wifi

weights

SEND_SMS READ_SMS CAMERA

Attacker’s goal: evasion The Attacker can evade easily the classifier by manipulating a few features if weights are sparse !

Attacking a linear classifier

http://pralab.diee.unica.it 23

Securing linear classifiers

We learn feature weights evenly-distributed by solving this optimization problem:

[A. Demontis et al., S+SSPR 2016]

There is a trade off between sparsity and security !

http://pralab.diee.unica.it 24

Third issue

http://pralab.diee.unica.it 25

Interpretability comes at a price ?

• A. Demontis et al.

On security and sparsity of linear classifiers for adversarial settings S+SSPR 2016

Supposed that sparsity can facilitate interpretability, there is a trade

• ff between interpretability and security in machine learning ?

http://pralab.diee.unica.it 26

People are disappointed, and worried, for such errors… Can we trust in algorithms as well as we trust in humans ?

Last issue

http://pralab.diee.unica.it 27

Algorithms are biased, but also humans are as well… When should you trust in humans and when in algorithms?

Trust in humans or machines ?

http://pralab.diee.unica.it 28

Biases of humans and machines

A bat and a ball together cost $ 1.10 The bat costs $ 1.0 more than the ball How much does the ball cost ?

http://pralab.diee.unica.it 29

Maybe we should reconsider again the distinction between two operation modes, System 1 and System 2 ? The discussion of the early days

• f machine learning about

symbolic and sub-symbolic processing ?

CoCo @ NIPS 2016 - Cognitive Computation: Integrating Neural and Symbolic Approaches

Back to the future ?

http://pralab.diee.unica.it

Thanks for listening ! Any questions ?

30

Engineering isn't about perfect solutions; it's about doing the best you can with limited resources (Randy Pausch, 1960-2008)