[PPT] - EMBC Tutorial on Interpretable and Transparent Deep Learning PowerPoint Presentation

SLIDE 1

1 / 5 4

EMBC Tutorial on Interpretable and Transparent Deep Learning

Wojciech Samek (Fraunhofer HHI) Grégoire Montavon (TU Berlin) Klaus-Robert Müller (TU Berlin) 13:30 - 14:00 Introduction KRM 14:00 - 15:00 T echniques for Interpretability GM 15:00 - 15:30 Cofgee Break ALL 15:30 - 16:15 Evaluating Interpretability & Applications WS 16:15 - 17:15 Applications in BME & the Sciences and Wrap-Up KRM

SLIDE 2

2 / 5 4

N a r r

w

i n g t h e C

n

c e p t

f

E x p l a n a t i

n

SLIDE 3

3 / 5 4

E x p l a i n i n g M L M

d

e l s : T w

V

i e w s

me c h a n i s t i c u n d e r s t a n d i n g f u n c t i

n

a l u n d e r s t a n d i n g

U n d e r s t a n d i n g w h a t m e c h a n i s m t h e n e t w

r

k u s e s t

s
l

v e a p r

b

l e m

r

i m p l e m e n t a f u n c t i

n

. U n d e r s t a n d i n g h

w

t h e n e t w

r

k s r e l a t e s t h e i n p u t t

t

h e

u

t p u t v a r i a b l e s .

SLIDE 4

4 / 5 4

E x p l a i n i n g M L M

d

e l s : T w

P

r

b

l e ms

mo d e l a n a l y s i s d e c i s i

n

a n a l y s i s

p

s

s i b l e a p p r

a

c h

b

u i l d p r

t
t

y p e s

f

" t y p i c a l " e x a m p l e s

f

a c e r t a i n c l a s s . p

s

s i b l e a p p r

a

c h

i

d e n t i f y w h i c h i n p u t v a r i a b l e s c

n

t r i b u t e t

t

h e p r e d i c t i

n

.

SLIDE 5

5 / 5 4

E x p l a i n i n g M L M

d

e l s : T w

P

r

b

l e ms

M

d

e l A n a l y s i s “ w h a t d

e

s s

m

e t h i n g p r e d i c t e d a s a p

l

t a b l e t y p i c a l l y l

k

l i k e . “ D e c i s i

n

A n a l y s i s “ w h y a g i v e n i m a g e i s c l a s s i fi e d a s a p

l

t a b l e ”

m

d

e l ’ s p r

t
t

y p i c a l p

l

t a b l e s

m

e p

l

t a b l e w h y i t i s c l a s s i fi e d a s a p

l

t a b l e

SLIDE 6

6 / 5 4

A S u r v e y

f

E x p l a n a t i

n

T e c h n i q u e s

SLIDE 7

7 / 5 4

O v e r v i e w

f

E x p l a n a t i

n

M e t h

d

s

1 . P e r t u r b a t i

n
B

a s e d Me t h

d

s 2 . Me a n i n g f u l P e r t u r b a t i

n

s 3 . S i m p l e T a y l

r

E x p a n s i

n

4 . G r a d i e n t x I n p u t 4 . L a y e r

Wi

s e R e l e v a n c e P r

p

a g a t i

n

( L R P )

SLIDE 8

8 / 5 4

A p p r

a

c h 1 : P e r t u r b a t i

n

D N N D N N D N N

castle … still a castle not a castle Idea: Assess features relevance by testing the model response to their removal or perturbation.

SLIDE 9

9 / 5 4

A p p r

a

c h 1 : P e r t u r b a t i

n

A d v a n t a g e s

S

i m p l e .

A

p p l i c a b l e t

a

n y ML m

d

e l . D i s a d v a n t a g e s

N

e e d t

r

e e v a l u a t e t h e f u n c t i

n

f

r

m a n y p e r t u r b a t i

n

s → s l

w
P

e r t u r b a t i

n

p r

c

e s s m a y i n t r

d

u c e a r t e f a c t s i n t h e i m a g e → u n r e l i a b l e input Building an explanation

i=1 i=2 i=3 ... i=1 i=2 i=3 ... i=6 i=6

heatmap

SLIDE 10

1 / 5 4

A p p r

a

c h 2 : M e a n i n g f u l P e r t u r b a t i

n

s

Idea: Don’t iterate over all possible perturbation, search locally for the best perturbation m* (or mask).

Fong and Vedaldi 2017, Interpretable Explanations of Black Boxes by Meaningful Perturbation

SLIDE 11

1 1 / 5 4

A p p r

a

c h 2 : M e a n i n g f u l P e r t u r b a t i

n

s

A d v a n t a g e s

C

a n b e a p p l i e d t

a

n y ( d i fg e r e n t i a b l e ) ML m

d

e l . L i mi t a t i

n

s

N

e e d t

r

u n a n

p

t i m i z a t i

n

p r

c

e d u r e

SLIDE 12

1 2 / 5 4

A p p r

a

c h 3 : ( S i mp l e ) T a y l

r

E x p a n s i

n

s

T a y l

r

E x p a n s i

n

I d e a : i d e n t i f y t h e c

n

t r i b u t i

n
f

i n p u t f e a t u r e s a s t h e fj r s t

r

d e r t e r m s

f

a T a y l

r

e x p a n s i

n

SLIDE 13

1 3 / 5 4

A p p r

a

c h 3 : ( S i mp l e ) T a y l

r

E x p a n s i

n

s

A d v a n t a g e s

C

a n b e a p p l i e d t

a

n y ( d i fg e r e n t i a b l e a n d m i l d l y n

n

l i n e a r ) ML m

d

e l . L i mi t a t i

n

s

N

e e d t

fj

n d a m e a n i n g f u l r

t

p

i

n t w h e r e t

p

e r f

r

m t h e e x p a n s i

n

. ( →

p

t i m i z a t i

n

,

r

h e u r i s t i c s )

SLIDE 14

1 4 / 5 4

A p p r

a

c h 4 : G r a d i e n t x I n p u t

Gradient x Input

Mo t i v a t i

n
C
m

p u t e a n e x p l a n a t i

n

i n a s i n g l e p a s s w i t h

u

t h a v i n g t

p

t i m i z e

r

s e a r c h f

r

a r

t

p

i

n t .

SLIDE 15

1 5 / 5 4

A p p r

a

c h 4 : G r a d i e n t x I n p u t

Taylor Expansions Gradient x Input

O b s e r v a t i

n

: C

m

p l e x a n a l y s e s r e d u c e t

g

r a d i e n t x i n p u t f

r

s i m p l e c a s e s .

Perturbation Analysis

Q u e s t i

n

: D

e

s i t w

r

k i n p r a c t i c e ?

SLIDE 16

1 6 / 5 4

A p p r

a

c h 4 : G r a d i e n t x I n p u t

A l b e r e t a l . i N N v e s t i g a t e N e u r a l N e t w

r

k s , J M L R S

f

t w a r e , 2 1 9

I n p u t E x p l a n a t i

n

P r e d i c t i

n

( c l a s s : b a s e b a l l )

SLIDE 17

1 7 / 5 4

A p p r

a

c h 4 : G r a d i e n t x I n p u t

A l b e r e t a l . i N N v e s t i g a t e N e u r a l N e t w

r

k s , J M L R S

f

t w a r e , 2 1 9

O b s e r v a t i

n

: E x p l a n a t i

n

s a r e n

i

s y . I n p u t V G G

1

6 I n c e p t i

n

V 3 R e s N e t 5 E x p l a n a t i

n

Mo d e l

SLIDE 18

1 8 / 5 4

A p p r

a

c h 4 : G r a d i e n t x I n p u t

N

t

l

c

a l e n

u

g h . T

m

u c h c

n

t e x t i n t r

d

u c e d w h e n m u l t i p l y i n g b y t h e i n p u t . S h a t t e r e d g r a d i e n t p r

b

l e m → g r a d i e n t

f

d e e p n e t s h a s l

w

i n f

r

m a t i v e v a l u e T w

r

e a s

n

s w h y e x p l a n a t i

n

s a r e n

i

s y :

SLIDE 19

1 9 / 5 4

A p p r

a

c h 4 : G r a d i e n t x I n p u t

The Shattered gradients problem [Montufar’14, Balduzzi’17]

SLIDE 20

2 / 5 4

O v e r v i e w

f

E x p l a n a t i

n

M e t h

d

s

R

e c a p

1 . P e r t u r b a t i

n
B

a s e d Me t h

d

s → u n i v e r s a l l y a p p l i c a b l e b u t s l

w

2 . Me a n i n g f u l P e r t u r b a t i

n

s → w i d e l y a p p l i c a b l e b u t r e q u i r e s

p

t i m i z a t i

n

3 . T a y l

r

E x p a n s i

n

s → q u i t e w i d e l y a p p l i c a b l e b u t r e q u i r e s t

fj

n d a r

t

p

i

n t 4 . G r a d i e n t x I n p u t → a p p l i c a b l e w i t h s

m

e r e s t r i c t i

n

s → f a s t , O ( f

r

w a r d p a s s ) → d

e

s n

t

w

r

k w e l l

n

h i g h l y n

n

l i n e a r f u n c t i

n

s ( e . g . D N N s )

SLIDE 21

2 1 / 5 4

L a y e r

Wi

s e R e l e v a n c e P r

p

a g a t i

n

SLIDE 22

2 2 / 5 4

I d e a : R e u s i n g M

d

e l S t r u c t u r e

mo d e l i s a c

mp
s

i t i

n
f

n e u r

n

s . T h i s c a n b e e x p l

i

t e d t

ma

k e e x p l a n a t i

n

e a s i e r .

SLIDE 23

2 3 / 5 4

L a y e r

w

i s e R e l e v a n c e P r

p

a g a t i

n

( L R P )

i n p u t

u

t p u t e x p l a n a t i

n

1 . f

r

w a r d p a s s 2 . c

n

s e r v a t i v e p r

p

a g a t i

n

SLIDE 24

2 4 / 5 4

V a r i

u

s L R P P r

p

a g a t i

n

R u l e s

L R P

L

R P

ϵ

L R P

γ

SLIDE 25

2 5 / 5 4

V a r i

u

s L R P P r

p

a g a t i

n

R u l e s

L R P

L

R P

ϵ

L R P

γ

E q u i v a l e n t t

g

r a d i e n t x i n p u t , n

i

s y R e d u c e s n

i

s e , i n c r e a s e s s p a r s i t y R e d u c e s n

i

s e , r e d u c e s s p a r s i t y

SLIDE 26

2 6 / 5 4

T r i c k : U s e a D i f f e r e n t R u l e a t e a c h L a y e r

SLIDE 27

2 7 / 5 4

I mp l e me n t i n g L R P E f fi c i e n t l y

L R P

/

ϵ / γ Mo r e g e n e r a l L R P r u l e

SLIDE 28

2 8 / 5 4

I mp l e me n t i n g L R P i n P y T

r

c h

SLIDE 29

2 9 / 5 4

V a r i

u

s L R P R u l e s U s e d i n P r a c t i c e

SLIDE 30

3 / 5 4

J u s t i f y i n g L R P a s a ‘ D e e p T a y l

r

D e c

mp
s

i t i

n

’

SLIDE 31

3 1 / 5 4

S i mp l e T a y l

r

D e c

mp
s

i t i

n

SLIDE 32

3 2 / 5 4

D e e p T a y l

r

D e c

mp
s

i t i

n

h a r d t

a

n a l y z e

SLIDE 33

3 3 / 5 4

D e e p T a y l

r

D e c

mp
s

i t i

n

K e y I d e a : U s e a “ r e l e v a n c e mo d e l ” t h a t i s e a s y t

a

n a l y z e

SLIDE 34

3 4 / 5 4

D e e p T a y l

r

D e c

mp
s

i t i

n

1 . R e l e v a n c e mo d e l 2 . T a y l

r

e x p a n s i

n

( L R P

)

( L R P

ϵ

) ( L R P

γ

) 3 . C h

s

i n g t h e r e f e r e n c e p

i

n t

SLIDE 35

3 5 / 5 4

L R P Wh a t ’ s N e w

SLIDE 36

3 6 / 5 4

L R P Wh a t ’ s N e w

1 . N e u r a l i z a t i

n

P r

p

a g a t i

n

( N E O N ) 2 . D a t a s e t

Wi

d e A n a l y s i s w i t h S p R A y

SLIDE 37

3 7 / 5 4

N E O N ( N e u r a l i z a t i

n
P

r

p

a g a t i

n

)

L R P ’ s i d e a : T

r
b

u s t l y e x p l a i n a m

d

e l , l e v e r a g e t h e n e u r a l n e t w

r

k s t r u c t u r e

f

t h e d e c i s i

n

f u n c t i

n

. N E O N ’ s i d e a : Wh e n t h e ML m

d

e l i s n

t

a n e u r a l n e t w

r

k ( e . g . a k e r n e l m a c h i n e ) , c

n

v e r t i t i n t

a

n e u r a l n e t w

r

k fj r s t ( ‘ n e u r a l i z e ’ i t ) .

SLIDE 38

3 8 / 5 4

N e u r a l i z i n g t h e O n e

C

l a s s S V M

O r i g i n a l

n

e

c

l a s s S V M s t r u c t u r a t i

n

: N e u r a l i z e d

n

e

c

l a s s S V M:

SLIDE 39

3 9 / 5 4

N e u r a l i z i n g t h e O n e

C

l a s s S V M

L R P

SLIDE 40

4 / 5 4

N e u r a l i z e d O n e

C

l a s s S V M

SLIDE 41

4 1 / 5 4

N e u r a l i z i n g K

me

a n s

SLIDE 42

4 2 / 5 4

L R P Wh a t ’ s N e w

1 . N e u r a l i z a t i

n

P r

p

a g a t i

n

( N E O N ) 2 . D a t a s e t

Wi

s e A n a l y s i s w i t h S p R A y

SLIDE 43

4 3 / 5 4

D a t a s e t

Wi

d e A n a l y s i s

L R P ’ s i d e a : E x p l a i n i n d i v i d u a l d e c i s i

n

s

f

a ML m

d

e l i n a w a y t h a t i s r e l i a b l e a n d i n t e r p r e t a b l e f

r

a h u m a n . S p R A y ’ s i d e a : E x p l a i n w h

l

e d a t a s e t d e c i s i

n

s

f

a ML m

d

e l b y s y s t e m a t i c a l l y a n a l y z i n g d i s t r i b u t i

n

s

f

L R P h e a t m a p s .

LRP SpRAy

SLIDE 44

4 4 / 5 4

D a t a s e t

Wi

d e A n a l y s i s

This analysis is possible due to the conservation property of LRP. Idea: detect different strategies

f classifiers on dataset-wide

basis.

SLIDE 45

4 5 / 5 4

D a t a s e t

Wi

d e A n a l y s i s

This analysis is possible due to the conservation property of LRP. Idea: detect different strategies

f classifiers on dataset-wide

basis.

mo r e i n P a r t 3

SLIDE 46

4 6 / 5 4

S p R A y ( S p e c t r a l R e l e v a n c e A n a l y s i s )

SLIDE 47

4 7 / 5 4

S p R A y ( S p e c t r a l R e l e v a n c e A n a l y s i s )

Lapuschkin et al. Unmasking Clever Hans predictors and assessing what machines really learn (2019)

mo r e i n P a r t 4

SLIDE 48

4 8 / 5 4

O p e n C h a l l e n g e s

SLIDE 49

4 9 / 5 4

O p e n C h a l l e n g e s : S y s t e ma t i c A p p l i c a t i

n
H
w

mu c h ma n u a l t u n i n g i s n e e d e d t

a

d a p t L R P t

n

e w a r c h i t e c t u r e s ?

C

a n e x p l a n a t i

n

t e c h n i q u e s b e i mp l e me n t e d i n a mo d u l a r w a y ?

C

a n e x p l a n a t i

n

b e ma d e d i fg e r e n t i a b l e a n d l e a r n e d ?

SLIDE 50

5 / 5 4

O p e n C h a l l e n g e s : S y s t e ma t i c E v a l u a t i

n
H
w

t

e

v a l u a t e t h e q u a l i t y

f

a n e x p l a n a t i

n

?

I

s t h e r e a t r a d e

fg

b e t w e e n e x p l a n a t i

n

f a i t h f u l n e s s a n d i n t e r p r e t a b i l t y ?

Wh

a t a r e t h e l i mi t s

f

e x p l a n a t i

n

s .

SLIDE 51

5 1 / 5 4

O p e n C h a l l e n g e s : S y s t e ma t i c E v a l u a t i

n

mo r e i n P a r t 3

H
w

t

e

v a l u a t e t h e q u a l i t y

f

a n e x p l a n a t i

n

?

I

s t h e r e a t r a d e

fg

b e t w e e n e x p l a n a t i

n

f a i t h f u l n e s s a n d i n t e r p r e t a b i l t y ?

Wh

a t a r e t h e l i mi t s

f

e x p l a n a t i

n

s ?

SLIDE 52

5 2 / 5 4

S u mma r y

B

e f

r

e e x p l a i n i n g a ML m

d

e l , i t i s i m p

r

t a n t t

a

s k w h e t h e r a g i v e n e x p l a n a t i

n

t e c h n i q u e s p r

v

i d e s t h e d e s i r e d t y p e

f

e x p l a n a t i

n

( e . g . l

c

a l v s . g l

b

a l e x p l a n a t i

n

) .

Ma

n y m e t h

d

s h a v e b e e n p r

p
s

e d e x p l a i n i n g i n d i v i d u a l p r e d i c t i

n

s . L R P r e q u i r e s t

c

a r e f u l l y t u n e p r

p

a g a t i

n

r u l e s . A f t e r t h i s i n i t i a l s t e p , L R P w

r

k s q u i c k l y a n d r e l i a b l y .

L

R P i s n

t

s i m p l y h e u r i s t i c s , L R P r u l e s c a n b e d e r i v e d f

r

m t h e d e e p T a y l

r

d e c

m

p

s

i t i

n

f r a m e w

r

k .

E

x p l a n a t i

n

m e t h

d

s s u c h a s L R P c a n b e c

m

b i n e d w i t h

t

h e r t e c h n i q u e s t

e

x t e n d t h e i r s c

p

e

f

a p p l i c a t i

n

( e . g . N E O N f

r

u s e w i t h k e r n e l s , S p R A y f

r

d a t a s e t

w

i d e a n a l y s i s ) .

SLIDE 53

5 3 / 5 4

C h e c k

u

r w e b s i t e

O n l i n e d e m

s

, t u t

r

i a l s , c

d

e e x a m p l e s , e t c .

n

a n d t u t

r

i a l p a p e r s

SLIDE 54

5 4 / 5 4

R e f e r e n c e s

S

B a c h , A B i n d e r , G Mo n t a v

n

, F K l a u s c h e n , K R Mü l l e r , W S a m e k . O n P i x e l

w

i s e E x p l a n a t i

n

s [ . . . ] b y L a y e r

w

i s e R e l e v a n c e P r

p

a g a t i

n

, P L O S O N E , 1 ( 7 ) : e 1 3 1 4 , 2 1 5

J

K a u fg m a n n , K R Mü l l e r , G Mo n t a v

n

. T

w

a r d s E x p l a i n i n g A n

m

a l i e s : A D e e p T a y l

r

D e c

m

p

s

i t i

n
f

O n e

C

l a s s Mo d e l s , a r X i v : 1 8 5 . 6 2 3 , 2 1 8

J

K a u fg m a n n , M E s d e r s , G Mo n t a v

n

, W S a m e k , K R Mü l l e r . F r

m

C l u s t e r i n g t

C

l u s t e r E x p l a n a t i

n

s v i a N e u r a l N e t w

r

k s , a r X i v : 1 9 6 . 7 6 3 3 , 2 1 9

S

L a p u s c h k i n , S Wä l d c h e n , A B i n d e r , G Mo n t a v

n

, W S a m e k , K R Mü l l e r . U n m a s k i n g C l e v e r H a n s P r e d i c t

r

s a n d A s s e s s i n g Wh a t Ma c h i n e s R e a l l y L e a r n , N a t u r e C

m

m u n i c a t i

n

s , 1 : 1 9 6 , 2 1 9

S

L a p u s c h k i n , A B i n d e r , G Mo n t a v

n

, K R Mü l l e r , W S a m e k . A n a l y z i n g C l a s s i fj e r s : F i s h e r V e c t

r

s a n d D e e p N e u r a l N e t w

r

k s , I E E E C V P R , 2 9 1 2

2

9 2 , 2 1 6

G

Mo n t a v

n

, S L a p u s c h k i n , A B i n d e r , W S a m e k , K R Mü l l e r . E x p l a i n i n g N

n

L i n e a r C l a s s i fj c a t i

n

D e c i s i

n

s w i t h D e e p T a y l

r

D e c

m

p

s

i t i

n

, P a t t e r n R e c

g

n i t i

n

, 6 5 : 2 1 1 – 2 2 2 , 2 1 7

G

Mo n t a v

n

, W S a m e k , K R Mü l l e r : Me t h

d

s f

r

I n t e r p r e t i n g a n d U n d e r s t a n d i n g D e e p N e u r a l N e t w

r

k s D i g i t a l S i g n a l P r

c

e s s i n g , 7 3 : 1

1

5 , 2 1 8

G

Mo n t a v

n

, A B i n d e r , S L a p u s c h k i n , W S a m e k , K R Mü l l e r : L a y e r

w

i s e R e l e v a n c e P r

p

a g a t i

n

: A n O v e r v i e w , S p r i n g e r L N C S 1 1 7 , 2 1 9 ( t

a

p p e a r )