Interpretability and functional transparency Tommi Jaakkola in - - PowerPoint PPT Presentation

Interpretability and functional transparency

Tommi Jaakkola in collaboration with David Alvarez Melis, Guang-He Lee, et al.

Many facets of “interpretability”

Many facets of “interpretability”

uncover causal mechanisms

[Garg et al. 2018]

Many facets of “interpretability”

uncover causal mechanisms

[Garg et al. 2018]

learn to highlight relevance

[Lei et al. 2016; Jin et al. 2017]

Many facets of “interpretability”

uncover causal mechanisms

[Garg et al. 2018]

learn to highlight relevance

[Lei et al. 2016; Jin et al. 2017]

learn functional transparency

[Lee et al. 2018; Alvarez et al. 2018]

Many facets of “interpretability”

uncover causal mechanisms

[Garg et al. 2018]

learn to highlight relevance

[Lei et al. 2016; Jin et al. 2017]

learn functional transparency

[Lee et al. 2018; Alvarez et al. 2018]

summarize by causal relations

[Alvarez et al. 2017]

Many facets of “interpretability”

uncover causal mechanisms

[Garg et al. 2018]

learn to highlight relevance

[Lei et al. 2016; Jin et al. 2017]

learn functional transparency

[Lee et al. 2018; Alvarez et al. 2018]

summarize by causal relations

[Alvarez et al. 2017]

from phenomena to models

“Interpretability”

• (Human) interpretability
• features (that make sense)
• relevance (what information is used to make a decision)
• reasoning (mechanism used to arrive at the decision)
• etc.

?

“Interpretability”

• (Human) interpretability
• features (that make sense)
• relevance (what information is used to make a decision)
• reasoning (mechanism used to arrive at the decision)
• etc.
• Functional transparency
• guaranteed properties, including robustness

?

Molding for transparency

… … … … … …

• deep locally linear

models

• ReLU networks with

large linear regions

• temporal models

with desired local behavior

…

Deep locally linear models

• A self-explaining architecture from a deep linear model

h ⇣

⌘

·

simple linear coefficients simple feature transformation θ

f ⇣

=

⌘

A linear model

• A self-explaining architecture from a deep linear model
• Arbitrarily powerful, but not (linearly) interpretable

h ⇣

⌘

⌘

θ ⇣

·

f ⇣

=

⌘

A deep linear model deep linear coefficients simple feature transformation

Deep locally linear models

• A self-explaining architecture from a deep linear model
• Arbitrarily powerful, but not (linearly) interpretable

h ⇣

⌘

⌘

θ ⇣

·

f ⇣

=

⌘

A deep linear model deep linear coefficients simple feature transformation

Deep locally linear models

… …

deep linear coefficients

• A self-explaining architecture from a deep linear model
• We can regularize the model for local interpretability

h ⇣

⌘

⌘

θ ⇣

·

f ⇣

=

⌘

A deep linear model deep linear coefficients simple feature transformation

(Alvarez et al. 2018)

R(θ) = krf(x) θ(x)T Jh;xk2

Deep locally linear models

• A self-explaining architecture from a deep linear model
• We can regularize the model for local interpretability

h ⇣

⌘

⌘

θ ⇣

·

f ⇣

=

⌘

A deep linear model deep linear coefficients simple feature transformation

(Alvarez et al. 2018)

R(θ) = krf(x) θ(x)T Jh;xk2

Deep locally linear models

locally linear “witness”

• A self-explaining architecture from a deep linear model
• We can regularize the model for local interpretability
• Generalizable beyond linear (monotone, separable)

h ⇣

⌘

⌘

θ ⇣

·

f ⇣

=

⌘

A deep linear model deep linear coefficients simple feature transformation

(Alvarez et al. 2018)

R(θ) = krf(x) θ(x)T Jh;xk2

Deep locally linear models

• A self-explaining architecture from a deep linear model

… …

locally interpretable deep linear coefficients

[Alvarez et al. 2018]

Deep locally linear models

Expanding linear regions

• E.g., a ReLU network (locally linear)

x

y

… … …

Expanding linear regions

• E.g., a ReLU network (locally linear)

x

y

… … …

• “activation pattern”

Expanding linear regions

• E.g., a ReLU network (locally linear)

x

y

… … …

x

• “activation pattern”

Expanding linear regions

• E.g., a ReLU network (locally linear)

x

y

… … …

x

• “activation pattern”

Expanding linear regions

• E.g., a ReLU network (locally linear)

x

y

… … …

x

• we can learn the network so

as to encourage large linear regions (gradient stability)

• “activation pattern”

Expanding linear regions

• We can aim to maximize the margin for each neuron

… …

x

…

zi

rxzi

effective linear weights

x

Expanding linear regions

• We can aim to maximize the margin for each neuron

… …

x

…

zi

rxzi

effective linear weights margin

|zi| krxzik

x

Expanding linear regions

• We can aim to maximize the margin for each neuron

… …

x

…

zi

rxzi

effective linear weights margin

|zi| krxzik

x

relaxed margin regularizer

krxzik2 + C max(0, 1 |zi|)

Expanding linear regions

• We maximize a relaxed margin loss
• A toy example

(a) Vanilla loss

min

θ

X

(x,y)∈D

L(fθ(x), y) +

• |ˆ

I(x, )| X

(i,j)∈ˆ I(x,γ)

 krxzi

jk2 2 + C max(0, 1 |zi j|)

• [Lee et al. 2018]

Molding temporal models

• Introducing a local “explainer” as a witness of desired

local behavior

• For example:

deep temporal models that are locally ARMA (witness: ARMA) deep sequence models that are locally bigram (witness: bigram)

g(t) = ˆ a1 g(t − 1) + ˆ a2 g(t − 2) + ✏t

estimated evaluated

f(t)

local witness

We focus in this paper on molding complex predictors towards exhibiting a chosen local functional behavior. We coin the problem functional transparency. The proposed ap- proach is setup as a co-operative game be- tween an unrestricted predictor such as a neural network, and a witness chosen from the desired transparent family. The goal of the witness is to highlight, locally, how well the predictor conforms to the chosen family

estimated evaluated local witness

ˆ P(wt|wt−1)

Molding temporal models

• Introducing a local “explainer” as a witness of desired

local behavior

• For example:

deep temporal models that are locally ARMA (witness: ARMA) deep sequence models that are locally bigram (witness: bigram)

g(t) = ˆ a1 g(t − 1) + ˆ a2 g(t − 2) + ✏t

estimated evaluated

f(t)

local witness

We focus in this paper on molding complex predictors towards exhibiting a chosen local functional behavior. We coin the problem functional transparency. The proposed ap- proach is setup as a co-operative game be- tween an unrestricted predictor such as a neural network, and a witness chosen from the desired transparent family. The goal of the witness is to highlight, locally, how well the predictor conforms to the chosen family

estimated evaluated local witness

ˆ P(wt|wt−1)

A co-operative witness…

• We can mold a complex function to agree locally with the

corresponding local witness

• (an asymmetric game, information sets do not agree)

n

X

i=1

 L(f(xi), yi) + λ d(f(xi), ˆ g(xi))

• <latexit sha1_base64="apk8BWj2FTOCdbr+XBNBtHaCftc=">ACjHicfZBvaxQxEMZz65/WU9urvRN8BSuepTdUlCQwkFfKF4gtcWLusym53dC02yS5KVHst+Lb9Lwbf6Ocxtr6CtOJDwY56ZSeZJKymsC8OLXnDr9p27G5v3+vcfPNzaHuw8OrZlbTjOeClLc5qCRSk0zpxwEk8rg6BSiSfp2dFKP/mGxopSf3HLCmMFhRa54OB8KhlMma1V0ojDqP2qKUtFUcxpwzhI+qEd5aPzROyOl/6iLymTfm4GbEyzK4UtwNGi492uOU4Gw3Av7ILehGgNQ7KOabLTe8ayktcKteMSrJ1HYeXiBowTXGLbZ7XFCvgZFDj3qEGhjZtu9ZY+95mM5qXxRzvaZf/saEBZu1Spr1TgFva6tkr+S5vXLn8dN0JXtUPNLx/Ka0ldSVc+0kwY5E4uPQA3wv+V8gUY4M673Wdv0e9i8KOf+6lCA640LxoGplBw3vrdCjZe0f8Khb4q9NT3vkbXbwJx/t7kefPB8PJZO3wJnlCnpIRicgrMiHvyZTMCfyQ/yk/wKtoKD4E1weFka9NY9j8lfEbz7DXUGxhw=</latexit><latexit sha1_base64="apk8BWj2FTOCdbr+XBNBtHaCftc=">ACjHicfZBvaxQxEMZz65/WU9urvRN8BSuepTdUlCQwkFfKF4gtcWLusym53dC02yS5KVHst+Lb9Lwbf6Ocxtr6CtOJDwY56ZSeZJKymsC8OLXnDr9p27G5v3+vcfPNzaHuw8OrZlbTjOeClLc5qCRSk0zpxwEk8rg6BSiSfp2dFKP/mGxopSf3HLCmMFhRa54OB8KhlMma1V0ojDqP2qKUtFUcxpwzhI+qEd5aPzROyOl/6iLymTfm4GbEyzK4UtwNGi492uOU4Gw3Av7ILehGgNQ7KOabLTe8ayktcKteMSrJ1HYeXiBowTXGLbZ7XFCvgZFDj3qEGhjZtu9ZY+95mM5qXxRzvaZf/saEBZu1Spr1TgFva6tkr+S5vXLn8dN0JXtUPNLx/Ka0ldSVc+0kwY5E4uPQA3wv+V8gUY4M673Wdv0e9i8KOf+6lCA640LxoGplBw3vrdCjZe0f8Khb4q9NT3vkbXbwJx/t7kefPB8PJZO3wJnlCnpIRicgrMiHvyZTMCfyQ/yk/wKtoKD4E1weFka9NY9j8lfEbz7DXUGxhw=</latexit><latexit sha1_base64="apk8BWj2FTOCdbr+XBNBtHaCftc=">ACjHicfZBvaxQxEMZz65/WU9urvRN8BSuepTdUlCQwkFfKF4gtcWLusym53dC02yS5KVHst+Lb9Lwbf6Ocxtr6CtOJDwY56ZSeZJKymsC8OLXnDr9p27G5v3+vcfPNzaHuw8OrZlbTjOeClLc5qCRSk0zpxwEk8rg6BSiSfp2dFKP/mGxopSf3HLCmMFhRa54OB8KhlMma1V0ojDqP2qKUtFUcxpwzhI+qEd5aPzROyOl/6iLymTfm4GbEyzK4UtwNGi492uOU4Gw3Av7ILehGgNQ7KOabLTe8ayktcKteMSrJ1HYeXiBowTXGLbZ7XFCvgZFDj3qEGhjZtu9ZY+95mM5qXxRzvaZf/saEBZu1Spr1TgFva6tkr+S5vXLn8dN0JXtUPNLx/Ka0ldSVc+0kwY5E4uPQA3wv+V8gUY4M673Wdv0e9i8KOf+6lCA640LxoGplBw3vrdCjZe0f8Khb4q9NT3vkbXbwJx/t7kefPB8PJZO3wJnlCnpIRicgrMiHvyZTMCfyQ/yk/wKtoKD4E1weFka9NY9j8lfEbz7DXUGxhw=</latexit><latexit sha1_base64="apk8BWj2FTOCdbr+XBNBtHaCftc=">ACjHicfZBvaxQxEMZz65/WU9urvRN8BSuepTdUlCQwkFfKF4gtcWLusym53dC02yS5KVHst+Lb9Lwbf6Ocxtr6CtOJDwY56ZSeZJKymsC8OLXnDr9p27G5v3+vcfPNzaHuw8OrZlbTjOeClLc5qCRSk0zpxwEk8rg6BSiSfp2dFKP/mGxopSf3HLCmMFhRa54OB8KhlMma1V0ojDqP2qKUtFUcxpwzhI+qEd5aPzROyOl/6iLymTfm4GbEyzK4UtwNGi492uOU4Gw3Av7ILehGgNQ7KOabLTe8ayktcKteMSrJ1HYeXiBowTXGLbZ7XFCvgZFDj3qEGhjZtu9ZY+95mM5qXxRzvaZf/saEBZu1Spr1TgFva6tkr+S5vXLn8dN0JXtUPNLx/Ka0ldSVc+0kwY5E4uPQA3wv+V8gUY4M673Wdv0e9i8KOf+6lCA640LxoGplBw3vrdCjZe0f8Khb4q9NT3vkbXbwJx/t7kefPB8PJZO3wJnlCnpIRicgrMiHvyZTMCfyQ/yk/wKtoKD4E1weFka9NY9j8lfEbz7DXUGxhw=</latexit>

loss on observations discrepancy with the local witness

ˆ f ← arg min

f

ˆ g ← arg min

g

X

xj∈B(xi)

d( ˆ f(xj), g(xj))

locally tailored witness witness at xi

[Lee et al. 2018]

Interpretability - the broader view

• The overall goal is about two-way communication, more

formal view of “interpretability”

Explain Guide

Interpretability - the broader view

• The overall goal is about two-way communication, more

formal view of “interpretability”

• Self-explaining models: models are trained to exhibit

desirable properties (causal, functional, relevance, etc)

• Multi-resolution explanations

Explain Guide