EXPLAINABLE AI (AND RELATED CONCEPTS) A QUICK TOUR AI Present and - - PowerPoint PPT Presentation

EXPLAINABLE AI (AND RELATED CONCEPTS)

A QUICK TOUR

AI Present and future Jacques Fleuriot

STATE OF PLAY

• Generally, machine learning models are black boxes
• Not intuitive
• Difficult for humans to understand, often even by their designers do not

fully understand the decision-making procedures.

• Yet they are being widely deployed in ways that affect our daily lives
• Bad press when things go wrong
• We’ll look at a few areas/case studies quickly but this is an active, fast

growing research field and there is much, much more going on.

SOME OF THE ISSUES

• Fairness/Bias
• e.g. Amazon ML tool for recruitment was found to be biased against women (Reuters, Oct 2018)
• Even what one might consider benign e.g. Netflix serving artworks for movies that many thought were

based on racial profiling (Guardian, Oct 2018)

• We know that “bad” training data can result in biases and unfairness
• Challenge: How does one define “fairness” in a rigorous, concrete way (in order to model it)?
• Trust
• One survey: 9% of respondents said they trusted AI with their financials, and only 4% trusted it for hiring

processes (Can We Solve AI’s ‘Trust Problem’? MIT Sloan Management Review, November 2018)

• Safety
• Can one be sure that a self-driving car will not behave dangerously in situations never encountered before?
• Ethics (see previous lectures)

EXPLAINABLE AI

• Not a new topic
• Rule-based systems are generally viewed as explainable (but are not scalable, able to deal with data,

etc.)

• Example: MYCIN expert system (with ~500 rules) for diagnosing patients based on reported

symptoms (1970s)

• It could be asked to explain the reasoning leading to the diagnosis and recommendation
• It operated at the same level of competence as a specialist and better than a GP
• Poor interface and relatively limited compute power at the time
• Ethical and legal issues related to the use of computers in medicine were raised (even) at the

time

• Are there ways of marrying powerful blackbox/ML approaches with (higher-level) symbolic

reasoning?

DARPA’S XAI: ONE POSSIBLE VISION

Learning Process

Video & Social Media Learned Function Output

Today

This incident is a violation

(p = .93)

Video & Social Media

New Learning Process

Explainable Model Explanation Interface

Tomorrow

• I understand why
• I understand the

evidence for this recommendation

• This is clearly one to

investigate

What should I report? What should I report?

Detecting Ceasefire Violations

Incident

Detecting Ceasefire Violations

Incident

These events occur before tweet reports

This is a violation:

• Why do you say that?
• What is the evidence?
• Could it have been an

accident?

• I don’t know if this

should be reported or not

Source: DARPA XAI

PERFORMANCE VS EXPLAINABILITY

Learning Performance Explainability

Neural Nets Statistical Models Ensemble Methods Decision Trees Deep Learning

SVMs AOGs Bayesian Belief Nets Markov Models

HBNs MLNs

SRL

CRFs

Random Forests

Graphical Models

Source: DARPA XAI

Explainability

Neural Nets Statistical Models Ensemble Methods Decision Trees Deep Learning

SVMs AOGs Bayesian Belief Nets Markov Models

HBNs MLNs

Model Induction

Techniques to infer an explainable model from any model as a black box

Deep Explanation

Modified deep learning techniques to learn explainable features

SRL

Interpretable Models

Techniques to learn more structured, interpretable, causal models

CRFs

Random Forests

Graphical Models

Learning Performance

PERFORMANCE VS EXPLAINABILITY

Source: DARPA XAI

INTERPRETABLE MODELS

• Example: Human-level concept learning through probabilistic

program induction (Lake et al. 2015, Science)

• “Model represents concepts as simple programs that best explain observed

examples under a Bayesian criterion”

• Bayesian Program Learning: Learn visual concepts from just a single example and

genralise in a human-like fashion — one-shot learning

• Key ideas: compositionality, causality and learning to learn
• Note: Interpretable and explainable are not necessarily the same

INTERPRETABLE MODEL

Source: DARPA XAI

DATASET

• Dataset of 1623 characters from

50 writing systems

• Images and pen strokes collected
• Good for comparing humans

and machine learning:

• cognitively natural and are

used as benchmarks for ML algorithm

GENERATIVE MODEL

BAYESIAN PROGRAM LEARNING

• Lake et al. showed that it is possible to perform one-shot

learning at human-level accuracy

• Most judges couldn’t distinguish between the machine- and

human-generated characters in (“visual Turing”) tests.

• However, BPL still sees less structure in visual concepts

than humans

• Also what’s the relationship with explainability?

MODEL INDUCTION

• Example: Bayesian Rule Lists (Letham et al., Ann. of Applied

Statistics, 2015)

• Aim: Predictive models that are not only accurate, but are also interpretable to human

experts.

• Models are decision lists, which consist of a series of if...then...statements
• Approach: Produce a posterior distribution over permutations of if...then... rules, starting

from a large, pre-mined set of possible rules

• Used to develop interpretable patient-level predictive models using massive
• bservational medical data

BAYESIAN RULE LISTS

• BRLs can discretise a high-dimensional, multivariate feature space into a series of interpretable decision statements e.g. 4146

unique medications and condition for above + age and gender

• Experiments showed that BRLs have predictive accuracy on par with the top ML algorithms at the time (approx. 85- 90% as

effective) but with models that are much more interpretable

• For technical details of underlying generative model, MCMC sampling, etc. (see paper).


USING VISUALISATION

• Example: Interpretable Learning for Self-Driving Cars by Visualizing

Causal Attention (Kim and Canny, ICCV 2017)

• If deep neural perception and control networks are to be a key component of self-

driving vehicles, these models need to be explainable

• Visual explanations in the form of real-time highlighted regions of an image that causally

influence the car steering control (i.e. the deep network output)

USING VISUALISATION

• Method highlights regions that causally influence deep neural

perception and control networks for self-driving cars.

• The visual attention model is augmented with an additional layer of

causal filtering.

• Does this correspond to where a driver would gaze though?

USING VISUALISATION AND TEXT

• Video-to-text language model to produce

textual rationales that justify the model’s decision

• The explanation generator uses a spatio-

temporal attention mechanism, which is encouraged to match the controller’s attention

Show, Attend, Control, and Justify: Interpretable Learning for Self-Driving Cars. Kim et al, Interpretable ML Symposium (NIPS 2017)

COMBINING DEEP LEARNING AND SYMBOLIC/LOGICAL REASONING

• Example: DeepProbLog: Neural Probabilistic Logic

Programming, (Manhaeve et al., 2018)

• Deep learning + ProbLog = DeepProbLog !
• Approach that aims to “integrate low-level perception with high-level reasoning”
• Incorporate DL into probabilistic LP such that
• probabilistic/logical modelling and reasoning is possible
• general purpose NNs are possible
• end-to-end training is possible

(Just when you thought you were done with logic programming)

DEEP PROBLOG

Source: Manhaeve et al. DeepProbLog: Neural Probabilistic Programming

DEEP PROBLOG

Source: Manhaeve et al. DeepProbLog: Neural Probabilistic Programming

EXAMPLE

Source: Manhaeve et al. DeepProbLog: Neural Probabilistic Programming

DEEP PROBLOG

• Relies on the fact that there is a differentiable version of ProbLog that allows for parameters

update of the logic program using gradient descent

• Seemless integration with NN training (which uses backprop) is thus possible
• Allows for a combination of “symbolic and sub-symbolic reasoning and learning, program

induction and probabilistic reasoning”

• For technical details, consult the paper by Manhaeve et al.

DEEP + SYMBOLIC

• Recent exciting work (aside from one involving ProbLog!):
• Harnessing Deep Neural Networks with Logic Rules (Hu et al., 2016)
• End-to-End Differentiable Proving (Rocktäschel and Riedel, 2017)
• Logical Rule Induction and Theory Learning Using Neural Theorem

Proving (Campero et al., 2018)

• Planning Chemical Syntheses with Deep Neural Networks and Symbolic

AI (Segler et al., 2018)

• and many more…

CONCLUSION

• Current ML models tend to be opaque and there’s an urgent need for interpretability/

explainability to ensure fairness, trust, safety, etc.

• Rapidly moving research area that is still wide open because (among many other issues):
• It’s unclear to many ML/DL researchers how their models actually achieve their decisions

(yet alone come up with explanation or interpretation, or both)

• What is an explanation anyway?
• DARPA XAI is a step in the right direction (but there are other initiatives and views on the

topic)

• Logical/symbolic representations and inference provide high-level (abstract) means of

reasoning, these are usually explainable too

• Combining probabilistic, symbolic and sub-symbolic reasoning and learning seems promising
• Finally, as this course tried to emphasise, AI ≠ ML (or DL)