[PPT] - Outline 1 Introduction 2 Bayesian Networks 3 Neuroscience 4 PowerPoint Presentation

SLIDE 1

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

APPLICATIONS IN NEUROSCIENCE, INDUSTRY 4.0, AND SPORTS

Pedro Larra˜ naga

Computational Intelligence Group Artificial Intelligence Department Universidad Polit´ ecnica de Madrid

Bayesian Networks: From Theory to Practice International Black Sea University Autumn School on Machine Learning 3-11 October 2019, Tbilisi, Georgia Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 1 / 88

SLIDE 2

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

Outline

1 Introduction 2 Bayesian Networks 3 Neuroscience 4 Industry 5 Sport 6 Estimation of Distribution Algorithms 7 Conclusions 8 References

Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 2 / 88

SLIDE 3

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

Outline

1 Introduction 2 Bayesian Networks 3 Neuroscience 4 Industry 5 Sport 6 Estimation of Distribution Algorithms 7 Conclusions 8 References

Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 3 / 88

SLIDE 4

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

Machine learning

EVOLUTIONARIES Genetic programming CONNECTIONISTS Backpropagation SYMBOLISTS Inverse deduction BAYESIANS Probabilistic inference ANALOGIZERS Kernel machines

The five tribes of machine learning and their master algorithms (Domingos, 2015) Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 4 / 88

SLIDE 5

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

Machine learning

Clustering

Non-probabilistic

– Hierarchical

Agglomerative
Divisive

– Partitional

K-means
K-medians
K-modes
Fuzzy C-means
Self-organizing map
Spectral clustering
K-medoids
Affinity propagation
K-plane clustering
Fuzzy C-shell
DBSCAN
Probabilistic

– Finite-mixture models – Bayesian networks

Anomaly detection

Probabilistic approaches
Distance-based
Reconstruction-based
Domain-based
Information theory-based
Bayesian network likelihood

Multi-output regression

Problem transformation methods

– Single-target method – Multi-target regressor stacking – Regressor chains

Algorithm adaptation methods

– Multi-output suppor vector regression – Kernel methods – Multi-target regression trees – Rule method – Gaussian Bayesian networks

Multiview-clustering

Agglomerative clustering
Density-based
Principal component analysis

– Canonical correlation analysis

Spectral clustering
Co-regularization
Ensemble clustering
Bayesian networks

Discovery associations

Association rules
Bayesian networks

MACHINE LEARNING

Anomaly detection

Probabilistic approaches
Distance-based
Reconstruction-based
Domain-based
Information theory-based
Bayesian network likelihood

Supervised classification

Non-probabilistic

– Nearest neighbors – Classification trees – Rule induction – Artificial neural networks – Support vector machines

Probabilistic

– Discriminant analysis – Logistic regression – Bayesian network classifiers

Metaclassifiers

– Fusion of outputs – Stacked generalization – Cascading – Bagging – Random forest – Boosting – Hybrid classifiers

Multi-dimensional classifiers

Problem transformation methods

– Binary relevance – Classifier chains – RAkEL – LPBR – Label powerset

Algorithm adaptation methods
Multi-dimensional Bayesian network clas-

sifiers

Feature subset selection

Filter approaches

– Univariate filters – Multivariate filters

Wrapper methods

– Bayesian networks-based EDAs

Embedded methods
Hybrid methods

Machine learning methods for approaching the eight problems in this talk Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 5 / 88

SLIDE 6

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

Machine learning

Clustering

Non-probabilistic

– Hierarchical

Agglomerative
Divisive

– Partitional

K-means
K-medians
K-modes
Fuzzy C-means
Self-organizing map
Spectral clustering
K-medoids
Affinity propagation
K-plane clustering
Fuzzy C-shell
DBSCAN
Probabilistic

– Finite-mixture models – Bayesian networks

Anomaly detection

Probabilistic approaches
Distance-based
Reconstruction-based
Domain-based
Information theory-based
Bayesian network likelihood

Multi-output regression

Problem transformation methods

– Single-target method – Multi-target regressor stacking – Regressor chains

Algorithm adaptation methods

– Multi-output suppor vector regression – Kernel methods – Multi-target regression trees – Rule method – Gaussian Bayesian networks

Multiview-clustering

Agglomerative clustering
Density-based
Principal component analysis

– Canonical correlation analysis

Spectral clustering
Co-regularization
Ensemble clustering
Bayesian networks

Discovery associations

Association rules
Bayesian networks

MACHINE LEARNING

Anomaly detection

Probabilistic approaches
Distance-based
Reconstruction-based
Domain-based
Information theory-based
Bayesian network likelihood

Supervised classification

Non-probabilistic

– Nearest neighbors – Classification trees – Rule induction – Artificial neural networks – Support vector machines

Probabilistic

– Discriminant analysis – Logistic regression – Bayesian network classifiers

Metaclassifiers

– Fusion of outputs – Stacked generalization – Cascading – Bagging – Random forest – Boosting – Hybrid classifiers

Multi-dimensional classifiers

Problem transformation methods

– Binary relevance – Classifier chains – RAkEL – LPBR – Label powerset

Algorithm adaptation methods
Multi-dimensional Bayesian network

classifiers

Feature subset selection

Filter approaches

– Univariate filters – Multivariate filters

Wrapper methods

– Bayesian networks-based EDAs

Embedded methods
Hybrid methods

Machine learning methods for approaching the eight problems in this talk Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 6 / 88

SLIDE 7

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

The 23 Asilomar AI principles + Explainable Artificial Intelligence

Asilomar Conference on Beneficial AI (Future of Life Institute 2017)

1. Research goal
2. Research funding
3. Science-policy lik
4. Research culture
5. Race avoidance
6. Safety
7. Failure transparency
8. Judicial transparency
9. Responsability
10. Value alignment
11. Human values
12. Personal pricavy
13. Liberty and privacy
14. Shared Benefit
15. Shared prosperity
16. Human control
17. Non-subversion
18. AI arms race
19. Capacity caution
20. Importance
21. Risks
22. Recursive self-improvement
23. Common good

Explainable Artificial Intelligence (Rudin, 2019)

Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 7 / 88

SLIDE 8

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

Outline

1 Introduction 2 Bayesian Networks 3 Neuroscience 4 Industry 5 Sport 6 Estimation of Distribution Algorithms 7 Conclusions 8 References

Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 8 / 88

SLIDE 9

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

“Risk of dementia” (Bielza and Larra˜ naga, 2014a)

Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 9 / 88

SLIDE 10

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

“Risk of dementia” p(A, N, S, D, P) = p(A)p(N|A)p(S|A)p(D|N, S)p(P|S)

Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 10 / 88

SLIDE 11

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

“Risk of dementia” p(A, N, S, D, P) = p(A)p(N|A)p(S|A)p(D|N, S)p(P|S)

Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 10 / 88

SLIDE 12

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

Inference (reasoning) with Bayesian networks

No evidence

Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 11 / 88

SLIDE 13

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

Inference (reasoning) with Bayesian networks

Evidence: “Stroke = yes”

Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 12 / 88

SLIDE 14

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

Inference (reasoning) with Bayesian networks

Evidence: “‘Stroke = yes, Neuronal Atrophy=yes”

Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 13 / 88

SLIDE 15

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

Inference (reasoning) with Bayesian networks

Evidence: “‘Stroke = yes, Neuronal Atrophy=yes, Age= young”

Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 14 / 88

SLIDE 16

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

Bayesian networks (Pearl, 1988; Koller and Friedman, 2009)

Structure and parameters A Bayesian network consists of two components

1

Graphical structure G is a directed acyclic graph (DAG) Vertices → variables Directed edges → conditional (in)dependences

2

Set of parameters specifies the set of conditional probability distributions Joint probability distribution: P(x1, . . . , xn) = n

i=1 P

xi | pa(xi)
Pedro Larra˜

naga Neuroscience, Industry 4.0, and Sports 15 / 88

SLIDE 17

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

Bayesian networks Learning and inference

Bayesian network learning Structure learning Parameter learning Probabilistic inference Compute the conditional probability P(Query | Evidence)

Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 16 / 88

SLIDE 18

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

Learning Bayesian networks from data

STRUCTURE LEARNING

1

Detecting conditional independencies between triples of variables by hypothesis tests

2

Score and search methods

AIC, BIC BD, K2, BDe, BDeu Greedy, simulated annealing, EDAs, genetic algorithms, MCMC Dynamic programming, branch & bound, mathematical programming

Score and search Search spaces Scores Search DAGs Equiv. classes Orderings Penalized likelihood Bayesian Exact Approximate

PARAMETER LEARNING

1

Maximum likelihood estimation

2

Bayesian estimation

Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 17 / 88

SLIDE 19

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

Outline

1 Introduction 2 Bayesian Networks 3 Neuroscience 4 Industry 5 Sport 6 Estimation of Distribution Algorithms 7 Conclusions 8 References

Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 18 / 88

SLIDE 20

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

Human Brain Project (2013-2023)

Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 19 / 88

SLIDE 21

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

The human brain Brain lobes and layers

Weight = 1.3kg, width = 140mm, length = 167mm, height = 93mm

Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 20 / 88

SLIDE 22

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

The human brain Brain at microscopic level

Composed of neurons, blood vessels, glial cells Neuron is the basic structural and functional unit of the nervous system –neuron doctrine– (S. Ram´

n y Cajal, late 19th century)

Just 4 microns thick → could fit 30,000 neurons on the head of a pin ∼86,000 million neurons (more than known stars in the universe)

Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 21 / 88

SLIDE 23

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

The neuron 3 parts of a neuron: dendrites, soma and axon

Axons fill most of the space in the brain → >150,000 km in the human brain!! Each neuron connected to 1,000 neighboring neurons 10,000 synaptic connections each

Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 22 / 88

SLIDE 24

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

Clustering of dendritic spines

MIXTURE OF GAUSSIAN BAYESIAN NETWORKS LEARNT WITH THE STRUCTURAL EM (Luengo-Sanchez et al., 2018)

Standard clustering of dendritic spines (Peters and Kaiserman-Abramof, 1970) (a) 3D reconstruction process of human (b) Spine repair process and multiresolution Reeb graph dendritic spines computation Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 23 / 88

SLIDE 25

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

Clustering of dendritic spines

MIXTURE OF GAUSSIAN BAYESIAN NETWORKS LEARNT WITH THE STRUCTURAL EM (Luengo-Sanchez et al., 2018) The data set: 3D reconstruction of more than 7,900 spines from layer III pyramidal neuron human cingulate cortex (aged 40 and 85) Each spine is characterized with 54 morphological variables (some of them directional variables) The model: Multivariate Gaussian mixture model: f(x; θ) = K

k=1 πkfk(x; µk, Σk)

Each mixture density is given by: fk(x; µk, Σk) = (2π)− n

2 |Σk|− 1 2 exp{− 1

2 (x − µk)T Σ−1 k

(x − µk)} Mixture of Gaussian Bayesian networks: Each mixture is expressed as a Gaussian Bayesian network

Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 24 / 88

SLIDE 26

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

Clustering of dendritic spines

MIXTURE OF GAUSSIAN BAYESIAN NETWORKS LEARNT WITH THE STRUCTURAL EM (Luengo-Sanchez et al., 2018)

Virtual spines obtained from the simulation of the mixture model with six components Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 25 / 88

SLIDE 27

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

Morphological probabilistic clustering of neurons

MAXIMIZING THE BIC CRITERION (Rodriguez-Sanchez et al., work in progress)

Allen Cell Types Database: 436 mice neurons with reconstructions of soma, dendrites and axon; 19 variables discretized into 4 bins EM algorithm (Demspter et al. 1977) adapted to structure learning of Bayesian networks (Friedman, 1998) (a) First cluster (b) Second cluster Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 26 / 88

SLIDE 28

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

Electrophysiological probabilistic clustering of neurons

MAXIMIZING THE BIC CRITERION (Rodriguez-Sanchez et al., work in progress)

Allen Cell Types Data Base: the same 436 neurons, 42 electrophysiological variables discretized into 4 bins (a) First cluster (b) Second cluster Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 27 / 88

SLIDE 29

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

Morphological and electrophysiological (as independent)

MAXIMIZING THE BIC CRITERION (Rodriguez-Sanchez et al., work in progress)

BIC value = - 23,0178.38 Cluster Morpho cluster 1 cluster 2 cluster 1 104 129 233 Cluster Electro cluster 2 21 169 190 125 298 423 Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 28 / 88

SLIDE 30

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

Morphological and electrophysiological (multiview)

MAXIMIZING THE BIC CRITERION (Rodriguez-Sanchez et al., work in progress)

BIC value = - 22,940.32 Cluster Morpho cluster 1 cluster 2 cluster 1 141 97 238 Cluster Electro cluster 2 9 176 185 150 273 423 Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 29 / 88

SLIDE 31

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

Morphological classification of interneurons

WEB-BASED SYSTEM. 42 EXPERTS. 320 INTERNEURONS (DeFelipe et al., 2013)

Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 30 / 88

SLIDE 32

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

Morphological classification of interneurons

WEB-BASED SYSTEM. 42 EXPERTS. 320 INTERNEURONS (DeFelipe et al., 2013)

The five class variables to be predicted by the classification model Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 31 / 88

SLIDE 33

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

Morphological classification of interneurons

WEB-BASED SYSTEM. 42 EXPERTS. 320 INTERNEURONS (DeFelipe et al., 2013)

The dataset with the five class variables Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 32 / 88

SLIDE 34

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

Morphological classification of interneurons

WEB-BASED SYSTEM. 42 EXPERTS. 320 INTERNEURONS (DeFelipe et al., 2013)

Inter-expert agreement in each of the class variables Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 33 / 88

SLIDE 35

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

Morphological classification of interneurons

WEB-BASED SYSTEM. 42 EXPERTS. 320 INTERNEURONS (DeFelipe et al., 2013)

Bayesian network models of the choice behaviour of expert 16 and expert 27 when selecting Martinotti Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 34 / 88

SLIDE 36

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

Morphological classification of interneurons

WEB-BASED SYSTEM. 42 EXPERTS. 320 INTERNEURONS (DeFelipe et al., 2013)

Bayesian network models of the choice behaviour of expert 16 and expert 27 when selecting common basket Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 35 / 88

SLIDE 37

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

Morphological classification of interneurons

Three different approaches

A. NEURON LABEL AS ITS MOST FREQUENT CELL TYPE
B. DIFFERENT LABEL RELIABILITY THRESHOLDS
C. GEM ALGORITHM FOR PROBABILISTIC CLASS LABELS

Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 36 / 88

SLIDE 38

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

Morphological classification of interneurons

A. NEURON LABEL AS ITS MOST FREQUENT CELL TYPE (DeFelipe et al., 2013)

Supervised classification problem. The labels of C5 are the modes of the assignments provided by the 42 experts

Classifiers: Gaussian naive Bayes, discrete naive Bayes, radial basis function, support vector machines, 1-nearest neighbor, k-nearest neighbors, rule induction, classification trees, random forests, and random trees Feature selection methods: univariate filter (gain ratio), and multivariate filter (correlation feature selection) Best accuracies: 85.48% for C1, 81.33% for C2, 73.86% for C3, 60.17% for C4, and 62.24% for C5

Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 37 / 88

SLIDE 39

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

Morphological classification of interneurons

B. DIFFERENT LABEL RELIABILITY THRESHOLDS (Mihaljevi´

c et al., 2015)

D5,X

25,−

X1, . . . , X214 C5 2 0.2, . . . , 4.1 MA 3 1.2, . . . , 4.2 HT . . . . . . . . . 237 1.0, . . . , 2.2 CB

D5,1234

25,21

C1, . . . , C4 C5 2 Tl., . . . , As. MA 3 Tl., . . . , Ds. HT . . . . . . . . .

D5,X,1234

25,−,21

X1, . . . , X214, C1, . . . , C4 C5 2 0.2, . . . , 4.1, Tl., . . . , As. MA 3 1.2, . . . , 4.2, Tl., . . . , Ds. HT . . . . . . . . .

D

X1, . . . , X214 C1 C2 C3 C4 C5 1 0.5, . . . , 2.1

Il. (30)
Ic. (30)
Ce. (30)
Bo. (10)

CT (21) 2 0.2, . . . , 4.1

Tl. (40)
Tc. (29)
De. (40)
As. (39)

MA (40) 3 1.2, . . . , 4.2

Tl. (40)
Ic. (39)
De. (40)
Ds. (40)

HT (40) . . . . . . . . . . . . . . . . . . . . . 237 1.0, . . . , 2.2

Il. (21)
Ic. (24)
Ce. (35)
As. (7)

CB (26)

A schematic overview of our automatic categorization of interneurons according to 214 morphological variables and 4 high-level axonal features C1–C4 Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 38 / 88

SLIDE 40

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

Morphological classification of interneurons

B. DIFFERENT LABEL RELIABILITY THRESHOLDS (Mihaljevi´

c et al., 2015)

50 100 150 200 250 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41

Threshold Neurons

Class

CB CH CT HT LB MA NG

Number of neurons of different types of C5 versus label reliability threshold (C1–C4 with a threshold of 21) Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 39 / 88

SLIDE 41

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

Morphological classification of interneurons

B. DIFFERENT LABEL RELIABILITY THRESHOLDS (Mihaljevi´

c et al., 2015) Bayesian network classifiers (Bielza and Larra˜ naga, 2014b)

(a) Naive Bayes (NB) structure (b) Selective naive Bayes (NB-FSS) structure p(x, c) = p(c)p(x1|c)p(x2|c)p(x3|c)p(x4|c)p(x5|c) p(x, c) = p(c)p(x1|c)p(x2|c)p(x3|c)p(x5|c) (a) Attribute weighted naive Bayes (AWNB) structure (b) Tree augmented naive Bayes (TAN) structure p(x, c) = p(c)p(x1|c)0.9p(x2|c)1p(x3|c)0.4p(x4|c)0 p(x, c) = p(c)p(x1|x2, c)p(x2|x3, c)p(x3|c) p(x5|c)0.8 p(x4|x3, c)p(x5|x4, c) Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 40 / 88

SLIDE 42

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

Morphological classification of interneurons

B. DIFFERENT LABEL RELIABILITY THRESHOLDS (Mihaljevi´

c et al., 2015)

40 60 80 100 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28

Threshold Accuracy (%)

Classifier NB NB−FSS AWNB TAN

From D5,X

10,− until D5,X 28,−

40 60 80 100 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28

Threshold Accuracy (%)

From D5,1234

10,21

until D5,1234

28,21

40 60 80 100 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28

Threshold Accuracy (%)

From D5,X,1234

10,−,21

until D5,X,1234

28,−,21

Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 41 / 88

SLIDE 43

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

Morphological classification of interneurons

B. DIFFERENT LABEL RELIABILITY THRESHOLDS (Mihaljevi´

c et al., 2015)

Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 42 / 88

SLIDE 44

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

Morphological classification of interneurons

B. DIFFERENT LABEL RELIABILITY THRESHOLDS (Mihaljevi´

c et al., 2015)

Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 43 / 88

SLIDE 45

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

Morphological classification of interneurons

B. DIFFERENT LABEL RELIABILITY THRESHOLDS (Mihaljevi´

c et al., 2015)

Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 44 / 88

SLIDE 46

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

Morphological classification of interneurons

B. DIFFERENT LABEL RELIABILITY THRESHOLDS (Mihaljevi´

c et al., 2015)

Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 45 / 88

SLIDE 47

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

Morphological classification of interneurons

C. GEM ALGORITHM FOR PROBABILISTIC CLASS LABELS (L´
pez-Cruz, 2013)

Morpho var. C5 class variable values cell X1 ... X214 π1 π2 π3 π4 π5 π6 π7 π8 π9 π10 1 10.1 ... 6.6

4 42 7 42 42 10 42 2 42 4 42 3 42 4 42 4 42 4 42

2 3.7 ... 7.7

6 42 2 42 1 42 1 42 1 42 4 42 8 42 10 42 9 42 42

3 5.9 ... 9.2

10 42 10 42 1 42 1 42 42 42 42 2 42 8 42 10 42

4 11.2 ... 10.1

6 42 4 42 4 42 4 42 2 42 10 42 1 42 1 42 5 42 5 42

... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... 236 13.6 ... 5.7

3 42 2 42 5 42 6 42 4 42 2 42 2 42 2 42 4 42 12 42

237 8.9 ... 8.8

7 42 3 42 2 42 2 42 4 42 2 42 10 42 10 42 1 42 1 42 Probabilistic class labels for each cell according to the votes received for each interneuron type from the 42 experts. The ten values of C5 are denoted by: (1) arcade, (2) Cajal-Retzius, (3) Chandelier, (4) common basket, (5) horse-tail, (6) large basket, (7) Martinotti, (8) neurogliaform, (9) common type, (10) other

The generalized expectation maximization (GEM) algorithm is particularized to learn Bayesian classifiers with different structural complexities ℓ(DX,Π|Θ) =

N

i=1

ln  

c∈ΩC

πicpC(c; θC)fX|C(xi|c; θX|C)  

Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 46 / 88

SLIDE 48

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

EQ-5D health states from PDQ-39 in Parkinson disease

Parkinson disease motor symptoms Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 47 / 88

SLIDE 49

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

EQ-5D health states from PDQ-39 in Parkinson disease

PDQ-39 and EQ-5D: quality of life instruments to measure the degree of disability in PD 39-item Parkinson’s Disease Questionnaire: a specific instrument PDQ-39 captures patient’s perception of his illness covering 8 dimensions:

1

Mobility

2

Activities of daily living

3

Emotional well-being

4

Stigma

5

Social support

6

Cognitions

7

Communication

8

Bodily discomfort

Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 48 / 88

SLIDE 50

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

EQ-5D health states from PDQ-39 in Parkinson disease

European Quality of Life - 5 Dimensions: a generic instrument EQ-5D is a generic measure of health for clinical and economic appraisal

Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 49 / 88

SLIDE 51

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

EQ-5D health states from PDQ-39 in Parkinson disease

Mapping PDQ-39 to EQ-5D PDQ1 PDQ2 ... ... PDQ39 EQ1 EQ2 EQ3 EQ4 EQ5 3 1 ... ... 3 1 3 3 2 1 2 3 ... ... 2 1 1 2 3 2 5 2 ... ... 4 1 3 3 1 2 ... ... ... ... ... ... ... ... ... ... 4 4 ... ... 3 3 1 2 3 2 4 4 ... ... 3 3 1 2 3 2 5 5 ... ... 4 2 3 2 3 3

φ : (PDQ1, ..., PDQ39) → (EQ1, ..., EQ5)

Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 50 / 88

SLIDE 52

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

EQ-5D health states from PDQ-39 in Parkinson disease

MULTIPLE DIAGNOSIS PROBLEM. DIRECT APPROACH (Peng and Reggia, 1987a,1987b)

X1 . . . Xm C1 . . . Cd (x(1), c(1)) x(1)

1

. . . x(1)

m

c(1)

1

. . . c(1)

d

(x(2), c(2)) x(2)

1

. . . x(2)

m

c(2)

1

. . . c(2)

d

. . . . . . . . . (x(N), c(N)) x(N)

1

. . . x(N)

m

c(N)

1

. . . c(N)

d Optimal diagnosis as abductive inference: searching for the most probable explanation (MPE) (c∗

1 , . . . , c∗ d )

= arg max

(c1,...,cd ) p(C1 = c1, . . . , Cd = cd|X1 = x1, . . . , Xm = xm)

= arg max

(c1,..,cd ) p(C1 = c1, .., Cd = cd)p(X1 = x1, .., Xm = xm|C1 = c1, .., Cd = cd)

Number of parameters to be estimated for the case of binary predictors and classes: 2d − 1 + 2d(2m − 1)

Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 51 / 88

SLIDE 53

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

EQ-5D health states from PDQ-39 in Parkinson disease

MULTIPLE DIAGNOSIS WITH MULTI-DIMENSIONAL BAYESIAN NETWORK CLASSIFIERS (MBCs) (van der Gaag and de Waal, 2006) In a multi-dimensional Bayesian network classifier (MBC) the set of vertices V is partitioned into: VC = {C1, ..., Cd} of class variables and VX = {X1, ..., Xm} of feature variables with (d + m = n) Three subgraphs in the structure of a multi-dimensional Bayesian network classifier: Class subgraph: GC, Bridge subgraph: GCX and Feature subgraph: GX

Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 52 / 88

SLIDE 54

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

EQ-5D health states from PDQ-39 in Parkinson disease

TRACTABILITY OF MPE IN MBCS WITH CLASS BRIDGE DECOMPOSABLE MBCS (Bielza et al., 2011) MPE is generally NP-hard in Bayesian networks (Kwisthout, 2011) An MBC is class-bridge decomposable MBC if:

1

GC ∪ GCX can be decomposed as: GC ∪ GCX = r

i=1(GCi ∪ G(CX)i) where

GCi ∪ G(CX)i with i = 1, ..., r are its maximal connected components

2

Non-shared children: Ch(VCi) ∩ Ch(VCj) = ∅, with i, j = 1, ..., r and i = j, where Ch(VCi) denotes the children of all the variables in VCi

(a) A CB-decomposable MBC (b) Its two maximal connected components Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 53 / 88

SLIDE 55

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

EQ-5D health states from PDQ-39 in Parkinson disease

COLLABORATION WITH ABBOTT LABORATORIES. 488 PARKINSON’S PATIENTS (Borchani et al., 2012) Empirical comparison with state of the art methods in a 5-fold cross-validation Method Mean accuracy Exact match MB-MBC 0.7119 ± 0.0338 0.2030 ± 0.0718 CB-MBC 0.6807 ± 0.0285 0.1865 ± 0.0429 MNL 0.6926 ± 0.0430 0.1802 ± 0.0713 OLS 0.4201 ± 0.0252 0.0123 ± 0.0046 CLAD 0.4254 ± 0.0488 0.0143 ± 0.0171

Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 54 / 88

SLIDE 56

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

EQ-5D health states from PDQ-39 in Parkinson disease

MB-MBC graphical structure

Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 55 / 88

SLIDE 57

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

Our book: Bielza and Larra˜ naga (2019). Cambridge University Press

Data-Driven Computational Neuroscience

I INTRODUCTION IV UNSUPERVISED PATTERN RECOGNITION

1. Neuroscience
11. Non-probabilistic clustering
12. Probabilistic clustering

II STATISTICS

2. Exploratory data analysis

V PROBABILISTIC GRAPHICAL MODELS

3. Probability theory and random variables
13. Bayesian networks
4. Probabilistic inference
14. Markov networks

III SUPERVISED PATTERN RECOGNITION VI SPATIAL STATISTICS

5. Performance evaluation
15. Spatial statistics
6. Feature subset selection
7. Non-probabilistic classifiers
8. Probabilistic classifiers
9. Metaclassifiers
10. Multi-dimensional classifiers

> 700 pages, > 1, 300 references, 117 tables, 249 figures, ... Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 56 / 88

SLIDE 58

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

Outline

1 Introduction 2 Bayesian Networks 3 Neuroscience 4 Industry 5 Sport 6 Estimation of Distribution Algorithms 7 Conclusions 8 References

Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 57 / 88

SLIDE 59

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

The fourth industrial revolution

4 3 2 1

Industrial technology shifts (Larra˜ naga et al. (2018)) Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 58 / 88

SLIDE 60

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

Laser surface heat treatment

Time-temperature-transformation curve with a possible cooling trajectory of a hardening process (Larra˜ naga et al. (2018)) Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 59 / 88

SLIDE 61

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

Monitoring the laser surface heat treatment

Physical arrangement of the different elements used to carry out and monitor the laser surface heat treatment of the steel cylinders (Gabilondo et al. (2015)) Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 60 / 88

SLIDE 62

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

The obstacle on the surface of the cylinders

During the heat treatment the spot was programmed to avoid an obstacle on the surface of the cylinders. There were three different variants: (a) at the top, (b) in the middle, or (c) at the bottom (Larra˜ naga et al. (2018)) Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 61 / 88

SLIDE 63

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

Segmentation of the frame into 14 regions

The 14 regions into which the frame was segmented. The regions adjacent to the edges were considered to be background (Larra˜ naga et al. (2018)) Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 62 / 88

SLIDE 64

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

Dimensionality reduction

R[t] = (R1[t], ..., Rm[t]) Q[t] = (Q1[t], ..., Qk·s[t]), where k · s ≪ m

Extraction of s statistical measures from the pixel values of each cluster Ri[t] t t + 1 T ...

Video with frames of m pixels Video with frames of k clusters

1 m 1 2 c

Dimensionality reduction of the feature vector R[t] to Q[t] based on segmenting the frames into k different regions and extracting s statistical measures from their pixel values (Larra˜ naga et al. (2018)) Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 63 / 88

SLIDE 65

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

Anomaly detection method by likelihood

Normality Model

TRAINING DATASET

Anomaly Score Anomaly Threshold

Normal Abnormal Anomaly Score Anomaly Threshold

1 Compute a probabilistic model (based on dynamic Bayesian networks) for the normal instances 2 Establish a threshold in this joint probability distribution 3 Compare the likelihood of the new instance with the likelihood threshold Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 64 / 88

SLIDE 66

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

Dynamic Bayesian networks Assumptions

Dynamic Bayesian networks (Dean and Kanazawa, 1989) are the temporal extension of Bayesian networks to stochastic processes observed at discrete time periods Two assumptions: first-order Markov process and stationarity P(X[0], ..., X[T]) = n

i=1 P(Xi[0] | Pa(Xi[0])) T t=1

n

i=1 P(Xi[t] | Pa(Xi[t])) X1[0] X2[0] X3[0] X1[t − 1] X2[t − 1] X3[t − 1] X1[t] X2[t] X3[t]

X1[0] X2[0] X3[0] X1[1] X2[1] X3[1] X1[2] X2[2] X3[2]

Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 65 / 88

SLIDE 67

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

Each region and its parents

Region 1 Region 2 Region 3 Region 4 Region 5 Region 6 Region 7 Region 8 Region 9 Illustration of the nine regions with variables that were parents (in green) of at least one variable of the target region (in yellow). There were two types of parent regions: regions that produced instantaneous influences only (light green) and regions that produced temporal influences only (dark green) (Larra˜ naga et al. (2018)) Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 66 / 88

SLIDE 68

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

Our book: Larra˜ naga et al. (2018). CRC Press

Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 67 / 88

SLIDE 69

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

Outline

1 Introduction 2 Bayesian Networks 3 Neuroscience 4 Industry 5 Sport 6 Estimation of Distribution Algorithms 7 Conclusions 8 References

Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 68 / 88

SLIDE 70

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

TCT Scout

Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 69 / 88

SLIDE 71

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

Outline

1 Introduction 2 Bayesian Networks 3 Neuroscience 4 Industry 5 Sport 6 Estimation of Distribution Algorithms 7 Conclusions 8 References

Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 70 / 88

SLIDE 72

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

Evolutionary Computation

Techniques

Pioneer work on computer simulation of evolution: Barriceli (1954), Fraser (1957) Evolutionary programming (Fogel, 1964) Evolution strategies (Rechenberg, 1973 and Schwefel, 1974) Genetic algorithms (Holland, 1975) Ant colony optimization (Dorigo, 1992) Cultural algorithms (Reynolds, 1994) Particle swarm optimization (Kennedy and Eberhart, 1995) Estimation of distribution algorithms (H. M¨ uhlenbein and Paaβ, 1996 and Larra˜ naga and Lozano (2002)) And more: differential evolution, harmony search, artificial immune systems ...

Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 71 / 88

SLIDE 73

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

Evolutionary Computation

Techniques

Pioneer work on computer simulation of evolution: Barriceli (1954), Fraser (1957) Evolutionary programming (Fogel, 1964) Evolution strategies (Rechenberg, 1973 and Schwefel, 1974) Genetic algorithms (Holland, 1975) Ant colony optimization (Dorigo, 1992) Cultural algorithms (Reynolds, 1994) Particle swarm optimization (Kennedy and Eberhart, 1995) Estimation of distribution algorithms (H. M¨ uhlenbein and Paaβ, 1996 and Larra˜ naga and Lozano (2002)) And more: differential evolution, harmony search, artificial immune systems ...

Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 71 / 88

SLIDE 74

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

EDAs. A Toy Example

max O(x) =

6

i=1

xi with xi = 0, 1

Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 72 / 88

SLIDE 75

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

EDAs. A Toy Example

max O(x) =

6

i=1

xi with xi = 0, 1

X1 X2 X3 X4 X5 X6 O(x) 1 1 1 1 3 2 1 1 2 3 1 1 4 1 1 1 1 4 5 1 1 6 1 1 1 1 4 7 1 1 1 1 1 5 8 1 1 9 1 1 1 3 10 1 1 2 11 1 1 1 1 4 12 1 1 1 3 13 1 1 2 14 1 1 2 15 1 1 1 1 1 5 16 1 1 17 1 1 1 1 1 5 18 1 1 1 3 19 1 1 1 1 1 5 20 1 1 1 3

Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 72 / 88

SLIDE 76

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

EDAs. A Toy Example

max O(x) =

6

i=1

xi with xi = 0, 1

X1 X2 X3 X4 X5 X6 O(x) 1 1 1 1 3 2 1 1 2 3 1 1 4 1 1 1 1 4 5 1 1 6 1 1 1 1 4 7 1 1 1 1 1 5 8 1 1 9 1 1 1 3 10 1 1 2 11 1 1 1 1 4 12 1 1 1 3 13 1 1 2 14 1 1 2 15 1 1 1 1 1 5 16 1 1 17 1 1 1 1 1 5 18 1 1 1 3 19 1 1 1 1 1 5 20 1 1 1 3

Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 72 / 88

SLIDE 77

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

EDAs. A Toy Example

Learning the probability distribution from the selected individuals

X1 X2 X3 X4 X5 X6 1 1 1 1 4 1 1 1 1 6 1 1 1 1 7 1 1 1 1 1 11 1 1 1 1 12 1 1 1 15 1 1 1 1 1 17 1 1 1 1 1 18 1 1 1 19 1 1 1 1 1

Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 73 / 88

SLIDE 78

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

EDAs. A Toy Example

Learning the probability distribution from the selected individuals

X1 X2 X3 X4 X5 X6 1 1 1 1 4 1 1 1 1 6 1 1 1 1 7 1 1 1 1 1 11 1 1 1 1 12 1 1 1 15 1 1 1 1 1 17 1 1 1 1 1 18 1 1 1 19 1 1 1 1 1 p(x) = p(x1, . . . , x6) = p(x1)p(x2)p(x3)p(x4)p(x5)p(x6)

Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 73 / 88

SLIDE 79

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

EDAs. A Toy Example

Learning the probability distribution from the selected individuals

X1 X2 X3 X4 X5 X6 1 1 1 1 4 1 1 1 1 6 1 1 1 1 7 1 1 1 1 1 11 1 1 1 1 12 1 1 1 15 1 1 1 1 1 17 1 1 1 1 1 18 1 1 1 19 1 1 1 1 1 p(x) = p(x1, . . . , x6) = p(x1)p(x2)p(x3)p(x4)p(x5)p(x6) p(X1 = 1) =

7 10 Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 73 / 88

SLIDE 80

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

EDAs. A Toy Example

Learning the probability distribution from the selected individuals

X1 X2 X3 X4 X5 X6 1 1 1 1 4 1 1 1 1 6 1 1 1 1 7 1 1 1 1 1 11 1 1 1 1 12 1 1 1 15 1 1 1 1 1 17 1 1 1 1 1 18 1 1 1 19 1 1 1 1 1 p(x) = p(x1, . . . , x6) = p(x1)p(x2)p(x3)p(x4)p(x5)p(x6) p(X1 = 1) =

7 10

p(X2 = 1) =

7 10

p(X3 = 1) =

6 10

p(X4 = 1) =

6 10

p(X5 = 1) =

8 10

p(X6 = 1) =

7 10 Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 73 / 88

SLIDE 81

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

EDAs. A Toy Example

Learning the probability distribution from the selected individuals

X1 X2 X3 X4 X5 X6 1 1 1 1 4 1 1 1 1 6 1 1 1 1 7 1 1 1 1 1 11 1 1 1 1 12 1 1 1 15 1 1 1 1 1 17 1 1 1 1 1 18 1 1 1 19 1 1 1 1 1 p(x) = p(x1, . . . , x6) = p(x1)p(x2)p(x3)p(x4)p(x5)p(x6) p(X1 = 1) =

7 10

p(X2 = 1) =

7 10

p(X3 = 1) =

6 10

p(X4 = 1) =

6 10

p(X5 = 1) =

8 10

p(X6 = 1) =

7 10 Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 73 / 88

SLIDE 82

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

EDAs. A Toy Example

Obtaining the new population by sampling from the probability distribution p(X1 = 1) = 7 10; p(X2 = 1) = 7 10; p(X3 = 1) = 6 10 p(X4 = 1) = 6 10; p(X5 = 1) = 8 10; p(X6 = 1) = 7 10 p(x) = p(x1, . . . , x6) = p(x1)p(x2)p(x3)p(x4)p(x5)p(x6) 0.23 p(X1 = 1) =

7 10 > 0.23 −

→ 1 0.65 p(X2 = 1) =

7 10 > 0.65 −

→ 1 0.89 p(X3 = 1) =

6 10 < 0.89 −

→ 0 0.12 p(X4 = 1) =

6 10 > 0.12 −

→ 1 0.48 p(X5 = 1) =

8 10 > 0.48 −

→ 1 0.54 p(X6 = 1) =

7 10 > 0.54 −

→ 1

Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 74 / 88

SLIDE 83

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

EDAs. A Toy Example

Obtaining the new population by sampling from the probability distribution X1 X2 X3 X4 X5 X6 O(x) 1 1 1 1 1 1 5 2 1 1 1 1 4 3 1 1 1 1 1 5 4 1 1 1 1 4 5 1 1 1 1 1 5 6 1 1 1 1 4 7 1 1 1 3 8 1 1 1 1 4 9 1 1 1 1 4 10 1 1 1 1 4 11 1 1 1 1 4 12 1 1 1 1 4 13 1 1 1 1 4 14 1 1 1 1 4 15 1 1 1 1 1 1 6 16 1 1 1 1 4 17 1 1 1 1 1 5 18 1 1 2 19 1 1 1 3 20 1 1 1 1 1 5

Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 75 / 88

SLIDE 84

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

EDAs. Larra˜

naga et al. (2012); Larra˜ naga et al. (2013)

Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 76 / 88

SLIDE 85

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

Graphical Representation of EDAs

Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 77 / 88

SLIDE 86

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

Probabilistic Models in EDAs

Univariate EDAs: Univariate Marginal Distribution Algorithm (UMDA). M¨ uhlenbein and Paaß (1996) Probabilistic model: pl (x) = n

i=1 pl (xi )

Structural learning: not necessary Bivariate EDAs: Mutual Information Maximization for Input Clustering (MIMIC). De Bonet et al. (1997) Probabilistic model: pπ

l (x) = pl (xi1 | xi2 )pl (xi2 | xi3 ) · · · pl (xin−1 | xin )pl (xin )

Structural learning: best permutation (factorization closest to the empirical distribution in the sense of Kullback-Leibler divergence) Multivariate EDAs: Etxeberria and Larra˜ naga (1999) (EBNA); Pelikan et al. (1999) (BOA); Harik et al. (1999) (EcGA); M¨ uhlenbein and Mahnig (1999) (LFDA) Probabilistic model: pl (x) = n

i=1 pl (xi |pai )

Structural learning: directed acyclic graph EDAs in continuous domains: Assuming Gaussianity Univariate: Larra˜ naga et al. (2000) (UMDAG

c )

Bivariate: Larra˜ naga et al. (2000) (MIMICG

c )

Multivariate: Larra˜ naga et al. (2000) (EMNAG

global , EMNAG ee, EGNAG)

Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 78 / 88

SLIDE 87

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

Probabilistic Models in EDAs

Univariate EDAs: Univariate Marginal Distribution Algorithm (UMDA). M¨ uhlenbein and Paaß (1996) Probabilistic model: pl (x) = n

i=1 pl (xi )

Structural learning: not necessary Bivariate EDAs: Mutual Information Maximization for Input Clustering (MIMIC). De Bonet et al. (1997) Probabilistic model: pπ

l (x) = pl (xi1 | xi2 )pl (xi2 | xi3 ) · · · pl (xin−1 | xin )pl (xin )

Structural learning: best permutation (factorization closest to the empirical distribution in the sense of Kullback-Leibler divergence) Multivariate EDAs: Etxeberria and Larra˜ naga (1999) (EBNA); Pelikan et al. (1999) (BOA); Harik et al. (1999) (EcGA); M¨ uhlenbein and Mahnig (1999) (LFDA) Probabilistic model: pl (x) = n

i=1 pl (xi |pai )

Structural learning: directed acyclic graph EDAs in continuous domains: Assuming Gaussianity Univariate: Larra˜ naga et al. (2000) (UMDAG

c )

Bivariate: Larra˜ naga et al. (2000) (MIMICG

c )

Multivariate: Larra˜ naga et al. (2000) (EMNAG

global , EMNAG ee, EGNAG)

Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 78 / 88

SLIDE 88

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

Graphical Representation of EDAs

Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 79 / 88

SLIDE 89

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

EDAs

Obtaining the new population by sampling with PLS (Henrion, 1988)

Given an ancestral ordering, π, of the nodes in the directed probabilistic graphical model (Bayesian network or Gaussian Bayesian network): for j = 1, 2, . . . , M for i = 1, 2, . . . , n xπ(i) ← generate a value from p(xπ(i)|paπ(i))

Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 80 / 88

SLIDE 90

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

Our book: Larra˜ naga and Lozano (2002). Kluwer Academics

Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 81 / 88

SLIDE 91

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

Outline

1 Introduction 2 Bayesian Networks 3 Neuroscience 4 Industry 5 Sport 6 Estimation of Distribution Algorithms 7 Conclusions 8 References

Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 82 / 88

SLIDE 92

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

BAYESIAN NETWORKS FOR DATA SCIENCE Bayesian networks: a machine learning paradigm based on probability theory and graph theory Advantages:

Transparency, interpretability Exact inference algorithms (reasoning) for predictive reasoning, diagnostic reasoning, intercausal reasoning and abductive inference Learning algorithms from data for clustering, multi-view clustering, supervised classification, multi-dimensional classification, anomaly detection, discovery associations, feature subset selection, multi-output regression, ... Inference and learning algorithms for temporal and data stream scenarios EDAs: evolutionary computation based on learning and simulation of Bayesian networks

Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 83 / 88

SLIDE 93

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

Outline

1 Introduction 2 Bayesian Networks 3 Neuroscience 4 Industry 5 Sport 6 Estimation of Distribution Algorithms 7 Conclusions 8 References

Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 84 / 88

SLIDE 94

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

References

C. Bielza and P

. Larra˜ naga (2014a). Discrete Bayesian network classifiers: A survey. ACM Computing Surveys, 47 (1), Article 5

C. Bielza and P

. Larra˜ naga (2014b). Bayesian networks in neuroscience: A survey. Frontiers in Computational Neuroscience, 8, Article 131

C. Bielza and P

. Larra˜ naga (2019). Data-Driven Computational Neuroscience. Cambridge University Press

C. Bielza, G. Li and P

. Larra˜ naga (2011). Multi-dimensional classification with Bayesian networks. International Journal of Approximate Reasoning, 52, 705-727

H. Borchani, C. Bielza, P

. Mart´ ınez-Mart´ ın and P . Larra˜ naga (2012). Multidimensional Bayesian network classifiers applied to predict the European quality of life-5 dimensions (EQ-5D) from the 39-item Parkinson’s disease questionnaire (PDQ-39), Journal of Biomedical Informatics, 45, 1175-1184

T. Dean and K. Kanazawa (1989). A model for reasoning about persistence and causation. Computational

Intelligence, 5(2), 142-150

J. DeFelipe, P

.L. L´

pez-Cruz, R. Benavides-Piccione, C. Bielza, P

. Larra˜ naga, S. Anderson, A. Burkhalter, B. Cauli, A. Fair´ en, D. Feldmeyer, G. Fishell, D. Fitzpatrick, T.F. Freund, G. Gonz´ alez-Burgos, S. Hestrin, S. Hill, P .R. Hof, J. Huang, E.G. Jones, Y. Kawaguchi, Z. Kisv´ arday, Y. Kubota, D.A. Lewis, O. Mar´ ın, H. Markram, C.J. McBain, H.S. Meyer, H. Monyer, S.B. Nelson, K. Rockland, J. Rossier, J.L.R. Rubenstein, B. Rudy, M. Scanziani, G.M. Shepherd, C.C. Sherwood, J.F. Staiger, G. Tam´ as, A. Thomson, Y. Wang, R. Yuste, and G.A. Ascoli (2013). New insights into the classification and nomenclature of cortical GABAergic

interneurons. Nature Reviews Neuroscience, 14(3), 202-216

Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 85 / 88

SLIDE 95

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

References

P . Domingos (2015). The Master Algorithm. Basic Books Future of Life Institute (2017). Asilomar conference on beneficial AI. https://ai-ethics.com/2017/08/11/future-of-life-institute-2017-asilomar-conference

A. Gabilondo, J. Dom´

ınguez, C. Soriano, and J.L. Oca˜ na (2015). Method and system for laser hardening of a surface of a workpiece. EP Patent App. EP20,130,758,815

D. Koller and N. Friedman (2009). Probabilistic Graphical Models: Principles and Techniques. The MIT

Press

J. Kwisthout (2011). Most probable explanations in Bayesian networks: Complexity and tractability.

International Journal of Approximate Reasoning, 52(9), 1452-1469 P . Larra˜ naga, J. A. Lozano (2002). Estimation of Distribution Algorithms. A New Tool for Evolutionary

Computation. Kluwer Academic Publishers

P . Larra˜ naga, H. Karshenas, C. Bielza, R. Santana (2013). A review on evolutionary algorithms in Bayesian network learning and inference tasks. Information Sciences, 233, 109–125 P . Larra˜ naga, H. Karshenas, C. Bielza, R. Santana (2012). A review on probabilistic graphical models in evolutionary computation. Journal of Heuristics, 18(5), 795–819 P . Larra˜ naga, D. Atienza, J. Diaz-Rojo, C. E. Puerto-Santana, A. Ogbechie and C. Bielza (2018). Industrial Applications of Machine Learning. CRC Press P .L. L´

pez-Cruz (2013). Contributions to Bayesian Network Learning with Applications to Neuroscience.

PhD Thesis. Technical University f Madrid

S. Luengo-Sanchez, I. Fernaud-Espinosa, C. Bielza, R. Benavides-Piccione, P

. Larra˜ naga, and J. DeFelipe (2018). 3D morphology-based clustering and simulation of human pyramidal cell dendritic spines. PLOS Computational Biology, 14(6), e1006221 Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 86 / 88

SLIDE 96

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

References

B. Mihaljevic, R. Benavides-Piccione, C. Bielza, J. DeFelipe, and P

. Larra˜ naga (2015). Bayesian network classifiers for categorizing cortical GABAergic interneurons. Neuroinformatics, 13(2), 192208

J. Pearl (1988). Probabilistic Reasoning in Intelligent Systems. Morgan Kaufmann
Y. Peng and J. A. Reggia (1987a). A probabilistic causal model for diagnostic problem solving. Part I:

Integrating symbolic causal inference with numeric probabilistic inference. IEEE Transactions on Systems, Man and Cybernetics, 17 (2), 146-162

Y. Peng and J. A. Reggia (1987b). A probabilistic causal model for diagnostic problem solving. Part II:

Diagnostic strategy. IEEE Transactions on Systems, Man and Cybernetics, 17 (3), 395-406

A. Peters and I.R. Kaiserman-Abramof (1970). The small pyramidal neuron of the rat cerebral cortex. The

perikaryon, dendrites and spines. American Journal of Anatomy, 127, 321-356

C. Rudin (2019). Stop explaining black box machine learning models for high stakes decisions and use

interpretable models instead. Nature Machine Intelligence, 1, 206-215

G. Trabelsi (2013). New Structure Learning Algorithms and Evaluation Methods for Large Dynamic

Bayesian Networks. PhD Thesis. Universit´ e de Nantes L.C. van der Gaag and P .R. de Waal (2006). Multi-dimensional Bayesian network classifiers. Proceedings of the 3rd European Workshop on Probabilistic Graphical Models, 107-114 Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 87 / 88

SLIDE 97

Intro Bayes Nets Neuro Industry Sport EDAs Concl Ref

APPLICATIONS IN NEUROSCIENCE, INDUSTRY 4.0, AND SPORTS

Pedro Larra˜ naga

Computational Intelligence Group Artificial Intelligence Department Universidad Polit´ ecnica de Madrid

Bayesian Networks: From Theory to Practice International Black Sea University Autumn School on Machine Learning 3-11 October 2019, Tbilisi, Georgia Pedro Larra˜ naga Neuroscience, Industry 4.0, and Sports 88 / 88