Part 2 MDL in L in Ac Actio ion Jilles V Vreeke ken 1 - - PowerPoint PPT Presentation

Jilles V Vreeke ken

Part 2 MDL in L in Ac Actio ion

1

Explicit Coding

Ad hoc sounds bad, but is it really?

• Bayesian learning for instance, is in

inherent ntly ly subjectiv ive, plus

• biasing search is a time-honoured tradition in data analysis

Using an expli licit encoding ing allows us to steer towards the type of structure we want nt to dis iscover We so also mitigate one of the practical weak spots of AIT

• all data is a string, but wouldn’t it be nice if the structure

you found would not depend on the order of the data?

2

Matrix Factorization

The rank of a matrix 𝑩 is

• number of rank-1 matrices that when summed form 𝑩 (Sche

hein n rank)

3

+ + … =

𝑑3

𝑐3

𝑑1

𝑐1 𝑐2

𝑑2

𝒄𝟐 ∘ 𝒅𝟐 𝒄𝟑 ∘ 𝒅𝟑 𝒄𝟒 ∘ 𝒅𝟒

𝑩

The rank of a Boolean matrix 𝑩 is

• number of rank-1 matrices that when summed form 𝑩 (Sche

hein n rank)

𝑑1

𝑐2

Boolean Matrix Factorization

(Miettinen et al 2006, 2008) 4

𝑑1

𝑐1 𝑐2

…

𝑑1 𝑐1 𝑑1 𝑑2 𝑑3 𝑐2 𝑐3

𝑩

𝒄𝟐 ∘ 𝒅𝟐 𝒄𝟑 ∘ 𝒅𝟑 𝒄𝟒 ∘ 𝒅𝟒

+ + =

The rank of a Boolean matrix 𝑩 is

• number of rank-1 matrices that when summed form 𝑩 (Sche

hein n rank)

• noise quickly inflate the ‘true’ latent rank to min 𝑜, 𝑛

𝑐1

Boolean Matrix Factorization

(Miettinen et al 2006, 2008) 5

= + + …

𝑩

𝒄𝟐 ∘ 𝒅𝟐 𝒄𝟑 ∘ 𝒅𝟑 𝒄𝟒 ∘ 𝒅𝟒

𝑐1 𝑐1

Boolean Matrix Factorization

Noise quickly inflates the rank to min (𝑜, 𝑛)

• how can we determine the ‘true’ latent rank?

(Miettinen & Vreeken 2012, 2014) 6

𝑩

𝑪 ∘ 𝑫

≈

Boolean Matrix Factorization

Separating structure and noise

• matrices 𝐶 and 𝐷 contain structure, matrix 𝐹 contains noise

(Miettinen & Vreeken 2012, 2014) 7

⊕

𝑩

𝑪 ∘ 𝑫 𝑭

=

Boolean Matrix Factorization

Encoding the structure

𝑀 𝑪 = log 𝑜 + log 𝑜 + log 𝑜 𝑐

𝑐∈𝑪

(Miettinen & Vreeken 2012, 2014) 8

⊕

𝑩

𝑪 ∘ 𝑫 𝑭

=

Boolean Matrix Factorization

Encoding the structure

𝑀 𝑫 = log 𝑛 + log 𝑛 + log 𝑛 𝑑

𝑑∈𝑫

(Miettinen & Vreeken 2012, 2014) 9

⊕

𝑩

𝑪 ∘ 𝑫 𝑭

=

Boolean Matrix Factorization

Encoding the noise

𝑀 𝑭 = log 𝑜𝑛 + log 𝑜𝑛 𝑭

(Miettinen & Vreeken 2012, 2014) 10

⊕

𝑩

𝑪 ∘ 𝑫 𝑭

=

Boolean Matrix Factorization

MDL for BMF 𝑀 𝐸, 𝐼 = 𝑀 𝑪 + 𝑀 𝑫 + 𝑀(𝑭)

(Miettinen & Vreeken 2012, 2014) 11

⊕

𝑩

𝑪 ∘ 𝑫 𝑭

=

Pattern Mining

The idea deal outcome of pattern mining

• patterns that show the structure of the data
• preferably a small set, without redundancy or noise

Frequent pattern mining does not

• t achieve this
• pattern explosion → overly many, overly redundant results

MDL allows us to effectively pursue the ideal

• we want a group of patterns that summarise the data well
• we take a patt

attern s set t mining approach

(Tatti & Vreeken 2012, Bertens et al. 2016, Bhattacharyya & Vreeken 2017) (for transaction data, Vreeken et al (2011), for graphs Koutra et al (2014) 12

Event sequences

(Tatti & Vreeken 2012, Bertens et al. 2016, Bhattacharyya & Vreeken 2017) 13

• ne, or

multiple sequences

a a a b d c d b a b c , a a b d c d b , a a a b d c d b a , … } { { a, b, c, d, … } a a a b d c d b a b c a b d a a b c Alphabet Ω Data 𝐸

Event sequences

(Tatti & Vreeken 2012, Bertens et al. 2016, Bhattacharyya & Vreeken 2017) 14

a a a b d c d b a b c , a a b d c d b , a a a b d c d b a , … } { Alphabet Ω { a, b, c, d, … } a a a b d c d b a b c a b d a a b c Data 𝐸 Patterns

serial episodes

a b a b a b a b

‘subsequences allowing gaps’

• ne, or

multiple sequences

Event sequences

(Tatti & Vreeken 2012, Bertens et al. 2016, Bhattacharyya & Vreeken 2017) 15

a a a b d c d b a b c , a a b d c d b , a a a b d c d b a , … } { { a, b, c, d, … } a a a b d c d b a b c a a d a a b c Patterns

serial episodes

a b a b a b a b

‘subsequences allowing gaps’

d c d c

• ne, or

multiple sequences

b b Alphabet Ω Data 𝐸

Models

As models we use code de tab tables

• dictionary of patterns & codes
• always contains all singletons

We use optimal prefix codes

• easy to compute,
• behave predictably,
• good results,
• more details follow

16

a b d c a b d c abc da p q ? ? ! !

Encoding Event Sequences

17

The length of the code for pattern 𝑌

𝑀 = − log 𝑞

= −log (

𝑣𝑣𝑣(𝑌) ∑𝑣𝑣𝑣(𝑍))

The length of the code stream 𝑀 𝐷𝑞 = ∑ 𝑣𝑣𝑣 𝑌 𝑀( )

𝑌∈𝐷𝐷

a a a b d c d b a b c a b d c a a a b d c d b a b c a b d c X X X X Data 𝐸:

𝐷𝑈

1:

Encoding 1: using only singletons

𝐷𝑞

Encoding Event Sequences

18

a d b a a a b d c d b a b c abc da p q ? ? ! ! p q p ! ? ? ! ! ! ! a a a b d c d b a b c Alignment: p ! ? ! q ? ! p ! !

gap gap

a b d c a b d c Data 𝐸: Encoding 2: using patterns

𝐷𝑞 𝐷𝑣 𝐷𝑈2:

Encoding Event Sequences

19

Data 𝐸: a a a b d c d b a b c

The length of a gap code for pattern 𝑌 𝑀 = − log(𝑞 )) and analogue for non-gap codes !

a d b abc da p q ? ? ! ! p q p ? ? p Encoding 2: using patterns

𝐷𝑞 𝐷𝑣

? a b d c a b d c ! ? ? ! ! ! !

𝐷𝑈2:

Encoding Event Sequences

20

which leaves us to define 𝑀(𝐷𝑈 ∣ 𝐷) By which, the encoded size of 𝐸 given 𝐷𝑈 and 𝐷 is 𝑀 𝐸 𝐷𝑈 = 𝑀 𝐷𝑞 𝐷𝑈 + 𝑀(𝐷𝑣 ∣ 𝐷𝑈)

Encoding a Code T able

21

X X ? ! a z a z … Y Y ? ! …

𝑀(𝐷𝑈 ∣ 𝐷, 𝐸) consists of

Encoding a Code T able

(Rissanen 1983) 22

X X ? ! a z a z … Y Y ? ! …

𝑀(𝐷𝑈 ∣ 𝐷, 𝐸) consists of

1) base singleton counts in 𝐸

𝑀ℕ Ω

+ 𝑀ℕ 𝐸 + log 𝐸 − 1 Ω − 1

Encoding a Code T able

(Rissanen 1983) 23

X X ? ! a z Y Y ? ! … a z …

𝑀(𝐷𝑈 ∣ 𝐷, 𝐸) consists of

1) base singleton counts in 𝐸

𝑀ℕ Ω

+ 𝑀ℕ 𝐸 + log 𝐸 − 1 Ω − 1

Encoding a Code T able

24

X X ? ! a z Y Y ? ! … a z …

𝑀(𝐷𝑈 ∣ 𝐷, 𝐸) consists of

1) base singleton counts in 𝐸

𝑀ℕ Ω

+ 𝑀ℕ 𝐸 + log 𝐸 − 1 Ω − 1

2) number of patterns, total, and per pattern usage 𝑀ℕ 𝒬 + 1 + 𝑀ℕ 𝑣𝑣𝑣 𝒬 + 1 + log 𝑣𝑣𝑣 𝒬 − 1

𝒬 − 1

𝒬

Encoding a Code T able

25

X X ? ! a z Y Y ? ! … a z …

𝑀(𝐷𝑈 ∣ 𝐷, 𝐸) consists of

1) base singleton counts in 𝐸

𝑀ℕ Ω

+ 𝑀ℕ 𝐸 + log 𝐸 − 1 Ω − 1

2) number of patterns, total, and per pattern usage 𝑀ℕ 𝒬 + 1 + 𝑀ℕ 𝑣𝑣𝑣 𝒬 + 1 + log 𝑣𝑣𝑣 𝒬 − 1

𝒬 − 1

𝒬

Encoding a Code T able

26

X X ? ! a z Y Y ? ! … a z …

𝑀(𝐷𝑈 ∣ 𝐷, 𝐸) consists of

1) base singleton counts in 𝐸

𝑀ℕ Ω

+ 𝑀ℕ 𝐸 + log 𝐸 − 1 Ω − 1

2) number of patterns, total, and per pattern usage 𝑀ℕ 𝒬 + 1 + 𝑀ℕ 𝑣𝑣𝑣 𝒬 + 1 + log 𝑣𝑣𝑣 𝒬 − 1

𝒬 − 1

3) per pattern 𝑌 : its length, elements, and number of gaps 𝑀ℕ 𝑌 − log 𝑞 𝑦 𝐸

𝑦∈𝑌

+ 𝑀ℕ 𝑣𝑕𝑞𝑣 𝑌 + 1

𝒬

Encoding a Code T able

27

X X ? ! a z Y Y ? ! … a z …

𝑀(𝐷𝑈 ∣ 𝐷, 𝐸) consists of

1) base singleton counts in 𝐸

𝑀ℕ Ω

+ 𝑀ℕ 𝐸 + log 𝐸 − 1 Ω − 1

2) number of patterns, total, and per pattern usage 𝑀ℕ 𝒬 + 1 + 𝑀ℕ 𝑣𝑣𝑣 𝒬 + 1 + log 𝑣𝑣𝑣 𝒬 − 1

𝒬 − 1

3) per pattern 𝑌 : its length, elements, and number of gaps 𝑀ℕ 𝑌 − log 𝑞 𝑦 𝐸

𝑦∈𝑌

+ 𝑀ℕ 𝑣𝑕𝑞𝑣 𝑌 + 1

𝒬

Encoding Event Sequences

Tatti & Vreeken (2012) Bertens et al. (2016), Bhattacharyya & Vreeken (2017) for transaction data, Vreeken et al (2011) Budhathoki & Vreeken (2015) for graphs Koutra et al (2014) 28

By which we have a lossless encoding. In other words, an objective function. By MDL, our goal is now to minimise

𝑀 𝐷𝑈, 𝐸 = 𝑀 𝐷𝑈 𝐷 + 𝑀(𝐸 ∣ 𝐷𝑈)

for how to do so, please see the papers

Experiments

• synt

nthe hetic data random  no structure found HMM  structure recovered

• real

l data text data for interpretation

(implementation available at http://eda.mmci.uni-saarland/sqs) 29

SQS-CANDS SQS-SEARCH Ω |𝐸| # 𝐷𝑜𝐷𝑣 || || Δ𝑀 Addresses 5 295 56 15 506 138 155 5k JMLR 3 846 788 40 879 563 580 30k Moby Dick 10 277 1 22 559 215 231 10k

Selected Results

(Tatti & Vreeken 2012; Bhattacharyya & Vreeken 2017, Grosse & Vreeken 2017) 30

JMLR

empirical, risk minimization structural indep, component analysis prinicipal Mahalanobis, distance edit, Euclidean, pairwise

• PRES. ADDRESSES

unit[ed] state[s] take oath army navy under circumst.

• econ. public expenditur
• exec. branch. governm.

LOTR

he Verb Conj he

[he said that he]

Conf _ the Noun of

[and even the end of]

the Adj Noun and

[the young Hobbits and]

a c b

Serial Episodes

a

• r

c b d

Choice-episode Ontological Episodes

A 𝒟 b

Clustering

The best clustering is the one that costs the least bits

• similar structure (patterns) within clusters
• different structure (patterns) between clusters

Partition your data such that 𝑀 𝐷 +

• 𝑀(𝐸𝑗, 𝐼𝑗)

(𝐸𝑗,𝐼𝑗)∈𝐷

is minimal

(similar to mixture modelling, but descriptive instead of predictive)

for itemsets, see Van Leeuwen et al (2009) 31

Clustering

Mammals occurrences

• 2221 areas in Europe
• 50x50km each
• 123 mammals
• no location info

for itemsets, see Van Leeuwen et al (2009) 32

k=6, MDL ‘optimal’

Classification

Split your data per cla class

• induce model per class

Then, for uns unseen instances

• assign

n class label of model that encodes it shor

• rtest

st

𝑀 𝑦 𝐼1 < 𝑀 𝑦 𝐼2 → 𝑄 𝑦 𝐼1 > 𝑄(𝑦 ∣ 𝐼2)

(for itemsets, see ECML PKDD’06) 33

Classification by MDL

Van Leeuwen et al (2006) 34

𝑀 𝑦 𝐼1 < 𝑀 𝑦 𝐼2 → 𝑄 𝑦 𝐼1 > 𝑄(𝑦 ∣ 𝐼2)

Split per class Model 𝐼𝑑

∗

per class 𝑑 Encode unseen record Shortest code wins! Run algorithm Database (𝑜 classes)

Outlier Detection

One-Class Classification (aka anomaly detection)

• lots of data for no

normal situation – insufficient data for tar target

Compression models the norm rm

• anomalies will have hig

igh description length 𝑀(𝑢 ∣ 𝐼𝑜𝑜𝑜𝑜

∗

)

Very nice properties

• performa

rmance nce high accuracy

• ve

versa satile no distance measure needed

• chara

ract cter erisa sation ‘this part of t is incompressible’

Smets & Vreeken (2011) Akoglu et al (2012) 35

CompreX on Images

Akoglu et al (2012) 36

Catholic church, Vatican Washington Memorial, D.C. Thames river, Buckingham palace, plain fields, London

Causal Discovery

37

X Y Q S R Z V W

Causal Discovery

Spirtes et al (2000) Marx & Vreeken, Conditional Independence Testing by Stochastic Complexity (2019) 38

We can find the causal al ske keleton using condit itional indepen enden dence tests, but only few few edge edge dir irections ns

X Y Q S R Z V W

Causal Inference

Spirtes et al (2000) Marx & Vreeken, Conditional Independence Testing by Stochastic Complexity (2019) 39

We can find the causal al ske keleton using condit itional indepen enden dence tests, but only few few edge edge dir irections ns

Q S R Z V W

?

X Y

Algorithmic Markov Condition

If 𝑌 → 𝑍, we have,

up to an additive constant,

𝐿 𝑄 𝑌 + 𝐿 𝑄 𝑍 𝑌 ≤ 𝐿 𝑄 𝑍 + 𝐿 𝑄 𝑌 𝑍 That is, we can do ca causal in l inference by identifying the factorization of the joint with the lowe west Kol

• lmog

mogor

• rov c

comp

• mplexity

(Janzing & Schölkopf, IEEE TIT 2012) 40

MDL and Regression

(Grünwald 2007) 41

a1 x + a0

𝑀 𝑁 + 𝑀(𝐸|𝑁)

a10 x10 + a9 x9 + … + a0 errors { }

Modelling the Data

We model 𝑍 as 𝑍 = 𝑔 𝑌 + 𝒪 As 𝑔 we consider lin linear, quad adrat atic, cu cubi bic, expone nent ntial, and recipr proca cal functions, and model the noise using a 0-mean Gaussian. We choose the 𝑔 that minimizes 𝑀 𝑍 𝑌 = 𝑀 𝑔 + 𝑀(𝒪)

Marx & Vreeken (2017) 42

Confidence and Significance

How certain are we?

Marx & Vreeken (2017) 43

ℂ = 𝑀 𝑌 + 𝑀 𝑍 𝑌) − 𝑀 𝑍 + 𝑀 𝑌 𝑍)

 the higher the more certain

𝑀(𝑌 → 𝑍) 𝑀(𝑍 → 𝑌)

Confidence and Significance

How certain are we? Is a given inference significant?

• ur null hypothesis 𝑀0 is that 𝒀 and 𝒁 are only correlated,

we have 𝑀0 =

|𝑀 𝑌→𝑍 −𝑀 𝑍→𝑌 | 2

• we can use the no-hypercompression inequality to test significance

P 𝑀0 𝐸 − 𝑀 𝐸 ≥ 𝑙 ≤ 2−𝑙

Grünwald (2007), Marx & Vreeken (2017) 44

ℂ = 𝑀 𝑌 + 𝑀 𝑍 𝑌) 𝑀 𝑌 + 𝑀 𝑍 − 𝑀 𝑍 + 𝑀 𝑌 𝑍) 𝑀 𝑌 + 𝑀 𝑍

 the higher the more certain  robust w.r.t. sample size

Performance on Benchmark Data

(Tübingen 97 univariate numeric cause-effect pairs, weighted)

45 Marx & Vreeken (2017)

Performance on Benchmark Data

(Tübingen 97 univariate numeric cause-effect pairs, weighted)

Marx & Vreeken (2017) 46

Inferences of state of the art algorithms orde dered ed by confi fide dence values. SLOPE is 85% accurate with 𝛽 = 0.001

Deep Learning

Model selection in deep learning is hard

• way too many ‘free’ parameters for standard regularizers,
• no meaningful prior over networks, and
• uniform prior will lead to overfitting

How about an MDL approach?

• what is the description length of a neural network?

47

MDL for Neural Networks

Suppose neural network 𝐼 ∈ ℋ predicts target 𝑧 given 𝑦 𝑧 = 𝐼(𝑦) How do we encode data given the model?

• if 𝐼(𝑦) is probabilistic, we have 𝑀 𝒛

𝐼 𝒚 = − ∑ log 𝑞(𝑧𝑗|𝑦𝑗)

𝑧𝑗∈𝒛

• else we can simply encode the residual error,



e.g. if 𝒛 is binary, we have 𝒇 = 𝒛 ⊕ 𝒛 , and 𝑀 𝒛 𝐼 𝒚 = log 𝑜 + log 𝑜 |𝒇|



e.g. if 𝒛 is continuous, we can encode using a zero-mean Gaussian

48

MDL for Neural Networks

Suppose neural network 𝐼 ∈ ℋ predicts target 𝑧 given 𝑦 𝑧 = 𝐼(𝑦) How do we encode the model?

• we could encode all of the parameters, but that’s highly ad hoc
• instead, we can use the notion of prequentia

ial coding ing

Dawid (1984) Barron et al (1998) Grünwald (2007) Blier & Ollivier (2018) 49

Prequential Coding

Simple, elegant idea: “Update your model after every message” That is, we re-train our network after ‘every’ new label

• we initialize topology 𝐼 ∈ ℋ with fixed weights
• we transmit the first 𝑙 labels using 𝐼0
• we now train 𝐼 on this first batch of 𝑙 labelled points, we obtain 𝐼1
• we transmit the second 𝑙 labels using 𝐼1
• we now train 𝐼 on the first two batches, and obtain 𝐼3

Dawid (1984) Barron et al (1998) Grünwald (2007) Blier & Ollivier (2018) 50

Prequential Coding

Simple, elegant idea: “Update your model after every message” 𝑀 𝐸 ℋ = 𝑀(𝐸𝑗 ∣ 𝐼𝑗−1)

𝐸𝑗

Best of all, this is not a crude, but a refined MDL code!

• depends fully on how 𝐼 behaves on the data
• no arbitrary choices on how to encode 𝐼
• within a constant of 𝑀(𝐸|𝐼∗), and this constant only depends on ℋ

Dawid (1984) Barron et al (1998) Grünwald (2007) 51

Schedule

Opening Introduction to MDL MDL in Action ––––––break –––––– Stochastic Complexity MDL in Dynamic Settings

52

8:00am 8:10am 8:50am 9:30am 10:00am 11:00am