Interesting Patterns Jilles Vreeken 15 May 2015 Questions of the - - PowerPoint PPT Presentation

Interesting Patterns

Jilles Vreeken

15 May 2015

Questions of the Day

What is interestingness? what is a pattern? and how can we mine interest esting patterns?

Data ata Pattern ern

What is a pattern?

y = x - 1

Recurring structure

Dat ata Pat attern

What is a pattern?

For a database 𝑒𝑒

 a pattern language 𝑄 and a set of constraints 𝐷

the go goal al is to find the set of patterns 𝑇 ⊆ 𝑄 such that

 each 𝑞 ∈ 𝑄 satisfies each 𝑑 ∈ 𝐷 on 𝑒𝑒, and 𝑇 is maximal

That is, find all ll patterns that satisfy the constraints

Pattern mining, formally

Suppose a supermarket,

 which sells it

items, 𝐽, and

 logs every trans

nsaction n 𝑢 ⊆ 𝐽 in a database db

 an interesting question to ask is,

‘What products are often sold together?’

 pattern language:

all possible sets of items 𝑄 = (𝐽)

 pattern:

an itemse mset, 𝑌 ⊆ 𝐽, 𝑌 ∈ 𝑄

Frequent Pattern Mining

Frequent Itemsets

𝑡𝑡𝑞𝑞𝑡𝑡𝑢( ) = 3

Frequent Conjunctive Formulas

4.9, 3.1, 1.5, 0.1, Iris-setosa 5.0, 3.2, 1.2, 0.2, Iris-setosa 5.5, 3.5, 1.3, 0.2, Iris-setosa 4.9, 3.1, 1.5, 0.1, Iris-setosa 4.4, 3.0, 1.3, 0.2, Iris-setosa 5.1, 3.4, 1.5, 0.2, Iris-setosa 5.0, 3.5, 1.3, 0.3, Iris-setosa 4.5, 2.3, 1.3, 0.3, Iris-setosa 4.4, 3.2, 1.3, 0.2, Iris-setosa 5.0, 3.5, 1.6, 0.6, Iris-setosa 5.1, 3.8, 1.9, 0.4, Iris-setosa 4.8, 3.0, 1.4, 0.3, Iris-setosa 5.1, 3.8, 1.6, 0.2, Iris-setosa 4.6, 3.2, 1.4, 0.2, Iris-setosa 5.3, 3.7, 1.5, 0.2, Iris-setosa 5.0, 3.3, 1.4, 0.2, Iris-setosa 7.0, 3.2, 4.7, 1.4, Iris-versicolor 6.4, 3.2, 4.5, 1.5, Iris-versicolor 6.9, 3.1, 4.9, 1.5, Iris-versicolor 5.5, 2.3, 4.0, 1.3, Iris-versicolor 6.5, 2.8, 4.6, 1.5, Iris-versicolor 5.7, 2.8, 4.5, 1.3, Iris-versicolor 6.3, 3.3, 4.7, 1.6, Iris-versicolor 4.9, 2.4, 3.3, 1.0, Iris-versicolor 6.6, 2.9, 4.6, 1.3, Iris-versicolor 5 2 2 7 3 9 1 4 Iris-versicolor

Petal length >= 2.0 and Petal width <= 0.5

Frequent Subgraphs

The Frequent Pattern Problem

The task is to find all frequent patterns ‘how often is 𝑌 sold’ ↔ sup

𝑒𝑒

𝑌 = | 𝑢 ∈ 𝑒𝑒 𝑌 ⊆ 𝑢 |

 the number of transactions in 𝑒𝑒 that ‘support’ the pattern

‘often enough’ ↔ sup

𝑒𝑒

≥ 𝑛𝑛𝑛𝑡𝑡𝑞

 have a support higher than the minimal-support threshold

So, the problem is to find all 𝑌 ∈with sup

𝑒𝑒

𝑌 ≥ 𝑛𝑛𝑛𝑡𝑡𝑞

 how can do we do this?

Monotonicity

The number of possible patterns is exponential, and hence exhaustive search is not a feasible option. However, in 1994 it was discovered that support exhibits mono notonic

• nicity. That is, for two itemsets 𝑌 and 𝑍, we know

𝑌 ⊂ 𝑍 → 𝑡𝑡𝑞𝑞 𝑌 ≥ 𝑡𝑡𝑞𝑞 𝑍 This is known as the A Priori property. It allows efficient search for frequent itemsets over the lattice of all itemsets.

The Itemset Lattice

a b c d 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

data

abcd (1) abc (2) abd (3) acd (1) bcd (1) ab (4) ac (2) ad (3) bc (2) bd (3) cd (1) a (4) b (4) c (3) d (3) ∅ (6)

itemset lattice

The Itemset Lattice

a b c d 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

data

abcd (1) abc (2) abd (3) acd (1) bcd (1) ab (4) ac (2) ad (3) bc (2) bd (3) cd (1) a (5) b (4) c (3) d (3) ∅ (6) frequent

itemset lattice

Levelwise search

1.

𝐺

1 = {𝑛 ∈ 𝐽 ∣ 𝑡𝑡𝑞𝑞 𝑛 ≥ 𝑛𝑛𝑛𝑡𝑡𝑞}

2. 2.

while le 𝐺𝑙 not empty

3.

𝐷𝑙+1 = 𝑌 ∈ 𝑄 𝑌 = 𝑙 + 1, ∀𝑍⊂𝑌, 𝑍 =𝑙𝑍 ∈ 𝐺𝑙

4.

𝐺𝑙+1 = 𝑌 ∈ 𝐷𝑙+1 𝑡𝑡𝑞𝑞 𝑌 ≥ 𝑛𝑛𝑛𝑡𝑡𝑞

5. 5.

retu eturn 𝐺

1 ∪ 𝐺2 ∪ ⋯

The A Priori algorithm can be applied to mine patterns for any enumerable pattern language 𝑄 for any monotonic constraint 𝑑. Many algorithms exist that are more efficient, but none so versatile.

Problems in pattern paradise

The pattern explosion

 high thresholds

few, but well-known patterns

 low thresholds

a gazillion patterns

Many patterns are redundant Unstable

 small data change,

yet different results

 even when distribution

did not really change

The Wine Explosion

the Wine dataset has 178 rows, 14 columns

T

• the Max!

Why not just report only patterns for which there is no extension that is frequent? These patterns are called maxim imall lly frequent.

abcd (1) abc (2) abd (3) acd (1) bcd (1) ab (4) ac (2) ad (3) bc (2) bd (3) cd (1) a (5) b (4) c (3) d (3) ∅ (6) frequent

(Bayardo, 1998)

Closure!

Why throw away so much information? If we keep all 𝑌 that cannot be extended without 𝑡𝑡𝑞𝑞 𝑌 dropping, all frequent itemsets and their frequencies can be reconstructed without loss! These are called closed d frequent itemsets.

abcd (1) abc (2) abd (3) acd (1) bcd (1) ab (4) ac (2) ad (3) bc (2) bd (3) cd (1) a (5) b (4) c (3) d (3) ∅ (6) frequent

(Pasquier, 1999)

Non-Derivable Patterns

Through inclusion/exclusion, we can derive the support of 𝑏𝑒𝑑. As 𝑡𝑡𝑞𝑞 𝑒𝑑 = 𝑡𝑡𝑞𝑞 𝑑 = 2, we know 𝑒 and 𝑑 always co-occur. Then, knowing that 𝑡𝑡𝑞𝑞 𝑏𝑑 = 2, we can derive 𝑡𝑡𝑞𝑞 𝑏𝑒𝑑 = 2.

abcd (1) abc (2) abd (3) acd (1) bcd (1) ab (4) ac (2) ad (3) bc (2) bd (3) cd (1) a (5) b (4) c (3) d (3) ∅ (6) frequent

(Calders & Goethals, 2003)

non-derivable

Margin-Closed

Who cares that we can reconstruct all frequencies exactly? Why not allow a little bit of slack and zap more patterns? That is the main of idea of margin-closed ed frequent itemsets.

abcd (1) abc (2) abd (3) acd (1) bcd (1) ab (4) ac (2) ad (3) bc (2) bd (3) cd (1) a (5) b (4) c (3) d (3) ∅ (6) frequent

(Moerchen et al, 2011)

Associations

Why is a frequent pattern 𝑌 interesting? Because it identifies assoc sociation

• ns between elements of 𝑌.

Many people buy both and . What’s going on? Many patients have active genes A, B and C. What’s going on? Many molecules share this structure. What’s going on? Okay… but does higher frequency mean more interesting?

Expectation

Frequency alone is deceiving, and leads to redundant results. Say that many many people buy . Then all `real’ patterns, such as can be extended with and we likely find that is also

• frequent. Do we want it to be reported?

Not unless its support deviates strongly from our expectation.

What did you expect?

What do we expect? How do we model this? How can we measure whether expectation and reality are different enough? Let’s start simple. Let’s assume all ll it items are in independ ndent nt.

Independence!

Under the assumption that all items 𝑛 ∈ 𝐽 are independent, the expected frequency of an itemset 𝑌 is simply 𝑛𝑛𝑒 𝑌 = 𝑔𝑡(𝑦)

𝑦∈𝑌

where we write 𝑔𝑡 𝑦 =

𝑡𝑡𝑡𝑡 𝑦 𝑒𝑒

for the frequency – the relative support – of an item 𝑦 ∈ 𝑌 in our database. Item frequencies can easily be extracted from data, as well as reasonably expected to be known by your domain expert.

Bro, do you even lift?

We want to identify patterns for which their frequency in the data deviates strongly from our expectation. One way to measure this deviation is lift. 𝑚𝑛𝑔𝑢 𝑌 = 𝑔𝑡 𝑌 𝑛𝑛𝑒 𝑌 Patterns with a lift higher than 1 are more frequent than

• expected. Those with lift lower than 1 are less frequent.

In our data/lattice example 𝑚𝑛𝑔𝑢 𝐵𝐵 = 1.2 and 𝑚𝑛𝑔𝑢 𝐵𝐵𝐷 = 1.83,

(IBM, 1996)

Example: Lift

𝑔𝑡 𝐵𝐵 =

4 6

𝑛𝑛𝑒 𝐵𝐵 =

5 6 ∗ 4 6 = 0.55

𝑚𝑛𝑔𝑢 𝐵𝐵 =

0.66 0.55 = 1.2

𝑔𝑡 𝐵𝐵𝐷 =

2 6

𝑛𝑛𝑒 𝐵𝐵𝐷 =

5 6 ∗ 4 6 ∗ 3 6 = 0.28

𝑚𝑛𝑔𝑢 𝐵𝐵𝐷 =

0.33 0.28 = 1.83

That is, according to 𝑚𝑛𝑔𝑢, 𝐵𝐵𝐷 is more interesting than 𝐵𝐵.

A B C D 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

Lift

Lift is ad hoc.

Lift strongly over-estimates, or under-estimates how surprising the frequency of a pattern is. It is a ba bad interestingness measure.

Somewhat more formally:

Lift is ad hoc because it comp mpares s scores dir irectly, it does not consider how likely scores are, and doe

• es not
• t use

se a prop

• per

statistic ical l test to determine how significant the deviation is.

Better Lift

The probability of a random

• m transa

saction

• n to suppor
• rt 𝑌 is 𝑛𝑛𝑒(𝑌).

Assume our dataset contains 𝑂 transactions, and let 𝑎 be a random variable to state how many transactions support 𝑌. Then, 𝑄(𝑎 = 𝑁) is the probability that the support of 𝑌 is 𝑁, and is given by the binomial distribution, with 𝑟 = 𝑛𝑛𝑒 𝑌 𝑞 𝑎 = 𝑁 = 𝑂 𝑁 𝑟𝑁 1 − 𝑟 𝑂−𝑁 We can now calculate how likely it is to observe a support of 𝑡𝑡𝑞𝑞(𝑌) or hig ighe her, and decide whether the 𝑞-value 𝑞 𝑎 ≥ 𝑡𝑡𝑞𝑞 𝑌 𝑂 is significant (eg. < 0.05)

Aside Aside: using Surprisingness

While we’re in the business of unrealistic assumptions, say we have a good way to calculate a p-value for 𝑌, What can we do with it? There are two main approaches,

1)

mine all patterns up to a certain threshold

2)

mine the top-𝑙 most surprising patterns

T

• o much of a good thing

Under the independence assumption, we compare 𝑄 𝐵𝐵𝐷𝐵 ↔ 𝑄 𝐵 𝑄 𝐵 𝑄 𝐷 𝑄(𝐵) And hence, any deviation from total independence is gauged as

• interesting. Say that 𝐵𝐵𝐷 is the true pattern, then it will have a

high lift, but so will an any extension of it! In other words, all supersets of a pattern are also scored highly. Why? How can we avoid this? Which ones should we report?

Partitions

Webb proposed that we should report only those patterns for which the frequency is surprising with regard to all ll its partitions. 𝑄 𝐵𝐵𝐷 ↔ 𝑄 𝐵 𝑄 𝐵 𝑄 𝐷 ↔ 𝑄 𝐵𝐵 𝑄 𝐷 ↔ 𝑄 𝐵𝐷 𝑄 𝐵 ↔ 𝑄 𝐵 𝑄 𝐵𝐷 Sounds like a good idea! But, how many partitions are there? And, how do we test for surprisingness? When we a consider 2-partition we can use Fisher’s exact test. Webb tests against the partition of 𝑌 = 𝑍 ∪ 𝑎 s.t. 𝑍 ∩ 𝑎 = ∅ and 𝑔𝑡 𝑍 𝑔𝑡(𝑎) is closest to 𝑔𝑡 𝑌

(Webb, 2010…)

Applying Fisher’s Exact T est

Let’s test 𝑔𝑡 𝐵𝐵𝐷 vs. 𝑔𝑡 𝐵𝐵 𝑔𝑡(𝐷) 𝑞 = 𝑏 + 𝑒 𝑏 𝑑 + 𝑒 𝑑 𝑛 𝑏 + 𝑑 = 𝑏 + 𝑒 ! 𝑑 + 𝑒 ! 𝑏 + 𝑑 ! 𝑒 + 𝑒 ! 𝑏! 𝑒! 𝑑! 𝑒! 𝑛! We get 𝑞 𝐵𝐵, 𝐷 = 0.6, meaning that 𝐵𝐵𝐷 is not interesting, yay!

(Fisher 1922; Webb 2010; Hamalainen 2012; Webb & Vreeken 2014)

𝑩𝑩 −𝑩𝑩 𝐷 2 1 3 −𝐷 2 1 3 4 2 6 𝒒𝟐 −𝒒𝟐 𝑞2 a b a+b −𝑞2 c d c+d a+c b+d a+b+c+d

More Elaborate Models

Although the math and stats get more and more complicated, the models and tests we saw so far really are straightforward. How can we infuse more background knowledge? We can test against

 Bayesian Networks (Jaroszwicz et al. 2004),  Maximum Entropy models (Wang et al 2006, Mampaey et al 2012, …)

Goes (probably) too deep for today. Let’s re-consider this in a few weeks.

Tiles

Only considering how of

• ften something occurs biases us towards

patterns with low cardinality. Instead, we can consider how much

• f the data a pattern cover

ers. That is, a pattern 𝑌 now consists of a row-set, and a column-set, and is regarded as more interesting the larger 𝑏𝑡𝑏𝑏 𝑌 = |𝑡𝑡𝑠𝑡 𝑌 | × |𝑑𝑡𝑚𝑡(𝑌)|

(Geerts et al 2014)

Patterns: Large Tiles

Genes Conditions

Example: Tiles

Mining Large Tiles

Sadly, 𝑏𝑡𝑏𝑏 is not (anti-)monotonic. Extending the column set of a given tile 𝑌 may result in both an increase or decrease in 𝑏𝑡𝑏𝑏. How can we mine tiles efficiently? Through depth-first search, using branch-and-bound. If you keep the row-set maximal, you can keep track of the conditiona nal l support of all not-yet-included items. Assuming maximal correlation, you have an upper bound.

Mining Tilings

A big pile of Tiles is as bad as a big pile of Frequent Itemsets. Way too many, way too redundant. Instead, we can also ask to find a set of tiles that together cover as many of the ones in the data as possible. This means we are doing set et cover er, which is well-known to be NP-hard, but for which the greedy algorithm is known to be the best possible polynomial time approximation algorithm.

Exact and Noisy Tiles

Let 𝑔𝑡(𝑌) for a tile 𝑌 be the relative number of 1s in the tile, 𝑡𝑛𝑏𝑡 𝑌 =

• 𝐽(𝐵𝑗,𝑘 = 1)

𝑘∈𝑠𝑠𝑠𝑡(𝑌) 𝑗∈𝑑𝑠𝑑𝑡(𝑌)

𝑔𝑡 𝑌 = 𝑡𝑛𝑏𝑡(𝑌) 𝑑𝑡𝑚𝑡 𝑌 × 𝑡𝑡𝑠𝑡 𝑌 For 𝑔𝑡 𝑌 = 1 or 0 we say the tile is exact ct. . Otherwise, it is noisy. y.

(Tatti & Vreeken 2011)

Noisy Tiles

Boolean Matrix Factorisation (BMF) aims to find a low-rank decomposition of a given binary matrix A into row and column factor matrices B and C, such that 𝐵 ≈ 𝐵 ° 𝐷 where ° is the binary product, i.e. 0 + 0 = 0, 0 + 1 = 1, 1 + 1 = 1, and we want to minimize the error between 𝐵 and 𝐵 ° 𝐷 When restricted to exact tiles (factors), BMF and Tiling are

• equivalent. BMF, however, is more general, as it allows for errors.

(Miettinen et al 2006, …)

Patterns are a powerful concept that can show a lot of insight in how your data is lo loca cally lly distributed. Monotonic constraints allow for efficient mining

 levelwise search alway

ays works – more elegant algorithms exist

 works for other pattern types equally well:

itemsets, sequences, trees, streams, low-entropy sets

Measuring interestingness is in inherently ly d dif iffic icult lt

 frequency alone is a bad measure  independence models are too weak  stronger models are computionally expensive

Conclusions

Patterns are a powerful concept that can show a lot of insight in how your data is lo loca cally lly distributed. Monotonic constraints allow for efficient mining

 levelwise search alway

ays works – more elegant algorithms exist

 works for other pattern types equally well:

itemsets, sequences, trees, streams, low-entropy sets

Measuring interestingness is in inherently ly d dif iffic icult lt

 frequency alone is a bad measure  independence models are too weak  stronger models are computionally expensive

Thank you!