Bridging the Gap between Data Diversity and Data Dependencies - - PowerPoint PPT Presentation

Bridging the Gap between Data Diversity and Data Dependencies

Bridging the Gap between Data Diversity and Data Dependencies

Jean-Marc Petit

INSA Lyon, Universit´ e de Lyon LIRIS CNRS (UMR 5205)

24th International Symposium on Methodologies for Intelligent Systems (ISMIS 2018) Limassol, Cyprus

1

Bridging the Gap between Data Diversity and Data Dependencies Introduction

Data diversity

2

Bridging the Gap between Data Diversity and Data Dependencies Introduction

Data Diversity: not only a gender question !

3

Bridging the Gap between Data Diversity and Data Dependencies Introduction

Example from the astrophysics domain

The Sloan Digital Sky Survey (SDSS): Mapping the Universe !

Class

u erru g errg r errr i erri z errz

STAR 16.56 14.62 13.94 13.79 13.48 0.01 0.00 0.01 0.01 0.00 Galaxie 19.79 17.77 16.59 16.07 15.63 0.06 0.01 0.00 0.00 0.01 STAR 15.64 14.04 14.57 12.83 13.12 0.01 0.00 0.01 0.00 0.01 Galaxie 21.61 20.81 19.87 19.30 19.03 0.15 0.04 0.02 0.02 0.05 STAR 20.09 17.28 15.79 14.31 13.49 0.04 0.00 0.00 0.00 0.00

5 magnitudes (u, g, r, i, and z) catalog database ⇒ Require to deal with numerical interval data as first class citizen See http://www.sdss.org/dr12/ for details

4

Bridging the Gap between Data Diversity and Data Dependencies Introduction

Data and metadata from SDSS

5

Bridging the Gap between Data Diversity and Data Dependencies Introduction

Data diversity

To cope with data diversity, key notions have be studied for years in computer science: data and metadata representation, data uncertainty, data inconsistency, data heterogeneity . . . Dealing with data diversity remains the hardest thing in practise ⇒ Require to understand what’s hidden behind the data: Where do they come from ? How are they produced ? ⇒ Be as close as possible of the available data sources and experts to better match their intended meaning

6

Bridging the Gap between Data Diversity and Data Dependencies Introduction

Data diversity

To cope with data diversity, key notions have be studied for years in computer science: data and metadata representation, data uncertainty, data inconsistency, data heterogeneity . . . Dealing with data diversity remains the hardest thing in practise ⇒ Require to understand what’s hidden behind the data: Where do they come from ? How are they produced ? ⇒ Be as close as possible of the available data sources and experts to better match their intended meaning

6

Bridging the Gap between Data Diversity and Data Dependencies Introduction

Data dependencies

7

Bridging the Gap between Data Diversity and Data Dependencies Introduction

Classical example of data dependencies: functional dependencies

r | = X → Y iff for all t1, t2 ∈ r If for all A ∈ X, t1[A] = t2[A] then for all B ∈ Y , t1[B] = t2[B] Turns out to be a very general notion, related to implications. a b a → b 1 1 1 1 1 1 1 Many connections with lattice theory, formal concept analysis (Galois connection) and logics (see for ex [11]) Crucial to understand relational database design

8

Bridging the Gap between Data Diversity and Data Dependencies Introduction

Beyond database design

New and timely applications require some forms of FD: Data quality: Analysing existing data to identify data quality problems [17, 9] Machine learning over relational databases: FD-aware

• ptimization for in-database learning [19]

Semantic query optimization: Query rewriting techniques based on data dependencies [12] ⇒ Many extensions of FD have been proposed to take into account some forms of data diversity (e.g. see [10, 18] for a survey) Matching Dependencies, Denial constraints . . . [17, 9, 15] Implications in Formal Concept Analysis (FCA) [7, 6] Association rules . . . in Data mining [5]

9

Bridging the Gap between Data Diversity and Data Dependencies Introduction

Data diversity and data dependencies

10

Bridging the Gap between Data Diversity and Data Dependencies Introduction

Questions and Contributions

How to take into account data diversity for data dependencies ? Does there exist unifying frameworks ? Two contributions: RQL: a query language to express implications over relational databases (ISMIS 2005 [3], demo ICDM 2014 [13], TCS 2017 [14]) Structural properties on attribute domains (ongoing work)

11

Bridging the Gap between Data Diversity and Data Dependencies RQL query language

Contents

1

RQL query language Preliminaries Main result underlying RQL The RQL language RQL implementation Summary

2

Structural properties on attribute domains Similarity map: a semilattice version Data Dependencies with similarity maps Main results

3

Conclusion and perspective

12

Bridging the Gap between Data Diversity and Data Dependencies RQL query language Preliminaries

Important known results for FD

Let F be a set of FD over a schema R CL(F) = {X ⊆ R|X +

F = X} : a closure system of F

IRR(F) the set of irreducible elements of CL(F) by intersection Reasoning on F is equivalent to reasoning on CL(F), for instance: X +

F = {A ∈ R | F |

= X → A} = ∩{Y ∈ CL(F) | X ⊆ Y } Let r be a relation over R. The agree set of r is ag(r) = {ag(t1, t2) | t1, t2 ∈ r} where ag(t1, t2) = {A ∈ R | t1[A] = t2[A]} r is an Armstrong relation for F iff IRR(F) ⊆ ag(r) ⊆ CL(F) [8]

13

Bridging the Gap between Data Diversity and Data Dependencies RQL query language Preliminaries

Example

Bar(B) Beer(Be) Price(P) t1 Nota bene Adelscott 2 t2 Montagne 1664 1.5 t3 Nota bene 1664 2 t4 Ritz Adelscott 5 t5 Caf´ e Flore Affligen 6 F = {B → P, P → B} CL(F) = {∅, Be, BP, BBeP} IRR(F) = {Be, BP} ag(r) = {∅, Be, BP}, often represented as: B Be P 1 1 1

14

Bridging the Gap between Data Diversity and Data Dependencies RQL query language Preliminaries

Towards a rule query language

Focus on rules equivalent to implications (or FD) ⇒ Armstrong axioms (reflexivity, augmentation, transitivity) have to be sound and complete Idea: Defining a rule query language (RQL) such that every RQL statement turns out to deliver implications Require to identify syntactic constraints such that we remain within the reasoning of implications

15

Bridging the Gap between Data Diversity and Data Dependencies RQL query language Preliminaries

Semantics of implications

Let b0 be a binary relation (given by a {0, 1}-relation) b0 | = X → Y ⇔ ∀t ∈ b0 (∀A ∈ X t.A = 1) ⇒ (∀A ∈ Y t.A = 1) Let d = {r0, r1, ..., rn} be a relational database r0 | = X → Y ⇔ ∀t1, t2 ∈ r0 (∀A ∈ X t1.A = t2.A) ⇒ (∀A ∈ Y t1.A = t2.A) d | = X → Y ⇔ ∀t1, t2 ∈ πX(σF(ri0 ⊲ ⊳ . . . ⊲ ⊳ rip)) (∀A ∈ X t1.A = t2.A) ⇒ (∀A ∈ Y t1.A = t2.A) d | = X → Y ⇔ ∀t1 ∈ πX(σF(ri0 ⊲ ⊳ . . . ⊲ ⊳ rin)), ∀t2 ∈ πX(σF ′(rj0 ⊲ ⊳ . . . ⊲ ⊳ rin)) such that (t1.rank = t2.rank + 1) (∀A ∈ X t1.A = t2.A) ⇒ (∀A ∈ Y t1.A = t2.A)

16

Bridging the Gap between Data Diversity and Data Dependencies RQL query language Preliminaries

Semantics of implications

Let b0 be a binary relation (given by a {0, 1}-relation) b0 | = X → Y ⇔ ∀t ∈ b0 (∀A ∈ X t.A = 1) ⇒ (∀A ∈ Y t.A = 1) Let d = {r0, r1, ..., rn} be a relational database r0 | = X → Y ⇔ ∀t1, t2 ∈ r0 (∀A ∈ X t1.A = t2.A) ⇒ (∀A ∈ Y t1.A = t2.A) d | = X → Y ⇔ ∀t1, t2 ∈ πX(σF(ri0 ⊲ ⊳ . . . ⊲ ⊳ rip)) (∀A ∈ X t1.A = t2.A) ⇒ (∀A ∈ Y t1.A = t2.A) d | = X → Y ⇔ ∀t1 ∈ πX(σF(ri0 ⊲ ⊳ . . . ⊲ ⊳ rin)), ∀t2 ∈ πX(σF ′(rj0 ⊲ ⊳ . . . ⊲ ⊳ rin)) such that (t1.rank = t2.rank + 1) (∀A ∈ X t1.A = t2.A) ⇒ (∀A ∈ Y t1.A = t2.A)

16

Bridging the Gap between Data Diversity and Data Dependencies RQL query language Preliminaries

Semantics of implications

Let b0 be a binary relation (given by a {0, 1}-relation) b0 | = X → Y ⇔ ∀t ∈ b0 (∀A ∈ X t.A = 1) ⇒ (∀A ∈ Y t.A = 1) Let d = {r0, r1, ..., rn} be a relational database r0 | = X → Y ⇔ ∀t1, t2 ∈ r0 (∀A ∈ X t1.A = t2.A) ⇒ (∀A ∈ Y t1.A = t2.A) d | = X → Y ⇔ ∀t1, t2 ∈ πX(σF(ri0 ⊲ ⊳ . . . ⊲ ⊳ rip)) (∀A ∈ X t1.A = t2.A) ⇒ (∀A ∈ Y t1.A = t2.A) d | = X → Y ⇔ ∀t1 ∈ πX(σF(ri0 ⊲ ⊳ . . . ⊲ ⊳ rin)), ∀t2 ∈ πX(σF ′(rj0 ⊲ ⊳ . . . ⊲ ⊳ rin)) such that (t1.rank = t2.rank + 1) (∀A ∈ X t1.A = t2.A) ⇒ (∀A ∈ Y t1.A = t2.A)

16

Bridging the Gap between Data Diversity and Data Dependencies RQL query language Preliminaries

Semantics of implications

Let b0 be a binary relation (given by a {0, 1}-relation) b0 | = X → Y ⇔ ∀t ∈ b0 (∀A ∈ X t.A = 1) ⇒ (∀A ∈ Y t.A = 1) Let d = {r0, r1, ..., rn} be a relational database r0 | = X → Y ⇔ ∀t1, t2 ∈ r0 (∀A ∈ X t1.A = t2.A) ⇒ (∀A ∈ Y t1.A = t2.A) d | = X → Y ⇔ ∀t1, t2 ∈ πX(σF(ri0 ⊲ ⊳ . . . ⊲ ⊳ rip)) (∀A ∈ X t1.A = t2.A) ⇒ (∀A ∈ Y t1.A = t2.A) d | = X → Y ⇔ ∀t1 ∈ πX(σF(ri0 ⊲ ⊳ . . . ⊲ ⊳ rin)), ∀t2 ∈ πX(σF ′(rj0 ⊲ ⊳ . . . ⊲ ⊳ rin)) such that (t1.rank = t2.rank + 1) (∀A ∈ X t1.A = t2.A) ⇒ (∀A ∈ Y t1.A = t2.A)

16

Bridging the Gap between Data Diversity and Data Dependencies RQL query language Preliminaries

Semantics of implications (cont’ed)

d | = X → Y ⇔ ∀t1, t2 ∈ πX(σF(r0 ⊲ ⊳ . . . ⊲ ⊳ rn)) (∀A ∈ X(2 ∗ ABS(t1.A − t2.A)/(t1.A + t2.A) < 0.1)) ⇒ (∀A ∈ Y (2 ∗ ABS(t1.A − t2.A)/(t1.A + t2.A) < 0.1)) d | = X → Y ⇔ ∀t1, t2 ∈ πX(σF(r0 ⊲ ⊳ . . . ⊲ ⊳ rn)) (∀A ∈ X t1.A ≤ t2.A) ⇒ (∀A ∈ Y t1.A ≤ t2.A) r0 | = X → Y ⇔ ∀t1, t2, t3 ∈ r0 (∀A ∈ X(t1.A ≤ t2.A) ∧ (t3.A ≤ t2.A)) ⇒ (∀A ∈ Y (t1.A ≤ t2.A) ∧ (t3.A ≤ t2.A))

17

Bridging the Gap between Data Diversity and Data Dependencies RQL query language Preliminaries

Semantics of implications (cont’ed)

d | = X → Y ⇔ ∀t1, t2 ∈ πX(σF(r0 ⊲ ⊳ . . . ⊲ ⊳ rn)) (∀A ∈ X(2 ∗ ABS(t1.A − t2.A)/(t1.A + t2.A) < 0.1)) ⇒ (∀A ∈ Y (2 ∗ ABS(t1.A − t2.A)/(t1.A + t2.A) < 0.1)) d | = X → Y ⇔ ∀t1, t2 ∈ πX(σF(r0 ⊲ ⊳ . . . ⊲ ⊳ rn)) (∀A ∈ X t1.A ≤ t2.A) ⇒ (∀A ∈ Y t1.A ≤ t2.A) r0 | = X → Y ⇔ ∀t1, t2, t3 ∈ r0 (∀A ∈ X(t1.A ≤ t2.A) ∧ (t3.A ≤ t2.A)) ⇒ (∀A ∈ Y (t1.A ≤ t2.A) ∧ (t3.A ≤ t2.A))

17

Bridging the Gap between Data Diversity and Data Dependencies RQL query language Preliminaries

Semantics of implications (cont’ed)

d | = X → Y ⇔ ∀t1, t2 ∈ πX(σF(r0 ⊲ ⊳ . . . ⊲ ⊳ rn)) (∀A ∈ X(2 ∗ ABS(t1.A − t2.A)/(t1.A + t2.A) < 0.1)) ⇒ (∀A ∈ Y (2 ∗ ABS(t1.A − t2.A)/(t1.A + t2.A) < 0.1)) d | = X → Y ⇔ ∀t1, t2 ∈ πX(σF(r0 ⊲ ⊳ . . . ⊲ ⊳ rn)) (∀A ∈ X t1.A ≤ t2.A) ⇒ (∀A ∈ Y t1.A ≤ t2.A) r0 | = X → Y ⇔ ∀t1, t2, t3 ∈ r0 (∀A ∈ X(t1.A ≤ t2.A) ∧ (t3.A ≤ t2.A)) ⇒ (∀A ∈ Y (t1.A ≤ t2.A) ∧ (t3.A ≤ t2.A))

17

Bridging the Gap between Data Diversity and Data Dependencies RQL query language Main result underlying RQL

Approach and contribution

Replaying part of the story underlying SQL and relational languages, especially through Tuple Relational Calculus (TRC) What we did: Extend TRC to support rule expression (SafeRL logical language, see [14] for details) Propose a new syntactic practical language (RQL) from SafeRL Q = { X → Y | ∀t1 . . . ∀tn

• ψ(t1, . . . , tn) →
• ∀A ∈ X(δ(A, t1, ..., tn)) → ∀A ∈ Y (δ(A, t1, ..., tn))
• }

18

Bridging the Gap between Data Diversity and Data Dependencies RQL query language Main result underlying RQL

Main result.

THM Let Q be a RQL query over a database d.

• 1. ans(Q, d) defines a closure system CL(Q) over sch(Q)
• 2. There exists a SQL query Q′ over d such that Q′ computes a

base B(Q) of CL(Q), i.e. IRR(Q) ⊆ B(Q) ⊆ CL(Q) B(Q): agree sets for FD and binary relation for implications Proof of 1. similar to the proof given for Functional Dependencies by Mannila and Raiha 1994 [21], Demetrovics and Thi 1995 [16]. Proof of 2. a bit more elaborated

19

Bridging the Gap between Data Diversity and Data Dependencies RQL query language The RQL language

RQL: a Practical Language

RQL has 5 clauses (with the ”look and feel” of SQL): FINDRULES OVER A1, ..., An SCOPE t1(SQL1), ..., tn(SQLn) WHERE condition(t1, ..., tn) CONDITION ON A IS δ(A, t1, ..., tn)

20

Bridging the Gap between Data Diversity and Data Dependencies RQL query language The RQL language

Examples

FINDRULES OVER Empno,Lastname,Workdept,Job,Sex,Bonus SCOPE t1,t2 Emp CONDITION ON A IS t1.A = t2.A; FINDRULES OVER Empno, Lastname, Workdept, Job, Sex, Bonus, Mgrno SCOPE t1 Emp CONDITION ON A IS t1.A IS NULL

21

Bridging the Gap between Data Diversity and Data Dependencies RQL query language The RQL language

Examples

FINDRULES OVER ... SCOPE t1,t2,t3 sensors WHERE t2.time = t1.time+interval 1 minute AND t3.time = t2.time+interval 1 minute CONDITION ON A IS t1.A < t2.A AND t2.A > t3.A;

22

Bridging the Gap between Data Diversity and Data Dependencies RQL query language RQL implementation

RQL query processing.

RQL parser SQL gen- erator Rule generator Rule verifier Optimizer Query pro- cessor DB RQL engine DBMS SQL query Base RQL query Rules

Figure: RQL queries processing overview

23

Bridging the Gap between Data Diversity and Data Dependencies RQL query language RQL implementation

RQL Web Interface

Figure: RQL Interface

24

Bridging the Gap between Data Diversity and Data Dependencies RQL query language RQL implementation

RQL Web Interface

Figure: Counter-example with RQL

25

Bridging the Gap between Data Diversity and Data Dependencies RQL query language Summary

Summary

RQL: a practical language to express different semantics for implication Discovery of implications seen as a query processing problem Side effect: data analysts may interact with their data through counter-examples Advantages

Easy to learn for SQL-aware data analysts (especially CS students !) http://rql.insa-lyon.fr

26

Bridging the Gap between Data Diversity and Data Dependencies Structural properties on attribute domains

Contents

1

RQL query language Preliminaries Main result underlying RQL The RQL language RQL implementation Summary

2

Structural properties on attribute domains Similarity map: a semilattice version Data Dependencies with similarity maps Main results

3

Conclusion and perspective

27

Bridging the Gap between Data Diversity and Data Dependencies Structural properties on attribute domains

Come back to functional dependencies

r | = X → Y iff for all t1, t2 ∈ r If for all A ∈ X, t1[A] = t2[A] then for all B ∈ Y , t1[B] = t2[B] Let us focus on the equality t1[A] = t2[A] without defining new predicates on t1[A] and t2[A] values

28

Bridging the Gap between Data Diversity and Data Dependencies Structural properties on attribute domains

From equality to similarity

Two possibilities: Replace “t1[A] = t2[A]” by “t1[A] is similar to t2[A]′′ ⇒ Similarity seen as a reflexive and symmetric binary relation Replace “t1[A] = t2[A]” by “t1[A] and t2[A] are similar to some similarity value s” ⇒ Similarity seen as an idempotent and commutative map ⇒ Focus on similarity map which appears to be less restrictive than similarity relation

29

Bridging the Gap between Data Diversity and Data Dependencies Structural properties on attribute domains

Similarity relation

Let DA be the domain of attribute A and u, v ∈ DA Let S be a binary relation on DA Similarity S is a similarity relation if S is reflexive (S(u, u) = 1) and symmetric (S(u, v) = S(v, u)). S subsumes the equality operator Two meaningful values: true (1) and false (0)

30

Bridging the Gap between Data Diversity and Data Dependencies Structural properties on attribute domains

Assumptions on similarity map

Notations: A is an attribute, DA its domain SA new values denoting similarities for A (disjoint from DA) Assumption: For any subset of DA ∪ SA, there is a unique similarity value.

31

Bridging the Gap between Data Diversity and Data Dependencies Structural properties on attribute domains Similarity map: a semilattice version

Similarity map: a semilattice version

Let A be an attribute, S = DA ∪ SA and mA : S × S → S a similarity map that is: Idempotent (mA(a, a) = a for all a ∈ S), Commutative (mA(a, a′) = mA(a′, a) for all a, a′ ∈ S), Associative (mA(a, mA(a′, a′′)) = mA(mA(a, a′), a′′) for all a, a′, a′′ ∈ S). mA induces a partial order on S: for every a, a′ ∈ S, a a′ whenever mA(a, a′) = a. (S, ) is a semilattice where glb(a, a′) = mA(a, a′) for all a, a′ ∈ S.

32

Bridging the Gap between Data Diversity and Data Dependencies Structural properties on attribute domains Similarity map: a semilattice version

Illustration

33

Bridging the Gap between Data Diversity and Data Dependencies Structural properties on attribute domains Similarity map: a semilattice version

Example with numerical interval values

Consider an attribute A whose domain is intervals of integer, i.e. DA = {[i, j]|i, j ∈ 1..n, i ≤ j} What would be the similarity values SA ? ⇒ The set of closed sets of DA by intersection Let {I1, . . . , Im} ⊆ DA ∪ SA. Similarity value of {I1, . . . , Im} ? ⇒ its intersection I = {I1, . . . , Im} ⇒ I is clearly unique

34

Bridging the Gap between Data Diversity and Data Dependencies Structural properties on attribute domains Similarity map: a semilattice version

Two examples of similarity map

Equality can be defined as:

mA(x, y) = x if x = y ⊥

• therwise

⊥ means ”not similar” or 0 (false) Similarity over intervals can be defined as:

mA(I1, I2) = I1 ∩ I2 if I1 ∩ I2 = ∅ ⊥

• therwise

35

Bridging the Gap between Data Diversity and Data Dependencies Structural properties on attribute domains Similarity map: a semilattice version

Underlying assumption

A dataset r has to be equipped with a semilattice structure for every attribute domain ⇒ Allow to be as close as possible of data values to quantify their similarities and differences ⇒ Require an important data pre-processing task, that could be partially automated using data mining techniques A different approach to address data diversity

36

Bridging the Gap between Data Diversity and Data Dependencies Structural properties on attribute domains Data Dependencies with similarity maps

Running example

r A B C t1 0.4 [1,2] 0.6 t2 0.5 [2,4] 0.5 t3 0.6 [3,5] 0.6 t4 0.4 [2,2] 0.4 t5 0.5 [3,5] 0.4

⇒ Semantics for mA and mC The values L and H qualify the different values ⊥ otherwise, i.e. not similar.

37

Bridging the Gap between Data Diversity and Data Dependencies Structural properties on attribute domains Data Dependencies with similarity maps

Application to functional dependencies

r | = X → Y iff for all t1, t2 ∈ r for all A ∈ X, t1[A] = t2[A] ⇒ for all B ∈ Y , t1[B] = t2[B] can be reformulated as follows: for all A ∈ X, glb(t1[A], t2[A]) = ⊥ ⇒ for all B ∈ Y , glb(t1[B], t2[B]) = ⊥ glb(t1[A], t2[A]) = ⊥ means there exists a similarity between the values of A on t1, t2

38

Bridging the Gap between Data Diversity and Data Dependencies Structural properties on attribute domains Data Dependencies with similarity maps

Minimal degree of similarities

Assume now an expert provides for each attribute A a minimal degree of similarity she expects. Let sim : sch(r) → (DA ∪ SA) \ {⊥} be such a map.

39

Bridging the Gap between Data Diversity and Data Dependencies Structural properties on attribute domains Data Dependencies with similarity maps

Examples

r | = X → Y iff for all t1, t2 ∈ r for all A ∈ X, t1[A] = t2[A] ⇒ for all B ∈ Y , t1[B] = t2[B] becomes r | =sim X → Y iff for all t1, t2 ∈ r for all A ∈ X, sim(A) glb(t1[A], t2[A]) ⇒ for all B ∈ Y , sim(B) glb(t1[B], t2[B]) sim(A) glb(t1[A], t2[A]) means the similarity level between the values of A on t1, t2 is above the mimimum

40

Bridging the Gap between Data Diversity and Data Dependencies Structural properties on attribute domains Data Dependencies with similarity maps

Example

Assume the expert tags those similarities: sim(A) = sim(C) = H and sim(B) = [3,4]

r A B C t1 0.4 [1,2] 0.6 t2 0.5 [2,4] 0.5 t3 0.6 [3,5] 0.6 t4 0.4 [2,2] 0.4 t5 0.5 [3,5] 0.4

r | =sim A → B (or r | =sim A, High → B, [3, 4]) r | =sim C → B ⇒ for ex. see counter-example t1, t2

41

Bridging the Gap between Data Diversity and Data Dependencies Structural properties on attribute domains Main results

Many results follow ...

Many well-known results on FD can be re-defined in this new setting

42

Bridging the Gap between Data Diversity and Data Dependencies Structural properties on attribute domains Main results

Agree sets

Agree sets can be extended naturally: instead of getting a set of attributes (due to 0 and 1 interpretation values based on equality), we obtain a set of similarities ag(r) = {ag(t1, t2) | t1, t2 ∈ r} ag(t1, t2) = {ag(t1[A], t2[A]) | A ∈ sch(r)} ag(t1[A], t2[A]) = glb(t1[A], t2[A]) Example: ag(t1, t2) =< L, [2, 2], H >

43

Bridging the Gap between Data Diversity and Data Dependencies Structural properties on attribute domains Main results

Example

r A B C t1 0.4 [1,2] 0.6 t2 0.5 [2,4] 0.5 t3 0.6 [3,5] 0.6 t4 0.4 [2,2] 0.4 t5 0.5 [3,5] 0.4 ag(r) A B C ag(t1, t2) L [2,2] H ag(t1, t3) ⊥ ⊥ 0.6 ag(t1, t4) 0.4 [2,2] ⊥ ag(t1, t5) L ⊥ ⊥ ag(t2, t3) H [3,4] H ag(t2, t4) L [2,2] L ag(t2, t5) 0.5 [3,4] L ag(t3, t4) ⊥ ⊥ ⊥ ag(t3, t5) H [3,5] ⊥ ag(t4, t5) L ⊥ 0.4

From ag(r), two interesting cases: replacing all values occurring in r by 1 and all other values by 0 ⇒ classical FD with equality replacing ⊥ by 0 (or false) and all other values by 1 (true) ⇒ classical FD extended to similarities

44

Bridging the Gap between Data Diversity and Data Dependencies Structural properties on attribute domains Main results

Closures and agree sets

From the agree set of r, the family Fr of closed sets by the glb

• peration is:

Fr = {glbsch(r)(T)|T ⊆ ag(r)} Lemma (Fr, sch(r)) is a semilattice Let M(Fr) be the meet irreducible elements of Fr Theorem M(Fr) ⊆ ag(r) ⊆ Fr

45

Bridging the Gap between Data Diversity and Data Dependencies Structural properties on attribute domains Main results

Similarity, attribute closure and implications

Let F be a family of closed sets, X ⊆ sch(r) and sim(X) = {sim(A)|A ∈ X} X +

sim(X) = glb({Y ∈ F | sim(X) X Y })

Theorem r | =sim X → Y iff sim(Y ) Y X +

sim(X)

46

Bridging the Gap between Data Diversity and Data Dependencies Structural properties on attribute domains Main results

Example with r | =sim A → B with sim(A) = H and sim(B) = [3, 4]

r A B C t1 0.4 [1,2] 0.4 t2 0.5 [2,4] 0.5 t3 0.6 [3,5] 0.6 t4 0.4 [2,2] 0.4 t5 0.5 [3,4] 0.4 ag(r) A B C ag(t1, t2) L [2,2] H ag(t1, t3) ⊥ ⊥ 0.6 ag(t1, t4) 0.4 [2,2] ⊥ ag(t1, t5) L ⊥ L ag(t2, t3) H [3,4] H ag(t2, t4) L [2,2] L ag(t2, t5) 0.5 [3,4] L ag(t3, t4) ⊥ ⊥ ⊥ ag(t3, t5) H [3,4] ⊥ ag(t4, t5) L ⊥ 0.4

A+

sim(A) = glbABC {< H, [3, 4], H >, < 0.5, [3, 4], L >, <

H, [3, 4], ⊥ >} =< H, [3, 4], ⊥ > sim(B) B< H, [3, 4], ⊥ > ⇒ r | =sim A, High → B, [3, 4]

47

Bridging the Gap between Data Diversity and Data Dependencies Structural properties on attribute domains Main results

Summary

Using similarity maps on attribute domains allows to reconsider classical data dependencies Require to change our mind: most of the effort has to be done at the attribute domain level to define similarity map After this, the problem is embedded into a lattice structure allowing to revisit many known results

48

Bridging the Gap between Data Diversity and Data Dependencies Conclusion and perspective

Contents

1

RQL query language Preliminaries Main result underlying RQL The RQL language RQL implementation Summary

2

Structural properties on attribute domains Similarity map: a semilattice version Data Dependencies with similarity maps Main results

3

Conclusion and perspective

49

Bridging the Gap between Data Diversity and Data Dependencies Conclusion and perspective

Conclusion

Two propositions to extend data dependencies First, through RQL, a query language devoted to implications (or FD) Second, through assumptions on attribute domains using semilattice structure induced by similarity maps ⇒ Both are elegant formalisms to extend functional dependencies by taking into account data diversity

50

Bridging the Gap between Data Diversity and Data Dependencies Conclusion and perspective

Perspective

Theoretical question ⇒ Under which conditions the second approach leads to implications (Armstrong axioms) ? Practical question ⇒ Given a dataset D equipped with semilattice structures, how to discover implications satisfied in D ?

51

Bridging the Gap between Data Diversity and Data Dependencies Conclusion and perspective

Acknowledgments

Joint work with Brice Chardin, Emmanuel Coquery, Marie Pailloux

• n RQL (partly funded by ANR, DAG project)

and Lhouari Nourine on structural properties on attribute domains (partly funded by the CNRS Mastodon program on Data Quality)

52

Bridging the Gap between Data Diversity and Data Dependencies Appendix For Further Reading

For Further Reading I

• S. Abiteboul, R. Hull, and V. Vianu,

Foundations of Databases. Addison-Wesley, 1995.

• B. Ganter and R. Wille,

Formal Concept Analysis. Springer, 1999. Marie Agier-Pailloux, Jean-Marc Petit, Einoshin Suzuki: Towards Ad-Hoc Rule Semantics for Gene Expression Data. ISMIS 2005: 494-503

• M. Agier-Pailloux, J.-M. Petit, and E. Suzuki,

Unifying framework for rule semantics: Application to gene expression data,

• Fundam. Inform., vol. 78, no. 4, pp. 543–559, 2007.

Rakesh Agrawal, Ramakrishnan Srikant: Fast Algorithms for Mining Association Rules in Large Databases. VLDB 1994: 487-499 Jaume Baixeries and Victor Codocedo and Mehdi Kaytoue and Amedeo Napoli, Characterizing approximate-matching dependencies in formal concept analysis with pattern structures, Discrete Applied Mathematics, 2018

53

Bridging the Gap between Data Diversity and Data Dependencies Appendix For Further Reading

For Further Reading II

• F. Baklouti and G. Levy and R. Emilion

A fast algorithm for general Galois lattices building.

• Elec. J. Symbolic data analysis, Vol. 2, N 1, 19-31, 2005
• C. Beeri, M. Dowd, R. Fagin, and R. Statman.

On the structure of Armstrong relations for functional dependencies. JACM, 31(1):30–46, 1984. Bertossi, Leopoldo and Kolahi, Solmaz and Lakshmanan, Laks V. S., Data Cleaning and Query Answering with Matching Dependencies and Matching Functions, ICDT 2011, pp. 268–279 Loredana Caruccio ; Vincenzo Deufemia ; Giuseppe Polese Relaxed Functional Dependencies: A Survey of Approaches IEEE TKDE, vol 28, issue 1, 2016

• N. Caspard and B. Monjardet,

The lattices of closure systems, closure operators, and implicational systems on a finite set: A survey, Discrete Applied Mathematics, vol. 127, no. 2, pp. 241–269, 2003.

54

Bridging the Gap between Data Diversity and Data Dependencies Appendix For Further Reading

For Further Reading III

Upen S. Chakravarthy, John Grant and Jack Minker Logic-based approach to semantic query optimization ACM Transactions on Database Systems (TODS) TODS Homepage archive Volume 15 Issue 2, June 1990, pages 162-207 Brice Chardin, Emmanuel Coquery, Marie Pailloux, Jean-Marc Petit: RQL: A SQL-Like Query Language for Discovering Meaningful Rules. IEEE ICDM demo 2014: 1203-1206 Brice Chardin and Emmanuel Coquery and Marie Pailloux and JM Petit, RQL: A Query Language for Rule Discovery in Databases,

• Theor. Comput. Sci., vol. 658, pp. 357–374, 2017,

Xu Chu, Ihab F. Ilyas, Paolo Papotti: Discovering Denial Constraints. PVLDB 6(13): 1498-1509 (2013)

• J. Demetrovics and V. D. Thi

Some remarks on generating Armstrong and inferring functional dependencies relation, Acta Cybernetica, vol. 12, no. 2, pp. 167–180, 1995.

55

Bridging the Gap between Data Diversity and Data Dependencies Appendix For Further Reading

For Further Reading IV

Wenfei Fan and Xibei Jia and Jianzhong Li and Shuai Ma, Reasoning about Record Matching Rules, PVLDB, vol 2, num 1, pp. 407–418, 2009

• L. Jezkov´

a, Pablo Cordero, and Manuel Enciso. Fuzzy functional dependencies: A comparative survey. Fuzzy Sets and Systems, 317:88–120, 2017. Abo Khamis, Mahmoud and Ngo, Hung Q. and Nguyen, XuanLong and Olteanu, Dan and Schleich, Maximilian: In-Database Learning with Sparse Tensors, PODS ’18, pp. 325–340

• S. Lopes, J.-M. Petit, and L. Lakhal,

Functional and approximate dependency mining: database and FCA points of view,

• J. Exp. Theor. Artif. Intell., vol. 14, no. 2-3, pp. 93–114, 2002.

Heikki Mannila, Kari-Jouko Rih: Algorithms for Inferring Functional Dependencies from Relations. Data Knowl. Eng. 12(1): 83-99 (1994)

56

Bridging the Gap between Data Diversity and Data Dependencies Appendix For Further Reading

Thank you Merci

57