Integrating inconsistent data in a probabilistic model Ji r - - PowerPoint PPT Presentation

Integrating inconsistent data in a probabilistic model

Jiˇ r´ ı Vomlel

This presentation is available at http://www.utia.cas.cz/vomlel/

Knowledge integration

• Discrete random variables Xi indexed by natural numbers from

V = {1, . . . , n} ⊂ N.

• Low-dimensional probability distributions Pj, j = 1, . . . , k defined
• n variables {Xℓ}ℓ∈Ej, Ej ⊆ V.
• Knowledge integration is the process of building a joint

probability distribution Q(X1, . . . , Xn) from a set of low-dimensional probability distributions P = {P1, . . . , Pk}.

α ? ? ?

1 2 −α

α α

1 2 −α

? ? ? ? ?

X2 X1 X3

α

1 2 −α 1 2 −α

α

1 2 −α

α

1 2 −α

1 2 −α

≤

? ?

1 2 −α

α α

1 2 −α

? ? ? ?

X2 X1 X3

α

1 2 −α 1 2 −α

α

1 2 −α

α

1 2 −α

α

≤ α ≤ α

α

≤ +

α

α

1 6

Consistent case

6 20 3 20 1 20 3 20 3 20 3 20 6 20 4 20 6 20 4 20

α =

4 20

4 20 6 20 4 20 4 20 6 20 6 20 4 20 1 20 3 20

X1 X3

3 20

X2

• input set

P = {P1, . . . , Pk}

• set of all distributions having Pj as its marginal

Sj = {Q : QEj = Pj}

• set of all distributions having {P1, . . . , Pk} as its marginals

S = ∩k

j=1Sj

• I-projection of Q0 to S

π(Q0, S) = arg minQ∈S I(Q Q0)

• Kullback-Leibler divergence

I(P Q)

= ∑

x

P(X = x) log P(X = x) Q(X = x)

Iterative Proportional Fitting Procedure (IPFP) Deming & Stephan, 1940

π(Q(2), S3) Q(0) π(Q(0), S1) Q(1) Q(2) π(Q(1), S2) Q(3) Q(4)

S1 S2 S3

π(Q, Sj)

=

Q Pj QEj

IPFP on the consistent input

0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18 0.2 3 6 9 12 15 Probability value Iteration number IPFP with ordering P1,P2,P3 IPFP with ordering P1,P3,P2

Inconsistent case

α =

1 10

? ? ? ? ? ? ?

1 10 4 10 1 10 4 10 4 10 1 10 4 10 1 10 1 10 4 10 1 10 4 10

X1 X3 ? X2

IPFP on the inconsistent input

0.05 0.1 0.15 0.2 0.25 0.3 0.35 3 6 9 12 15 18 21 Probability value Iteration number IPFP with ordering P1,P2,P3 IPFP with ordering P1,P3,P2

Let r =

√ −3α2 + 2α, β = 0.5(1 −α − r), and γ = 0.5(−α + r).

The limit cycle for the ordering P1, P2, P3 x 000 001 010 011 100 101 110 111 limn→∞Q3n+1(x) α γ β β γ α limn→∞Q3n+2(x) γ β α α β γ limn→∞Q3n+3(x) β α γ γ α β

• arithm. average

1 6 1 6 1 6 1 6 1 6 1 6

The limit cycle for the ordering P1, P3, P2 x 000 001 010 011 100 101 110 111 limn→∞Q3n+1(x) α β γ γ β α limn→∞Q3n+2(x) γ α β β α γ limn→∞Q3n+3(x) β γ α α γ β

• arithm. average

1 6 1 6 1 6 1 6 1 6 1 6

For α = 0.1 we get β .

= 0.244 and γ . = 0.156.

Inconsistent input set P = {P1, . . . , Pk}

It means that S = ∩k

j=1Sj = ∅.

Q(X1, . . . , Xn) is required to:

• minimize a distance aggregate with respect to P:

∑

Pj∈P

wj · d(Pj QEj)

• factorize with respect to E = {E1, . . . , Ek}:

there exist potentials ψEi : XEi → R, i = 1, 2, . . . , k such that for all x ∈ X Q(x)

=

∏

Ei∈E

ψEi(xEi) .

Distance

• measured by the Kullback-Leibler divergence

d(Pj QEj)

= (Pj QEj) = ∑

xEj

Pj(xEj) log Pj(xEj) QEj(xEj)

• measured by the total variance

d(Pj QEj)

= |Pj − QEj| = ∑

xEj

|Pj(xEj) − QEj(xEj)|

IPFP properties in the inconsistent case

• “converges” to a limit cycle
• distributions in the limit cycle are different for different orderings
• f the input set
• in the example, the average of the distributions in the limit cycle

does not depend on the ordering – but generally, it is not true

• in the example, the distributions in the limit cycles minimized the

aggregate of the total variance – but generally it is not known

• there are also other distributions that minimize the aggregate of

the total variance that are not computed with IPFP

• generally, the distributions in the limit cycles do not minimize the

aggregate of the Kullback-Leibler divergence

• distributions computed within a finite number of iterations

factorize with respect to E = {E1, . . . , Ek}

GEMA

Q(0) Q(0)

S′

1

S′

3

π(Q(2), S′

3)

π(Q(0), S1) π(Q(1), S′

2)

π(Q(0), S2)

S3 S′

2

π(Q(0), S′

1)

Q(0) Q(2) Q(0)

S1

Q(1) Q(3) Q(3)

S2

π(Q(0), S3)

GEMA on the consistent input

0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18 0.2 30 60 90 120 150 180 210 Probability value Iteration number GEMA

GEMA on the inconsistent input set

0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18 0.2 30 60 90 120 150 180 210 Probability value Iteration number GEMA

GEMA properties

• converges also in the inconsistent case
• the limit distribution satisfies the necessary condition for the local

minima of the aggregate of the Kullback-Leibler divergence

• the distributions computed within a finite number of iterations

factorize with respect to E = {E1, . . . , Ek}