preliminaries: Kolmogorov complexity U(p) = T i (p) K(x) = min p { p - - PowerPoint PPT Presentation

A safe & computable approximation to Kolmogorov complexity

Peter Bloem, Francisco Mota, Steven de Rooij, Luìs Antunes & Pieter Adriaans

preliminaries: Kolmogorov complexity

U(ıp) = Ti(p) K(x) = minp { p : U(p) = x }

preliminaries: Kolmogorov complexity

U(ıp) = Ti(p) K(x) = minp { p : U(p) = x }

ӫ U is a formalisation of the notion of a description ӫ K is invariant to the choice of U (up to a constant)

preliminaries: Kolmogorov complexity

U(ıp) = Ti(p) K(x) = minp { p : U(p) = x }

ӫ U is a formalisation of the notion of a description ӫ K is invariant to the choice of U (up to a constant)

preliminaries: Kolmogorov complexity

U(ıp) = Ti(p) K(x) = minp { p : U(p) = x }

ӫ U is a formalisation of the notion of a description ӫ K is invariant to the choice of U (up to a constant)

motivation “Kolmogorov is not computable, it’s only of theoretical use” No, approximations are usually correct

preliminaries: Probabilities & codes ӫ L(x): (prefjx) code length function ӫ p(x): probability (semi) measure

• log p(x) = L(x)

step 1: computable probabilities From TMs to probabilities: T(p) = x pT(x) = Σp:T(p) = x 2-|p| m(x) = pU(x) equivalent to the lower semicomputable semimeasures

step 2: model classes A model class C is an effectively enumerable subset of all Turing machines. UC(ıp) = Ti(p) KC(x) = minp { p : UC(p) = x } mC(x) = Σp:UC(p) = x 2-|p|

step 3: safe approximation ӫ L(x): approximating code-length function ӫ L(x) is safe against p when p(L(x) - K(x) ≥ k) ≤ cb-k for some c and b > 1

Is KC safe against p∈C?

no.

x x x x x x x x not x

x x x x x x x x not x

• log mC(x)

KC(x)

Is -log mC safe against p∈C?

yes.

• log mC is safe against mC

mC − log mC(x) − K(x) k

• = mC

mC(x) 2−k2−K(x) =

• x:mC(x)2−k2−K(x) mC(x)
• 2−k2−K(x)

= 2−k 2−K(x) 2−k

• log mC is safe against mC

mC − log mC(x) − K(x) k

• = mC

mC(x) 2−k2−K(x) =

• x:mC(x)2−k2−K(x) mC(x)
• 2−k2−K(x)

= 2−k 2−K(x) 2−k

• log mC is safe against mC

mC − log mC(x) − K(x) k

• = mC

mC(x) 2−k2−K(x) =

• x:mC(x)2−k2−K(x) mC(x)
• 2−k2−K(x)

= 2−k 2−K(x) 2−k

• log mC is safe against mC

mC − log mC(x) − K(x) k

• = mC

mC(x) 2−k2−K(x) =

• x:mC(x)2−k2−K(x) mC(x)
• 2−k2−K(x)

= 2−k 2−K(x) 2−k

• log mC is safe against mC

mC − log mC(x) − K(x) k

• = mC

mC(x) 2−k2−K(x) =

• x:mC(x)2−k2−K(x) mC(x)
• 2−k2−K(x)

= 2−k 2−K(x) 2−k

• log mC is safe against mC

mC − log mC(x) − K(x) k

• = mC

mC(x) 2−k2−K(x) =

• x:mC(x)2−k2−K(x) mC(x)
• 2−k2−K(x)

= 2−k 2−K(x) 2−k

• log mC is safe against members of C

mC(·) =

• q∈C

cqpq(·) cqpq(·) cqpq

• − log mC(x) − K(x) k
• mC

− log mC(x) − K(x) k

• 2−k

• log mC is safe against members of C

mC(·) =

• q∈C

cqpq(·) cqpq(·) cqpq

• − log mC(x) − K(x) k
• mC

− log mC(x) − K(x) k

• 2−k

• log mC is safe against members of C

mC(·) =

• q∈C

cqpq(·) cqpq(·) cqpq

• − log mC(x) − K(x) k
• mC

− log mC(x) − K(x) k

• 2−k

• log mC is safe against members of C

mC(·) =

• q∈C

cqpq(·) cqpq(·) cqpq

• − log mC(x) − K(x) k
• mC

− log mC(x) − K(x) k

• 2−k

can we compute mC? ӫ We can if it’s upper and lower semicomputable ӫ lower: dovetail all programs for UC ӫ upper: dovetail until (1-s)/sx ≤ 2c − 1 ӫ If C is complete, this algorithm is com- putable

KC(x) κC(x) =

• log mC(x)

κC(x) =

• log mC(x)
• log m(x)

K(x)

dominates unsafe bounds 2-safe dominates bounds bounds dominates dominates

What does this buy us? ӫ bridge between the practical and the platonic ӫ Bayesian ↔ MDL ↔ Algorithmic ӫ corollary: Kt ӫ Additional results: ID, NID

Questions?