Strong Asymptotic Assertions for Discrete MDL in Regression and - - PowerPoint PPT Presentation

IDSIA Lugano Switzerland

Strong Asymptotic Assertions for Discrete MDL in Regression and Classification Jan Poland and Marcus Hutter A Strange Way of Proving Consistency of MDL Learning

• r

2

Focus of this Talk

Regression Classification Sequence Prediction

this talk

this paper COLT‘04

3

Why Consistency?

4

Setup

! xn, yn xi yi , i n x " # y $ %& # yx '& '($" " )))

5

Bayesian Framework

* ν ! # ! , ! + ν wν > , %&

ν∈C wν

+!&

6

Proper/Online Learning

• &

xt, y<t x<t, y<t . !& x ! !! y #

7

Bayes Mixture

" xn, yn" ξynxn, yn

• ν wν

n

t νytxt

• ν wν

n

t νytxt

/ ! / !"

8

Static MDL

" ! $*.& ̺ynxn, yn ν∗

x,yynxn

• ν∗

x,y ν∈C wννynxn

0! ! )

9

Dynamic MDL

! $0 #!1 !1 " & ̺yn, y<n ̺ynxn ̺y<nx<n

• ̺yn, y<n

̺ynxn

• y ̺ynxn

̺ynxn

ν∈C wννynxn.

!& ! ! yn) ! $0 ! ! ! 2)

10

Distance and Convergence

& h

t, ψ

• y∈{,}
• ytxt, y<t
• ψytxt, y<t
• .

ψ# # H

x∞, ψ ∞

• t

h

t, ψ < .

! h

t )

11

Other Distance Measures

st, ψ

• y∈{,}
• ytxt, y<t ψytxt, y<t
• %

at, ψ

• y∈{,}
• ytxt, y<t ψytxt, y<t
• dt, ψ
• y∈{,}

ytxt, y<t ytxt, y<t ψytxt, y<t ,3#

12

Distance Measures: Properties

4 ht&

• %

at dt ! 5 st&

• %

at dt ! * at& % dt ,3# dt& % at

13

Convergence Theorem

̺ ̺

• ̺
• t at w−
• t at w−
• t dt w−
• H,̺ w−
• 6 "

w "

14

Properties of the Proof

.

• #!

'!7 #

15

Loss bounds

*! ℓy, yx x" y" # y ) /#! L ! & L̺ L w−

• w−

L

# 1 # !

16

Discussion

w−

• !
• −
• −
• !

17

Discussion

" !# Thank you!