Ev aluating Hyp otheses Read Ch Recommended exercises - - PDF document

SLIDE 1 Ev aluating Hyp

• theses

Read Ch

• Recommended

exercises

• Sample

error true error

• Condence

in terv als for

• bserv

ed h yp

• thesis

error

• Estimators
• Binomial

distribution Normal distribution Cen tral Limit Theorem

• P

aired t tests

• Comparing

learning metho ds

• lecture

slides for textb

• k

Machine L e arning T Mitc hell McGra w Hill

SLIDE 2 Tw

• Denitions
• f

Error The true error

• f

h yp

• thesis

h with resp ect to target function f and distribution D is the probabilit y that h will misclassify an instance dra wn at random according to D

• er

r

• r

D h

• Pr

xD f x

• hx

The sample error

• f

h with resp ect to target function f and data sample S is the prop

• rtion
• f

examples h misclassies er r

• r

S h

• n

X xS

• f

x

• hx

Where

• f

x

• hx

is

• if

f x

• hx

and

• therwise

Ho w w ell do es er r

• r

S h estimate er r

• r

D h

• lecture

slides for textb

• k

Machine L e arning T Mitc hell McGra w Hill

SLIDE 3 Problems Estimating Error

• Bias

If S is training set er r

• r

S h is

• ptimisticall

y biased bias

• E

er r

• r

S h

• er

r

• r

D h F

• r

un biased estimate h and S m ust b e c hosen indep enden tly

• V

arianc e Ev en with un biased S

• er

r

• r

S h ma y still vary from er r

• r

D h

• lecture

slides for textb

• k

Machine L e arning T Mitc hell McGra w Hill

SLIDE 4 Example Hyp

• thesis

h misclassies

• f

the

• examples

in S er r

• r

S h

• What

is er r

• r

D h

• lecture

slides for textb

• k

Machine L e arning T Mitc hell McGra w Hill

SLIDE 5 Estimators Exp erimen t

• c

ho

• se

sample S

• f

size n according to distribution D

• measure

er r

• r

S h er r

• r

S h is a random v ariable ie result

• f

an exp erimen t er r

• r

S h is an un biased estimator for er r

• r

D h Giv en

• bserv

ed er r

• r

S h what can w e conclude ab

• ut

er r

• r

D h

• lecture

slides for textb

• k

Machine L e arning T Mitc hell McGra w Hill

SLIDE 6 Condence In terv als If

• S

con tains n examples dra wn indep enden tly

• f

h and eac h

• ther
• n
• Then
• With

appro ximately

• probabilit

y

• er

r

• r

D h lies in in terv al er r

• r

S h

• v

u u u u t er r

• r

S h

• er

r

• r

S h n

• lecture

slides for textb

• k

Machine L e arning T Mitc hell McGra w Hill

SLIDE 7 Condence In terv als If

• S

con tains n examples dra wn indep enden tly

• f

h and eac h

• ther
• n
• Then
• With

appro ximately N probabilit y

• er

r

• r

D h lies in in terv al er r

• r

S h

• z

N v u u u u t er r

• r

S h

• er

r

• r

S h n where N

• z

N

• lecture

slides for textb

• k

Machine L e arning T Mitc hell McGra w Hill

SLIDE 8 er r

• r

S h is a Random V ariable Rerun the exp erimen t with dieren t randomly dra wn S

• f

size n Probabilit y

• f
• bserving

r misclassied examples

0.02 0.04 0.06 0.08 0.1 0.12 0.14 5 10 15 20 25 30 35 40 P(r) Binomial distribution for n = 40, p = 0.3

P r

• n

r n

• r
• er

r

• r

D h r

• er

r

• r

D h nr

• lecture

slides for textb

• k

Machine L e arning T Mitc hell McGra w Hill

SLIDE 9 Binomial Probabilit y Distributi

• n

0.02 0.04 0.06 0.08 0.1 0.12 0.14 5 10 15 20 25 30 35 40 P(r) Binomial distribution for n = 40, p = 0.3

P r

• n

r n

• r
• p

r

• p

nr Probabilit y P r

• f

r heads in n coin ips if p

• Pr

heads

• Exp

ected

• r

mean v alue

• f

X

• E

X

• is

E X

• n

X i iP i

• np
• V

ariance

• f

X is V ar X

• E

X

• E

X

• np
• p
• Standard

deviation

• f

X

• X
• is
• X
• r

E X

• E

X

• r

np

• p
• lecture

slides for textb

• k

Machine L e arning T Mitc hell McGra w Hill

SLIDE 10 Normal Distributi

• n

Appro ximates Bino mial er r

• r

S h follo ws a Binomial distribution with

• mean
• er

r

• r

S h

• er

r

• r

D h

• standard

deviation

• er

r

• r

S h

• er

r

• r

S h

• v

u u u u t er r

• r

D h

• er

r

• r

D h n Appro ximate this b y a Normal distribution with

• mean
• er

r

• r

S h

• er

r

• r

D h

• standard

deviation

• er

r

• r

S h

• er

r

• r

S h

• v

u u u u t er r

• r

S h

• er

r

• r

S h n

• lecture

slides for textb

• k

Machine L e arning T Mitc hell McGra w Hill

SLIDE 11 Normal Probabilit y Distributi

• n

0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4

• 3
• 2
• 1

1 2 3 Normal distribution with mean 0, standard deviation 1

px

• p
• e
• x
• The

probabilit y that X will fall in to the in terv al a b is giv en b y Z b a pxdx

• Exp

ected

• r

mean v alue

• f

X

• E

X

• is

E X

• V

ariance

• f

X is V ar X

• Standard

deviation

• f

X

• X
• is
• X
• lecture

slides for textb

• k

Machine L e arning T Mitc hell McGra w Hill

SLIDE 12 Normal Probabilit y Distributi

• n

0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4

• 3
• 2
• 1

1 2 3

• f

area probabilit y lies in

• N
• f

area probabilit y lies in

• z

N

• N
• z

N

• lecture

slides for textb

• k

Machine L e arning T Mitc hell McGra w Hill

SLIDE 13 Condence In terv als More Correctly If

• S

con tains n examples dra wn indep enden tly

• f

h and eac h

• ther
• n
• Then
• With

appro ximately

• probabilit

y

• er

r

• r

S h lies in in terv al er r

• r

D h

• v

u u u u t er r

• r

D h

• er

r

• r

D h n equiv alen tl y

• er

r

• r

D h lies in in terv al er r

• r

S h

• v

u u u u t er r

• r

D h

• er

r

• r

D h n whic h is appro ximately er r

• r

S h

• v

u u u u t er r

• r

S h

• er

r

• r

S h n

• lecture

slides for textb

• k

Machine L e arning T Mitc hell McGra w Hill

SLIDE 14 Cen tral Limit Theorem Consider a set

• f

indep enden t iden ticall y distributed random v ariables Y

• Y

n

• all

go v erned b y an arbitrary probabilit y distribution with mean

• and

nite v ariance

• Dene

the sample mean

• Y
• n

n X i Y i Cen tral Limit Theorem As n

• the

distribution go v erning

• Y

approac hes a Normal distribution with mean

• and

v ariance

• n
• lecture

slides for textb

• k

Machine L e arning T Mitc hell McGra w Hill

SLIDE 15 Calculating Condence In terv als

• Pic

k parameter p to estimate

• er

r

• r

D h

• Cho
• se

an estimator

• er

r

• r

S h

• Determine

probabilit y distribution that go v erns estimator

• er

r

• r

S h go v erned b y Binomial distribution appro ximated b y Normal when n

• Find

in terv al L U

• suc

h that N

• f

probabilit y mass falls in the in terv al

• Use

table

• f

z N v alues

• lecture

slides for textb

• k

Machine L e arning T Mitc hell McGra w Hill

SLIDE 16 Dierence Bet w een Hyp

• theses

T est h

• n

sample S

• test

h

• n

S

• Pic

k parameter to estimate d

• er

r

• r

D h

• er

r

• r

D h

• Cho
• se

an estimator

• d
• er

r

• r

S

• h
• er

r

• r

S

• h
• Determine

probabilit y distribution that go v erns estimator

• d
• s

error S

• h
• error

S

• h
• n
• error

S

• h
• error

S

• h
• n
• Find

in terv al L U

• suc

h that N

• f

probabilit y mass falls in the in terv al

• dz

N v u u u u u t er r

• r

S

• h
• er

r

• r

S

• h
• n
• er

r

• r

S

• h
• er

r

• r

S n

• lecture

slides for textb

• k

Machine L e arning T Mitc hell McGra w Hill

SLIDE 17 P aired t test to compare h A h B

• P

artition data in to k disjoin t test sets T

• T
• T

k

• f

equal size where this size is at least

• F
• r

i from

• to

k

• do
• i
• er

r

• r

T i h A

• er

r

• r

T i h B

• Return

the v alue

• where
• k

k X i

• i

N

• condence

in terv al estimate for d

• t

N k

• s
• s
• v

u u u u u t

• k

k

• k

X i

• i
• Note
• i

appr

• ximately

Normal ly distribute d

• lecture

slides for textb

• k

Machine L e arning T Mitc hell McGra w Hill

SLIDE 18 Comparing learning algorithms L A and L B What w ed lik e to estimate E S D er r

• r

D L A S

• er

r

• r

D L B S

• where

LS

• is

the h yp

• thesis
• utput

b y learner L using training set S ie the exp ected dierence in true error b et w een h yp

• theses
• utput

b y learners L A and L B

• when

trained using randomly selected training sets S dra wn according to distribution D

• But

giv en limited data D

• what

is a go

• d

estimator

• could

partition D

• in

to training set S and training set T

• and

measure er r

• r

T

• L

A S

• er

r

• r

T

• L

B S

• ev

en b etter rep eat this man y times and a v erage the results next slide

• lecture

slides for textb

• k

Machine L e arning T Mitc hell McGra w Hill

SLIDE 19 Comparing learning algorithms L A and L B

• P

artition data D

• in

to k disjoin t test sets T

• T
• T

k

• f

equal size where this size is at least

• F
• r

i from

• to

k

• do

use T i for the test set and the r emaining data for tr aining set S i

• S

i

• fD
• T

i g

• h

A

• L

A S i

• h

B

• L

B S i

• i
• er

r

• r

T i h A

• er

r

• r

T i h B

• Return

the v alue

• where
• k

k X i

• i
• lecture

slides for textb

• k

Machine L e arning T Mitc hell McGra w Hill

SLIDE 20 Comparing learning algorithms L A and L B Notice w ed lik e to use the paired t test

• n
• to
• btain

a condence in terv al but not really correct b ecause the training sets in this algorithm are not indep enden t they

• v

erlap more correct to view algorithm as pro ducing an estimate

• f

E S D

• er

r

• r

D L A S

• er

r

• r

D L B S

• instead
• f

E S D er r

• r

D L A S

• er

r

• r

D L B S

• but

ev en this appro ximation is b etter than no comparison

• lecture

slides for textb

• k

Machine L e arning T Mitc hell McGra w Hill