Ev aluating Hyp otheses Read Ch Recommended exercises - - PDF document

ev aluating hyp otheses read ch recommended exercises
SMART_READER_LITE
LIVE PREVIEW

Ev aluating Hyp otheses Read Ch Recommended exercises - - PDF document

Ev aluating Hyp otheses Read Ch Recommended exercises Sample error true error Condence in terv als for observ ed h yp othesis error Estimators


slide-1
SLIDE 1 Ev aluating Hyp
  • theses
Read Ch
  • Recommended
exercises
  • Sample
error true error
  • Condence
in terv als for
  • bserv
ed h yp
  • thesis
error
  • Estimators
  • Binomial
distribution Normal distribution Cen tral Limit Theorem
  • P
aired t tests
  • Comparing
learning metho ds
  • lecture
slides for textb
  • k
Machine L e arning T Mitc hell McGra w Hill
slide-2
SLIDE 2 Tw
  • Denitions
  • f
Error The true error
  • f
h yp
  • thesis
h with resp ect to target function f and distribution D is the probabilit y that h will misclassify an instance dra wn at random according to D
  • er
r
  • r
D h
  • Pr
xD f x
  • hx
The sample error
  • f
h with resp ect to target function f and data sample S is the prop
  • rtion
  • f
examples h misclassies er r
  • r
S h
  • n
X xS
  • f
x
  • hx
Where
  • f
x
  • hx
is
  • if
f x
  • hx
and
  • therwise
Ho w w ell do es er r
  • r
S h estimate er r
  • r
D h
  • lecture
slides for textb
  • k
Machine L e arning T Mitc hell McGra w Hill
slide-3
SLIDE 3 Problems Estimating Error
  • Bias
If S is training set er r
  • r
S h is
  • ptimisticall
y biased bias
  • E
er r
  • r
S h
  • er
r
  • r
D h F
  • r
un biased estimate h and S m ust b e c hosen indep enden tly
  • V
arianc e Ev en with un biased S
  • er
r
  • r
S h ma y still vary from er r
  • r
D h
  • lecture
slides for textb
  • k
Machine L e arning T Mitc hell McGra w Hill
slide-4
SLIDE 4 Example Hyp
  • thesis
h misclassies
  • f
the
  • examples
in S er r
  • r
S h
  • What
is er r
  • r
D h
  • lecture
slides for textb
  • k
Machine L e arning T Mitc hell McGra w Hill
slide-5
SLIDE 5 Estimators Exp erimen t
  • c
ho
  • se
sample S
  • f
size n according to distribution D
  • measure
er r
  • r
S h er r
  • r
S h is a random v ariable ie result
  • f
an exp erimen t er r
  • r
S h is an un biased estimator for er r
  • r
D h Giv en
  • bserv
ed er r
  • r
S h what can w e conclude ab
  • ut
er r
  • r
D h
  • lecture
slides for textb
  • k
Machine L e arning T Mitc hell McGra w Hill
slide-6
SLIDE 6 Condence In terv als If
  • S
con tains n examples dra wn indep enden tly
  • f
h and eac h
  • ther
  • n
  • Then
  • With
appro ximately
  • probabilit
y
  • er
r
  • r
D h lies in in terv al er r
  • r
S h
  • v
u u u u t er r
  • r
S h
  • er
r
  • r
S h n
  • lecture
slides for textb
  • k
Machine L e arning T Mitc hell McGra w Hill
slide-7
SLIDE 7 Condence In terv als If
  • S
con tains n examples dra wn indep enden tly
  • f
h and eac h
  • ther
  • n
  • Then
  • With
appro ximately N probabilit y
  • er
r
  • r
D h lies in in terv al er r
  • r
S h
  • z
N v u u u u t er r
  • r
S h
  • er
r
  • r
S h n where N
  • z
N
  • lecture
slides for textb
  • k
Machine L e arning T Mitc hell McGra w Hill
slide-8
SLIDE 8 er r
  • r
S h is a Random V ariable Rerun the exp erimen t with dieren t randomly dra wn S
  • f
size n Probabilit y
  • f
  • bserving
r misclassied examples

0.02 0.04 0.06 0.08 0.1 0.12 0.14 5 10 15 20 25 30 35 40 P(r) Binomial distribution for n = 40, p = 0.3

P r
  • n
r n
  • r
  • er
r
  • r
D h r
  • er
r
  • r
D h nr
  • lecture
slides for textb
  • k
Machine L e arning T Mitc hell McGra w Hill
slide-9
SLIDE 9 Binomial Probabilit y Distributi
  • n

0.02 0.04 0.06 0.08 0.1 0.12 0.14 5 10 15 20 25 30 35 40 P(r) Binomial distribution for n = 40, p = 0.3

P r
  • n
r n
  • r
  • p
r
  • p
nr Probabilit y P r
  • f
r heads in n coin ips if p
  • Pr
heads
  • Exp
ected
  • r
mean v alue
  • f
X
  • E
X
  • is
E X
  • n
X i iP i
  • np
  • V
ariance
  • f
X is V ar X
  • E
X
  • E
X
  • np
  • p
  • Standard
deviation
  • f
X
  • X
  • is
  • X
  • r
E X
  • E
X
  • r
np
  • p
  • lecture
slides for textb
  • k
Machine L e arning T Mitc hell McGra w Hill
slide-10
SLIDE 10 Normal Distributi
  • n
Appro ximates Bino mial er r
  • r
S h follo ws a Binomial distribution with
  • mean
  • er
r
  • r
S h
  • er
r
  • r
D h
  • standard
deviation
  • er
r
  • r
S h
  • er
r
  • r
S h
  • v
u u u u t er r
  • r
D h
  • er
r
  • r
D h n Appro ximate this b y a Normal distribution with
  • mean
  • er
r
  • r
S h
  • er
r
  • r
D h
  • standard
deviation
  • er
r
  • r
S h
  • er
r
  • r
S h
  • v
u u u u t er r
  • r
S h
  • er
r
  • r
S h n
  • lecture
slides for textb
  • k
Machine L e arning T Mitc hell McGra w Hill
slide-11
SLIDE 11 Normal Probabilit y Distributi
  • n

0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4

  • 3
  • 2
  • 1

1 2 3 Normal distribution with mean 0, standard deviation 1

px
  • p
  • e
  • x
  • The
probabilit y that X will fall in to the in terv al a b is giv en b y Z b a pxdx
  • Exp
ected
  • r
mean v alue
  • f
X
  • E
X
  • is
E X
  • V
ariance
  • f
X is V ar X
  • Standard
deviation
  • f
X
  • X
  • is
  • X
  • lecture
slides for textb
  • k
Machine L e arning T Mitc hell McGra w Hill
slide-12
SLIDE 12 Normal Probabilit y Distributi
  • n

0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4

  • 3
  • 2
  • 1

1 2 3

  • f
area probabilit y lies in
  • N
  • f
area probabilit y lies in
  • z
N
  • N
  • z
N
  • lecture
slides for textb
  • k
Machine L e arning T Mitc hell McGra w Hill
slide-13
SLIDE 13 Condence In terv als More Correctly If
  • S
con tains n examples dra wn indep enden tly
  • f
h and eac h
  • ther
  • n
  • Then
  • With
appro ximately
  • probabilit
y
  • er
r
  • r
S h lies in in terv al er r
  • r
D h
  • v
u u u u t er r
  • r
D h
  • er
r
  • r
D h n equiv alen tl y
  • er
r
  • r
D h lies in in terv al er r
  • r
S h
  • v
u u u u t er r
  • r
D h
  • er
r
  • r
D h n whic h is appro ximately er r
  • r
S h
  • v
u u u u t er r
  • r
S h
  • er
r
  • r
S h n
  • lecture
slides for textb
  • k
Machine L e arning T Mitc hell McGra w Hill
slide-14
SLIDE 14 Cen tral Limit Theorem Consider a set
  • f
indep enden t iden ticall y distributed random v ariables Y
  • Y
n
  • all
go v erned b y an arbitrary probabilit y distribution with mean
  • and
nite v ariance
  • Dene
the sample mean
  • Y
  • n
n X i Y i Cen tral Limit Theorem As n
  • the
distribution go v erning
  • Y
approac hes a Normal distribution with mean
  • and
v ariance
  • n
  • lecture
slides for textb
  • k
Machine L e arning T Mitc hell McGra w Hill
slide-15
SLIDE 15 Calculating Condence In terv als
  • Pic
k parameter p to estimate
  • er
r
  • r
D h
  • Cho
  • se
an estimator
  • er
r
  • r
S h
  • Determine
probabilit y distribution that go v erns estimator
  • er
r
  • r
S h go v erned b y Binomial distribution appro ximated b y Normal when n
  • Find
in terv al L U
  • suc
h that N
  • f
probabilit y mass falls in the in terv al
  • Use
table
  • f
z N v alues
  • lecture
slides for textb
  • k
Machine L e arning T Mitc hell McGra w Hill
slide-16
SLIDE 16 Dierence Bet w een Hyp
  • theses
T est h
  • n
sample S
  • test
h
  • n
S
  • Pic
k parameter to estimate d
  • er
r
  • r
D h
  • er
r
  • r
D h
  • Cho
  • se
an estimator
  • d
  • er
r
  • r
S
  • h
  • er
r
  • r
S
  • h
  • Determine
probabilit y distribution that go v erns estimator
  • d
  • s
error S
  • h
  • error
S
  • h
  • n
  • error
S
  • h
  • error
S
  • h
  • n
  • Find
in terv al L U
  • suc
h that N
  • f
probabilit y mass falls in the in terv al
  • dz
N v u u u u u t er r
  • r
S
  • h
  • er
r
  • r
S
  • h
  • n
  • er
r
  • r
S
  • h
  • er
r
  • r
S n
  • lecture
slides for textb
  • k
Machine L e arning T Mitc hell McGra w Hill
slide-17
SLIDE 17 P aired t test to compare h A h B
  • P
artition data in to k disjoin t test sets T
  • T
  • T
k
  • f
equal size where this size is at least
  • F
  • r
i from
  • to
k
  • do
  • i
  • er
r
  • r
T i h A
  • er
r
  • r
T i h B
  • Return
the v alue
  • where
  • k
k X i
  • i
N
  • condence
in terv al estimate for d
  • t
N k
  • s
  • s
  • v
u u u u u t
  • k
k
  • k
X i
  • i
  • Note
  • i
appr
  • ximately
Normal ly distribute d
  • lecture
slides for textb
  • k
Machine L e arning T Mitc hell McGra w Hill
slide-18
SLIDE 18 Comparing learning algorithms L A and L B What w ed lik e to estimate E S D er r
  • r
D L A S
  • er
r
  • r
D L B S
  • where
LS
  • is
the h yp
  • thesis
  • utput
b y learner L using training set S ie the exp ected dierence in true error b et w een h yp
  • theses
  • utput
b y learners L A and L B
  • when
trained using randomly selected training sets S dra wn according to distribution D
  • But
giv en limited data D
  • what
is a go
  • d
estimator
  • could
partition D
  • in
to training set S and training set T
  • and
measure er r
  • r
T
  • L
A S
  • er
r
  • r
T
  • L
B S
  • ev
en b etter rep eat this man y times and a v erage the results next slide
  • lecture
slides for textb
  • k
Machine L e arning T Mitc hell McGra w Hill
slide-19
SLIDE 19 Comparing learning algorithms L A and L B
  • P
artition data D
  • in
to k disjoin t test sets T
  • T
  • T
k
  • f
equal size where this size is at least
  • F
  • r
i from
  • to
k
  • do
use T i for the test set and the r emaining data for tr aining set S i
  • S
i
  • fD
  • T
i g
  • h
A
  • L
A S i
  • h
B
  • L
B S i
  • i
  • er
r
  • r
T i h A
  • er
r
  • r
T i h B
  • Return
the v alue
  • where
  • k
k X i
  • i
  • lecture
slides for textb
  • k
Machine L e arning T Mitc hell McGra w Hill
slide-20
SLIDE 20 Comparing learning algorithms L A and L B Notice w ed lik e to use the paired t test
  • n
  • to
  • btain
a condence in terv al but not really correct b ecause the training sets in this algorithm are not indep enden t they
  • v
erlap more correct to view algorithm as pro ducing an estimate
  • f
E S D
  • er
r
  • r
D L A S
  • er
r
  • r
D L B S
  • instead
  • f
E S D er r
  • r
D L A S
  • er
r
  • r
D L B S
  • but
ev en this appro ximation is b etter than no comparison
  • lecture
slides for textb
  • k
Machine L e arning T Mitc hell McGra w Hill