ev aluating hyp otheses read ch recommended exercises
play

Ev aluating Hyp otheses Read Ch Recommended exercises - PDF document

Ev aluating Hyp otheses Read Ch Recommended exercises Sample error true error Condence in terv als for observ ed h yp othesis error Estimators


  1. Ev aluating Hyp otheses �Read Ch� �� �Recommended exercises� ���� ���� ���� � Sample error� true error � Con�dence in terv als for observ ed h yp othesis error � Estimators � Binomial distribution� Normal distribution� Cen tral Limit Theorem � P aired tests t � Comparing learning metho ds �� lecture slides for textb o ok Machine L e arning � T� Mitc hell� McGra w Hill� ����

  2. Tw o De�nitions of Error The of h yp othesis with resp ect to true error h target function and distribution D is the f probabilit y that will misclassify an instance h dra wn at random according to D � er r or � h � � Pr � f � x � � � h � x �� D x �D The of with resp ect to target sample error h function and data sample is the prop ortion of f S examples misclassi�es h � � h � � � f � x � � � h � x �� er r or � X S n x � S Where � f � x � � � h � x �� is � if � x � � � h � x �� and � � f otherwise� Ho w w ell do es � h � estimate � h �� er r or er r or S D �� lecture slides for textb o ok Machine L e arning � T� Mitc hell� McGra w Hill� ����

  3. Problems Estimating Error �� If is training set� � h � is Bias� S er r or S optimisticall y biased � � er � h �� � � h � bias E r or er r or S D F or un biased estimate� and m ust b e c hosen h S indep enden tly �� Ev en with un biased S � er r or � h � ma y V arianc e� S still from er r or � h � vary D �� lecture slides for textb o ok Machine L e arning � T� Mitc hell� McGra w Hill� ����

  4. Example Hyp othesis misclassi�es �� of the �� examples in h S �� � h � � � � �� er r or S �� What is � h �� er r or D �� lecture slides for textb o ok Machine L e arning � T� Mitc hell� McGra w Hill� ����

  5. Estimators Exp erimen t� �� c ho ose sample of size according to S n distribution D �� measure er r or � h � S � h � is a random v ariable �i�e�� result of an er r or S exp erimen t� � h � is an un biased for � h � er r or estimator er r or S D Giv en observ ed � h � what can w e conclude er r or S ab out � h �� er r or D �� lecture slides for textb o ok Machine L e arning � T� Mitc hell� McGra w Hill� ����

  6. Con�dence In terv als If � con tains examples� dra wn indep enden tly of S n and eac h other h � � �� n Then � With appro ximately ��� probabilit y � � h � er r or D lies in in terv al v � h ��� � � h �� er r or er r or u u S S er r or � h � � � � �� u u S t n �� lecture slides for textb o ok Machine L e arning � T� Mitc hell� McGra w Hill� ����

  7. Con�dence In terv als If � con tains examples� dra wn indep enden tly of S n and eac h other h � � �� n Then � With appro ximately N� probabilit y � � h � er r or D lies in in terv al v � h ��� � � h �� er r or er r or u u S S er r or � h � � z u u S N t n where N �� ��� ��� ��� ��� ��� ��� ��� � ���� ���� ���� ���� ���� ���� ���� z N �� lecture slides for textb o ok Machine L e arning � T� Mitc hell� McGra w Hill� ����

  8. � h � is a Random V ariable er r or S Rerun the exp erimen t with di�eren t randomly dra wn S �of size n � Probabilit y of observing misclassi�ed examples� r Binomial distribution for n = 40, p = 0.3 0.14 0.12 0.1 0.08 P(r) 0.06 0.04 0.02 0 0 5 10 15 20 25 30 35 40 n � r n � r � r � � � h � �� � � h �� P er r or er r or D D r �� n � r �� �� lecture slides for textb o ok Machine L e arning � T� Mitc hell� McGra w Hill� ����

  9. Binomial Probabilit y Distributi on Binomial distribution for n = 40, p = 0.3 0.14 0.12 0.1 0.08 P(r) 0.06 0.04 0.02 0 0 5 10 15 20 25 30 35 40 n � r n � r � r � � �� � p � P p �� n � �� r r Probabilit y P � r � of r heads in n coin �ips� if p � Pr � heads � � Exp ected� or mean v alue of X � E � X �� is n � X � � � i � � E iP np X i �� � V ariance of is X � � X � � �� X � � X �� � � np �� � p � V ar E E � Standard deviation of X � � � is X r r � � � E �� X � E � X �� � � np �� � p � X �� lecture slides for textb o ok Machine L e arning � T� Mitc hell� McGra w Hill� ����

  10. Normal Distributi on Appro ximates Bino� mial � h � follo ws a distribution� with er r or Binomial S � mean � � h � � er r or er r or � h � D S � standard deviation � er r or � h � S v � h ��� � � h �� er r or er r or u u D D � u � u � h � er r or t S n Appro ximate this b y a distribution with Normal � mean � � h � � er r or er r or � h � D S � standard deviation � � h � er r or S v er r or � h ��� � er r or � h �� u u S S � � u u � h � er r or t S n �� lecture slides for textb o ok Machine L e arning � T� Mitc hell� McGra w Hill� ����

  11. Normal Probabilit y Distributi on Normal distribution with mean 0, standard deviation 1 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0 -3 -2 -1 0 1 2 3 � x � � � � � � � p p � x � � e � � � � � � The probabilit y that will fall in to the in terv al X � a� b � is giv en b y b Z p � x � dx a � Exp ected� or mean v alue of � � X �� is X E � X � � E � � V ariance of is X � � X � � V ar � � Standard deviation of � � is X � X � � � X �� lecture slides for textb o ok Machine L e arning � T� Mitc hell� McGra w Hill� ����

  12. Normal Probabilit y Distributi on 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0 -3 -2 -1 0 1 2 3 ��� of area �probabilit y� lies in � � � �� � � N� of area �probabilit y� lies in � � z � N �� ��� ��� ��� ��� ��� ��� ��� N � ���� ���� ���� ���� ���� ���� ���� z N �� lecture slides for textb o ok Machine L e arning � T� Mitc hell� McGra w Hill� ����

  13. Con�dence In terv als� More Correctly If � con tains examples� dra wn indep enden tly of S n and eac h other h � � �� n Then � With appro ximately ��� probabilit y � � h � er r or S lies in in terv al v � h ��� � � h �� er r or er r or u u D D � h � � � � �� u er r or u D t n equiv alen tl y � er r or � h � lies in in terv al D v er r or � h ��� � er r or � h �� u u D D � h � � � � �� u er r or u S t n whic h is appro ximately v � h ��� � � h �� er r or er r or u u S S � h � � � � �� u er r or u S t n �� lecture slides for textb o ok Machine L e arning � T� Mitc hell� McGra w Hill� ����

  14. Cen tral Limit Theorem Consider a set of indep enden t� iden ticall y distributed random v ariables � all go v erned Y � � � Y � n b y an arbitrary probabilit y distribution with mean � and �nite v ariance � De�ne the sample mean� � � � n � � Y Y X i n i �� As � � � the Cen tral Limit Theorem� n � distribution go v erning approac hes a Normal Y � � distribution� with mean and v ariance � � n �� lecture slides for textb o ok Machine L e arning � T� Mitc hell� McGra w Hill� ����

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend


More recommend