USING DOMAIN KNO WLEDGE IN MA CHINE LEARNING Inductiv e vs - PDF document

USING DOMAIN KNO WLEDGE IN MA CHINE LEARNING� Inductiv e vs� Analytical Learning Carolina Ruiz �

Motiv ation Inductiv e Learning induction Examples ������������������ �� � hypothesis �� Prior Knowledge� �generalization� � Statistical reasoning is used to iden tify features that empirically distinguish from examples� M K �e�g� decision trees� NNs� ILP � GAs� � F undamen tal b ounds on accuracy dep end on n um� b er of training examples� Analytical Learning deduction Examples ������������������ �� � hypothesis � Prior Knowledge �generalization� � Logical reasoning is used to iden tify features that distinguish from examples� M K � W orks w ell ev en when training examples are scarce� Inductiv e � Analytical Learning � Com bines the b est of b oth w orlds �

Learning L e arning is to impr ove with exp erienc e at some task Learning from Examples� Inductiv e Learning � Giv en� A set of training examples h x � i � h x � i � f � x � � � � f � x � � n n � Find� A h yp othesis h de�ned o v er the whole space of instances that coincides with f o v er the training � � data� i�e� h � x � � f � x � for all � i n � i i Learning from Examples and Prior Kno wledge � Giv en� A set of training examples h x � i � h x � i � � f � x � � � � f � x and � � n n a set B of rules expressing prior kno wledge � Find� A h yp othesis h de�ned o v er the whole space of instances s�t� B � h � x � f � x � for all � � i � n� i i �

P art �� Motiv ation Inductiv e Learning induction Examples ������������������ �� � hypothesis �� Prior Knowledge� �generalization� � Statistical reasoning is used to iden tify features that empirically distinguish from examples� M K �e�g� decision trees� NNs� ILP � GAs� � F undamen tal b ounds on accuracy dep end on n um� b er of training examples� � When the output h yp othesis is a set of rules then this learning is called Inductive L o gic Pr o gr am� ming �ILP� � ILP� uses prior kno wledge to augmen t the input description of instances �whic h increases the com� plexit y of the h yp othesis space�� �

Inductiv e Logic Programming INDUCTIVE LOGIC PROGRAMMING � � INDUCTIVE MACHINE LEARNING COMPUTATIONAL LOGIC F rom Inductiv e Mac hine Learning� inherits its goal�s�� � to dev elop to ols and tec hniques to induce h yp othe� ses from observ ations �examples�� or � to syn thesize new kno wledge from exp erience� F rom Computational Logic� inherits� � abilit y to o v ercome limitations of classical induc� tiv e learners� �� use of limited kno wledge represen tation formal� ism �e�g� prop ositional logic� �� inabilit y to use substan tial domain kno wledge during learning pro cess� � theory � orien tation� and tec hniques � in terest in prop erties of rules� con v ergence �sound� ness and completeness� of algorithms� computa� tional complexit y � �

EXAMPLE �tak en from Luc De Raedt and Nada La vrac� ����� Giv en � � a set of examples D � f f l ies � tw eety � g M � � K � a bac kground theory B bir d�twe ety� bir d�oliver� Find � � a h yp othesis h that explains the examples D w�r�t� B � f � g � h � f l ies � X � bir d � X � �

EXAMPLE �tak en from Luc De Raedt and Nada La vrac� ����� Giv en � � a set of examples D � sor t ���� � ���� � � � � M sor t ��� � � � �� � �� � � � ��� � � � � sor t ��� � �� � ���� � � � � K sor t ��� � � � �� � �� � � � ��� � � � � a bac kground theory B � per mutation� � � � B � � sor ted� � � � � Find � � a h yp othesis h that satis�es the language bias and explains the examples D w�r�t� B � h � f sor t � X � Y � � per mutation � X � Y � � sor ted � Y � g � �

Sequen tial Co v ering T emplate � Giv en a set of examples � M K fg �� le arne d�rules �� �� while there are examples still to b e explained do �a� rule �� Learn�one�rule �b� remo v e from examples those explained b y rule �c� le arne d�rules �� le arne d�rules � rule �� Output le arne d�rules �

ILP implemen tations of Learn�one�rule General to Sp eci�c Searc h �top do wn� e�g� F OIL �� Start with the most general clause e�g � sor t � X � Y � � �� sp ecialize the rule b y adding the b est p ossible lit� eral to the b o dy of the rule � e�g � sor t � X � Y � per mutation � X � Y � un til the h yp othesis en tails no examples en� K suring that h yp othesis en tails at least a threshold n um b er of examples� M Sp eci�c to General Searc h �b ottom up� e�g� GOLEM �� Start with the most sp eci�c clause that implies a giv en example �� generalize �� un til the h yp othesis cannot b e further generalized without implying negativ e examples� �

Searc hing for the b est literal � F oil �Quinlan� ����� Candidate Literals� All literals that can b e constructed using predicates and terms that app ear in B and�or in the examples� Ev aluation F unction� F oil�s information�based no� tion of gain Note� This searc h can b e computationally explosiv e Imp ortan t� to prune the searc h space� Matthew Berub e�s MQP �advised b y C� Ruiz and S� Alv arez� prunes the searc h space b y using typ e in� formation making F OIL applicable to some large do� mains� ��

Summarizing Inductiv e Learning � Uses statistical inference to compute a h yp othesis that �t the data � Uses bac kground kno wledge to augmen t the input de� scription of instances �whic h ma y increase the com� plexit y of the h yp othesis space�� � Requires little bac kground kno wledge � Requires large amoun ts of training data ��

P art I I� Motiv ation Analytical Learning deduction Examples ������������������ �� � hypothesis � Prior Knowledge �generalization� � Logical reasoning is used to iden tify features that distinguish from examples� M K � W orks w ell ev en when training examples are scarce� � Analytical Learning Metho d� Explanation�Based Learning �EBL� � EBL� uses prior kno wledge to reduce the complex� it y of the h yp othesis space and to impro v e the ac� curacy of the output h yp othesis� � Prior Kno wledge is assumed to b e correct and com� plete ��

Analytical Learning The Analytical Learning T ask� � h x � i Giv en� A set D of training examples � f � x i i and Bac kground Kno wledge B �represen ted as a set of rules � Horn clauses� � Find� a h yp othesis h suc h that� � � �� D B h �� for all � � i � n� h � x � f � x � i i P art ��� ab o v e reduces the size of the h yp othesis space and mak es the output h yp othesis more accurate� Assumptions� The bac kground kno wledge is Correct� eac h of its assertions is a truthful statemen t ab out the w orld Complete� ev ery instance that satis�es the target concept can b e pro v en b y the domain kno wledge to satisfy it� Are these reasonable assumptions� Wh y b other learn� ing if the bac kground kno wledge is already complete� ��

Analytical Learning � Motiv ation � Chess example �tak en from Mitc hell�s b o ok� � Man y features are presen t� Prior kno wledge helps us tell apart relev an t features from irrelev an t ones� � F or the c hess domain� it is easy to write do wn rules that completely describ e all legal mo v es� but it is v ery hard to write do wn rules that completely describ e an optimal mo v e strategy � ��