computational learning theory an analysis of a
play

Computational Learning Theory: An Analysis of a Conjunction Learner - PowerPoint PPT Presentation

Oct 10, 2023 •347 likes •845 views

Computational Learning Theory: An Analysis of a Conjunction Learner Machine Learning Slides based on material from Dan Roth, Avrim Blum, Tom Mitchell and others 1 This lecture: Computational Learning Theory The Theory of Generalization

  1. Computational Learning Theory: An Analysis of a Conjunction Learner Machine Learning Slides based on material from Dan Roth, Avrim Blum, Tom Mitchell and others 1

  2. This lecture: Computational Learning Theory • The Theory of Generalization • Probably Approximately Correct (PAC) learning • Positive and negative learnability results • Agnostic Learning • Shattering and the VC dimension 2

  3. Where are we? • The Theory of Generalization – When can be trust the learning algorithm? – What functions can be learned? – Batch Learning • Probably Approximately Correct (PAC) learning • Positive and negative learnability results • Agnostic Learning • Shattering and the VC dimension 3

  4. This section 1. Analyze a simple algorithm for learning conjunctions 2. Define the PAC model of learning 3. Make formal connections to the principle of Occam’s razor 4

  5. This section 1. Analyze a simple algorithm for learning conjunctions 2. Define the PAC model of learning 3. Make formal connections to the principle of Occam’s razor 5

  6. Learning Conjunctions The true function f = x 2 ∧ x 3 ∧ x 4 ∧ x 5 ∧ x 100 Training data – <(1,1,1,1,1,1,…,1,1), 1> – <(1,1,1,0,0,0,…,0,0), 0> – <(1,1,1,1,1,0,...0,1,1), 1> – <(1,0,1,1,1,0,...0,1,1), 0> – <(1,1,1,1,1,0,...0,0,1), 1> – <(1,0,1,0,0,0,...0,1,1), 0> – <(1,1,1,1,1,1,…,0,1), 1> – <(0,1,0,1,0,0,...0,1,1), 0> 6

  7. Learning Conjunctions f = x 2 ∧ x 3 ∧ x 4 ∧ x 5 ∧ x 100 Training data – <(1,1,1,1,1,1,…,1,1), 1> A simple learning algorithm ( Elimination ) – <(1,1,1,0,0,0,…,0,0), 0> – <( 1 , 1 , 1 , 1 , 1 ,0,...0,1, 1 ), 1> Discard all negative examples • – <(1,0,1,1,1,0,...0,1,1), 0> – <( 1 , 1 , 1 , 1 , 1 ,0,...0,0, 1 ), 1> – <(1,0,1,0,0,0,...0,1,1), 0> – <( 1 , 1 , 1 , 1 , 1 ,1,…,0, 1 ), 1> – <(0,1,0,1,0,0,...0,1,1), 0> 7

  8. Learning Conjunctions f = x 2 ∧ x 3 ∧ x 4 ∧ x 5 ∧ x 100 Training data – <( 1 , 1 , 1 , 1 , 1 ,1,…,1, 1 ), 1> A simple learning algorithm ( Elimination ) – <(1,1,1,0,0,0,…,0,0), 0> – <( 1 , 1 , 1 , 1 , 1 ,0,...0,1, 1 ), 1> Discard all negative examples • – <(1,0,1,1,1,0,...0,1,1), 0> Build a conjunction using the features • – <( 1 , 1 , 1 , 1 , 1 ,0,...0,0, 1 ), 1> that are common to all positive – <(1,0,1,0,0,0,...0,1,1), 0> conjunctions – <( 1 , 1 , 1 , 1 , 1 ,1,…,0, 1 ), 1> h = x 1 ∧ x 2 ∧ x 3 ∧ x 4 ∧ x 5 ∧ x 100 – <(0,1,0,1,0,0,...0,1,1), 0> 8

  9. Learning Conjunctions f = x 2 ∧ x 3 ∧ x 4 ∧ x 5 ∧ x 100 Training data – <( 1 , 1 , 1 , 1 , 1 ,1,…,1, 1 ), 1> A simple learning algorithm ( Elimination ) – <(1,1,1,0,0,0,…,0,0), 0> – <( 1 , 1 , 1 , 1 , 1 ,0,...0,1, 1 ), 1> Discard all negative examples • – <(1,0,1,1,1,0,...0,1,1), 0> Build a conjunction using the features • – <( 1 , 1 , 1 , 1 , 1 ,0,...0,0, 1 ), 1> that are common to all positive – <(1,0,1,0,0,0,...0,1,1), 0> conjunctions – <( 1 , 1 , 1 , 1 , 1 ,1,…,0, 1 ), 1> h = x 1 ∧ x 2 ∧ x 3 ∧ x 4 ∧ x 5 ∧ x 100 – <(0,1,0,1,0,0,...0,1,1), 0> Positive examples eliminate irrelevant features 9

  10. Learning Conjunctions f = x 2 ∧ x 3 ∧ x 4 ∧ x 5 ∧ x 100 Training data – <( 1 , 1 , 1 , 1 , 1 ,1,…,1, 1 ), 1> A simple learning algorithm: – <(1,1,1,0,0,0,…,0,0), 0> – <( 1 , 1 , 1 , 1 , 1 ,0,...0,1, 1 ), 1> Discard all negative examples • – <(1,0,1,1,1,0,...0,1,1), 0> Build a conjunction using the features • – <( 1 , 1 , 1 , 1 , 1 ,0,...0,0, 1 ), 1> that are common to all positive – <(1,0,1,0,0,0,...0,1,1), 0> conjunctions – <( 1 , 1 , 1 , 1 , 1 ,1,…,0, 1 ), 1> h = x 1 ∧ x 2 ∧ x 3 ∧ x 4 ∧ x 5 ∧ x 100 – <(0,1,0,1,0,0,...0,1,1), 0> Clearly this algorithm produces a conjunction that is consistent with the data, that is err S (h) = 0, if the target function is a monotone conjunction Exercise: Why? 10

  11. Learning Conjunctions: Analysis h = x 1 ∧ x 2 ∧ x 3 ∧ x 4 ∧ x 5 ∧ x 100 f = x 2 ∧ x 3 ∧ x 4 ∧ x 5 ∧ x 100 Claim 1: Any hypothesis consistent with the training data will only make mistakes on positive future examples Why? 11

  12. Learning Conjunctions: Analysis h = x 1 ∧ x 2 ∧ x 3 ∧ x 4 ∧ x 5 ∧ x 100 f = x 2 ∧ x 3 ∧ x 4 ∧ x 5 ∧ x 100 Claim 1: Any hypothesis consistent with the training data will only make mistakes on positive future examples Why? A mistake will occur only if some literal z (in our example x 1 ) is present in h but not in f This mistake can cause a positive example to be predicted as negative by h Specifically: x 1 = 0, x 2 =1, x 3 =1, x 4 =1, x 5 =1, x 100 =1 The reverse situation can never happen For an example to be predicted as positive in the training set, every relevant literal must have been present 12

  13. Learning Conjunctions: Analysis h = x 1 ∧ x 2 ∧ x 3 ∧ x 4 ∧ x 5 ∧ x 100 f = x 2 ∧ x 3 ∧ x 4 ∧ x 5 ∧ x 100 Claim 1: Any hypothesis consistent with the training data will only make mistakes on positive future examples Why? A mistake will occur only if some literal z (in our example x 1 ) is present in h but not in f This mistake can cause a positive example to be predicted as negative by h Specifically: x 1 = 0, x 2 =1, x 3 =1, x 4 =1, x 5 =1, x 100 =1 The reverse situation can never happen For an example to be predicted as positive in the training set, every relevant literal must have been present 13

  14. Learning Conjunctions: Analysis h = x 1 ∧ x 2 ∧ x 3 ∧ x 4 ∧ x 5 ∧ x 100 f = x 2 ∧ x 3 ∧ x 4 ∧ x 5 ∧ x 100 Claim 1: Any hypothesis consistent with the training data will only make mistakes on positive future examples - Why? f h + + - A mistake will occur only if some literal z (in our - example x 1 ) is present in h but not in f This mistake can cause a positive example to be predicted as negative by h Specifically: x 1 = 0, x 2 =1, x 3 =1, x 4 =1, x 5 =1, x 100 =1 The reverse situation can never happen For an example to be predicted as positive in the training set, every relevant literal must have been present 14

  15. Learning Conjunctions: Analysis Theorem: Suppose we are learning a conjunctive concept with n dimensional Boolean features using m training examples. If then, with probability > 1 - ± , the error of the learned hypothesis err D (h) will be less than ² . 15

  16. Learning Conjunctions: Analysis Theorem: Suppose we are learning a conjunctive concept with n dimensional Boolean features using m training examples. If Poly in n, 1/ ± , 1/ ² then, with probability > 1 - ± , the error of the learned hypothesis err D (h) will be less than ² . If we see these many training examples, then the algorithm will produce a conjunction that, with high probability, will make few errors 16

  17. Learning Conjunctions: Analysis Theorem: Suppose we are learning a conjunctive concept with n dimensional Boolean features using m training examples. If then, with probability > 1 - ± , the error of the learned hypothesis err D (h) will be less than ² . Let’s prove this assertion 17

  18. Proof Intuition h = x 1 ∧ x 2 ∧ x 3 ∧ x 4 ∧ x 5 ∧ x 100 f = x 2 ∧ x 3 ∧ x 4 ∧ x 5 ∧ x 100 What kinds of examples would drive a hypothesis to make a mistake? Positive examples, where x 1 is absent f would say true and h would say false None of these examples appeared during training Otherwise x 1 would have been eliminated If they never appeared during training, maybe their appearance in the future would also be rare! Let’s quantify our surprise at seeing such examples 18

  19. Proof Intuition h = x 1 ∧ x 2 ∧ x 3 ∧ x 4 ∧ x 5 ∧ x 100 f = x 2 ∧ x 3 ∧ x 4 ∧ x 5 ∧ x 100 What kinds of examples would drive a hypothesis to make a mistake? Positive examples, where x 1 is absent f would say true and h would say false None of these examples appeared during training Otherwise x 1 would have been eliminated If they never appeared during training, maybe their appearance in the future would also be rare! Let’s quantify our surprise at seeing such examples 19

  20. Proof Intuition h = x 1 ∧ x 2 ∧ x 3 ∧ x 4 ∧ x 5 ∧ x 100 f = x 2 ∧ x 3 ∧ x 4 ∧ x 5 ∧ x 100 What kinds of examples would drive a hypothesis to make a mistake? Positive examples, where x 1 is absent f would say true and h would say false None of these examples appeared during training Otherwise x 1 would have been eliminated If they never appeared during training, maybe their appearance in the future would also be rare! Let’s quantify our surprise at seeing such examples 20

  21. Proof Intuition h = x 1 ∧ x 2 ∧ x 3 ∧ x 4 ∧ x 5 ∧ x 100 f = x 2 ∧ x 3 ∧ x 4 ∧ x 5 ∧ x 100 What kinds of examples would drive a hypothesis to make a mistake? Positive examples, where x 1 is absent f would say true and h would say false None of these examples appeared during training Otherwise x 1 would have been eliminated If they never appeared during training, maybe their appearance in the future would also be rare! Let’s quantify our surprise at seeing such examples 21

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Applying Computational Learning Theory to Software Testing Neil Walkinshaw Computational

Applying Computational Learning Theory to Software Testing Neil Walkinshaw Computational

Applying Computational Learning Theory to Software Testing Neil Walkinshaw Computational Learning Theory (CLT) Theoretical analysis of questions of learnability . Is it feasible to learn a given type of concept from a set of examples?

1.14k views • 4 slides
Computational Learning Theory: Probably Approximately Correct (PAC) Learning Machine Learning 1

Computational Learning Theory: Probably Approximately Correct (PAC) Learning Machine Learning 1

Computational Learning Theory: Probably Approximately Correct (PAC) Learning Machine Learning 1 Slides based on material from Dan Roth, Avrim Blum, Tom Mitchell and others Computational Learning Theory The Theory of Generalization

411 views • 28 slides
CSCE 478/878 Lecture 3: Computational Learning Theory Stephen D. Scott (Adapted from Tom

CSCE 478/878 Lecture 3: Computational Learning Theory Stephen D. Scott (Adapted from Tom

CSCE 478/878 Lecture 3: Computational Learning Theory Stephen D. Scott (Adapted from Tom Mitchells slides) September 8, 2003 1 Introduction Combines machine learning with: Algorithm design and analysis Computational complexity

393 views • 27 slides
CSCE 478/878 Lecture 3: Computational Learning Theory Examines the worst-case minimum and

CSCE 478/878 Lecture 3: Computational Learning Theory Examines the worst-case minimum and

Introduction Combines machine learning with: Algorithm design and analysis Computational complexity CSCE 478/878 Lecture 3: Computational Learning Theory Examines the worst-case minimum and maximum data and time requirements for

222 views • 7 slides
Computational Learning Theory: Positive and negative learnability results Machine Learning 1

Computational Learning Theory: Positive and negative learnability results Machine Learning 1

Computational Learning Theory: Positive and negative learnability results Machine Learning 1 Slides based on material from Dan Roth, Avrim Blum, Tom Mitchell and others Computational Learning Theory The Theory of Generalization Probably

534 views • 40 slides
Computational Learning Theory: Agnostic Learning Machine Learning 1 Slides based on material

Computational Learning Theory: Agnostic Learning Machine Learning 1 Slides based on material

Computational Learning Theory: Agnostic Learning Machine Learning 1 Slides based on material from Dan Roth, Avrim Blum, Tom Mitchell and others This lecture: Computational Learning Theory The Theory of Generalization Probably

653 views • 44 slides
Computational Learning Theory: The Theory of Generalization Machine Learning 1 Slides based on

Computational Learning Theory: The Theory of Generalization Machine Learning 1 Slides based on

Computational Learning Theory: The Theory of Generalization Machine Learning 1 Slides based on material from Dan Roth, Avrim Blum, Tom Mitchell and others Checkpoint: The bigger picture Supervised learning: instances, concepts, and

895 views • 42 slides
Computational Learning Theory: Shattering and VC Dimensions Machine Learning 1 Slides based on

Computational Learning Theory: Shattering and VC Dimensions Machine Learning 1 Slides based on

Computational Learning Theory: Shattering and VC Dimensions Machine Learning 1 Slides based on material from Dan Roth, Avrim Blum, Tom Mitchell and others This lecture: Computational Learning Theory The Theory of Generalization Probably

1.09k views • 81 slides
Computational Learning Theory: Occams Razor Machine Learning 1 Slides based on material from

Computational Learning Theory: Occams Razor Machine Learning 1 Slides based on material from

Computational Learning Theory: Occams Razor Machine Learning 1 Slides based on material from Dan Roth, Avrim Blum, Tom Mitchell and others This lecture: Computational Learning Theory The Theory of Generalization Probably

585 views • 35 slides
Computational Learning Theory For which tasks is successful learning possible? Under what

Computational Learning Theory For which tasks is successful learning possible? Under what

Computational Learning Theory For which tasks is successful learning possible? Under what conditions is successful learning guaranteed? What is successful learning? Probably approximately correct (PAC) framework Bounds on number

422 views • 27 slides
Computational Learning Theory Based on Machine Learning, T. Mitchell, McGRAW Hill, 1997, ch.

Computational Learning Theory Based on Machine Learning, T. Mitchell, McGRAW Hill, 1997, ch.

0. Computational Learning Theory Based on Machine Learning, T. Mitchell, McGRAW Hill, 1997, ch. 7 Acknowledgement: The present slides are an adaptation of slides drawn by T. Mitchell 1. Main Questions in Computational Learning Theory

682 views • 42 slides
Computational Learning Theory 1 / 22 Decidability Computation Decidability which

Computational Learning Theory 1 / 22 Decidability Computation Decidability which

Computational Learning Theory 1 / 22 Decidability Computation Decidability which problems have algorithmic solutions Machine Learning Feasibility what assumptions must we make to trust that we can learn an unknown target

246 views • 22 slides
Computational and Statistical Learning Theory TTIC 31120 Prof. Nati Srebro Lecture 7:

Computational and Statistical Learning Theory TTIC 31120 Prof. Nati Srebro Lecture 7:

Computational and Statistical Learning Theory TTIC 31120 Prof. Nati Srebro Lecture 7: Computational Complexity of Learning Hardness of Improper Learning (continued) Agnostic Learning Hardness of Learning via Crypto Easy to generate

433 views • 24 slides
Computational Learning Theory - HT 2018 formerly Advanced Machine Learning (HT 2017) Introduction

Computational Learning Theory - HT 2018 formerly Advanced Machine Learning (HT 2017) Introduction

Computational Learning Theory - HT 2018 formerly Advanced Machine Learning (HT 2017) Introduction and Course Details Varun Kanade University of Oxford January 15, 2018 What is Machine Learning? 200-basic level categories Here: Six

466 views • 18 slides
Statistical and Computational Statistical and Computational Learning Theory Learning Theory

Statistical and Computational Statistical and Computational Learning Theory Learning Theory

Statistical and Computational Statistical and Computational Learning Theory Learning Theory Fundamental Question: Predict Error Rates Fundamental Question: Predict Error Rates Given: Given: The space H of hypotheses The space H of

497 views • 29 slides
Computational Learning Theory [read Chapter 7] [Suggested exercises: 7.1, 7.2, 7.5, 7.8]

Computational Learning Theory [read Chapter 7] [Suggested exercises: 7.1, 7.2, 7.5, 7.8]

Computational Learning Theory [read Chapter 7] [Suggested exercises: 7.1, 7.2, 7.5, 7.8] Computational learning theory Setting 1: learner p oses queries to teac her Setting 2: teac her c ho oses examples

491 views • 24 slides
Observational equivalence using scheduler for quantum processes K. Yasuda / T. Kubota / Y.

Observational equivalence using scheduler for quantum processes K. Yasuda / T. Kubota / Y.

Observational equivalence using scheduler for quantum processes K. Yasuda / T. Kubota / Y. Kakutani Dept. of Computer Science University of Tokyo Outline 1. Introduction 2. Quantum process calculus qCCS 3. Open bisimulation on qCCS 4. Our

599 views • 42 slides
Bisimulation Congruences in the Calculus of Looping Sequences Roberto Barbuti Andrea Maggiolo

Bisimulation Congruences in the Calculus of Looping Sequences Roberto Barbuti Andrea Maggiolo

Bisimulation Congruences in the Calculus of Looping Sequences Roberto Barbuti Andrea Maggiolo Schettini Paolo Milazzo Angelo Troina Dipartimento di Informatica, Universit` a di Pisa, Italy Tunis ICTAC 2006 1/22 Introduction Formal

242 views • 22 slides
YangBaxter equation and a congruence of biracks Pemysl Jedlika with Agata Pilitowska and

YangBaxter equation and a congruence of biracks Pemysl Jedlika with Agata Pilitowska and

YangBaxter equation and a congruence of biracks 1 / 12 YangBaxter equation and a congruence of biracks Pemysl Jedlika with Agata Pilitowska and Anna Zamojska-Dzienio Department of Mathematics Faculty of Engineering (former Technical

542 views • 26 slides
Congruence permutable Fregean varieties Katarzyna Somczyska Pedagogical University in Krakw

Congruence permutable Fregean varieties Katarzyna Somczyska Pedagogical University in Krakw

Outline Introduction The structure of finite algebras Free algebras Congruence permutable Fregean varieties Katarzyna Somczyska Pedagogical University in Krakw Katarzyna Somczyska Congruence permutable Fregean varieties Outline

516 views • 47 slides
Poetic Figures 1 THE OMISSION OF CERTAIN WORDS Ellipsis: leaving out omission of

Poetic Figures 1 THE OMISSION OF CERTAIN WORDS Ellipsis: leaving out omission of

Poetic Figures 1 THE OMISSION OF CERTAIN WORDS Ellipsis: leaving out omission of words necessary in grammar e.g., happy the students who study hard; happy their teacher, too it accelerates the passage Poetic Figures 1

251 views • 13 slides
Hoare Logic and Model Checking In the previous lecture we saw the informal concepts that

Hoare Logic and Model Checking In the previous lecture we saw the informal concepts that

Introduction Hoare Logic and Model Checking In the previous lecture we saw the informal concepts that Separation Logic is based on. This lecture will Kasper Svendsen University of Cambridge introduce a formal proof system for Separation

740 views • 10 slides
Structural Ambiguity How different parse trees may be produced from the same sentence or phrase

Structural Ambiguity How different parse trees may be produced from the same sentence or phrase

Constituency Parsing Structural Ambiguity How different parse trees may be produced from the same sentence or phrase Attachment ambiguity: where a constituent may attach to the rest of the tree I saw a man with a telescope

183 views • 17 slides
Revision: Negation of propositional formulae Conjunctive and disjunctive normal forms of

Revision: Negation of propositional formulae Conjunctive and disjunctive normal forms of

Revision: Negation of propositional formulae Conjunctive and disjunctive normal forms of propositional formulae Valentin Goranko DTU Informatics September 2010 V Goranko Important equivalences for negations of propositional formulae

180 views • 7 slides

More recommend