Department of Computer Science CSCI 5622: Machine Learning Chenhao - PowerPoint PPT Presentation

Department of Computer Science CSCI 5622: Machine Learning Chenhao Tan Lecture 14: PAC learnability Slides adapted from Jordan Boyd-Graber, Chris Ketelsen 1

Announcements • Proposal due tomorrow night • HW2 regrade requests • Peer feedback assignment • Midterm example • HW4 due in a month 2

Learning objectives • Learn about basics of learning theory • Prove some simple bounds on errors and sample sizes • Gain some intuition about complexity and overfitting 3

Simple Example 4

Simple Example 5

Simple Example 6

PAC Learnability 7

PAC Learnability 8

PAC Learnability 9

PAC Learnability 10

PAC Learnability 11

PAC Learnability 12

PAC Learnability 13

Alien Example 14

Alien Example 15

Alien Example 16

Alien Example 17

Alien Example 18

Alien Example 19

Alien Example 20

Alien Example 21

Alien Example 22

Alien Example 23

Alien Example 24

Alien Example 25

Alien Example 26

Alien Example 27

Alien Example 28

General case, Finite Hypothesis class • OK, so we saw an example proving PAC learnability for a specific problem with specific hypothesis and specific algorithm • Can we be more general than this? • Yes! • Today, H is finite • Next time, H is infinite • Distinction • H is finite and c is in H • H is finite and c is not in H 29

Finite Consistent Hypothesis Class 30

Finite Consistent Hypothesis Class 31

Finite Consistent Hypothesis Class 32

Finite Consistent Hypothesis Class 33

Finite Consistent Hypothesis Class 34

Finite Consistent Hypothesis Class 35

Finite Consistent Hypothesis Class 36

Finite Consistent Hypothesis Class 37

Finite Consistent Hypothesis Class 38

Finite Consistent Hypothesis Class 39

Finite Consistent Hypothesis Class 40

Finite Consistent Hypothesis Class 41

Finite Inconsistent Hypothesis Class 42

Finite Inconsistent Hypothesis Class 43

Finite Inconsistent Hypothesis Class 44

Finite Inconsistent Hypothesis Class 45

Finite Inconsistent Hypothesis Class 46

Finite Inconsistent Hypothesis Class 47

Finite Inconsistent Hypothesis Class 48

Finite Inconsistent Hypothesis Class 49

Finite Inconsistent Hypothesis Class 50

Finite Inconsistent Hypothesis Class 51

Finite Inconsistent Hypothesis Class 52

PAC Learnability • OK, that was a lot of Math! • I expect you to • Know the bounds we’ve proved • Know which bounds apply to which situations • Know how to apply a particular bound to a problem • I do not expect you to • Know the details of the proofs • Be able to prove PAC bounds yourself 53