SLIDE 1
For Monday
- Read FOIL paper
- No homework
Program 2
- Questions?
For Monday Read FOIL paper No homework Program 2 Questions? Rule - - PowerPoint PPT Presentation
For Monday Read FOIL paper No homework Program 2 Questions? Rule - - PowerPoint PPT Presentation
For Monday Read FOIL paper No homework Program 2 Questions? Rule Learning Why learn rules? Proposition Rule Learning Basic if-then rules Condition is typically a conjunction of attribute tests Basic Approaches
SLIDE 2
SLIDE 3
Rule Learning
- Why learn rules?
SLIDE 4
Proposition Rule Learning
- Basic if-then rules
- Condition is typically a conjunction of
attribute tests
SLIDE 5
Basic Approaches
- Decision tree -> rules
- Neural network -> rules (TREPAN)
- Sequential covering algorithms
– Top-down – Bottom-up – Hybrid
SLIDE 6
Decision Tree Rules
- Resulting rules may contain unnecessary antecedents,
resulting in over-fitting.
- Rules are post-pruned.
- Resulting rules may lead to conflicting conclusions on
some instances.
- Sort rules by training (validation) accuracy to create
an ordered decision list.
- The first rule that applies is used to classify a test
instance.
red circle → A (97% train accuracy) red big → B (95% train accuracy) : : Test case: <big, red, circle> assigned to class A
SLIDE 7
Sequential Covering
SLIDE 8
Minimum Set Cover
SLIDE 9
9
Greedy Sequential Covering Example
X Y
+ + + + + + + + + + + + +
SLIDE 10
10
Greedy Sequential Covering Example
X Y
+ + + + + + + + + + + + +
SLIDE 11
11
Greedy Sequential Covering Example
X Y
+ + + + + +
SLIDE 12
12
Greedy Sequential Covering Example
X Y
+ + + + + +
SLIDE 13
13
Greedy Sequential Covering Example
X Y
+ + +
SLIDE 14
14
Greedy Sequential Covering Example
X Y
+ + +
SLIDE 15
15
Greedy Sequential Covering Example
X Y
SLIDE 16
16
No-optimal Covering Example
X Y
+ + + + + + + + + + + + +
SLIDE 17
17
Greedy Sequential Covering Example
X Y
+ + + + + + + + + + + + +
SLIDE 18
18
Greedy Sequential Covering Example
X Y
+ + + + + +
SLIDE 19
19
Greedy Sequential Covering Example
X Y
+ + + + + +
SLIDE 20
20
Greedy Sequential Covering Example
X Y
+ +
SLIDE 21
21
Greedy Sequential Covering Example
X Y
+ +
SLIDE 22
22
Greedy Sequential Covering Example
X Y
+
SLIDE 23
23
Greedy Sequential Covering Example
X Y
+
SLIDE 24
24
Greedy Sequential Covering Example
X Y
SLIDE 25
Learning a Rule
- Two basic approaches:
– Top-down – Bottom-up
SLIDE 26
26
Top-Down Rule Learning Example
X Y
+ + + + + + + + + + + + +
SLIDE 27
27
Top-Down Rule Learning Example
X Y
+ + + + + + + + + + + + +
Y>C1
SLIDE 28
28
Top-Down Rule Learning Example
X Y
+ + + + + + + + + + + + +
Y>C1 X>C2
SLIDE 29
29
Top-Down Rule Learning Example
X Y
+ + + + + + + + + + + + +
Y>C1 X>C2 Y<C3
SLIDE 30
30
Top-Down Rule Learning Example
X Y
+ + + + + + + + + + + + +
Y>C1 X>C2 Y<C3 X<C4
SLIDE 31
31
Bottom-Up Rule Learning Example
X Y
+
+ + + + + + + + + + + +
SLIDE 32
32
Bottom-Up Rule Learning Example
X Y
+
+ + + +
+
+ + + + + + +
SLIDE 33
33
Bottom-Up Rule Learning Example
X Y
+
+ + + +
+
+ + + + + + +
SLIDE 34
34
Bottom-Up Rule Learning Example
X Y
+
+ + + +
+
+ + +
+
+ + +
SLIDE 35
35
Bottom-Up Rule Learning Example
X Y
+
+ + + +
+
+ + +
+
+ + +
SLIDE 36
36
Bottom-Up Rule Learning Example
X Y
+
+ + + +
+
+ + + + + + +
SLIDE 37
37
Bottom-Up Rule Learning Example
X Y
+
+ + + +
+
+
+ +
+ + + +
SLIDE 38
38
Bottom-Up Rule Learning Example
X Y
+
+ + + +
+
+
+ +
+ + + +
SLIDE 39
39
Bottom-Up Rule Learning Example
X Y
+
+ + + +
+
+ + + + + + +
SLIDE 40
40
Bottom-Up Rule Learning Example
X Y
+
+ + + +
+
+ + + + + +
+
SLIDE 41
41
Bottom-Up Rule Learning Example
X Y
+
+ + + +
+
+ + + + + +
+
SLIDE 42
Algorithm Specifics
- Metrics
– How do we pick literals to add to our rules?
- Handling continuous features
- Pruning
SLIDE 43
Rules vs. Trees
SLIDE 44
Top-down vs Bottom-up
SLIDE 45
45
Rule Learning vs. Knowledge Engineering
- An influential experiment with an early rule-learning
method (AQ) by Michalski (1980) compared results to knowledge engineering (acquiring rules by interviewing experts).
- People known for not being able to articulate their
knowledge well.
- Knowledge engineered rules:
– Weights associated with each feature in a rule – Method for summing evidence similar to certainty factors. – No explicit disjunction
- Data for induction:
– Examples of 15 soybean plant diseases descried using 35 nominal and discrete ordered features, 630 total examples. – 290 “best” (diverse) training examples selected for training. Remainder used for testing
- What is wrong with this methodology?
SLIDE 46
46
“Soft” Interpretation of Learned Rules
- Certainty of match calculated for each category.
- Scoring method:
– Literals: 1 if match, -1 if not – Terms (conjunctions in antecedent): Average of literal scores. – DNF (disjunction of rules): Probabilistic sum: c1 + c2 – c1*c2
- Sample score for instance A B ¬C D ¬ E F
A B C → P (1 + 1 + -1)/3 = 0.333 D E F → P (1 + -1 + 1)/3 = 0.333 Total score for P: 0.333 + 0.333 – 0.333* 0.333 = 0.555
- Threshold of 0.8 certainty to include in possible diagnosis set.
SLIDE 47
47
Experimental Results
- Rule construction time:
– Human: 45 hours of expert consultation – AQ11: 4.5 minutes training on IBM 360/75
- What doesn’t this account for?
- Test Accuracy: