SLIDE 1
Data set
Training set Used up in training
wixi y x2 x1 x0 = −1 xn y′ w0 w
1
Testing and Error Estimation Machine Learning Prof Hans Georg - - PowerPoint PPT Presentation
Testing and Error Estimation Machine Learning Prof Hans Georg - - PowerPoint PPT Presentation
Testing and Error Estimation Machine Learning Prof Hans Georg Schaathun Hgskolen i lesund 5th February 2016 1 Data set Training set Used up in training Test set x 0 = 1 w 0 Find error probability w x 1 1 w 2 w i x i y x 2
SLIDE 2
SLIDE 3
Testing the Neuron
f(x) sign y x2 x1 xn −1 y′ Features Class label Input
3
SLIDE 4
Testing the Neuron
f(x) sign y x2 x1 xn −1 y′ Features Class label Input t
=?
3
SLIDE 5
Testing the Neuron
f(x) sign y x2 x1 xn −1 y′ Features Class label Input t
=?
Error Event Yes Correct Event No
3
SLIDE 6
Testing the Neuron
f(x) sign y x2 x1 xn −1 y′ Features Class label Input t
=?
Error Event Yes Correct Event No
Run for each input vector, and count events.
3
SLIDE 7
Error Rate
- 1. error rate:
e e+c
- 2. e error events
- 3. c correct classifications
What is the error probability?
4
SLIDE 8
Error Rate
- 1. error rate:
e e+c
- 2. e error events
- 3. c correct classifications
What is the error probability? — Error probability pe — Estimator: ˆ pe (stochastic variable) — Estimate: observation re (error rate) — Confidence interval: (ˆ pl, ˆ pu)
4
SLIDE 9
Confusion Matrix
Prediction Malign Benign Actual Class Malign True Positive False Negative Benign False Positive True Negative
5
SLIDE 10
The ROC curve
— We can vary w0 — Receiver operating characteristic (ROC)
0.0 0.2 0.4 0.6 0.8 1.0 False positive rate 0.0 0.2 0.4 0.6 0.8 1.0 True positive rate
6
SLIDE 11
The ROC curve
— We can vary w0 — Receiver operating characteristic (ROC)
0.0 0.2 0.4 0.6 0.8 1.0 False positive rate 0.0 0.2 0.4 0.6 0.8 1.0 True positive rate
6
SLIDE 12
The ROC curve
— We can vary w0 — Receiver operating characteristic (ROC)
0.0 0.2 0.4 0.6 0.8 1.0 False positive rate 0.0 0.2 0.4 0.6 0.8 1.0 True positive rate
6
SLIDE 13
The ROC curve
— We can vary w0 — Receiver operating characteristic (ROC)
0.0 0.2 0.4 0.6 0.8 1.0 False positive rate 0.0 0.2 0.4 0.6 0.8 1.0 True positive rate
6
SLIDE 14
The ROC curve
— We can vary w0 — Receiver operating characteristic (ROC)
0.0 0.2 0.4 0.6 0.8 1.0 False positive rate 0.0 0.2 0.4 0.6 0.8 1.0 True positive rate
6
SLIDE 15
Summary
Testing
- 1. Use a test set independent of the training set
- 2. Test set with known class labels
- 3. Do recall, and compare to known labels
Evaluation
- 1. Statistical analysis of test results
- 2. How large test set?