measuring model performance or error
play

Measuring model performance or error Introduction to Machine - PowerPoint PPT Presentation

Mar 08, 2024 •530 likes •1.31k views

INTRODUCTION TO MACHINE LEARNING Measuring model performance or error Introduction to Machine Learning Is our model any good? Context of task Accuracy Computation time Interpretability 3 types of tasks

  1. INTRODUCTION TO MACHINE LEARNING Measuring model performance or error

  2. Introduction to Machine Learning Is our model any good? ● Context of task ● Accuracy ● Computation time ● Interpretability ● 3 types of tasks ● Classification ● Regression ● Clustering

  3. Introduction to Machine Learning Classification ● Accuracy and Error ● System is right or wrong ● Accuracy goes up when Error goes down correctly classified instances Accuracy = total amount of classified instances Error = 1 - Accuracy

  4. Introduction to Machine Learning Example ● Squares with 2 features: small/big and solid/do � ed ● Label: colored/not colored ● Binary classification problem

  5. Introduction to Machine Learning Example Truth Predicted ✔ ✘ ✘ ✔ ✔ ✔ ✔ ✔ 3 = = 60% 5 ✔ ✔ ✔ ✘ ✘

  6. Introduction to Machine Learning Example Truth Predicted ✔ ✘ ✘ ✔ ✔ ✔ ✔ ✔ 3 = = 60% 5 ✔ ✔ ✔ ✘ ✘

  7. Introduction to Machine Learning Limits of accuracy ● Classifying very rare heart disease ● Classify all as negative (not sick) ● Predict 99 correct (not sick) and miss 1 ● Accuracy: 99% ● Bogus… you miss every positive case!

  8. Introduction to Machine Learning Confusion matrix ● Rows and columns contain all available labels ● Each cell contains frequency of instances that are classified in a certain way

  9. Introduction to Machine Learning Confusion matrix ● Binary classifier: positive or negative (1 or 0) Prediction P N p TP FN Truth n FP TN

  10. Introduction to Machine Learning Confusion matrix ● Binary classifier: positive or negative (1 or 0) Prediction True Positives P N Prediction: P Truth: P p TP FN Truth n FP TN

  11. Introduction to Machine Learning Confusion matrix ● Binary classifier: positive or negative (1 or 0) Prediction True Negatives P N Prediction: N Truth: N p TP FN Truth n FP TN

  12. Introduction to Machine Learning Confusion matrix ● Binary classifier: positive or negative (1 or 0) Prediction False Negatives P N Prediction: N Truth: P p TP FN Truth n FP TN

  13. Introduction to Machine Learning Confusion matrix ● Binary classifier: positive or negative (1 or 0) Prediction False Positives P N Prediction: P Truth: N p TP FN Truth n FP TN

  14. Introduction to Machine Learning Ratios in the confusion matrix ● Accuracy ● Precision ● Recall Prediction P N p TP FN Truth n FP TN

  15. Introduction to Machine Learning Ratios in the confusion matrix ● Accuracy ● Precision ● Recall Prediction Precision P N TP/(TP+FP) p TP FN Truth n FP TN

  16. Introduction to Machine Learning Ratios in the confusion matrix ● Accuracy ● Precision ● Recall Prediction Precision P N TP/(TP+FP) p TP FN Truth n FP TN

  17. Introduction to Machine Learning Ratios in the confusion matrix ● Accuracy ● Precision ● Recall Prediction Recall P N TP/(TP+FN) p TP FN Truth n FP TN

  18. Introduction to Machine Learning Ratios in the confusion matrix ● Accuracy ● Precision ● Recall Prediction Recall P N TP/(TP+FN) p TP FN Truth n FP TN

  19. Introduction to Machine Learning Back to the squares Prediction P N Truth p 1 1 Truth Predicted n 1 2

  20. Introduction to Machine Learning Back to the squares Prediction P N Truth p 1 1 Truth Predicted n 1 2 ✔

  21. Introduction to Machine Learning Back to the squares Prediction P N Truth p 1 1 Truth Predicted n 1 2 ✔ ✔

  22. Introduction to Machine Learning Back to the squares Prediction P N Truth p 1 1 Truth Predicted n 1 2 ✘

  23. Introduction to Machine Learning Back to the squares Prediction P N Truth p 1 1 Truth Predicted n 1 2 ✘

  24. Introduction to Machine Learning Back to the squares ● Accuracy: (TP+TN)/(TP+FP+FN+TN) = (1+2)/(1+2+1+1) = 60% ● Precision: TP/(TP+FP) = 1/(1+1) = 50% ● Recall: TP/(TP+FN) = 1/(1+1) = 50% Prediction P N Truth p 1 1 Truth Predicted n 1 2 ✔ ✔ ✔

  25. Introduction to Machine Learning Back to the squares ● Accuracy: (TP+TN)/(TP+FP+FN+TN) = (1+2)/(1+2+1+1) = 60% ● Precision: TP/(TP+FP) = 1/(1+1) = 50% ● Recall: TP/(TP+FN) = 1/(1+1) = 50% Prediction P N Truth p 1 1 Truth Predicted n 1 2 ✔ ✘ ✘ ✔ ✔

  26. Introduction to Machine Learning Back to the squares ● Accuracy: (TP+TN)/(TP+FP+FN+TN) = (1+2)/(1+2+1+1) = 60% ● Precision: TP/(TP+FP) = 1/(1+1) = 50% ● Recall: TP/(TP+FN) = 1/(1+1) = 50% Prediction P N Truth p 1 1 Truth Predicted n 1 2 ✘ ✔

  27. Introduction to Machine Learning Back to the squares ● Accuracy: (TP+TN)/(TP+FP+FN+TN) = (1+2)/(1+2+1+1) = 60% ● Precision: TP/(TP+FP) = 1/(1+1) = 50% ● Recall: TP/(TP+FN) = 1/(1+1) = 50% Prediction P N Truth p 1 1 Truth Predicted n 1 2 ✘ ✔

  28. Introduction to Machine Learning Rare heart disease ● Accuracy: 99/(99+1) = 99% ● Recall: 0/1 = 0% ● Precision: undefined — no positive predictions Prediction P N p 0 1 Truth n 0 99

  29. Introduction to Machine Learning Regression: RMSE ● Root Mean Squared Error (RMSE) ● Mean distance between estimates and regression line ● 12 ● ● ● ● ● ● ● 11 ● ● ● ● ● ● ● ● ● 10 ● ● X2 ● ● ● 9 ● ● ● ● ● ● 8 ● ● ● ● ● ● ● ● ● 7 ● ● ● 6 6 7 8 9 10 11 12 X1

  30. Introduction to Machine Learning Clustering ● No label information ● Need distance metric between points

  31. Introduction to Machine Learning Clustering ● Performance measure consists of 2 elements ● Similarity within each cluster ● Similarity between clusters 


  32. Introduction to Machine Learning Within cluster similarity ● Within sum of squares (WSS) ● Diameter 10 ● ● ● ● Minimize ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 5 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● X2 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● − 5 − 5 0 5 10 X1

  33. Introduction to Machine Learning Between cluster similarity ● Between cluster sum of squares (BSS) ● Intercluster distance 10 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 5 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Maximize ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● X2 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● − 5 − 5 0 5 10 X1

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Introduction Questions: Assessment of the expected error of a learning algorithm: Is the error

Introduction Questions: Assessment of the expected error of a learning algorithm: Is the error

Introduction Training Measuring Classifier Performance Comparing Classifiers Introduction Training Measuring Classifier Performance Comparing Classifiers Introduction Questions: Assessment of the expected error of a learning algorithm: Is

291 views • 5 slides
Synchronization in duet performance: Testing the two-person phase error correction model Dirk

Synchronization in duet performance: Testing the two-person phase error correction model Dirk

Synchronization in duet performance: Testing the two-person phase error correction model Dirk Vorberg Institut fr Psychologie Technische Universitt Braunschweig Braunschweig, Germany RPPW2005, Alden Biesen Overview 1. How do ensemble

574 views • 29 slides
Kentucky State Model roup June 20, 2007 Users G The List Model Error Report

Kentucky State Model roup June 20, 2007 Users G The List Model Error Report

Possible User Interface Add-Ons Kentucky State Model roup June 20, 2007 Users G The List Model Error Report System Performance Report Mapping Macros Accessibility Index Report Air Quality Post-Processor Select /

261 views • 14 slides
Key Performance I ndicators in Measuring PAI s & PLO Performance For Electrical Engineering

Key Performance I ndicators in Measuring PAI s & PLO Performance For Electrical Engineering

Key Performance I ndicators in Measuring PAI s & PLO Performance For Electrical Engineering Department Curriculum of Malaysian Polytechnics (The he role o of KPIs Is to assess a and improve the he Ou Outcomes-based E Educati tion)

741 views • 47 slides
Overview ROI: Framework ROI: Logic Model ROI: Model ROI: Framework A performance

Overview ROI: Framework ROI: Logic Model ROI: Model ROI: Framework A performance

Measuring Return On Investment (ROI) for State Government Overview ROI: Framework ROI: Logic Model ROI: Model ROI: Framework A performance measure used to evaluate the efficiency of an investment ROI = Impact from program Impact

180 views • 4 slides
Measurement of Timing Error Detection Performance of Software-based Error Detection Mechanisms

Measurement of Timing Error Detection Performance of Software-based Error Detection Mechanisms

Measurement of Timing Error Detection Performance of Software-based Error Detection Mechanisms and Its Correlation with Simulation Yutaka Masuda, Masanori Hashimoto, Takao Onoye Dept. of Information Systems Eng., Osaka University

376 views • 18 slides
Does Certificate Transparency Break the Web? Measuring Adoption and Error Rate Emily Stark , Ryan

Does Certificate Transparency Break the Web? Measuring Adoption and Error Rate Emily Stark , Ryan

Does Certificate Transparency Break the Web? Measuring Adoption and Error Rate Emily Stark , Ryan Sleevi, Rijad Muminovic, Devon OBrien, Eran Messeri, Adrienne Porter Felt, Brendan McMillion, Parisa Tabriz estark@chromium.org How

484 views • 31 slides
Measuring and Evaluating Computer System Performance CSE 141, S2'06 Jeff Brown Performance

Measuring and Evaluating Computer System Performance CSE 141, S2'06 Jeff Brown Performance

Measuring and Evaluating Computer System Performance CSE 141, S2'06 Jeff Brown Performance Marches On ... But what is performance ? CSE 141, S2'06 Jeff Brown The bottom line: Performance Time to Throughput Car Speed Passengers Bay

274 views • 23 slides
Overfi'ng and Model selec1on Aar$ Singh Machine Learning

Overfi'ng and Model selec1on Aar$ Singh Machine Learning

Overfi'ng and Model selec1on Aar$ Singh Machine Learning 10-701/15-781 Feb 20, 2014 True vs. Empirical Error True Error : Target performance

451 views • 32 slides
Measuring Performance: Chapter 4! Or My computer is faster than your computer with thanks to

Measuring Performance: Chapter 4! Or My computer is faster than your computer with thanks to

Measuring Performance: Chapter 4! Or My computer is faster than your computer with thanks to Larry Carter, UCSD 1 Performance Marches On ... 1200 DEC Alpha 21264/600 1100 1000 900 800 700 Performance 600 500 DEC Alpha 5/500 400

728 views • 29 slides
k -Step Ahead Prediction Error Model 1. k -Step Ahead Prediction Error Model 1. ARMAX model is

k -Step Ahead Prediction Error Model 1. k -Step Ahead Prediction Error Model 1. ARMAX model is

k -Step Ahead Prediction Error Model 1. k -Step Ahead Prediction Error Model 1. ARMAX model is ARMA plus eXogeneous signal model: A ( z ) y ( n ) = B ( z ) u ( n k ) + C ( z ) ( n ) k -Step Ahead Prediction Error Model 1. ARMAX model is

1.52k views • 127 slides
How to Get Good Performance by Using OpenMP ! ! Loop optimizations ! ! Measuring OpenMP

How to Get Good Performance by Using OpenMP ! ! Loop optimizations ! ! Measuring OpenMP

Agenda ! How to Get Good Performance by Using OpenMP ! ! Loop optimizations ! ! Measuring OpenMP performance ! ! Best practices ! ! Task parallelism !! !" #" Memory access patterns ! Correctness versus performance ! It

166 views • 14 slides
Error model Independent errors - error probability per bit p Error probability per word

Error model Independent errors - error probability per bit p Error probability per word

379 views • 8 slides
k -Step Ahead Prediction Error Model 1. ARMAX model is ARMA plus eXogeneous signal model: A ( z )

k -Step Ahead Prediction Error Model 1. ARMAX model is ARMA plus eXogeneous signal model: A ( z )

k -Step Ahead Prediction Error Model 1. ARMAX model is ARMA plus eXogeneous signal model: A ( z ) y ( n ) = B ( z ) u ( n k ) + C ( z ) ( n ) u - input y - output - white noise k - delay A , B , C are polynomials in z 1 All

280 views • 16 slides
Performance measures Nicolas Papageorgiou AQF-2005 Measuring hedge fund performance What

Performance measures Nicolas Papageorgiou AQF-2005 Measuring hedge fund performance What

Performance measures Nicolas Papageorgiou AQF-2005 Measuring hedge fund performance What is the best method? Risk adjusted performance measures Multi-factor models Peer group factors Return based factor Asset based

416 views • 24 slides
Measuring Performance November 17, 2008 Measuring Performance Introduction CPU Peformance and

Measuring Performance November 17, 2008 Measuring Performance Introduction CPU Peformance and

Introduction CPU Peformance and Its Factors Evaluating Performance Measuring Performance November 17, 2008 Measuring Performance Introduction CPU Peformance and Its Factors Evaluating Performance Outline 1 Introduction 2 CPU Peformance and

988 views • 65 slides
Writing and Identifying Statements and Exclamations Teaching Input Types of Sentences There are

Writing and Identifying Statements and Exclamations Teaching Input Types of Sentences There are

Writing and Identifying Statements and Exclamations Teaching Input Types of Sentences There are four basic types of sentence in the English language. They are Commands Questions Let me drive your car. How much does it cost?

300 views • 18 slides
Malware Datasets Aleieldin Salem and Alexander Pretschner Technische Universitt Mnchen

Malware Datasets Aleieldin Salem and Alexander Pretschner Technische Universitt Mnchen

Poking the Bear: Lessons Learned from Probing Three Android Malware Datasets Aleieldin Salem and Alexander Pretschner Technische Universitt Mnchen Garching bei Mnchen {salem, pretschn @in.tum.de} Montpellier, 04.09.2018 Abstract

590 views • 34 slides
The 4 th Competition on Syntax-Guided Synthesis Rajeev Alur, Dana Fisman, Rishabh Singh and

The 4 th Competition on Syntax-Guided Synthesis Rajeev Alur, Dana Fisman, Rishabh Singh and

The 4 th Competition on Syntax-Guided Synthesis Rajeev Alur, Dana Fisman, Rishabh Singh and Armando Solar-Lezama SyGuS Idea and Definition in a Nutshell New Trends in Synthesis Syntactic restrictions Specification S R on the High Level

562 views • 25 slides
Why Im NOT Why Im NOT Why Im NOT a Jew... Or Am I? Why Im NOT a Jew... Or Am

Why Im NOT Why Im NOT Why Im NOT a Jew... Or Am I? Why Im NOT a Jew... Or Am

Why Im NOT Why Im NOT Why Im NOT a Jew... Or Am I? Why Im NOT a Jew... Or Am I? The importance of definitions What is a Jew? A citizen of Israel? What is a Jew? A citizen A cultural of Israel? Jew? What

415 views • 28 slides
Around the World Unit 8 Lesson 11 What will we be learning this class? A look into a new

Around the World Unit 8 Lesson 11 What will we be learning this class? A look into a new

Around the World Unit 8 Lesson 11 What will we be learning this class? A look into a new country A review of punctuation Today we will be looking into another country. Can you guess which country it is? Here are some hints: This

479 views • 24 slides
cse 311: foundations of computing Spring 2015 Lecture 10: Functions, Modular arithmetic

cse 311: foundations of computing Spring 2015 Lecture 10: Functions, Modular arithmetic

cse 311: foundations of computing Spring 2015 Lecture 10: Functions, Modular arithmetic functions A funct unctio ion from to . Every element of is assigned to exactly one element of . We write

399 views • 19 slides
User Interface - Hall of Shame SWEN-610 Foundations of Software Engineering Department of

User Interface - Hall of Shame SWEN-610 Foundations of Software Engineering Department of

User Interface - Hall of Shame SWEN-610 Foundations of Software Engineering Department of Software Engineering Rochester Institute of Technology Contempt & Loathing Dont Insult Users Dont Shout at Users !!!! There is no need for

875 views • 44 slides
Methodology, and Tools to Write Bug-Free Programs Presented by Paolo Antonucci 04/05/13 About

Methodology, and Tools to Write Bug-Free Programs Presented by Paolo Antonucci 04/05/13 About

K. Rustan M. Leino, Peter Mller Using the Spec# Language, Methodology, and Tools to Write Bug-Free Programs Presented by Paolo Antonucci 04/05/13 About this paper This paper is more similar to a practical tutorial than a research paper.

628 views • 19 slides

More recommend