LEARNING Outline Confusion Matrix F1 Score Gain and Lift Charts - PowerPoint PPT Presentation

Feb 20, 2024 •238 likes •357 views

Measuring Performance CSCI 447/547 MACHINE LEARNING Outline Confusion Matrix F1 Score Gain and Lift Charts Kolmogorov Smirnov Chart ROC / AUC Regression Metrics Kappa Statistic Confusion Matrix Confusion Matrix

Measuring Performance CSCI 447/547 MACHINE LEARNING
Outline  Confusion Matrix  F1 Score  Gain and Lift Charts  Kolmogorov Smirnov Chart  ROC / AUC  Regression Metrics  Kappa Statistic
Confusion Matrix Confusion Matrix Actual Positive Negative Predict Positive a b Precision a/(a+b) Negative Predictive Negative c d d/(d+c) Value Sensitivity / Recall Specificity Accuracy = (a+d)/(a+b+c+d) a/(a+c) d/(d+b) Confusion Matrix Actual 1 0 Predict 1 3,384 639 Precision 85.7% Negative Predictive 0 16 951 98.3% Value Sensitivity / Recall Specificity Accuracy = 88% 99.6% 59.8%
F1 Score  Good F1 score means you have low false positives and low false negatives  F1 = 2*(Precision * Recall)/(Precision + Recall)  Ranges from 0 to 1  Higher values are better
Gain and Lift Charts  Calculate probability for each observation  Sort in descending order  Split into 10 partitions (deciles)  Calculate correct predictions for each partition
Kolomogorov Smirnov Chart  Measure of the degree of separation between positive and negative distributions
ROC / AUC Curves  Advantage over lift charts is that ROC is (almost) independent of the (possibly fluctuating) accuracy rate  Measures model’s ability to discriminate between positive and negative classes
Regression Metrics  Mean Absolute Error  Gives an idea of magnitude of error but not direction  Mean Squared Error (MSE)  Root Mean Squared Error (RMSE)  Converts MSE back to original magnitude  R 2 (and Adjusted R 2 )  Indication of correlation of predictions to actual values  Range between 0 and 1 with higher being better
Logarithmic Loss (Logloss)  Evaluating the predictions of probabilities of membership in a given class  Smaller is better, 0 is perfect
Kappa Statistic  Cohen’s Kappa  How much better than chance a model is  Range is -1 to 1, with higher being better  Some advise against using this  Dependent on distribution of correct and incorrect predictions, Cohen’s Kappa can be misleading  Power’s Kappa ( Informedness)  Likelihood of making an informed decision over a random guess  Recall + TNR – 1  Range is -1 to 1  1: model is always correct; 0: model is random; -1: model is always incorrect
Summary  Confusion Matrix  F1 Score  Gain and Lift Charts  Kolmogorov Smirnov Chart  ROC / AUC  Regression Metrics  Kappa Statistic

Download Presentation

Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Federated Learning Min Du Postdoc, UC Berkeley Outline q Preliminary: deep learning and SGD q

Federated Learning Min Du Postdoc, UC Berkeley Outline q Preliminary: deep learning and SGD q Federated learning: FedSGD and FedAvg q Related research in federated learning q Open problems Outline q Preliminary: deep learning and SGD q Federated

616 views • 50 slides

Hao Su July 6, 2017 Outline Overview of 3D deep learning 3D deep learning algorithms

Deep 3D Representation Learning for Visual Computing Hao Su July 6, 2017 Outline Overview of 3D deep learning 3D deep learning algorithms Conclusion 2 Outline Overview of 3D deep learning Background 3D deep learning tasks 3D deep

1.66k views • 122 slides

Intro to Online Learning Instructor: Haifeng Xu Outline Online Learning/Optimization

CS6501: T opics in Learning and Game Theory (Fall 2019) Intro to Online Learning Instructor: Haifeng Xu Outline Online Learning/Optimization Measure Algorithm Performance via Regret Warm-up: A Simple Example 2 Overview of Machine

629 views • 37 slides

Machine Learning Classifiers and Boosting Reading Ch 18.6-18.12, 20.1-20.3.2 Outline

Machine Learning Classifiers and Boosting Reading Ch 18.6-18.12, 20.1-20.3.2 Outline Different types of learning problems Different types of learning algorithms Supervised learning Decision trees Nave Bayes

658 views • 54 slides

Statistical learning Chapter 20, Sections 13 Chapter 20, Sections 13 1 Outline

Statistical learning Chapter 20, Sections 13 Chapter 20, Sections 13 1 Outline Bayesian learning Maximum a posteriori and maximum likelihood learning Bayes net learning ML parameter learning with complete data linear

399 views • 13 slides

Multi-Task Active Learning Yi Zhang Outline Active Learning Multi-Task Active Learning

Multi-Task Active Learning Yi Zhang Outline Active Learning Multi-Task Active Learning Linguistic Annotations (ACL 08) Image Classification (CVPR 08) Current Work and Discussions Constraint-Driven Active Learning

945 views • 47 slides

At the Confluence of Logic and Learning Guy Van den Broeck Dagstuhl September 3, 2019 Outline

At the Confluence of Logic and Learning Guy Van den Broeck Dagstuhl September 3, 2019 Outline 1. The AI dilemma: logic vs. learning 2. Deep learning with symbolic knowledge 3. Efficient reasoning during learning 4. New machine learning

1.43k views • 78 slides

Machine Learning CS 486/686: Introduction to Artificial Intelligence 1 Outline Forms of

Machine Learning CS 486/686: Introduction to Artificial Intelligence 1 Outline Forms of Learning Inductive Learning - Decision Trees 2 What is Machine Learning Definition: - A computer program is said to learn from experience E with

754 views • 40 slides

10. Learning to Rank Outline 10.1. Why Learning to Rank (LeToR)? 10.2. Pointwise, Pairwise,

10. Learning to Rank Outline 10.1. Why Learning to Rank (LeToR)? 10.2. Pointwise, Pairwise, Listwise 10.3. Gathering User Input 10.4. LeToR Evaluation 10.5. Beyond Search Advanced Topics in Information Retrieval / Learning to Rank 2 10.1.

1.01k views • 34 slides

Learning from Observations Chapter 18, Sections 13 Chapter 18, Sections 13 1 Outline

Learning from Observations Chapter 18, Sections 13 Chapter 18, Sections 13 1 Outline Learning agents Inductive learning Decision tree learning Measuring learning performance Chapter 18, Sections 13 2 Learning

653 views • 32 slides

Learning from Observations Chapter 18, Sections 13 Chapter 18, Sections 13 1 Outline

569 views • 30 slides

Reinforcement Learning Stephen D. Scott (Adapted from Tom Mitchells slides) 1 Outline

Reinforcement Learning Stephen D. Scott (Adapted from Tom Mitchells slides) 1 Outline Control learning Control policies that choose optimal actions Q learning Convergence Temporal difference learning 2 Control Learning

576 views • 20 slides

Introduction to Deep Learning Outline Deep Learning RNN CNN Attention

Introduction to Deep Learning Outline Deep Learning RNN CNN Attention Transformer Pytorch Introduction Basics Examples RNNs Some slides borrowed from Fei-Fei Li & Justin Johnson &

1.1k views • 55 slides

Outline Wh y Mac hine Learning? What is a w ell-dened learning problem?

Outline Wh y Mac hine Learning? What is a w ell-dened learning problem? An example: learning to pla y c hec k ers What questions should w e ask ab out Mac hine Learning? 1 lecture slides

625 views • 21 slides

Todays Outline Reinforcement Learning Dan Weld Reinforcement Learning Q-value

5/7/2012 CSE 473: Artificial Intelligence Todays Outline Reinforcement Learning Dan Weld Reinforcement Learning Q-value iteration Q-learning Exploration / exploitation Linear function approximation Linear function

159 views • 12 slides

Bayesian Learning 1 Outline MLE, MAP vs. Bayesian Learning Bayesian Linear Regression

Bayesian Learning 1 Outline MLE, MAP vs. Bayesian Learning Bayesian Linear Regression Bayesian Gaussian Mixture Models Non-parametric Bayes 2 Take Away ... 1. Maximum Likelihood Estimate (MLE) = arg max p ( D| )

609 views • 27 slides

CSE217 INTRODUCTION TO DATA SCIENCE LECTURE 6: LEARNING PRINCIPLES Spring 2019 Marion Neumann

CSE217 INTRODUCTION TO DATA SCIENCE LECTURE 6: LEARNING PRINCIPLES Spring 2019 Marion Neumann RECAP: MACHINE LEARNING Workflow 2 NOISE noisy samples from true function 3 WHY IS NOISE A PROBLEM? small random sample from the noisy

746 views • 19 slides

New Langevin based algorithms for MCMC in high dimensions Alain Durmus Joint work with Gareth O.

New Langevin based algorithms for MCMC in high dimensions Alain Durmus Joint work with Gareth O. Roberts, Gilles Vilmart and Konstantinos Zygalakis. Dpartement TSI, Telecom ParisTech Siximes rencontres des jeunes statisticiens Main themes

426 views • 41 slides

Joint stateparameter estimation for nonlinear stochastic energy balance models Fei Lu 1 Nils

Joint stateparameter estimation for nonlinear stochastic energy balance models Fei Lu 1 Nils Weitzel 2 Adam Monahan 3 1 Department of Mathematics, Johns Hopkins 2 Meteorological Institute, University of Bonn, Germany 3 School of Earth and Ocean

502 views • 20 slides

Differentially Private Markov Chain Monte Carlo o 2 , Onur Dikmen 3 and Antti Honkela 1 a 1

Differentially Private Markov Chain Monte Carlo o 2 , Onur Dikmen 3 and Antti Honkela 1 a 1 , Joonas J Mikko Heikkil alk Equal contribution 1 University of Helsinki 2 Aalto University 3 Halmstad University NeurIPS, 12 December

497 views • 11 slides

Linear Estimation Problem Formulation Basic ideas Goal for much of this class is to

Linear Estimation Problem Formulation Basic ideas Goal for much of this class is to estimate a random variable Types of inputs Usual assumptions Is a signal, y ( n ) C 1 1 Error criterion Much of what we discuss

513 views • 14 slides

Forecast verification 4th VALUE Training School Jonas Bhend, Sven Kotlarski Forecast verification

Federal Department of Home Affairs FDHA Federal Office of Meteorology and Climatology MeteoSwiss Forecast verification 4th VALUE Training School Jonas Bhend, Sven Kotlarski Forecast verification is the process of comparing forecasts with

379 views • 26 slides

Deep Learning-based Short Video Recommendation and Prefetching for Mobile Commuting Users Qian Li

RAPID708 Deep Learning-based Short Video Recommendation and Prefetching for Mobile Commuting Users Qian Li 1 , Yuan Zhang 1 , Hong Huang 2 , Jinyao Yan 1 1 Communication University of China 2 Huazhong University of Science and

187 views • 16 slides

IN5550: Neural Methods in Natural Language Processing Lecture 2 Supervised Machine Learning:

IN5550: Neural Methods in Natural Language Processing Lecture 2 Supervised Machine Learning: from Linear Models to Neural Networks Andrey Kutuzov, Vinit Ravishankar, Jeremy Barnes, Lilja vrelid, Stephan Oepen, & Erik Velldal University

1.95k views • 191 slides