lecture 3 method evaluation and tuning parameter selection
play

Lecture 3: Method evaluation and tuning parameter selection Felix - PowerPoint PPT Presentation

Nov 18, 2023 β€’438 likes β€’716 views

Lecture 3: Method evaluation and tuning parameter selection Felix Held, Mathematical Sciences MSA220/MVE440 Statistical Learning for Big Data 29th March 2019 Evaluating performance of a statistical method Goals structure, e.g. in kNN

  1. Lecture 3: Method evaluation and tuning parameter selection Felix Held, Mathematical Sciences MSA220/MVE440 Statistical Learning for Big Data 29th March 2019

  2. Evaluating performance of a statistical method

  3. Goals structure, e.g. 𝑙 in kNN regression/classification, or β€œChoose between logistic regression, LDA and kNN” set? 1/25 β–Ά Model selection: Choose a hyper-parameter or model β–Ά Model assessment: How well did a model do on a data

  4. How to choose the best 𝑙 for kNN? 1 https://archive.ics.uci.edu/ml/datasets/Breast+Cancer+Wisconsin+(Diagnostic) 2/25 k = 1 k = 10 k = 100 0.30 ● ● ● ● ● ● ● ● ● Symmetry ● ● ● ● ● ● ● ● ● ● ● ● 0.25 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0.20 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0.15 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0.10 ● ● ● 0.0 0.1 0.2 0.3 0.0 0.1 0.2 0.3 0.0 0.1 0.2 0.3 Compactness Diagnosis Benign Malignant ● ● β–Ά UCI breast cancer wisconsin (diagnostic) data set 1 β–Ά Which 𝑙 will do best for class prediction of new data?

  5. Error rates (I) or classifier we looked at expected prediction loss Note that 𝑔 was thought to be an arbitrary unknown function. assumption 𝑔(β‹…|𝒰) is fixated after estimation but dependent on the training samples 𝒰 𝑔(𝐲|𝒰)] 3/25 β–Ά Remember: To determine the optimal regression function 𝐾(𝑔) = 𝔽 π‘ž(𝐲,𝑧) [𝑀(𝑧, 𝑔(𝐲))] β–Ά Now: 𝑔 is estimated from data under some model β–Ά The resulting regressor/classifier Λ† β–Ά Expected prediction error for a fixed training set 𝒰 𝑆(𝒰) = 𝔽 π‘ž(𝐲,𝑧) [𝑀(𝑧, Λ†

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

On Information-Maximization On Information-Maximization Clustering: Tuning Parameter Clustering:

On Information-Maximization On Information-Maximization Clustering: Tuning Parameter Clustering:

ICML2011 Jun. 28-Jul. 2, 2011 On Information-Maximization On Information-Maximization Clustering: Tuning Parameter Clustering: Tuning Parameter Selection and Analytic Solution Selection and Analytic Solution Masashi Sugiyama, Makoto Yamada,

773 views β€’ 28 slides
Variable selection and parameter tuning in high-dimensional prediction Christoph Bernau and

Variable selection and parameter tuning in high-dimensional prediction Christoph Bernau and

Setup Results Discussion Variable selection and parameter tuning in high-dimensional prediction Christoph Bernau and Anne-Laure Boulesteix Institut f ur Medizinische Informationsverarbeitung, Biometrie und Epidemiologie

186 views β€’ 14 slides
Tuning parameter selection for voxel-wise brain connectivity estimation via low dimensional

Tuning parameter selection for voxel-wise brain connectivity estimation via low dimensional

Introduction Covariance matrix Precision matrix Theoretical Results A feasible approach Tuning parameter selection for voxel-wise brain connectivity estimation via low dimensional submatrices Bin Nan Department of Biostatistics University

641 views β€’ 27 slides
10/16/19 Parameters and Parameter Tuning Genetic Algorithms History Taxonomy

10/16/19 Parameters and Parameter Tuning Genetic Algorithms History Taxonomy

10/16/19 Parameters and Parameter Tuning Genetic Algorithms History Taxonomy Parameter Tuning vs Parameter Control EA calibration Parameter Tuning Testing Effort Parameters and Recommendations Parameter

569 views β€’ 11 slides
Data Mining II Optimization & Parameter Tuning Heiko Paulheim Why Parameter Tuning?

Data Mining II Optimization & Parameter Tuning Heiko Paulheim Why Parameter Tuning?

Data Mining II Optimization & Parameter Tuning Heiko Paulheim Why Parameter Tuning? What we have seen so far many learning algorithms for classification, regression, ... Many of those have parameters k and distance

586 views β€’ 54 slides
Data Mining II Optimization & Parameter Tuning Heiko Paulheim Why Parameter Tuning?

Data Mining II Optimization & Parameter Tuning Heiko Paulheim Why Parameter Tuning?

Data Mining II Optimization & Parameter Tuning Heiko Paulheim Why Parameter Tuning? What we have seen so far many learning algorithms for classification, regression, ... Many of those have parameters k and distance

797 views β€’ 52 slides
Discussion Regarding Uneconomic Adjustment Policy & Parameter Tuning Market and Product

Discussion Regarding Uneconomic Adjustment Policy & Parameter Tuning Market and Product

Discussion Regarding Uneconomic Adjustment Policy & Parameter Tuning Market and Product Development Team Joint MSC/Stakeholder Meeting September 25, 2008 Uneconomic Adjustment Policy/Parameter Tuning Overview/Context Greg Cook Manager,

876 views β€’ 66 slides
Scalable Bandit Methods for Hyper-parameter Tuning Kirthevasan Kandasamy Carnegie Mellon

Scalable Bandit Methods for Hyper-parameter Tuning Kirthevasan Kandasamy Carnegie Mellon

Scalable Bandit Methods for Hyper-parameter Tuning Kirthevasan Kandasamy Carnegie Mellon University Guest Lecture - Scalable Machine Learning for Big Data Biology University of Pittsburgh, Pittsburgh, PA November 3, 2017 Hyper-parameter

1.42k views β€’ 127 slides
Linkages Between Parameter Tuning and Scarcity Pricing Frank A. Wolak Chair, Market Surveillance

Linkages Between Parameter Tuning and Scarcity Pricing Frank A. Wolak Chair, Market Surveillance

Linkages Between Parameter Tuning and Scarcity Pricing Frank A. Wolak Chair, Market Surveillance Committee (MSC) Joint Stakeholder/MSC Meeting July 30, 2008 Common Features of Scarcity Pricing and Parameter Tuning Scarcity pricing occurs

731 views β€’ 13 slides
Parameter Tuning. Automatic Algorithm Configuration Petr Po s k P. Po s k c

Parameter Tuning. Automatic Algorithm Configuration Petr Po s k P. Po s k c

CZECH TECHNICAL UNIVERSITY IN PRAGUE Faculty of Electrical Engineering Department of Cybernetics Parameter Tuning. Automatic Algorithm Configuration Petr Po s k P. Po s k c 2016 A0M33EOA: Evolutionary Optimization

747 views β€’ 32 slides
Parameter Tuning for Influence Maximization Manqing Ma Last Updated: 11/19/2018 (CSCI6250 FNS

Parameter Tuning for Influence Maximization Manqing Ma Last Updated: 11/19/2018 (CSCI6250 FNS

Parameter Tuning for Influence Maximization Manqing Ma Last Updated: 11/19/2018 (CSCI6250 FNS Presentation) Outline Objective: Param Tuning for BI /GPI ( ref: Karampourniotis, P. D., Szymanski, B. K., & Korniss, G. (2018). Influence

535 views β€’ 20 slides
A Short Introduction to Bayesian Optimization With applications to parameter tuning on

A Short Introduction to Bayesian Optimization With applications to parameter tuning on

A Short Introduction to Bayesian Optimization With applications to parameter tuning on accelerators Johannes Kirschner 28th February 2018 ICFA Workshop on Machine Learning for Accelerator Control Solve x = arg max f ( x ) x X 0

511 views β€’ 36 slides
Hyper-parameter tuning to improve existing software Alexander Brownlee, University of Stirling

Hyper-parameter tuning to improve existing software Alexander Brownlee, University of Stirling

Hyper-parameter tuning to improve existing software Alexander Brownlee, University of Stirling Collaborators 2 Outline The software What to improve? A systematic approach: Statistical analysis Single-objective tuning

501 views β€’ 24 slides
10/16/19 Parameter Control Genetic Algorithms Motivation Parameter setting Tuning

10/16/19 Parameter Control Genetic Algorithms Motivation Parameter setting Tuning

10/16/19 Parameter Control Genetic Algorithms Motivation Parameter setting Tuning Control Examples Where to apply parameter control How to apply parameter control Parameter Control Motivation Motivation An EA

659 views β€’ 7 slides
On Automated Parameter Tuning, with Applications in Next-Generation Manufacturing Lars Kotthofg

On Automated Parameter Tuning, with Applications in Next-Generation Manufacturing Lars Kotthofg

On Automated Parameter Tuning, with Applications in Next-Generation Manufacturing Lars Kotthofg and Patrick Johnson Artifjcially Intelligent Manufacturing Center University of Wyoming larsko,pjohns27@uwyo.edu UCC, 02 April 2019 1 Big

567 views β€’ 41 slides
Measurement-Based TCP Parameter Tuning in Cloud Data Centers Simon Jouet

Measurement-Based TCP Parameter Tuning in Cloud Data Centers Simon Jouet

Measurement-Based TCP Parameter Tuning in Cloud Data Centers Simon Jouet University of Glasgow [s.jouet.1@research.gla.ac.uk] Background TCP CongesHon Control For a transport endpoint embedded in a

546 views β€’ 9 slides
Deep Nets with http://vem.quantumunlimited.org/the-gates-of-horn/ Keras Professor

Deep Nets with http://vem.quantumunlimited.org/the-gates-of-horn/ Keras Professor

docs https://keras.io Deep Nets with http://vem.quantumunlimited.org/the-gates-of-horn/ Keras Professor Marie Roch These slides only cover enough to get started with feed-forward networks and do not cover regularization which is

134 views β€’ 10 slides
Applied Machine Learning Applied Machine Learning Regularization Siamak Ravanbakhsh Siamak

Applied Machine Learning Applied Machine Learning Regularization Siamak Ravanbakhsh Siamak

Applied Machine Learning Applied Machine Learning Regularization Siamak Ravanbakhsh Siamak Ravanbakhsh COMP 551 COMP 551 (winter 2020) (winter 2020) 1 Learning objectives Learning objectives Basic idea of overfitting and underfitting

404 views β€’ 39 slides
A major risk in classification: overfitting Assume we have a small data set We fit a model that

A major risk in classification: overfitting Assume we have a small data set We fit a model that

A major risk in classification: overfitting Assume we have a small data set We fit a model that separates red and blue red blue When more data becomes available, we see that the model is poor red blue A simpler model might have worked

211 views β€’ 17 slides
RECSM Summer School: Machine Learning for Social Sciences Session 1.3: Supervised Learning and

RECSM Summer School: Machine Learning for Social Sciences Session 1.3: Supervised Learning and

RECSM Summer School: Machine Learning for Social Sciences Session 1.3: Supervised Learning and Model Accuracy Reto West Department of Political Science and International Relations University of Geneva 1 Supervised Learning Supervised

436 views β€’ 42 slides
#8: Flux and Gauss Law Flux (in physics) refers to the product of a field crossing an area

#8: Flux and Gauss Law Flux (in physics) refers to the product of a field crossing an area

23.1-23.3 #8: Flux and Gauss Law Flux (in physics) refers to the product of a field crossing an area Consider air flowing through a window (volume flux) = vA cos If we define an area vector, where the magnitude is equal to the area of

392 views β€’ 9 slides
A bit of STC history 1969/1970 1969/1970 1999/2000 M P K 1999/2000, M-P Kwan 2002/2003 MJ K

A bit of STC history 1969/1970 1969/1970 1999/2000 M P K 1999/2000, M-P Kwan 2002/2003 MJ K

Space-Time Cube in Visual Analytics Gennady Andrienko Natalia Andrienko Natalia Andrienko http://geoanalytics.net/and in cooperation with P.Gatalsky, G.Fuchs, K.Vrotsou, I.Peca, C.Tominski, H.Schumann inspired by T.Hagerstrand, M-J Kraak , M-P

534 views β€’ 35 slides
A small chunk of material (the tan cube) is placed above a solenoid. It magnetizes, weakly,

A small chunk of material (the tan cube) is placed above a solenoid. It magnetizes, weakly,

A small chunk of material (the tan cube) is placed above a solenoid. It magnetizes, weakly, as shown by small arrows inside. What kind of material must the cube be? A. Dielectric B. Conductor C. Diamagnetic D. Paramagnetic E.

287 views β€’ 7 slides
Renormalization of random quantum magnets Ferenc Igl oi (Budapest) in collaboration with acs ,

Renormalization of random quantum magnets Ferenc Igl oi (Budapest) in collaboration with acs ,

Renormalization of random quantum magnets Ferenc Igl oi (Budapest) in collaboration with acs , R Istv an Kov obert Juh asz Cargese, August 25- September 6, 2014 1 Introduction Quantum phase transitions quantum spin

796 views β€’ 28 slides

More recommend