Introduction Example: optimizing hyperparameters for an SVM classifier Challenges in hyperparameter search State-of-the-art References Hyperparameter Search in Machine Learning Marc Claesen and Bart De Moor marc.claesen@esat.kuleuven.be ESAT-STADIUS, KU Leuven iMinds Medical IT Department STADIUS Center for Dynamical Systems, Signal Processing and Data Analytics Marc Claesen and Bart De Moor Hyperparameter Search in Machine Learning
Introduction Example: optimizing hyperparameters for an SVM classifier Challenges in hyperparameter search State-of-the-art References Outline Introduction 1 Example: optimizing hyperparameters for an SVM classifier 2 Challenges in hyperparameter search 3 State-of-the-art 4 Marc Claesen and Bart De Moor Hyperparameter Search in Machine Learning
Introduction Example: optimizing hyperparameters for an SVM classifier Challenges in hyperparameter search State-of-the-art References Machine learning Methods capable of learning patterns of interest from data. by formulating the learning task as an optimization problem Marc Claesen and Bart De Moor Hyperparameter Search in Machine Learning
Introduction Example: optimizing hyperparameters for an SVM classifier Challenges in hyperparameter search State-of-the-art References Machine learning Methods capable of learning patterns of interest from data. by formulating the learning task as an optimization problem Machine learning is situated on the intersection of various fields: statistics, computer science, optimization, (biology), . . . Marc Claesen and Bart De Moor Hyperparameter Search in Machine Learning
Introduction Example: optimizing hyperparameters for an SVM classifier Challenges in hyperparameter search State-of-the-art References Machine learning Methods capable of learning patterns of interest from data. by formulating the learning task as an optimization problem Machine learning is situated on the intersection of various fields: statistics, computer science, optimization, (biology), . . . The field encompasses learning methods with various origins, e.g.: biology, e.g. neural networks [1] convex optimization, e.g. support vector machines [2] statistics, e.g. hidden Markov models [3] tensor decompositions, e.g. recommender systems [4] Marc Claesen and Bart De Moor Hyperparameter Search in Machine Learning
Introduction Example: optimizing hyperparameters for an SVM classifier Challenges in hyperparameter search State-of-the-art References Hyperparameter search Most machine learning methods are (hyper)parameterized. e.g. Occam’s razor: model complexity and overfitting Marc Claesen and Bart De Moor Hyperparameter Search in Machine Learning
Introduction Example: optimizing hyperparameters for an SVM classifier Challenges in hyperparameter search State-of-the-art References Hyperparameter search Most machine learning methods are (hyper)parameterized. e.g. Occam’s razor: model complexity and overfitting Hyperparameters can significantly impact performance suitable hyperparameters must be determined for each task occurs in both supervised and unsupervised learning → need for disciplined, automated optimization methods Marc Claesen and Bart De Moor Hyperparameter Search in Machine Learning
Introduction Example: optimizing hyperparameters for an SVM classifier Challenges in hyperparameter search State-of-the-art References Hyperparameter search Most machine learning methods are (hyper)parameterized. e.g. Occam’s razor: model complexity and overfitting Hyperparameters can significantly impact performance suitable hyperparameters must be determined for each task occurs in both supervised and unsupervised learning → need for disciplined, automated optimization methods Some examples: SVM: regularization and kernel hyperparameters ANN: regularization, network architecture, transfer functions Marc Claesen and Bart De Moor Hyperparameter Search in Machine Learning
Introduction Example: optimizing hyperparameters for an SVM classifier Challenges in hyperparameter search State-of-the-art References Formalizing hyperparameter tuning In a general sense, tuning involves these components: a learning algorithm A , parameterized by hyperparameters λ Marc Claesen and Bart De Moor Hyperparameter Search in Machine Learning
Introduction Example: optimizing hyperparameters for an SVM classifier Challenges in hyperparameter search State-of-the-art References Formalizing hyperparameter tuning In a general sense, tuning involves these components: a learning algorithm A , parameterized by hyperparameters λ training and test data X ( tr ) , X ( te ) Marc Claesen and Bart De Moor Hyperparameter Search in Machine Learning
Introduction Example: optimizing hyperparameters for an SVM classifier Challenges in hyperparameter search State-of-the-art References Formalizing hyperparameter tuning In a general sense, tuning involves these components: a learning algorithm A , parameterized by hyperparameters λ training and test data X ( tr ) , X ( te ) a model M = A ( X ( tr ) | λ ) Marc Claesen and Bart De Moor Hyperparameter Search in Machine Learning
Introduction Example: optimizing hyperparameters for an SVM classifier Challenges in hyperparameter search State-of-the-art References Formalizing hyperparameter tuning In a general sense, tuning involves these components: a learning algorithm A , parameterized by hyperparameters λ training and test data X ( tr ) , X ( te ) a model M = A ( X ( tr ) | λ ) loss function L to assess quality of M , typically using X ( te ) : L ( M | X ( te ) ) Marc Claesen and Bart De Moor Hyperparameter Search in Machine Learning
Introduction Example: optimizing hyperparameters for an SVM classifier Challenges in hyperparameter search State-of-the-art References Formalizing hyperparameter tuning In a general sense, tuning involves these components: a learning algorithm A , parameterized by hyperparameters λ training and test data X ( tr ) , X ( te ) a model M = A ( X ( tr ) | λ ) loss function L to assess quality of M , typically using X ( te ) : L ( M | X ( te ) ) In optimization terms, we aim to find λ ∗ (assuming minimization): λ ∗ = arg min A ( X ( tr ) | λ ) | X ( te ) � � L λ Marc Claesen and Bart De Moor Hyperparameter Search in Machine Learning
Introduction Example: optimizing hyperparameters for an SVM classifier Challenges in hyperparameter search State-of-the-art References Formalizing hyperparameter tuning In a general sense, tuning involves these components: a learning algorithm A , parameterized by hyperparameters λ training and test data X ( tr ) , X ( te ) a model M = A ( X ( tr ) | λ ) loss function L to assess quality of M , typically using X ( te ) : L ( M | X ( te ) ) In optimization terms, we aim to find λ ∗ (assuming minimization): λ ∗ = arg min A ( X ( tr ) | λ ) | X ( te ) � � F ( λ | A , X ( tr ) , X ( te ) , L ) L = arg min � �� � λ λ objective function Marc Claesen and Bart De Moor Hyperparameter Search in Machine Learning
Introduction Example: optimizing hyperparameters for an SVM classifier Challenges in hyperparameter search State-of-the-art References Tuning in practice Most often done using a combination of grid and manual search: grid search suffers from the curse of dimensionality manual tuning leads to poor reproducibility Marc Claesen and Bart De Moor Hyperparameter Search in Machine Learning
Introduction Example: optimizing hyperparameters for an SVM classifier Challenges in hyperparameter search State-of-the-art References Tuning in practice Most often done using a combination of grid and manual search: grid search suffers from the curse of dimensionality manual tuning leads to poor reproducibility Better solutions exist but lack adoption because: potential performance improvements are underestimated lack of availability and/or ease of use Marc Claesen and Bart De Moor Hyperparameter Search in Machine Learning
Introduction Example: optimizing hyperparameters for an SVM classifier Challenges in hyperparameter search State-of-the-art References Outline Introduction 1 Example: optimizing hyperparameters for an SVM classifier 2 Challenges in hyperparameter search 3 State-of-the-art 4 Marc Claesen and Bart De Moor Hyperparameter Search in Machine Learning
Introduction Example: optimizing hyperparameters for an SVM classifier Challenges in hyperparameter search State-of-the-art References Support vector machine (SVM) classifiers Marc Claesen and Bart De Moor Hyperparameter Search in Machine Learning
Introduction Example: optimizing hyperparameters for an SVM classifier Challenges in hyperparameter search State-of-the-art References Support vector machine (SVM) classifiers Marc Claesen and Bart De Moor Hyperparameter Search in Machine Learning
Introduction Example: optimizing hyperparameters for an SVM classifier Challenges in hyperparameter search State-of-the-art References Support vector machine (SVM) classifiers n 1 � � � min α i α j y i y j κ ( x i , x j ) + C ξ i , 2 α,ξ, b i =1 i ∈ SV j ∈ SV � � � α i α j y i y j κ ( x i , x j ) + b ≥ 1 − ξ i , ξ i ≥ 0 , ∀ i . subject to y i j ∈ SV Marc Claesen and Bart De Moor Hyperparameter Search in Machine Learning
Recommend
More recommend
