Hyperparameter Tuning in Python Using Optunity - - PowerPoint PPT Presentation

Introduction Optunity overview Example: tuning an SVM References

Hyperparameter Tuning in Python Using Optunity

http://www.optunity.net Marc Claesen Jaak Simm Dusan Popovic Bart De Moor

ESAT-STADIUS, KU Leuven iMinds Medical IT Department

September 9, 2014 STADIUS

Center for Dynamical Systems, Signal Processing and Data Analytics

Marc Claesen, Jaak Simm, Dusan Popovic, Bart De Moor Hyperparameter Tuning in Python Using Optunity

Introduction Optunity overview Example: tuning an SVM References

Outline

S T A D I U S

Center for Dynamical Systems, Signal Processing and Data Analytics

1

Introduction

2

Optunity overview

3

Example: tuning an SVM

Marc Claesen, Jaak Simm, Dusan Popovic, Bart De Moor Hyperparameter Tuning in Python Using Optunity

Introduction Optunity overview Example: tuning an SVM References

Hyperparameter tuning

S T A D I U S

Center for Dynamical Systems, Signal Processing and Data Analytics

Tuning is the task of obtaining hyperparameters for a given learning approach that yield a good model for given data.

• ccurs in both supervised and unsupervised learning

hyperparameter choice can significantly impact performance Some examples: SVM: regularization and kernel hyperparameters ANN: regularization and network architecture KSC: number of clusters and kernel hyperparameters

Marc Claesen, Jaak Simm, Dusan Popovic, Bart De Moor Hyperparameter Tuning in Python Using Optunity

Introduction Optunity overview Example: tuning an SVM References

Tuning in practice

S T A D I U S

Center for Dynamical Systems, Signal Processing and Data Analytics

Most often done using a combination of grid and manual search: grid search suffers from the curse of dimensionality promotes ad hoc tuning approaches Manual tuning leads to poor reproducibility requires expert knowledge a black art (Snoek, Larochelle, & Adams, 2012) Better solutions exist but lack adoption because: potential performance improvements are underestimated lack of availability and/or ease of use

Marc Claesen, Jaak Simm, Dusan Popovic, Bart De Moor Hyperparameter Tuning in Python Using Optunity

Introduction Optunity overview Example: tuning an SVM References

Formalizing hyperparameter tuning

S T A D I U S

Center for Dynamical Systems, Signal Processing and Data Analytics

In a general sense, tuning involves these components: a learning algorithm A, parameterized by hyperparameters λ training and test data X(tr), X(te) a model M = A(X(tr) | λ) loss function L to assess quality of M, typically using X(te): L(M | X(te))

Marc Claesen, Jaak Simm, Dusan Popovic, Bart De Moor Hyperparameter Tuning in Python Using Optunity

Introduction Optunity overview Example: tuning an SVM References

Formalizing hyperparameter tuning

S T A D I U S

Center for Dynamical Systems, Signal Processing and Data Analytics

In a general sense, tuning involves these components: a learning algorithm A, parameterized by hyperparameters λ training and test data X(tr), X(te) a model M = A(X(tr) | λ) loss function L to assess quality of M, typically using X(te): L(M | X(te)) In optimization terms, we aim to find λ∗ (assuming minimization): λ∗ = arg min

λ

L

• A(X(tr) | λ) | X(te)

Marc Claesen, Jaak Simm, Dusan Popovic, Bart De Moor Hyperparameter Tuning in Python Using Optunity

Introduction Optunity overview Example: tuning an SVM References

Formalizing hyperparameter tuning

S T A D I U S

Center for Dynamical Systems, Signal Processing and Data Analytics

In a general sense, tuning involves these components: a learning algorithm A, parameterized by hyperparameters λ training and test data X(tr), X(te) a model M = A(X(tr) | λ) loss function L to assess quality of M, typically using X(te): L(M | X(te)) In optimization terms, we aim to find λ∗ (assuming minimization): λ∗ = arg min

λ

L

• A(X(tr) | λ) | X(te)

= arg min

λ

F(λ | A, X(tr), X(te))

• bjective function

Marc Claesen, Jaak Simm, Dusan Popovic, Bart De Moor Hyperparameter Tuning in Python Using Optunity

Introduction Optunity overview Example: tuning an SVM References

Outline

S T A D I U S

Center for Dynamical Systems, Signal Processing and Data Analytics

1

Introduction

2

Optunity overview

3

Example: tuning an SVM

Marc Claesen, Jaak Simm, Dusan Popovic, Bart De Moor Hyperparameter Tuning in Python Using Optunity

Introduction Optunity overview Example: tuning an SVM References

Design goals

S T A D I U S

Center for Dynamical Systems, Signal Processing and Data Analytics

Our key goals are: providing appropriate solvers for the tuning task terse user and dev APIs with easy learning curves compatibility with any version of Python (> 2.7) easy integration within alien environments Optunity’s implementation is not focused on efficiency. evaluating F is the performance bottleneck external dependencies are avoided (NumPy, scikit-learn, . . . ) We are all consenting adults.

Marc Claesen, Jaak Simm, Dusan Popovic, Bart De Moor Hyperparameter Tuning in Python Using Optunity

Introduction Optunity overview Example: tuning an SVM References

General approach

S T A D I U S

Center for Dynamical Systems, Signal Processing and Data Analytics

Optunity solves tuning problems using: methods for continuous black-box function optimization sequential function evaluations (with vectorization support) expert and simplified programming interfaces (optional) domain constraints on the objective function F Using the simplified interface entails specifying: F and the permitted number of evaluations box constraints for each hyperparameter (optional) the solver to be used

Marc Claesen, Jaak Simm, Dusan Popovic, Bart De Moor Hyperparameter Tuning in Python Using Optunity

Introduction Optunity overview Example: tuning an SVM References

Solvers

S T A D I U S

Center for Dynamical Systems, Signal Processing and Data Analytics

We offer a variety of fundamentally different approaches to be suitable for many problems (Wolpert & Macready, 1997). The following solvers are currently available: undirected search

grid search random search (Bergstra & Bengio, 2012)

Nelder-Mead simplex (Nelder & Mead, 1965) evolutionary methods (using DEAP (Fortin et al., 2012))

particle swarm optimization (Kennedy, 2010) CMA-ES (Hansen & Ostermeier, 2001)

Marc Claesen, Jaak Simm, Dusan Popovic, Bart De Moor Hyperparameter Tuning in Python Using Optunity

Introduction Optunity overview Example: tuning an SVM References

Cross-validation

S T A D I U S

Center for Dynamical Systems, Signal Processing and Data Analytics

Optunity provides k-fold cross-validation to estimate performance of supervised algorithms implemented as a function decorator Configurable features: number of folds and iterations static or dynamic folds support for strata and clusters

Marc Claesen, Jaak Simm, Dusan Popovic, Bart De Moor Hyperparameter Tuning in Python Using Optunity

Introduction Optunity overview Example: tuning an SVM References

Integrating Optunity in alien environments

S T A D I U S

Center for Dynamical Systems, Signal Processing and Data Analytics

Optunity can be integrated in other environments easily: launchable as standalone subprocess no language bindings necessary communication using JSON messages through pipes minimal data exchange (e.g. no data sets)

Solvers

           grid search random search Nelder-Mead particle swarm CMA-ES

API ⇔ JSON ⇔ Wrapper Method

R MATLAB Java C/C++ . . .            callback requests ⋆ final solution configuration ⋆ callback results

• Optunity in Python

working environment generic solvers arbitrary method

Marc Claesen, Jaak Simm, Dusan Popovic, Bart De Moor Hyperparameter Tuning in Python Using Optunity

Introduction Optunity overview Example: tuning an SVM References

Current development

S T A D I U S

Center for Dynamical Systems, Signal Processing and Data Analytics

Implementing additional solvers: coupled simulated annealing (Xavier-de Souza, Suykens, Vandewalle, & Boll´ e, 2010) efficient global optimization (Jones, Schonlau, & Welch, 1998)

• ther Bayesian approaches (Hutter, Hoos, & Leyton-Brown,

2011) Integrating Optunity in alien environments: nearly finished: R and MATLAB planned: Java and C/C++

Marc Claesen, Jaak Simm, Dusan Popovic, Bart De Moor Hyperparameter Tuning in Python Using Optunity

Introduction Optunity overview Example: tuning an SVM References

Outline

S T A D I U S

Center for Dynamical Systems, Signal Processing and Data Analytics

1

Introduction

2

Optunity overview

3

Example: tuning an SVM

Marc Claesen, Jaak Simm, Dusan Popovic, Bart De Moor Hyperparameter Tuning in Python Using Optunity

Introduction Optunity overview Example: tuning an SVM References

Task summary

S T A D I U S

Center for Dynamical Systems, Signal Processing and Data Analytics

Tuning an SVM classifier with RBF kernel κ(u, v) = e−γu−v2:

• ptimize regularization parameter C and kernel parameter γ

evaluate (C, γ) pair using 2× iterated 10-fold cross-validation

Marc Claesen, Jaak Simm, Dusan Popovic, Bart De Moor Hyperparameter Tuning in Python Using Optunity

Introduction Optunity overview Example: tuning an SVM References

Code

S T A D I U S

Center for Dynamical Systems, Signal Processing and Data Analytics

1 import optunity as opt 2 import sklearn.svm as svm Marc Claesen, Jaak Simm, Dusan Popovic, Bart De Moor Hyperparameter Tuning in Python Using Optunity

Introduction Optunity overview Example: tuning an SVM References

Code

S T A D I U S

Center for Dynamical Systems, Signal Processing and Data Analytics

1 import optunity as opt 2 import sklearn.svm as svm 3 4 @opt.cross_validated(x=data, y=labels, num_folds=10, num_iter=2) 5 def svm acc(x_train, y_train, x_test, y_test, C, gamma): Marc Claesen, Jaak Simm, Dusan Popovic, Bart De Moor Hyperparameter Tuning in Python Using Optunity

Introduction Optunity overview Example: tuning an SVM References

Code

S T A D I U S

Center for Dynamical Systems, Signal Processing and Data Analytics

1 import optunity as opt 2 import sklearn.svm as svm 3 4 @opt.cross_validated(x=data, y=labels, num_folds=10, num_iter=2) 5 def svm acc(x_train, y_train, x_test, y_test, C, gamma): 6

model = svm.SVC(C=C, gamma=gamma).fit(x_train, y_train)

Marc Claesen, Jaak Simm, Dusan Popovic, Bart De Moor Hyperparameter Tuning in Python Using Optunity

Introduction Optunity overview Example: tuning an SVM References

Code

S T A D I U S

Center for Dynamical Systems, Signal Processing and Data Analytics

1 import optunity as opt 2 import sklearn.svm as svm 3 4 @opt.cross_validated(x=data, y=labels, num_folds=10, num_iter=2) 5 def svm acc(x_train, y_train, x_test, y_test, C, gamma): 6

model = svm.SVC(C=C, gamma=gamma).fit(x_train, y_train)

7

y_pred = model.predict(x_test)

8

return opt.score_functions.accuracy(y_test, y_pred)

Marc Claesen, Jaak Simm, Dusan Popovic, Bart De Moor Hyperparameter Tuning in Python Using Optunity

Introduction Optunity overview Example: tuning an SVM References

Code

S T A D I U S

Center for Dynamical Systems, Signal Processing and Data Analytics

1 import optunity as opt 2 import sklearn.svm as svm 3 4 @opt.cross_validated(x=data, y=labels, num_folds=10, num_iter=2) 5 def svm acc(x_train, y_train, x_test, y_test, C, gamma): 6

model = svm.SVC(C=C, gamma=gamma).fit(x_train, y_train)

7

y_pred = model.predict(x_test)

8

return opt.score_functions.accuracy(y_test, y_pred)

9 10 optimal_pars, _, _ = opt.maximize(svm acc, num_evals=100, 11

C=[0, 20], gamma=[0, 20])

Marc Claesen, Jaak Simm, Dusan Popovic, Bart De Moor Hyperparameter Tuning in Python Using Optunity

Introduction Optunity overview Example: tuning an SVM References

Code

S T A D I U S

Center for Dynamical Systems, Signal Processing and Data Analytics

1 import optunity as opt 2 import sklearn.svm as svm 3 4 @opt.cross_validated(x=data, y=labels, num_folds=10, num_iter=2) 5 def svm acc(x_train, y_train, x_test, y_test, C, gamma): 6

model = svm.SVC(C=C, gamma=gamma).fit(x_train, y_train)

7

y_pred = model.predict(x_test)

8

return opt.score_functions.accuracy(y_test, y_pred)

9 10 optimal_pars, _, _ = opt.maximize(svm acc, num_evals=100, 11

C=[0, 20], gamma=[0, 20])

12 optimal_model = svm.SVC(**optimal_pars).fit(data, labels) Marc Claesen, Jaak Simm, Dusan Popovic, Bart De Moor Hyperparameter Tuning in Python Using Optunity

Introduction Optunity overview Example: tuning an SVM References

Response surface I

S T A D I U S

Center for Dynamical Systems, Signal Processing and Data Analytics

Marc Claesen, Jaak Simm, Dusan Popovic, Bart De Moor Hyperparameter Tuning in Python Using Optunity

Introduction Optunity overview Example: tuning an SVM References

Response surface II

S T A D I U S

Center for Dynamical Systems, Signal Processing and Data Analytics

Marc Claesen, Jaak Simm, Dusan Popovic, Bart De Moor Hyperparameter Tuning in Python Using Optunity

Introduction Optunity overview Example: tuning an SVM References

The end

S T A D I U S

Center for Dynamical Systems, Signal Processing and Data Analytics

Marc Claesen, Jaak Simm, Dusan Popovic, Bart De Moor Hyperparameter Tuning in Python Using Optunity

Introduction Optunity overview Example: tuning an SVM References

Acknowledgements

S T A D I U S

Center for Dynamical Systems, Signal Processing and Data Analytics

Research Council KU Leuven: GOA/10/09 MaNet Flemish Government:

FWO: projects: G.0871.12N (Neural circuits) IWT: TBM-Logic Insulin(100793), TBM Rectal Cancer(100783), TBM IETA(130256); PhD grants Industrial Research fund (IOF): IOF/HB/13/027 Logic Insulin iMinds Medical Information Technologies SBO 2014 VLK Stichting E. van der Schueren: rectal cancer

Federal Government: FOD: Cancer Plan 2012-2015 KPC-29-023 (prostate) COST: Action: BM1104: Mass Spectrometry Imaging

Marc Claesen, Jaak Simm, Dusan Popovic, Bart De Moor Hyperparameter Tuning in Python Using Optunity

Introduction Optunity overview Example: tuning an SVM References

References I

S T A D I U S

Center for Dynamical Systems, Signal Processing and Data Analytics

Bergstra, J., & Bengio, Y. (2012). Random search for hyper-parameter optimization. Journal of Machine Learning Research, 13(1), 281–305. Fortin, F.-A., Rainville, D., Gardner, M.-A. G., Parizeau, M., Gagn´ e, C., et al. (2012). DEAP: Evolutionary algorithms made easy. Journal of Machine Learning Research, 13(1), 2171–2175. Hansen, N., & Ostermeier, A. (2001). Completely derandomized self-adaptation in evolution strategies. Evolutionary computation, 9(2), 159–195.

Marc Claesen, Jaak Simm, Dusan Popovic, Bart De Moor Hyperparameter Tuning in Python Using Optunity

Introduction Optunity overview Example: tuning an SVM References

References II

S T A D I U S

Center for Dynamical Systems, Signal Processing and Data Analytics

Hutter, F., Hoos, H. H., & Leyton-Brown, K. (2011). Sequential model-based optimization for general algorithm

• configuration. In Learning and intelligent optimization (pp.

507–523). Springer. Jones, D. R., Schonlau, M., & Welch, W. J. (1998). Efficient global optimization of expensive black-box functions. Journal

• f Global optimization, 13(4), 455–492.

Kennedy, J. (2010). Particle swarm optimization. In Encyclopedia

• f machine learning (pp. 760–766). Springer.

Nelder, J. A., & Mead, R. (1965). A simplex method for function

• minimization. The computer journal, 7(4), 308–313.

Marc Claesen, Jaak Simm, Dusan Popovic, Bart De Moor Hyperparameter Tuning in Python Using Optunity

Introduction Optunity overview Example: tuning an SVM References

References III

S T A D I U S

Center for Dynamical Systems, Signal Processing and Data Analytics

Snoek, J., Larochelle, H., & Adams, R. P. (2012). Practical Bayesian optimization of machine learning algorithms. In Advances in neural information processing systems (pp. 2951–2959). Wolpert, D. H., & Macready, W. G. (1997). No free lunch theorems for optimization. Evolutionary Computation, IEEE Transactions on, 1(1), 67–82. Xavier-de Souza, S., Suykens, J. A., Vandewalle, J., & Boll´ e, D. (2010). Coupled simulated annealing. Systems, Man, and Cybernetics, Part B: Cybernetics, IEEE Transactions on, 40(2), 320–335.

Marc Claesen, Jaak Simm, Dusan Popovic, Bart De Moor Hyperparameter Tuning in Python Using Optunity