stochastic search using the natural gradient
play

Stochastic Search using the Natural Gradient Ecient Natural - PowerPoint PPT Presentation

Mar 31, 2023 •163 likes •1.24k views

Stochastic Search using the Natural Gradient Ecient Natural Evolution Strategies (eNES) Yi Sun, Daan Wierstra, Tom Schaul, and Jrgen Schmidhuber {yi,daan,tom,juergen}@idsia.ch IDSIA, Galleria 2, Manno 6928, Switzerland June 17th, 2009 Yi

  1. The Gaussian Search Distribution The search distribution is given by p ( z j θ ) = N ( z j x , C ) . We use the parameter set θ = h x , A i , with A being the Cholesky decomposition of C , i.e., A is an upper triangular matrix (UTM) and C = A > A . No redundancy in θ since C is symmetric. Yi Sun, et al. (IDSIA) E¢cient Natural Evolution Strategies 17/06/2009 6 / 22

  2. The Gaussian Search Distribution The search distribution is given by p ( z j θ ) = N ( z j x , C ) . We use the parameter set θ = h x , A i , with A being the Cholesky decomposition of C , i.e., A is an upper triangular matrix (UTM) and C = A > A . No redundancy in θ since C is symmetric. O θ log p ( z j θ ) can be computed in closed form: O x log p ( z j θ ) = C � ( z � x ) � A � � O A log p ( z j θ ) = A �> ( z � x ) ( z � x ) > C � � diag Yi Sun, et al. (IDSIA) E¢cient Natural Evolution Strategies 17/06/2009 6 / 22

  3. The Gaussian Search Distribution The search distribution is given by p ( z j θ ) = N ( z j x , C ) . We use the parameter set θ = h x , A i , with A being the Cholesky decomposition of C , i.e., A is an upper triangular matrix (UTM) and C = A > A . No redundancy in θ since C is symmetric. O θ log p ( z j θ ) can be computed in closed form: O x log p ( z j θ ) = C � ( z � x ) � A � � O A log p ( z j θ ) = A �> ( z � x ) ( z � x ) > C � � diag O s θ J ( θ ) can be computed from O θ log p ( z 1 j θ ) . . . O θ log p ( z 1 j θ ) . Yi Sun, et al. (IDSIA) E¢cient Natural Evolution Strategies 17/06/2009 6 / 22

  4. Stochastic Gradient Ascent θ J ( θ ) = θ + α θ θ + α O s n Gf Yi Sun, et al. (IDSIA) E¢cient Natural Evolution Strategies 17/06/2009 7 / 22

  5. Stochastic Gradient Ascent θ J ( θ ) = θ + α θ θ + α O s n Gf Yi Sun, et al. (IDSIA) E¢cient Natural Evolution Strategies 17/06/2009 7 / 22

  6. Stochastic Gradient Ascent θ J ( θ ) = θ + α θ θ + α O s n Gf Yi Sun, et al. (IDSIA) E¢cient Natural Evolution Strategies 17/06/2009 7 / 22

  7. Stochastic Gradient Ascent θ J ( θ ) = θ + α θ θ + α O s n Gf Yi Sun, et al. (IDSIA) E¢cient Natural Evolution Strategies 17/06/2009 7 / 22

  8. Stochastic Gradient Ascent θ J ( θ ) = θ + α θ θ + α O s n Gf Yi Sun, et al. (IDSIA) E¢cient Natural Evolution Strategies 17/06/2009 7 / 22

  9. Novel Ideas in eNES Use the Natural Gradient instead of the vanilla gradient. 1 Yi Sun, et al. (IDSIA) E¢cient Natural Evolution Strategies 17/06/2009 8 / 22

  10. Novel Ideas in eNES Use the Natural Gradient instead of the vanilla gradient. 1 The natural gradient is computed in an Exact and E¢cient way. 2 Yi Sun, et al. (IDSIA) E¢cient Natural Evolution Strategies 17/06/2009 8 / 22

  11. Novel Ideas in eNES Use the Natural Gradient instead of the vanilla gradient. 1 The natural gradient is computed in an Exact and E¢cient way. 2 Use Importance Mixing for reusing previously evaluated samples. 3 Yi Sun, et al. (IDSIA) E¢cient Natural Evolution Strategies 17/06/2009 8 / 22

  12. Novel Ideas in eNES Use the Natural Gradient instead of the vanilla gradient. 1 The natural gradient is computed in an Exact and E¢cient way. 2 Use Importance Mixing for reusing previously evaluated samples. 3 Introducing Optimal Fitness Baseline to reduce the variance of 4 gradient estimation. Yi Sun, et al. (IDSIA) E¢cient Natural Evolution Strategies 17/06/2009 8 / 22

  13. 1. Why Natural Gradient? Vanilla gradient doesn’t work: Yi Sun, et al. (IDSIA) E¢cient Natural Evolution Strategies 17/06/2009 9 / 22

  14. 1. Why Natural Gradient? Vanilla gradient doesn’t work: Over-aggressive steps on ridges. Yi Sun, et al. (IDSIA) E¢cient Natural Evolution Strategies 17/06/2009 9 / 22

  15. 1. Why Natural Gradient? Vanilla gradient doesn’t work: Over-aggressive steps on ridges. Too small steps on plateaus. Yi Sun, et al. (IDSIA) E¢cient Natural Evolution Strategies 17/06/2009 9 / 22

  16. 1. Why Natural Gradient? Vanilla gradient doesn’t work: Over-aggressive steps on ridges. Too small steps on plateaus. Slow or premature convergence, non-robust performance. Yi Sun, et al. (IDSIA) E¢cient Natural Evolution Strategies 17/06/2009 9 / 22

  17. 1. Why Natural Gradient? Vanilla gradient doesn’t work: Over-aggressive steps on ridges. Too small steps on plateaus. Slow or premature convergence, non-robust performance. Yi Sun, et al. (IDSIA) E¢cient Natural Evolution Strategies 17/06/2009 9 / 22

  18. 1. Why Natural Gradient? Vanilla gradient doesn’t work: Over-aggressive steps on ridges. Too small steps on plateaus. Slow or premature convergence, non-robust performance. Yi Sun, et al. (IDSIA) E¢cient Natural Evolution Strategies 17/06/2009 9 / 22

  19. 1. Why Natural Gradient? Vanilla gradient doesn’t work: Over-aggressive steps on ridges. Too small steps on plateaus. Slow or premature convergence, non-robust performance. Yi Sun, et al. (IDSIA) E¢cient Natural Evolution Strategies 17/06/2009 9 / 22

  20. 1. Why Natural Gradient? Vanilla gradient doesn’t work: Over-aggressive steps on ridges. Too small steps on plateaus. Slow or premature convergence, non-robust performance. Yi Sun, et al. (IDSIA) E¢cient Natural Evolution Strategies 17/06/2009 9 / 22

  21. 1. Why Natural Gradient? Vanilla gradient doesn’t work: Over-aggressive steps on ridges. Too small steps on plateaus. Slow or premature convergence, non-robust performance. Yi Sun, et al. (IDSIA) E¢cient Natural Evolution Strategies 17/06/2009 9 / 22

  22. 1. Why Natural Gradient? Vanilla gradient doesn’t work: Over-aggressive steps on ridges. Too small steps on plateaus. Slow or premature convergence, non-robust performance. Basic idea of natural gradient Yi Sun, et al. (IDSIA) E¢cient Natural Evolution Strategies 17/06/2009 9 / 22

  23. 1. Why Natural Gradient? Vanilla gradient doesn’t work: Over-aggressive steps on ridges. Too small steps on plateaus. Slow or premature convergence, non-robust performance. Basic idea of natural gradient Steepest ascent direction when considering correlations between elements in θ . Yi Sun, et al. (IDSIA) E¢cient Natural Evolution Strategies 17/06/2009 9 / 22

  24. 1. Why Natural Gradient? Vanilla gradient doesn’t work: Over-aggressive steps on ridges. Too small steps on plateaus. Slow or premature convergence, non-robust performance. Basic idea of natural gradient Steepest ascent direction when considering correlations between elements in θ . Re-weight gradient elements according to their uncertainties, resp. Yi Sun, et al. (IDSIA) E¢cient Natural Evolution Strategies 17/06/2009 9 / 22

  25. 1. Why Natural Gradient? Vanilla gradient doesn’t work: Over-aggressive steps on ridges. Too small steps on plateaus. Slow or premature convergence, non-robust performance. Basic idea of natural gradient Steepest ascent direction when considering correlations between elements in θ . Re-weight gradient elements according to their uncertainties, resp. Isotropic convergence on ill-shaped surface. Yi Sun, et al. (IDSIA) E¢cient Natural Evolution Strategies 17/06/2009 9 / 22

  26. 1. Why Natural Gradient? Vanilla gradient doesn’t work: Over-aggressive steps on ridges. Too small steps on plateaus. Slow or premature convergence, non-robust performance. Basic idea of natural gradient Steepest ascent direction when considering correlations between elements in θ . Re-weight gradient elements according to their uncertainties, resp. Isotropic convergence on ill-shaped surface. Yi Sun, et al. (IDSIA) E¢cient Natural Evolution Strategies 17/06/2009 9 / 22

  27. 1. Why Natural Gradient? Vanilla gradient doesn’t work: Over-aggressive steps on ridges. Too small steps on plateaus. Slow or premature convergence, non-robust performance. Basic idea of natural gradient Steepest ascent direction when considering correlations between elements in θ . Re-weight gradient elements according to their uncertainties, resp. Isotropic convergence on ill-shaped surface. Yi Sun, et al. (IDSIA) E¢cient Natural Evolution Strategies 17/06/2009 9 / 22

  28. 1. Why Natural Gradient? Vanilla gradient doesn’t work: Over-aggressive steps on ridges. Too small steps on plateaus. Slow or premature convergence, non-robust performance. Basic idea of natural gradient Steepest ascent direction when considering correlations between elements in θ . Re-weight gradient elements according to their uncertainties, resp. Isotropic convergence on ill-shaped surface. Yi Sun, et al. (IDSIA) E¢cient Natural Evolution Strategies 17/06/2009 9 / 22

  29. 1. Why Natural Gradient? Vanilla gradient doesn’t work: Over-aggressive steps on ridges. Too small steps on plateaus. Slow or premature convergence, non-robust performance. Basic idea of natural gradient Steepest ascent direction when considering correlations between elements in θ . Re-weight gradient elements according to their uncertainties, resp. Isotropic convergence on ill-shaped surface. Yi Sun, et al. (IDSIA) E¢cient Natural Evolution Strategies 17/06/2009 9 / 22

  30. 1. Why Natural Gradient? Vanilla gradient doesn’t work: Over-aggressive steps on ridges. Too small steps on plateaus. Slow or premature convergence, non-robust performance. Basic idea of natural gradient Steepest ascent direction when considering correlations between elements in θ . Re-weight gradient elements according to their uncertainties, resp. Isotropic convergence on ill-shaped surface. Yi Sun, et al. (IDSIA) E¢cient Natural Evolution Strategies 17/06/2009 9 / 22

  31. 1. Why Natural Gradient? Vanilla gradient doesn’t work: Over-aggressive steps on ridges. Too small steps on plateaus. Slow or premature convergence, non-robust performance. Basic idea of natural gradient Steepest ascent direction when considering correlations between elements in θ . Re-weight gradient elements according to their uncertainties, resp. Isotropic convergence on ill-shaped surface. Yi Sun, et al. (IDSIA) E¢cient Natural Evolution Strategies 17/06/2009 9 / 22

  32. 1. Why Natural Gradient? Vanilla gradient doesn’t work: Over-aggressive steps on ridges. Too small steps on plateaus. Slow or premature convergence, non-robust performance. Basic idea of natural gradient Steepest ascent direction when considering correlations between elements in θ . Re-weight gradient elements according to their uncertainties, resp. Isotropic convergence on ill-shaped surface. Yi Sun, et al. (IDSIA) E¢cient Natural Evolution Strategies 17/06/2009 9 / 22

  33. 1. Why Natural Gradient? Vanilla gradient doesn’t work: Over-aggressive steps on ridges. Too small steps on plateaus. Slow or premature convergence, non-robust performance. Basic idea of natural gradient Steepest ascent direction when considering correlations between elements in θ . Re-weight gradient elements according to their uncertainties, resp. Isotropic convergence on ill-shaped surface. Yi Sun, et al. (IDSIA) E¢cient Natural Evolution Strategies 17/06/2009 9 / 22

  34. 1. Formulation of Natural Gradient Assume the distance between two adjacent distributions p ( �j θ ) and p ( �j θ + δθ ) is de…ned by their KL divergence. The natural gradient O θ J ( θ ) is given by the necessary condition ˜ O θ J ( θ ) = O θ J ( θ ) . F ˜ Yi Sun, et al. (IDSIA) E¢cient Natural Evolution Strategies 17/06/2009 10 / 22

  35. 1. Formulation of Natural Gradient Assume the distance between two adjacent distributions p ( �j θ ) and p ( �j θ + δθ ) is de…ned by their KL divergence. The natural gradient O θ J ( θ ) is given by the necessary condition ˜ O θ J ( θ ) = O θ J ( θ ) . F ˜ F is the Fisher information matrix (FIM) of θ : (Intuitively, the normalized covariance of the gradient.) h ( O θ log p ( z j θ )) ( O θ log p ( z j θ )) > i F = E . Yi Sun, et al. (IDSIA) E¢cient Natural Evolution Strategies 17/06/2009 10 / 22

  36. 1. Formulation of Natural Gradient Assume the distance between two adjacent distributions p ( �j θ ) and p ( �j θ + δθ ) is de…ned by their KL divergence. The natural gradient O θ J ( θ ) is given by the necessary condition ˜ O θ J ( θ ) = O θ J ( θ ) . F ˜ F is the Fisher information matrix (FIM) of θ : (Intuitively, the normalized covariance of the gradient.) h ( O θ log p ( z j θ )) ( O θ log p ( z j θ )) > i F = E . F may not be invertible. Yi Sun, et al. (IDSIA) E¢cient Natural Evolution Strategies 17/06/2009 10 / 22

  37. 1. Formulation of Natural Gradient Assume the distance between two adjacent distributions p ( �j θ ) and p ( �j θ + δθ ) is de…ned by their KL divergence. The natural gradient O θ J ( θ ) is given by the necessary condition ˜ O θ J ( θ ) = O θ J ( θ ) . F ˜ F is the Fisher information matrix (FIM) of θ : (Intuitively, the normalized covariance of the gradient.) h ( O θ log p ( z j θ )) ( O θ log p ( z j θ )) > i F = E . F may not be invertible. If F is invertable, we can compute the (estimated) natural gradient as O θ J ( θ ) = F � O θ J ( θ ) , ˜ O s θ J ( θ ) = F � O s ˜ θ J ( θ ) . Yi Sun, et al. (IDSIA) E¢cient Natural Evolution Strategies 17/06/2009 10 / 22

  38. 2. Property of FIM in the Gaussian Case Let θ = h x , A i . Under this setting, we …nd (quite luckily): The Fisher information matrix is indeed invertible . Yi Sun, et al. (IDSIA) E¢cient Natural Evolution Strategies 17/06/2009 11 / 22

  39. 2. Property of FIM in the Gaussian Case Let θ = h x , A i . Under this setting, we …nd (quite luckily): The Fisher information matrix is indeed invertible . The Fisher information matrix is a block diagonal matrix 2 3 C � 6 7 F 1 6 7 F = 6 7 5 . ... 4 F d Yi Sun, et al. (IDSIA) E¢cient Natural Evolution Strategies 17/06/2009 11 / 22

  40. 2. Property of FIM in the Gaussian Case Let θ = h x , A i . Under this setting, we …nd (quite luckily): The Fisher information matrix is indeed invertible . The Fisher information matrix is a block diagonal matrix 2 3 C � 6 7 F 1 6 7 F = 6 5 . 7 ... 4 F d C � is the FIM for x . Yi Sun, et al. (IDSIA) E¢cient Natural Evolution Strategies 17/06/2009 11 / 22

  41. 2. Property of FIM in the Gaussian Case Let θ = h x , A i . Under this setting, we …nd (quite luckily): The Fisher information matrix is indeed invertible . The Fisher information matrix is a block diagonal matrix 2 3 C � 6 7 F 1 6 7 F = 6 5 . 7 ... 4 F d C � is the FIM for x . F k is the FIM for ( n � k + 1 non-zero elements in) the k -th row of A . Yi Sun, et al. (IDSIA) E¢cient Natural Evolution Strategies 17/06/2009 11 / 22

  42. 2. Property of FIM in the Gaussian Case Let θ = h x , A i . Under this setting, we …nd (quite luckily): The Fisher information matrix is indeed invertible . The Fisher information matrix is a block diagonal matrix 2 3 C � 6 7 F 1 6 7 F = 6 7 5 . ... 4 F d C � is the FIM for x . F k is the FIM for ( n � k + 1 non-zero elements in) the k -th row of A . The FIM suggest a natural grouping of elements in θ . Yi Sun, et al. (IDSIA) E¢cient Natural Evolution Strategies 17/06/2009 11 / 22

  43. 2. E¢cient Inverse of FIM The computation of natural gradient requires the inverse of F . � d 2 � , so computing F � requires Naively, F is a matrix of size O � d 6 � O . Yi Sun, et al. (IDSIA) E¢cient Natural Evolution Strategies 17/06/2009 12 / 22

  44. 2. E¢cient Inverse of FIM The computation of natural gradient requires the inverse of F . � d 2 � , so computing F � requires Naively, F is a matrix of size O � d 6 � O . We already …nd that F is block diagonal, so computing F � requires � d 4 � O . Yi Sun, et al. (IDSIA) E¢cient Natural Evolution Strategies 17/06/2009 12 / 22

  45. 2. E¢cient Inverse of FIM The computation of natural gradient requires the inverse of F . � d 2 � , so computing F � requires Naively, F is a matrix of size O � d 6 � O . We already …nd that F is block diagonal, so computing F � requires � d 4 � O . We can do better! Use the special form of each sub-block, the � d 3 � complexity is reduced to O . Yi Sun, et al. (IDSIA) E¢cient Natural Evolution Strategies 17/06/2009 12 / 22

  46. 2. E¢cient Inverse of FIM The computation of natural gradient requires the inverse of F . � d 2 � , so computing F � requires Naively, F is a matrix of size O � d 6 � O . We already …nd that F is block diagonal, so computing F � requires � d 4 � O . We can do better! Use the special form of each sub-block, the � d 3 � complexity is reduced to O . The estimated natural gradient is then computed as θ J ( θ ) = 1 O s n F � Gf . � d 3 � with complexity O . Yi Sun, et al. (IDSIA) E¢cient Natural Evolution Strategies 17/06/2009 12 / 22

  47. 3. Importance Mixing At each cycle, we need to evaluate n new samples. Yi Sun, et al. (IDSIA) E¢cient Natural Evolution Strategies 17/06/2009 13 / 22

  48. 3. Importance Mixing At each cycle, we need to evaluate n new samples. It is common that the updated θ ( t ) is close to θ ( t � 1 ) . Yi Sun, et al. (IDSIA) E¢cient Natural Evolution Strategies 17/06/2009 13 / 22

  49. 3. Importance Mixing At each cycle, we need to evaluate n new samples. It is common that the updated θ ( t ) is close to θ ( t � 1 ) . Problem: Redundant …tness evaluations in overlapping high density area. Yi Sun, et al. (IDSIA) E¢cient Natural Evolution Strategies 17/06/2009 13 / 22

  50. 3. Importance Mixing At each cycle, we need to evaluate n new samples. It is common that the updated θ ( t ) is close to θ ( t � 1 ) . Problem: Redundant …tness evaluations in overlapping high density area. Importance Mixing: Generate samples in less explored areas, while keeping the updated batch conformed to the new search distribution. Yi Sun, et al. (IDSIA) E¢cient Natural Evolution Strategies 17/06/2009 13 / 22

  51. 3. Importance Mixing At each cycle, we need to evaluate n new samples. It is common that the updated θ ( t ) is close to θ ( t � 1 ) . Problem: Redundant …tness evaluations in overlapping high density area. Importance Mixing: Generate samples in less explored areas, while keeping the updated batch conformed to the new search distribution. Yi Sun, et al. (IDSIA) E¢cient Natural Evolution Strategies 17/06/2009 13 / 22

  52. 3. Importance Mixing At each cycle, we need to evaluate n new samples. It is common that the updated θ ( t ) is close to θ ( t � 1 ) . Problem: Redundant …tness evaluations in overlapping high density area. Importance Mixing: Generate samples in less explored areas, while keeping the updated batch conformed to the new search distribution. Reusing samples: fewer …tness evaluations. Yi Sun, et al. (IDSIA) E¢cient Natural Evolution Strategies 17/06/2009 13 / 22

  53. 3. Importance Mixing Formally, importance mixing is carried out by two rejection samplings. Yi Sun, et al. (IDSIA) E¢cient Natural Evolution Strategies 17/06/2009 14 / 22

  54. 3. Importance Mixing Formally, importance mixing is carried out by two rejection samplings. Forward pass: For each sample z from the previous batch, accept with probability � z j θ ( t ) � 8 9 < = p � z j θ ( t � 1 ) � min : 1 , ; . p Yi Sun, et al. (IDSIA) E¢cient Natural Evolution Strategies 17/06/2009 14 / 22

  55. 3. Importance Mixing Formally, importance mixing is carried out by two rejection samplings. Forward pass: For each sample z from the previous batch, accept with probability � z j θ ( t ) � 8 9 < = p � z j θ ( t � 1 ) � min : 1 , ; . p Backward pass: Accept newly generated sample z with probability � z j θ ( t � 1 ) � 8 9 < = p � z j θ ( t ) � max : 0 , 1 � ; p until batch size reached. Yi Sun, et al. (IDSIA) E¢cient Natural Evolution Strategies 17/06/2009 14 / 22

  56. 4. Optimal Fitness Baseline A typical problem with the Monte-Carlo gradient estimation is that the variance is too big. The …tness baseline is introduced to reduce the variance. Z Z O θ J = O θ f ( z ) p ( z j θ ) d z � O θ bp ( z j θ ) d z | {z } = 0 Z = O θ [ f ( z ) � b ] p ( z j θ ) d z , b is called the …tness baseline. Yi Sun, et al. (IDSIA) E¢cient Natural Evolution Strategies 17/06/2009 15 / 22

  57. 4. Optimal Fitness Baseline A typical problem with the Monte-Carlo gradient estimation is that the variance is too big. The …tness baseline is introduced to reduce the variance. Z Z O θ J = O θ f ( z ) p ( z j θ ) d z � O θ bp ( z j θ ) d z | {z } = 0 Z = O θ [ f ( z ) � b ] p ( z j θ ) d z , b is called the …tness baseline. Adding the baseline b won’t a¤ect the expectation of O θ J . Yi Sun, et al. (IDSIA) E¢cient Natural Evolution Strategies 17/06/2009 15 / 22

  58. 4. Optimal Fitness Baseline A typical problem with the Monte-Carlo gradient estimation is that the variance is too big. The …tness baseline is introduced to reduce the variance. Z Z O θ J = O θ f ( z ) p ( z j θ ) d z � O θ bp ( z j θ ) d z | {z } = 0 Z = O θ [ f ( z ) � b ] p ( z j θ ) d z , b is called the …tness baseline. Adding the baseline b won’t a¤ect the expectation of O θ J . But it a¤ects the variance of the estimation: For natural gradient h i h i O θ J ( θ )] ∝ b 2 E u > u u > v V [ ˜ � 2 b E + const with u = F � O θ log p ( z j θ ) , v = f ( z ) u . Yi Sun, et al. (IDSIA) E¢cient Natural Evolution Strategies 17/06/2009 15 / 22

  59. 4. Optimal Fitness Baseline O θ J ( θ )] is of quadratic form, we can minimize it. The optimal V [ ˜ …tness baseline is given by � � E [ u > u ] ' ∑ n u > v i = 1 u > b � = E i v i . ∑ n i = 1 u > i u i Yi Sun, et al. (IDSIA) E¢cient Natural Evolution Strategies 17/06/2009 16 / 22

  60. 4. Optimal Fitness Baseline O θ J ( θ )] is of quadratic form, we can minimize it. The optimal V [ ˜ …tness baseline is given by � � E [ u > u ] ' ∑ n u > v i = 1 u > b � = E i v i . ∑ n i = 1 u > i u i The natural gradient is then estimated by θ J ( θ ) = 1 O s n F � G ( f � b � ) . ˜ Yi Sun, et al. (IDSIA) E¢cient Natural Evolution Strategies 17/06/2009 16 / 22

  61. 4. Optimal Fitness Baseline O θ J ( θ )] is of quadratic form, we can minimize it. The optimal V [ ˜ …tness baseline is given by � � E [ u > u ] ' ∑ n u > v i = 1 u > b � = E i v i . ∑ n i = 1 u > i u i The natural gradient is then estimated by θ J ( θ ) = 1 O s n F � G ( f � b � ) . ˜ Better: Di¤erent baselines b j for di¤erent (groups of) parameter θ j , further reducing the variance. Yi Sun, et al. (IDSIA) E¢cient Natural Evolution Strategies 17/06/2009 16 / 22

  62. 4. Optimal Fitness Baseline O θ J ( θ )] is of quadratic form, we can minimize it. The optimal V [ ˜ …tness baseline is given by � � E [ u > u ] ' ∑ n u > v i = 1 u > b � = E i v i . ∑ n i = 1 u > i u i The natural gradient is then estimated by θ J ( θ ) = 1 O s n F � G ( f � b � ) . ˜ Better: Di¤erent baselines b j for di¤erent (groups of) parameter θ j , further reducing the variance. The block diagonal structure of F suggests using a block …tness baseline , where di¤erent baseline values are computed for each group of parameters in θ . Yi Sun, et al. (IDSIA) E¢cient Natural Evolution Strategies 17/06/2009 16 / 22

  63. Putting Things Together Initialization Update population using importance mixing loop Yi Sun, et al. (IDSIA) E¢cient Natural Evolution Strategies 17/06/2009 17 / 22

  64. Putting Things Together Initialization Update population using importance mixing loop Evaluate newly generated samples Yi Sun, et al. (IDSIA) E¢cient Natural Evolution Strategies 17/06/2009 17 / 22

  65. Putting Things Together Initialization Update population using importance mixing loop Compute optimal baseline b � and Evaluate newly O s ˜ θ J ( θ ) generated samples Yi Sun, et al. (IDSIA) E¢cient Natural Evolution Strategies 17/06/2009 17 / 22

  66. Putting Things Together Initialization Update: Update population O s θ θ + α ˜ θ J ( θ ) using importance mixing loop Compute optimal baseline b � and Evaluate newly O s ˜ θ J ( θ ) generated samples Yi Sun, et al. (IDSIA) E¢cient Natural Evolution Strategies 17/06/2009 17 / 22

  67. Empirical Results - Standard Blackbox Benchmarks Unimodal-50 Cigar DiffPow Ellipsoid ParabR Schwefel SharpR Sphere Tablet -fitness number of evaluations Yi Sun, et al. (IDSIA) E¢cient Natural Evolution Strategies 17/06/2009 18 / 22

  68. Empirical Results - Multimodal 5 4.5 4 3.5 3 2.5 2 1.5 1 0.5 0 −0.5 −0.5 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 eNES is able to jump over deceptive local optima. Yi Sun, et al. (IDSIA) E¢cient Natural Evolution Strategies 17/06/2009 19 / 22

  69. Empirical Results - Multimodal 5 4.5 4 3.5 3 2.5 2 1.5 1 0.5 0 −0.5 −0.5 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 eNES is able to jump over deceptive local optima. Yi Sun, et al. (IDSIA) E¢cient Natural Evolution Strategies 17/06/2009 19 / 22

  70. Empirical Results - Multimodal 5 4.5 4 3.5 3 2.5 2 1.5 1 0.5 0 −0.5 −0.5 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 eNES is able to jump over deceptive local optima. Yi Sun, et al. (IDSIA) E¢cient Natural Evolution Strategies 17/06/2009 19 / 22

  71. Empirical Results - Multimodal 5 4.5 4 3.5 3 2.5 2 1.5 1 0.5 0 −0.5 −0.5 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 eNES is able to jump over deceptive local optima. Yi Sun, et al. (IDSIA) E¢cient Natural Evolution Strategies 17/06/2009 19 / 22

  72. Empirical Results - Multimodal 5 4.5 4 3.5 3 2.5 2 1.5 1 0.5 0 −0.5 −0.5 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 eNES is able to jump over deceptive local optima. Yi Sun, et al. (IDSIA) E¢cient Natural Evolution Strategies 17/06/2009 19 / 22

  73. Empirical Results - Multimodal 5 4.5 4 3.5 3 2.5 2 1.5 1 0.5 0 −0.5 −0.5 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 eNES is able to jump over deceptive local optima. Yi Sun, et al. (IDSIA) E¢cient Natural Evolution Strategies 17/06/2009 19 / 22

  74. Empirical Results - Multimodal 5 4.5 4 3.5 3 2.5 2 1.5 1 0.5 0 −0.5 −0.5 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 eNES is able to jump over deceptive local optima. Yi Sun, et al. (IDSIA) E¢cient Natural Evolution Strategies 17/06/2009 19 / 22

  75. Empirical Results - Double Pole Balancing β 1 β 2 F x Non-Markovian double pole balancing, average numbers of evaluations. Method SANE ESP NEAT CMA CoSyNE FEM NES Eval. 262 , 700 7 , 374 6 , 929 3 , 521 1 , 249 2 , 099 1 , 753 Yi Sun, et al. (IDSIA) E¢cient Natural Evolution Strategies 17/06/2009 20 / 22

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Adaptive Stochastic Natural Gradient Method for One-Shot Neural Architecture Search Youhei

Adaptive Stochastic Natural Gradient Method for One-Shot Neural Architecture Search Youhei

Adaptive Stochastic Natural Gradient Method for One-Shot Neural Architecture Search Youhei Akimoto (University of Tsukuba / RIKEN AIP) Shinichi Shirakawa (Yokohama National University) Nozomu Yoshinari (Yokohama National University) Kento

477 views • 6 slides
Painless Stochastic Gradient Descent : Interpolation, Line-Search, and Convergence Rates. NeurIPS

Painless Stochastic Gradient Descent : Interpolation, Line-Search, and Convergence Rates. NeurIPS

Painless Stochastic Gradient Descent : Interpolation, Line-Search, and Convergence Rates. NeurIPS 2019 Aaron Mishkin 1 21 Stochastic Gradient Descent: Workhorse of ML? Stochastic gradient descent (SGD) is today one of the main

1.22k views • 21 slides
Painless Stochastic Gradient Descent : Interpolation, Line-Search, and Convergence Rates. MLSS

Painless Stochastic Gradient Descent : Interpolation, Line-Search, and Convergence Rates. MLSS

Painless Stochastic Gradient Descent : Interpolation, Line-Search, and Convergence Rates. MLSS 2020 Aaron Mishkin, amishkin@cs.ubc.ca 1 21 Stochastic Gradient Descent: Workhorse of ML? Stochastic gradient descent (SGD) is today one of

634 views • 21 slides
Stochastic Gradient Descent (SGD) Todays Class Stochastic Gradient Descent (SGD) SGD Recap

Stochastic Gradient Descent (SGD) Todays Class Stochastic Gradient Descent (SGD) SGD Recap

CS6501: Deep Learning for Visual Recognition Stochastic Gradient Descent (SGD) Todays Class Stochastic Gradient Descent (SGD) SGD Recap Regression vs Classification Generalization / Overfitting / Underfitting Regularization

633 views • 30 slides
CSC2541 Lecture 5 Natural Gradient Roger Grosse Roger Grosse CSC2541 Lecture 5 Natural Gradient

CSC2541 Lecture 5 Natural Gradient Roger Grosse Roger Grosse CSC2541 Lecture 5 Natural Gradient

CSC2541 Lecture 5 Natural Gradient Roger Grosse Roger Grosse CSC2541 Lecture 5 Natural Gradient 1 / 12 Motivation Two classes of optimization procedures used throughout ML (stochastic) gradient descent, with momentum, and maybe

405 views • 12 slides
Overview of the Stochastic Gradient Method December 02, 2020 P. Carpentier Master Optimization

Overview of the Stochastic Gradient Method December 02, 2020 P. Carpentier Master Optimization

Stochastic Gradient Algorithm Connexion with Stochastic Approximation Asymptotic Efficiency and Averaging Practical Considerations Overview of the Stochastic Gradient Method December 02, 2020 P. Carpentier Master Optimization Stochastic

338 views • 33 slides
Stochastic Local Search Methods Dynamic Local Search Iterated Local Search Tabu Search Marco

Stochastic Local Search Methods Dynamic Local Search Iterated Local Search Tabu Search Marco

Outline DM811 HEURISTICS AND LOCAL SEARCH ALGORITHMS FOR COMBINATORIAL OPTIMZATION 1. Stochastic Local Search Methods (Metaheuristics) Randomized Iterative Improvement Lecture 12 Attribute Based Hill Climber Stochastic Local Search Methods

590 views • 9 slides
Applications of the Stochastic Gradient Method December 11, 2019 P. Carpentier Master

Applications of the Stochastic Gradient Method December 11, 2019 P. Carpentier Master

Two Exercices about Stochastic Gradient Option Pricing Problem and Variance Reduction Spatial Rendez-vous Under Probability Constraint Applications of the Stochastic Gradient Method December 11, 2019 P. Carpentier Master Optimization

819 views • 55 slides
Adaptive primal-dual stochastic gradient methods Yangyang Xu Mathematical Sciences, Rensselaer

Adaptive primal-dual stochastic gradient methods Yangyang Xu Mathematical Sciences, Rensselaer

Adaptive primal-dual stochastic gradient methods Yangyang Xu Mathematical Sciences, Rensselaer Polytechnic Institute October 26, 2019 1 / 22 Stochastic gradient method stochastic program: F ( x ; ) min x X f ( x ) = E N

959 views • 22 slides
Fitting Neural Networks Gradient Descent and Stochastic Gradient Descent CS109A Introduction to

Fitting Neural Networks Gradient Descent and Stochastic Gradient Descent CS109A Introduction to

Fitting Neural Networks Gradient Descent and Stochastic Gradient Descent CS109A Introduction to Data Science Pavlos Protopapas, Kevin Rader and Chris Tanner New requirement for the final project: For the first time ever, researchers who submit

619 views • 34 slides
On the steplength selection in Stochastic Gradient Methods Giorgia Franchini

On the steplength selection in Stochastic Gradient Methods Giorgia Franchini

Introduction Stochastic Gradient Methods and their properties A numerical experiment: the test problem Future developments On the steplength selection in Stochastic Gradient Methods Giorgia Franchini giorgia.franchini@unimore.it Universit

528 views • 19 slides
Exponential convergence of testing error for stochastic gradient methods Loucas Pillaud-Vivien

Exponential convergence of testing error for stochastic gradient methods Loucas Pillaud-Vivien

Exponential convergence of testing error for stochastic gradient methods Loucas Pillaud-Vivien INRIA - Ecole Normale Sup erieure, Paris, France Joint work with Alessandro Rudi and Francis Bach COLT - July 2018 1/10 Stochastic Gradient

376 views • 20 slides
Project: More Experiments on Stochastic Gradient Methods Last updated: May 25, 2020 May 25, 2020

Project: More Experiments on Stochastic Gradient Methods Last updated: May 25, 2020 May 25, 2020

Project: More Experiments on Stochastic Gradient Methods Last updated: May 25, 2020 May 25, 2020 1 / 17 Goal We want to know more the internal details of simpleNN We want to roughly compare the two stochastic gradient approaches: SG with

502 views • 17 slides
Stochastic Search and Diffusion Arturo Berrones Some basic physics: Newtons 2nd Law of

Stochastic Search and Diffusion Arturo Berrones Some basic physics: Newtons 2nd Law of

Stochastic Search and Diffusion Arturo Berrones Some basic physics: Newtons 2nd Law of motion: F = ma = m x The force F can be expressed as the gradient of a Born 4 January 1643 [OS: 25 December 1642] [1] Woolsthorpe-by-Colsterworth ,

375 views • 10 slides
Stochastic Gradient Method: Applications February 03, 2015 P. Carpentier Master MMMEF Cours

Stochastic Gradient Method: Applications February 03, 2015 P. Carpentier Master MMMEF Cours

Two Elementary Exercices on the Stochastic Gradient Option Pricing Problem and Variance Reduction Optimal Control Under Probability Constraint Stochastic Gradient Method: Applications February 03, 2015 P. Carpentier Master MMMEF Cours MNOS

702 views • 49 slides
Part 7.5 Stochastic Gradient Descent and Stochastic Newton 181 Wolfgang Bangerth Background

Part 7.5 Stochastic Gradient Descent and Stochastic Newton 181 Wolfgang Bangerth Background

Part 7.5 Stochastic Gradient Descent and Stochastic Newton 181 Wolfgang Bangerth Background In many practical applications, the objective function is a large sum: N f ( x ) = i = 1 f i ( x ) Issues and questions: Evaluating

381 views • 9 slides
NaturalLI: Natural Logic Inference for Common Sense Reasoning Angeli &amp; Manning (2014)

NaturalLI: Natural Logic Inference for Common Sense Reasoning Angeli &amp; Manning (2014)

NaturalLI: Natural Logic Inference for Common Sense Reasoning Angeli &amp; Manning (2014) (MacCartney 2007) Introduction: Motivation examples Natural Logic: Lexical Relations Monotonicity and Polarity Proof by alignment

899 views • 46 slides
CSE 473: Artificial Intelligence Autumn 2011 Search Luke Zettlemoyer Slides from Dan Klein,

CSE 473: Artificial Intelligence Autumn 2011 Search Luke Zettlemoyer Slides from Dan Klein,

CSE 473: Artificial Intelligence Autumn 2011 Search Luke Zettlemoyer Slides from Dan Klein, Stuart Russell, Andrew Moore Outline Agents that Plan Ahead Search Problems Uninformed Search Methods (part review for some)

616 views • 38 slides
The Hill We Must Die On: Cryptographers and Congress Shaanan Cohney Gabriel Kaptchuk University

The Hill We Must Die On: Cryptographers and Congress Shaanan Cohney Gabriel Kaptchuk University

The Hill We Must Die On: Cryptographers and Congress Shaanan Cohney Gabriel Kaptchuk University of Pennsylvania Johns Hopkins University January 9, 2019 1 / 32 Who Are We? Gabe Kaptchuk Shaanan Cohney 4 th year PhD Student at 5 th

676 views • 43 slides
Req equest uest for or Proposal oposal Tech echnical nical Ass ssis istance tance Web

Req equest uest for or Proposal oposal Tech echnical nical Ass ssis istance tance Web

Lang ngua uage ge Training aining Cen enter er Program ogram Req equest uest for or Proposal oposal Tech echnical nical Ass ssis istance tance Web ebinar nar Februar ruary 4, 2016 Webinar Participants Defen ense se La

229 views • 10 slides
Image Identification with Natural Language Specification Qi Feng, Donghyun Kim Department of

Image Identification with Natural Language Specification Qi Feng, Donghyun Kim Department of

Introduction Methods Results Saliency Map Image Identification with Natural Language Specification Qi Feng, Donghyun Kim Department of Computer Science, Boston University fung@bu.edu, donhk@bu.edu December 08, 2017 . . . . . . . .

165 views • 15 slides
Task-Oriented Query Reformulation with Reinforcement Learning Authors: Rodrigo Nogueira and

Task-Oriented Query Reformulation with Reinforcement Learning Authors: Rodrigo Nogueira and

Task-Oriented Query Reformulation with Reinforcement Learning Authors: Rodrigo Nogueira and Kyunghyun Cho Slides: Chris Benson Motivation Query: uiuc natural language processing class Search Engine Motivation Query: uiuc class ai

303 views • 27 slides
WELC LCOME ME TO JS JS101 Job Search ch Training Skills, Knowledge, and Information for the

WELC LCOME ME TO JS JS101 Job Search ch Training Skills, Knowledge, and Information for the

THE 7 HABITS WELC LCOME ME TO JS JS101 Job Search ch Training Skills, Knowledge, and Information for the Modern Job Seeker Based on the 7 Eleme ments of Job Search ch: Understand the Modern Job Search Personalize Your Plan

872 views • 13 slides
Stochastic Methods for Continuous Optimization Anne Auger and Dimo Brockhoff Paris-Saclay Master

Stochastic Methods for Continuous Optimization Anne Auger and Dimo Brockhoff Paris-Saclay Master

Stochastic Methods for Continuous Optimization Anne Auger and Dimo Brockhoff Paris-Saclay Master - Master 2 Informatique - Parcours Apprentissage, Information et Contenu (AIC) anne.auger@inria.fr 2015 Overview Problem Statement Black Box

1.2k views • 94 slides

More recommend