. . November 15th, 2012 Biostatistics 615/815 - Lecture 18 Hyun Min Kang November 15th, 2012 Hyun Min Kang Multi-dimensional optimization Biostatistics 615/815 Lecture 18: . . Summary Mixture . Implementation Simplex Mixture . . . . . . . . 1 / 46 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. Implementation November 15th, 2012 Biostatistics 615/815 - Lecture 18 Hyun Min Kang Multidimensional Optimization : A mixture distribution Summary . Mixture 2 / 46 . Simplex Mixture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. Mixture November 15th, 2012 Biostatistics 615/815 - Lecture 18 Hyun Min Kang k number of mixture components f the probability density function x observed data k . Summary . A general mixture distribution 3 / 46 Implementation . . . . . . . . Mixture Simplex . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ∑ p ( x ; π, φ, η ) = π i f ( x ; φ i , η ) i =1 π mixture proportion of each component φ parameters specific to each component η parameters shared among components
. . November 15th, 2012 Biostatistics 615/815 - Lecture 18 Hyun Min Kang i Or equivalently, consider log-likelihood to avoid underflow . . Calculating in log-space . i Find parameters that maximizes the likelihood of the entire sample . . Finding Maximum-likelihood . Problem : Maximum Likelihood Estimation Simplex . . . . . . . . Mixture Summary 4 / 46 Implementation Mixture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ∏ L = p ( x i | π, φ, η ) ∑ log p ( x i | π, φ, η ) l =
• Analytical solution does exist x i n . . November 15th, 2012 Biostatistics 615/815 - Lecture 18 Hyun Min Kang n x i i n • i n • Gaussian MLE in single-dimensional space Summary Mixture . . . . . . . . Simplex Implementation Mixture . 5 / 46 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . p ( x ; µ, σ 2 ) = N ( x ; µ, σ 2 ) Given x , what is the MLE parameters of µ and σ 2 ?
. Implementation November 15th, 2012 Biostatistics 615/815 - Lecture 18 Hyun Min Kang Gaussian MLE in single-dimensional space . . Mixture Summary 5 / 46 Simplex . Mixture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . p ( x ; µ, σ 2 ) = N ( x ; µ, σ 2 ) Given x , what is the MLE parameters of µ and σ 2 ? • Analytical solution does exist • ˆ µ = ∑ n i =1 x i / n σ 2 = ∑ n µ ) 2 / n • ˆ i =1 ( x i − ˆ
. MLE in Gaussian mixture November 15th, 2012 Biostatistics 615/815 - Lecture 18 Hyun Min Kang . . Possible approaches . k . . Parameter estimation in Gaussian mixture . . Summary . . . . . . . . . Mixture Simplex 6 / 46 Implementation Mixture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . • No analytical solution • Numerical optimization required • Multi-dimensional optimization problem • µ 1 , µ 2 , · · · , µ k • σ 2 1 , σ 2 2 , · · · , σ 2 • Simplex Method • Expectation Maximization • Markov-Chain Monte Carlo
. Implementation November 15th, 2012 Biostatistics 615/815 - Lecture 18 Hyun Min Kang The Simplex Method Summary . Mixture . 7 / 46 Simplex . Mixture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . • Calculate likelihoods at simplex vertexes • Geometric shape with k + 1 corners • A triangle in k = 2 dimensions • Simplex crawls • Towards minimum • Away from maximum • Probably the most widely used optimization method
. Implementation November 15th, 2012 Biostatistics 615/815 - Lecture 18 Hyun Min Kang How the Simplex Method Works Summary . Mixture 8 / 46 . Simplex Mixture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. Implementation November 15th, 2012 Biostatistics 615/815 - Lecture 18 Hyun Min Kang Simplex Method in Two Dimensions Summary . . Mixture 9 / 46 . . Mixture . . . . . . Simplex . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . • Evaluate functions at three vertexes • The highest (worst) point • The next highest point • The lowest (best) point • Intuition • Move away from high point, towards low point
. Implementation November 15th, 2012 Biostatistics 615/815 - Lecture 18 Hyun Min Kang Direction for Optimization Summary . Mixture 10 / 46 . Simplex Mixture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. Implementation November 15th, 2012 Biostatistics 615/815 - Lecture 18 Hyun Min Kang Reflection Summary . Mixture 11 / 46 . Simplex Mixture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. Implementation November 15th, 2012 Biostatistics 615/815 - Lecture 18 Hyun Min Kang Reflection and Expansion Summary . Mixture 12 / 46 . Simplex Mixture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. Implementation November 15th, 2012 Biostatistics 615/815 - Lecture 18 Hyun Min Kang Contraction (1-dimension) Summary . Mixture 13 / 46 . Simplex Mixture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. Implementation November 15th, 2012 Biostatistics 615/815 - Lecture 18 Hyun Min Kang Contact-ion Summary . Mixture 14 / 46 . Simplex Mixture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. Implementation November 15th, 2012 Biostatistics 615/815 - Lecture 18 Hyun Min Kang Summary : The Simplex Method Summary . Mixture 15 / 46 . Simplex Mixture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. Implementation November 15th, 2012 Biostatistics 615/815 - Lecture 18 Hyun Min Kang . Implementing the Simplex Method Summary . Mixture 16 / 46 . . Simplex Mixture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . template <class F> // F is a function object class simplex615 { // contains (dim+1) points of size (dim) protected: std::vector<std::vector<double> > X; // (dim+1)*dim matrix std::vector<double> Y; // (dim+1) vector std::vector<double> midPoint; // variables for update std::vector<double> thruLine; // variables for update int dim, idxLo, idxHi, idxNextHi; // dimension, min, max, 2ndmax values void evaluateFunction(F& foo); // evaluate function value at each point void evaluateExtremes(); // determine the min, max, 2ndmax void prepareUpdate(); // calculate midPoint, thruLine bool updateSimplex(F& foo, double scale); // for reflection/expansion.. void contractSimplex(F& foo); // for multiple contraction static int check_tol(double fmax, double fmin, double ftol); // check tolerance public: simplex615(double* p, int d); // constructor with initial points void amoeba(F& foo, double tol); // main function for optimization std::vector<double>& xmin(); // optimal x value double ymin(); // optimal y value };
• Reflection, Expansion and Contraction After calculating midPoint and thruLine Reflection Call updateSimplex(foo, -1.0) Expansion Call updateSimplex(foo, -2.0) Contraction Call updateSimplex(foo, 0.5) . . November 15th, 2012 Biostatistics 615/815 - Lecture 18 Hyun Min Kang Implementation overview . Summary Mixture Mixture . . . . . . . . Simplex Implementation 17 / 46 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . • Data representation • Each X[i] is point of the simplex • Y[i] corresponds to f ( X [ i ]) • midPoint is the average of all points (except for the worst point) • thruLine is vector from the worse point to the midPoint
. Implementation November 15th, 2012 Biostatistics 615/815 - Lecture 18 Hyun Min Kang Implementation overview . . Mixture Summary 17 / 46 Simplex . Mixture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . • Data representation • Each X[i] is point of the simplex • Y[i] corresponds to f ( X [ i ]) • midPoint is the average of all points (except for the worst point) • thruLine is vector from the worse point to the midPoint • Reflection, Expansion and Contraction After calculating midPoint and thruLine Reflection Call updateSimplex(foo, -1.0) Expansion Call updateSimplex(foo, -2.0) Contraction Call updateSimplex(foo, 0.5)
Recommend
More recommend
