MORE CONVEX OPTIMIZATION MORE CONVEX OPTIMIZATION Matthieu R Bloch Thursday, February 20, 2020 1
LOGISTICS LOGISTICS TAs and Office hours Tuesday: TJ (VL C449 Cubicle D) - 1:30pm - 2:45pm Wednesday: Matthieu (TSRB 423) - 12:00:pm-1:15pm Thursday: Hossein (VL C449 Cubicle B): 10:45pm - 12:00pm Friday: Brighton (TSRB 523a) - 12pm-1:15pm Homework 3 Due Wednesday February 19, 2020 11:59pm EST (Friday February 126, 2020 for DL) Please submit a standalone PDF with your solution, including plots and listings Please also submit your code (we check it randomly) Make sure you show your work, don’t leave gaps in logic 2
PROJECTS PROJECTS Form teams of 3-5 and report on the shared spreadsheet Link on canvas Topic is up to you! You can explore papers or play with scikit Not need to invent new things, but I want to see some depth Consult with me if you have ideas. It’s ok to use something related to your research Timeline Teams formed by Feb 23, 2020 Project proposal March 8 2020 Poster presentation April 14 and 16, 2020 Final report April 24, 2020 3
RECAP: CONSTRAINED OPTIMIZATION RECAP: CONSTRAINED OPTIMIZATION General form of constrained optimization problem g i ( x ) ≤ 0 ∀ i ∈ [1; m ] min f ( x ) such that { x ∈ R d h j ( x ) = 0 ∀ j ∈ [1; p ] is called the objective function f is called an inequality constraint g i ( x ) is called an equality constraint h j ( x ) if satisfies all the constraints, we say that it is feasible . x Definition. (Convex problem) A constrained optimization problem is convex if is convex, the ’s are convex, and the ’s are affine f g i h j Turn constrained optimization into unconstrained one using Lagrangian p m L ( x , λ , μ ) ≜ f ( x ) + ∑ λ i g i ( x ) + ∑ μ j h j ( x ) with λ ≥ 0 i =1 j =1 and are dual variables or Lagrange multipliers λ m ] ⊺ μ p ] ⊺ λ ≜ [ λ 1 , ⋯ , μ ≜ [ μ 1 , ⋯ , 4
RECAP PRIMAL AND DUAL PROBLEMS RECAP PRIMAL AND DUAL PROBLEMS The Lagrange dual function is L D ( λ , μ ) = min L ( x , λ , μ ) x The dual optimization problem is max L ( λ , μ ) = λ ≥ 0 , μ min max L ( x , λ , μ ) x λ ≥ 0 , μ . The primal function is L P ( x ) ≜ max L ( x , λ , μ ) λ ≥ 0 , μ The primal optimization problem is min x L P ( x ) = min max L ( x , λ , μ ) x λ ≥ 0 , μ Lemma. The (Lagrangian) dual function is concave. Lemma. Let denote the set of feasible values of the original problem. Then . F min x L P ( x ) = min x ∈ F f ( x ) 5
6
WEAK DUALITY WEAK DUALITY Theorem (Weak duality) d ∗ p ∗ ≜ λ ≥ 0 , μ min max L ( x , λ , μ ) ≤ ≜ min max L ( x , λ , μ ) x x λ ≥ 0 , μ We can solve the dual (concave maximization problem) and obtain a lower bound for the primal The gap is called the duality gap p ∗ d ∗ − ≥ 0 Strong duality is the situation in which p ∗ d ∗ − = 0 Question: how are the solutions of the primal and dual problems related? 7
8
9
KARUSH-KUHN TUCKER CONDITIONS KARUSH-KUHN TUCKER CONDITIONS Assume , , are all differentiable f { } g i { h j } Consider x , λ , μ Stationarity: p m 0 = ∇ f ( x ) + ∑ λ i ∇ ( x ) + g i ∑ μ j ∇ h j ( x ) i =1 j =1 Primal feasibility: ∀ i ∈ [1; m ] g i ( x ) ≤ 0 ∀ j ∈ [1; p ] h j ( x ) = 0 Dual feasibility: ∀ i ∈ [1; m ] λ i ≥ 0 Complementary slackness: ∀ i ∈ [1; m ] λ i g i ( x ) = 0 10
KKT CONDITIONS: NECESSITY AND SUFFICIENCY KKT CONDITIONS: NECESSITY AND SUFFICIENCY Theorem (KKT necessity) If and are primal and dual solutions with zero duality gap, then and satisfy x ∗ λ ∗ μ ∗ x ∗ λ ∗ μ ∗ ( , ) ( , ) the KKT conditions. Theorem (KKT sufficiency) If the original problem is convex and and satisfy the KKT conditions, then is primal optimal, ~ μ ~ ~ ~ x ( , λ ) x is dual optimal, and the duality gap is zero. ~ μ ~ ( , λ ) If a constrained optimization problem is differentiable and convex KKT conditions are necessary and sufficient for primal/dual optimality (with zero duality gap) we can use the KKT conditions to find a solution to our optimization problem We’re in luck: the optimal so�-margin hyperplane falls in this category! 11
12
13
Recommend
More recommend
