Convex Optimization ( EE227A: UC Berkeley ) Lecture 6 (Conic - PowerPoint PPT Presentation

Convex Optimization ( EE227A: UC Berkeley ) Lecture 6 (Conic optimization) 07 Feb, 2013 ◦ Suvrit Sra

Organizational Info ◮ Quiz coming up on 19th Feb. ◮ Project teams by 19th Feb ◮ Good if you can mix your research with class projects ◮ More info in a few days 2 / 31

Mini Challenge Kummer’s confluent hypergeometric function x j ( a ) j � M ( a, c, x ) := j ! , a, c, x ∈ R , ( c ) j j ≥ 0 and ( a ) 0 = 1 , ( a ) j = a ( a + 1) · · · ( a + j − 1) is the rising-factorial . 3 / 31

Mini Challenge Kummer’s confluent hypergeometric function x j ( a ) j � M ( a, c, x ) := j ! , a, c, x ∈ R , ( c ) j j ≥ 0 and ( a ) 0 = 1 , ( a ) j = a ( a + 1) · · · ( a + j − 1) is the rising-factorial . Claim: Let c > a > 0 and x ≥ 0 . Then the function h a,c ( µ ; x ) := µ �→ Γ( a + µ ) Γ( c + µ ) M ( a + µ, c + µ, x ) is strictly log-convex on [0 , ∞ ) ( note that h is a function of µ ). � ∞ 0 t x − 1 e − t dt is the Gamma function (which is Recall: Γ( x ) := known to be log-convex for x ≥ 1 ; see also Exercise 3.52 of BV). 3 / 31

LP formulation Write min � Ax − b � 1 as a linear program. x ∈ R n min � Ax − b � 1 � i | a T min i x − b i | � | a T min i t i , i x − b i | ≤ t i , i = 1 , . . . , m. x,t 1 T t, − t i ≤ a T i x − b i ≤ t i , min i = 1 , . . . , m. x,t 4 / 31

LP formulation Write min � Ax − b � 1 as a linear program. x ∈ R n min � Ax − b � 1 � i | a T min i x − b i | � | a T min i t i , i x − b i | ≤ t i , i = 1 , . . . , m. x,t 1 T t, − t i ≤ a T i x − b i ≤ t i , min i = 1 , . . . , m. x,t Exercise: Recast � Ax − b � 2 2 + λ � Bx � 1 as a QP. 4 / 31

Cone programs – overview ◮ Last time we briefly saw LP, QP, SOCP, SDP 5 / 31

Cone programs – overview ◮ Last time we briefly saw LP, QP, SOCP, SDP LP (standard form) f T x min s.t. Ax = b, x ≥ 0 . Feasible set X = { x | Ax = b } ∩ R n + (nonneg orthant) 5 / 31

Cone programs – overview ◮ Last time we briefly saw LP, QP, SOCP, SDP LP (standard form) f T x min s.t. Ax = b, x ≥ 0 . Feasible set X = { x | Ax = b } ∩ R n + (nonneg orthant) Input data: ( A, b, c ) Structural constraints: x ≥ 0 . 5 / 31

Cone programs – overview ◮ Last time we briefly saw LP, QP, SOCP, SDP LP (standard form) f T x min s.t. Ax = b, x ≥ 0 . Feasible set X = { x | Ax = b } ∩ R n + (nonneg orthant) Input data: ( A, b, c ) Structural constraints: x ≥ 0 . How should we generalize this model? 5 / 31

Cone programs – overview ◮ Replace linear map x �→ Ax by a nonlinear map? 6 / 31

Cone programs – overview ◮ Replace linear map x �→ Ax by a nonlinear map? ◮ Quickly becomes nonconvex, potentially intractable 6 / 31

Cone programs – overview ◮ Replace linear map x �→ Ax by a nonlinear map? ◮ Quickly becomes nonconvex, potentially intractable Generalize structural constraint R n + 6 / 31

Cone programs – overview ◮ Replace linear map x �→ Ax by a nonlinear map? ◮ Quickly becomes nonconvex, potentially intractable Generalize structural constraint R n + ♣ Replace nonneg orthant by a convex cone K ; 6 / 31

Cone programs – overview ◮ Replace linear map x �→ Ax by a nonlinear map? ◮ Quickly becomes nonconvex, potentially intractable Generalize structural constraint R n + ♣ Replace nonneg orthant by a convex cone K ; ♣ Replace ≥ by conic inequality � 6 / 31

Cone programs – overview ◮ Replace linear map x �→ Ax by a nonlinear map? ◮ Quickly becomes nonconvex, potentially intractable Generalize structural constraint R n + ♣ Replace nonneg orthant by a convex cone K ; ♣ Replace ≥ by conic inequality � ♣ Nesterov and Nemirovski developed nice theory in late 80s ♣ Rich class of cones for which cone programs are tractable 6 / 31

Conic inequalities ◮ We are looking for “good” vector inequalities � on R n 7 / 31

Conic inequalities ◮ We are looking for “good” vector inequalities � on R n ◮ Characterized by the set K := { x ∈ R n | x � 0 } of vector nonneg w.r.t. � x � y ⇔ x − y � 0 ⇔ x − y ∈ K . 7 / 31

Conic inequalities ◮ We are looking for “good” vector inequalities � on R n ◮ Characterized by the set K := { x ∈ R n | x � 0 } of vector nonneg w.r.t. � x � y ⇔ x − y � 0 ⇔ x − y ∈ K . ◮ Necessary and sufficient condition for a set K ⊂ R n to define a useful vector inequality � is: it should be a nonempty, pointed cone . 7 / 31

Cone programs – inequalities • K is nonempty: K � = ∅ • K is closed wrt addition: x, y ∈ K = ⇒ x + y ∈ K • K closed wrt noneg scaling: x ∈ K , α ≥ 0 = ⇒ αx ∈ K • K is pointed: x, − x ∈ K = ⇒ x = 0 8 / 31

Cone programs – inequalities • K is nonempty: K � = ∅ • K is closed wrt addition: x, y ∈ K = ⇒ x + y ∈ K • K closed wrt noneg scaling: x ∈ K , α ≥ 0 = ⇒ αx ∈ K • K is pointed: x, − x ∈ K = ⇒ x = 0 Cone inequality x � K y ⇐ ⇒ x − y ∈ K x ≻ K y ⇐ ⇒ x − y ∈ int ( K ) . 8 / 31

Conic inequalities ◮ Cone underlying standard coordinatewise vector inequalities: x ≥ y ⇔ x i ≥ y i ⇔ x i − y i ≥ 0 , is the nonegative orthant R n + . 9 / 31

Conic inequalities ◮ Cone underlying standard coordinatewise vector inequalities: x ≥ y ⇔ x i ≥ y i ⇔ x i − y i ≥ 0 , is the nonegative orthant R n + . ◮ Two more important properties that R n + has as a cone: x i ∈ R n � � ⇒ x ∈ R n It is closed → x = + + It has nonempty interior (contains Euclidean ball of positive radius) 9 / 31

Conic inequalities ◮ Cone underlying standard coordinatewise vector inequalities: x ≥ y ⇔ x i ≥ y i ⇔ x i − y i ≥ 0 , is the nonegative orthant R n + . ◮ Two more important properties that R n + has as a cone: x i ∈ R n � � ⇒ x ∈ R n It is closed → x = + + It has nonempty interior (contains Euclidean ball of positive radius) ◮ We’ll require our cones to also satisfy these two properties. 9 / 31

Conic optimization problems Standard form cone program f T x s.t. Ax = b, x ∈ K min f T x s.t. Ax � K b. min 10 / 31

Conic optimization problems Standard form cone program f T x s.t. Ax = b, x ∈ K min f T x s.t. Ax � K b. min ♣ The nonnegative orthant R n + ♣ The second order cone Q n := { ( x, t ) ∈ R n | � x � 2 ≤ t } X = X T � 0 ♣ The semidefinite cone: S n � � + := . 10 / 31

Conic optimization problems Standard form cone program f T x s.t. Ax = b, x ∈ K min f T x s.t. Ax � K b. min ♣ The nonnegative orthant R n + ♣ The second order cone Q n := { ( x, t ) ∈ R n | � x � 2 ≤ t } X = X T � 0 ♣ The semidefinite cone: S n � � + := . ♣ Other cones K given by Cartesian products of these 10 / 31

Conic optimization problems Standard form cone program f T x s.t. Ax = b, x ∈ K min f T x s.t. Ax � K b. min ♣ The nonnegative orthant R n + ♣ The second order cone Q n := { ( x, t ) ∈ R n | � x � 2 ≤ t } X = X T � 0 ♣ The semidefinite cone: S n � � + := . ♣ Other cones K given by Cartesian products of these ♣ These cones are “nice”: ♣ LP, QP, SOCP, SDP: all are cone programs 10 / 31

Conic optimization problems Standard form cone program f T x s.t. Ax = b, x ∈ K min f T x s.t. Ax � K b. min ♣ The nonnegative orthant R n + ♣ The second order cone Q n := { ( x, t ) ∈ R n | � x � 2 ≤ t } X = X T � 0 ♣ The semidefinite cone: S n � � + := . ♣ Other cones K given by Cartesian products of these ♣ These cones are “nice”: ♣ LP, QP, SOCP, SDP: all are cone programs ♣ Can treat them theoretically in a uniform way (roughly) 10 / 31

Conic optimization problems Standard form cone program f T x s.t. Ax = b, x ∈ K min f T x s.t. Ax � K b. min ♣ The nonnegative orthant R n + ♣ The second order cone Q n := { ( x, t ) ∈ R n | � x � 2 ≤ t } X = X T � 0 ♣ The semidefinite cone: S n � � + := . ♣ Other cones K given by Cartesian products of these ♣ These cones are “nice”: ♣ LP, QP, SOCP, SDP: all are cone programs ♣ Can treat them theoretically in a uniform way (roughly) ♣ Not all cones are nice! 10 / 31

Cone programs – tough case Copositive cone A ∈ S n × n | x T Ax ≥ 0 , ∀ x ≥ 0 � � Def. Let CP n := . Exercise: Verify that CP n is a convex cone. 11 / 31

Cone programs – tough case Copositive cone A ∈ S n × n | x T Ax ≥ 0 , ∀ x ≥ 0 � � Def. Let CP n := . Exercise: Verify that CP n is a convex cone. If someone told you convex is “easy” ... they lied! 11 / 31

Cone programs – tough case Copositive cone A ∈ S n × n | x T Ax ≥ 0 , ∀ x ≥ 0 � � Def. Let CP n := . Exercise: Verify that CP n is a convex cone. If someone told you convex is “easy” ... they lied! ◮ Testing membership in CP n is co-NP complete. (Deciding whether given matrix is not copositive is NP-complete.) ◮ Copositive cone programming: NP-Hard 11 / 31