Lecture 2 Convex Set CK Cheng Dept. of Computer Science and - PowerPoint PPT Presentation

CSE203B Convex Optimization Lecture 2 Convex Set CK Cheng Dept. of Computer Science and Engineering University of California, San Diego 1

Convex Optimization Problem: min 𝑦 𝑔 0 𝑦 Subject to 𝑔 𝑗 𝑦 ≤ 𝑐 𝑗 , 𝑗 = 1, ⋯ , 𝑛 1. 𝑔 0 𝑦 is a convex function 2. For 𝑔 𝑗 𝑦 ≤ 𝑐 𝑗 , 𝑗 = 1, … , 𝑛 𝑦|𝑔 𝑗 (𝑦) ≤ 𝑐 𝑗 , 𝑗 = 1, ⋯ , 𝑛 is a convex set 2

Convex Optimization Problem: A. Convex Function Definition: 𝑔 𝑗 𝛽𝑦 + 𝛾𝑧 ≤ 𝛽𝑔 𝑗 𝑦 + 𝛾𝑔 𝑗 𝑧 , ∀𝛽 + 𝛾 = 1, 𝛽, 𝛾 ≥ 0 B. Convex Set Definition: ∀𝑦, 𝑧 ∈ 𝐷 We have 𝛽𝑦 + 𝛾𝑧 ∈ 𝐷, ∀𝛽 + 𝛾 = 1, 𝛽, 𝛾 ≥ 0 3

Chapter 2 Convex Set 1. Set Convexity and Specification 1. Convexity 2. Implicit vs. Explicit Enumeration 2. Convex Set Terms and Definitions 3. Separating Hyperplanes 4. Dual Cones 4

1. Set Convexity and Specification: Convexity A set 𝑇 is convex if we have 𝛽𝑦 + 𝛾𝑧 ∈ 𝑇, ∀𝛽 + 𝛾 = 1, 𝛽, 𝛾 ≥ 0, ∀𝑦, 𝑧 ∈ 𝑇 Examples: 5

1. Set Convexity and Specification: Convexity A set 𝑇 is convex if we have 𝛽𝑦 + 𝛾𝑧 ∈ 𝑇, ∀𝛽 + 𝛾 = 1, 𝛽, 𝛾 ≥ 0, ∀𝑦, 𝑧 ∈ 𝑇 Remark: 1. Most used sets in the class 1. Scalar set: 𝑇 ⊂ 𝑆 2. Vector set: 𝑇 ⊂ 𝑆 𝑜 3. Matrix set: 𝑇 ⊂ 𝑆 𝑜×𝑛 2. Set S is convex if every two points in S has the connected straight segment in the set. 3. For convex sets 𝑇 1 and 𝑇 2 : 𝑇 1 ∩ 𝑇 2 is also convex 6

1. Set Convexity and Specification: Set Specification via Implicit or Explicit Enumeration 𝑇 𝐽 = {𝑦|𝐵𝑦 ≤ 𝑐, 𝑦 ∈ 𝑆 𝑜 } Implicit Expression Explicit Enumeration 𝑇 𝐹 = {𝐵𝑦 | 𝑦 ∈ 𝑆 𝑜 } Implicit Expression: Explicit Expression: Constraints Enumeration 𝑝 𝐵𝑦 , 𝑦 ∈ 𝑆 𝑜 Min 𝑔 𝑝 𝑦 Min 𝑔 Subject to 𝐵𝑦 ≤ 𝑐, 𝑦 ∈ 𝑆 𝑜 7

1. Implicit vs Explicit Enumeration of Convex Set Implicit Expression 𝑇 1 = {𝑦|𝐵𝑦 ≤ 𝑐} Example: { 𝑦 | 𝐵𝑦 ≤ 𝑐 } 𝑦 1 +2𝑦 2 +3𝑦 3 ≤ 4 Remark: Simultaneous linear 2𝑦 1 −𝑦 2 ≤ 3 constraints imply AND, 𝑦 2 +𝑦 3 ≤ 5 Intersection of the constraints 𝑦 3 ≤ 10 1 2 3 4 𝑦 1 2 −1 0 3 𝑦 2 𝐵 = , 𝑦 = , 𝑐 = 0 1 1 5 𝑦 3 0 0 1 10 8

1. Implicit vs Explicit Enumeration of Convex Set 𝑇 1 = {𝑦|𝐵𝑦 ≤ 𝑐, 𝑦 ∈ 𝑆 𝑜 } is a convex set Proof: Given two vectors 𝑣, 𝑤 ∈ 𝑇 1 , 𝑗. 𝑓. 𝐵𝑣 ≤ 𝑐 , 𝐵𝑤 ≤ 𝑐 For 𝑥 = 𝜄 1 𝑣 + 𝜄 2 𝑤, ∀𝜄 1 + 𝜄 2 = 1, 𝜄 1 , 𝜄 2 ≥ 0 we have 𝐵𝑥 ≤ 𝑐. ( 𝐵𝑥 = 𝜄 1 𝐵𝑣 + 𝜄 2 𝐵𝑤 ≤ 𝜄 1 𝑐 + 𝜄 2 𝑐 = 𝑐) The inequality implies 𝑥 ∈ 𝑇 1 By definition, set 𝑇 1 is convex. Remark: 1. Simultaneous linear constraints imply AND, Intersection of the constraints 2. Linear constraints constitute a convex set. 9

1. Specification of Convex Set: Implicit Expression Example: 𝑞 𝑦 (𝑢) ≤ 1 for 𝑢 ≤ 𝜌 𝑇 = 𝑦 ∈ 𝑆 𝑛 3 } 𝑥ℎ𝑓𝑠𝑓 𝑞 𝑦 𝑢 = 𝑦 1 cos 𝑢 + 𝑦 2 cos 2𝑢 + ⋯ + 𝑦 𝑛 cos 𝑛𝑢 10

1. Implicit vs Explicit Enumeration of Convex Set Example: 𝑇 2 = 𝑦 𝐵𝑦 ≥ 𝑐, 𝑦 ∈ 𝑆 𝑜 } 𝑇 3 = 𝑦 𝐵𝑦 = 𝑐, 𝑦 ∈ 𝑆 𝑜 } 11

1. Specification of Set: Explicit Expression Explicit Enumeration 𝑦 𝐵𝑦 ≤ 𝑐, 𝑦 ∈ 𝑆 𝑜 } 𝑤𝑡. 𝐵𝑦 𝑦 ∈ 𝑆 𝑜 } Example: 2 1 1 𝑐 = 𝐵 = 1 1 0 1 0 −1 12

1. Specification of Set: Explicit Expression Implicit and Explicit Conversion Example: x 𝐵𝑦 ≤ 𝑐, 𝑦 ∈ 𝑆 𝑜 } 𝑤𝑡 {𝑉𝜄| 1 𝑈 𝜄 = 1, 𝜄 ∈ 𝑆 + 𝑛 } 𝜄 1 1 1 2 𝜄 2 1 1 −1 −1 𝑦 1 1 0 1 𝑦 2 ≤ 𝜄 3 1 −1 −1 3 −1 0 1 𝜄 4 0 −1 1 13

1. Implicit vs Explicit Enumeration of Convex Set Remark: Implicit Expression: Constraints of the problem formulation Explicit Enumeration: Formulation of the objective function The interchange may not be trivial. min 𝑔 0 (𝑉𝜄) min 𝑔 0 (𝑦) 𝑡. 𝑢. 𝐽 𝑈 𝜄 ≤ 1 𝑡. 𝑢. 𝐵𝑦 ≤ 𝑐 𝑛 𝑦 ∈ 𝑆 𝑜 𝜄 ∈ 𝑆 + Every vector 𝑣 𝑗 in matrix 𝑉 is a solution of n equations in constraint 𝐵𝑦 ≤ 𝑐 𝑞 𝑓𝑟𝑣𝑏𝑢𝑗𝑝𝑜𝑡 𝑑𝑝𝑛𝑐 𝑞, 𝑜 𝑞𝑝𝑡𝑡𝑗𝑐𝑚𝑓 𝑜 𝑤𝑏𝑠𝑗𝑏𝑐𝑚𝑓𝑡 𝑤𝑓𝑠𝑢𝑓𝑦 𝑞𝑝𝑗𝑜𝑢𝑡. 14

1. Implicit vs Explicit Enumeration of Convex Set Explicit Enumeration 𝐵𝑦 + 𝑐 𝑑 𝑈 𝑦 + 𝑒 > 0, 𝑦 ∈ 𝐷 4 } (Projective Function) 𝑇 4 = 𝑑 𝑈 𝑦 + 𝑒 𝑨 𝑢 𝑨 ∈ 𝑆 𝑜 , 𝑢 > 0, 𝑨, 𝑢 ∈ 𝐷 5 } (Perspective Function) 𝑇 5 = 𝑇 4 is convex if 𝐷 4 is convex 𝑇 5 is convex if 𝐷 5 is convex 15

1. Implicit vs Explicit Enumeration of Convex Set Statement: 𝑇 5 𝑗𝑡 𝑑𝑝𝑜𝑤𝑓𝑦 . Proof: 𝑨 1 𝑨 2 Given ∈ 𝑇 5 , ∈ 𝑇 5 , 𝑢 1 𝑢 2 Let us set 𝑨 3 = 𝛽𝑨 1 + β𝑨 2 , 𝑢 3 = 𝛽𝑢 1 + 𝛾𝑢 2 , ∀𝛽 + 𝛾 = 1, 𝛽, 𝛾 ≥ 0 We then have 𝑨 3 = 𝛽𝑨 1 + β𝑨 2 𝛽𝑢 1 𝑨 1 𝛾𝑢 2 𝑨 2 = + 𝑢 3 𝛽𝑢 1 + β𝑢 2 𝛽𝑢 1 + β𝑢 2 𝑢 1 𝛽𝑢 1 + β𝑢 2 𝑢 2 𝛽𝑢 1 𝛾𝑢 2 Let 𝛽 ′ = , 𝛾 ′ = 𝛽𝑢 1 + β𝑢 2 𝛽𝑢 1 + β𝑢 2 (Note that 𝛽 ′ + 𝛾 ′ = 1, 𝛽 ′ , 𝛾 ′ ≥ 0), we have 𝑨 3 = 𝛽 ′ 𝑨 1 + 𝛾 ′ 𝑨 2 ∈ 𝑇 5 𝑢 3 𝑢 1 𝑢 2 Therefore, by definition 𝑇 5 is convex . 16

2. Convex Set Terms and Definitions Definitions: Affine Set, Cone, and Convex Hull Given 𝑣 1 , 𝑣 2 , ⋯ , 𝑣 𝑙 ∈ 𝑆 𝑜 , function 𝑔 𝑣, 𝜄 = 𝜄 1 𝑣 1 + 𝜄 2 𝑣 2 + ⋯ + 𝜄 𝑙 𝑣 𝑙 and two conditions 1. 𝜄 1 +𝜄 2 + ⋯ + 𝜄 𝑙 = 1 2. 𝜄 𝑗 ≥ 0 ∀𝑗 i. f u, θ condition 1}: Affine set ii. f u, θ condition 2}: Cone iii. f u, θ conditions 1 and 2}: Convex hull 𝐹𝑦1: 𝜄 1 𝑣 1 + 𝜄 2 𝑣 2 = 𝑣 1 + 𝜄 2 (𝑣 2 − 𝑣 1 ) 𝐹𝑦2: 𝜄 1 𝑣 1 + 𝜄 2 𝑣 2 + 𝜄 3 𝑣 3 17

2. Sets and Definitions Definitions: Hyperplane and Half Spaces 𝑦 𝑏 𝑈 𝑦 = 𝑐}, 𝑏 ∈ 𝑆 𝑜 , 𝑐 ∈ 𝑆 Hyperplane 𝑦 𝑏 𝑈 (𝑦 − 𝑦 0 ) = 0}, for any 𝑦 0 ∈ 𝑆 𝑜 , 𝑏 ∈ 𝑆 𝑜 , 𝑐 ∈ 𝑆 or 𝑦 𝑏 𝑈 𝑦 ≤ 𝑐} 𝑏 ∈ 𝑆 𝑜 , 𝑐 ∈ 𝑆 Half Space 𝑦 𝑏 𝑈 (𝑦 − 𝑦 0 ) ≤ 0} or 𝑦 1 𝑦 2 − 0.5 𝐹𝑦: 𝑦 𝑦 1 + 𝑦 2 = 1} 𝑝𝑠 𝑦 [1,1] = 0} 0.5 𝑔 𝑦 1 , 𝑦 2 = 𝑦 1 + 𝑦 2 − 1 𝑦 𝑏 𝑈 (𝑦 − 𝑦 0 ) ≤ 0}, 𝑏 𝑈 = [1,1], 𝑐 = 1, 𝑦 0 = [2, −1] 𝑏 𝑈 𝑐 normalize the expression: 𝑏 2 𝑦 = 𝑏 2 18

2. Sets and Definitions: Hyperplanes Ex : 3 variables 𝑦|𝑏 𝑈 𝑦 = 𝑐 , 𝑏 𝑈 = 1,1,1 , 𝑐 = 6 Ex : 4 variables 𝑦|𝑏 𝑈 𝑦 = 𝑐 , 𝑏 𝑈 = 1,1,1,1 , 𝑐 = 6 (1) degrees of freedom : 𝑜 − 1 𝑆 𝑜 . (2) all vectors ( 𝑦 − 𝑧) are orthogonal to direction 𝑏 , i.e. 𝑏 𝑈 𝑦 − 𝑧 = 0, ∀𝑦, 𝑧 in the hyperplane Proof: Exercise: Conceptually (visually) construct hyperplane. 19

2. Sets and Definitions: Hyperplanes Hyperplane : as an Equal potential of cost function 0 𝑦 = 𝑑 𝑈 𝑦 min𝑔 𝑦 1 𝑓. 𝑕. 1, 2 𝑦 2 𝜖𝑔 0 𝑦 𝜖𝑦 1 = 1 𝜖𝑔 0 𝑦 𝜖𝑦 2 = 2 Vector 𝑑 is the sensitivity or cost of vector 𝑦 1 𝑦 2 20

2. Sets and Definitions Hyperplane : as a linearized constraint 𝑏 𝑈 𝑦 ≤ 𝑐 𝑦 1 𝑓. 𝑕. 1, 2 𝑦 2 ≤ 10 Remark : • Hyperplane is one key building block of convex optimization. (theory, algorithms, applications for machine learning, deep learning, …) • Each hyperplane separates the space into two half spaces. • If 𝑜 ≥ 𝑞, 𝑞 hyperplanes can separate the space into 2 𝑞 disjoint regions. 21

2. Sets and Definitions Ⅴ. Polyhedra (plural) : Poly (many) Hedron (face) 𝑄 = 𝑦 𝐵𝑦 ≤ 𝑐, 𝐷𝑦 = 𝑒} 𝑈 𝑈 𝑑 1 𝑏 1 𝑈 𝑈 𝑑 2 𝑏 2 𝐵 = C = … … 𝑈 𝑈 𝑑 𝑞 𝑏 𝑛 22

2. Sets and Definitions ⅤI. Matrix Space : Positive Semidefinite Cone ① 𝑇 𝑜 = 𝑌 ∈ 𝑆 𝑜⨉𝑜 𝑌 = 𝑌 𝑈 } Symmetric Matrix 𝑜 = 𝑌 ∈ 𝑇 𝑜 𝑌 ⪰ 0} 𝑗. 𝑓. 𝑧 𝑈 𝑌𝑧 ≥ 0, ∀𝑧 ② 𝑇 + = 𝑌 ∈ 𝑇 𝑜 𝑌 ≻ 0} 𝑜 𝑗. 𝑓. 𝑧 𝑈 𝑌𝑧 > 0, ∀𝑧 ≠ 0 𝑇 ++ Ex: 𝑌 = 𝑦 𝑧 2 ⇔ 𝑦 ≥ 0, 𝑨 ≥ 0, 𝑦𝑨 ≥ 𝑧 2 𝑨 ∈ 𝑇 + 𝑧 [𝑏 𝑐]𝑌 𝑏 𝑐 = 𝑏 2 𝑦 + 𝑐 2 𝑨 + 2𝑏𝑐𝑧 ≥ 0, ∀𝑏, 𝑐 ∈ ℝ 23

2. Sets and Definitions 𝑦 𝑧 1 0 −𝑦 −1 𝑧 = 𝑦 0 1 Proof : −𝑦 −1 𝑧 𝑨 − 𝑦 −1 𝑧 2 𝑧 𝑨 1 0 0 1 −𝑦 −1 𝑧 Let 𝑆 = 1 0 1 We have 𝑏 𝑐 𝑌 𝑏 𝑐 = [𝑏 𝑐]𝑆 −𝑈 𝑆 𝑈 𝑌𝑆𝑆 −1 𝑏 𝑐 = 𝑏 𝑐 𝑆 −𝑈 𝑦 𝑨 − 𝑦 −1 𝑧 2 𝑆 −1 𝑏 0 0 𝑐 24