Lecture 2 Convex Set CK Cheng Dept. of Computer Science and - - PowerPoint PPT Presentation

1

CSE203B Convex Optimization Lecture 2 Convex Set

CK Cheng

• Dept. of Computer Science and Engineering

University of California, San Diego

Convex Optimization Problem:

2

• 1. 𝑔

0 𝑦 is a convex function

• 2. For 𝑔

𝑗 𝑦 ≤ 𝑐𝑗, 𝑗 = 1, … , 𝑛

min

𝑦 𝑔 0 𝑦

Subject to 𝑔

𝑗 𝑦 ≤ 𝑐𝑗 , 𝑗 = 1, ⋯ , 𝑛

𝑦|𝑔

𝑗(𝑦) ≤ 𝑐𝑗, 𝑗 = 1, ⋯ , 𝑛 is a convex set

Convex Optimization Problem:

3

• A. Convex Function Definition:

𝑔

𝑗 𝛽𝑦 + 𝛾𝑧 ≤ 𝛽𝑔 𝑗 𝑦 + 𝛾𝑔 𝑗 𝑧 , ∀𝛽 + 𝛾 = 1, 𝛽, 𝛾 ≥ 0

• B. Convex Set Definition: ∀𝑦, 𝑧 ∈ 𝐷

We have 𝛽𝑦 + 𝛾𝑧 ∈ 𝐷, ∀𝛽 + 𝛾 = 1, 𝛽, 𝛾 ≥ 0

Chapter 2 Convex Set

• 1. Set Convexity and Specification
• 1. Convexity
• 2. Implicit vs. Explicit Enumeration
• 2. Convex Set Terms and Definitions
• 3. Separating Hyperplanes
• 4. Dual Cones

4

• 1. Set Convexity and Specification: Convexity

5

A set 𝑇 is convex if we have 𝛽𝑦 + 𝛾𝑧 ∈ 𝑇, ∀𝛽 + 𝛾 = 1, 𝛽, 𝛾 ≥ 0, ∀𝑦, 𝑧 ∈ 𝑇 Examples:

• 1. Set Convexity and Specification: Convexity

6

A set 𝑇 is convex if we have 𝛽𝑦 + 𝛾𝑧 ∈ 𝑇, ∀𝛽 + 𝛾 = 1, 𝛽, 𝛾 ≥ 0, ∀𝑦, 𝑧 ∈ 𝑇 Remark:

• 1. Most used sets in the class
• 1. Scalar set: 𝑇 ⊂ 𝑆
• 2. Vector set: 𝑇 ⊂ 𝑆𝑜
• 3. Matrix set: 𝑇 ⊂ 𝑆𝑜×𝑛
• 2. Set S is convex if every two points in S has the

connected straight segment in the set.

• 3. For convex sets 𝑇1 and 𝑇2: 𝑇1 ∩ 𝑇2 is also convex

• 1. Set Convexity and Specification:

Set Specification via Implicit or Explicit Enumeration

Implicit Expression Explicit Enumeration

7

𝑇𝐽 = {𝑦|𝐵𝑦 ≤ 𝑐, 𝑦 ∈ 𝑆𝑜} 𝑇𝐹 = {𝐵𝑦 | 𝑦 ∈ 𝑆𝑜}

Implicit Expression: Constraints Min 𝑔

𝑝 𝑦

Subject to 𝐵𝑦 ≤ 𝑐, 𝑦 ∈ 𝑆𝑜 Explicit Expression: Enumeration Min 𝑔

𝑝 𝐵𝑦 , 𝑦 ∈ 𝑆𝑜

• 1. Implicit vs Explicit Enumeration of Convex Set

Implicit Expression

8

𝑇1 = {𝑦|𝐵𝑦 ≤ 𝑐} Example: {𝑦|𝐵𝑦≤𝑐}

𝑦1 +2𝑦2 +3𝑦3 ≤ 4 2𝑦1 −𝑦2 ≤ 3 𝑦2 +𝑦3 ≤ 5 𝑦3 ≤ 10 𝐵 = 1 2 3 2 −1 1 1 1 , 𝑦 = 𝑦1 𝑦2 𝑦3 , 𝑐 = 4 3 5 10 Remark: Simultaneous linear constraints imply AND, Intersection of the constraints

𝑇1 = {𝑦|𝐵𝑦 ≤ 𝑐, 𝑦 ∈ 𝑆𝑜} is a convex set Proof:

9

Given two vectors 𝑣, 𝑤 ∈ 𝑇1, 𝑗. 𝑓. 𝐵𝑣 ≤ 𝑐, 𝐵𝑤 ≤ 𝑐 For 𝑥 = 𝜄1𝑣 + 𝜄2𝑤, ∀𝜄1 + 𝜄2 = 1, 𝜄1, 𝜄2 ≥ 0 we have 𝐵𝑥 ≤ 𝑐. (𝐵𝑥 = 𝜄1𝐵𝑣 + 𝜄2𝐵𝑤 ≤ 𝜄1𝑐 + 𝜄2𝑐 = 𝑐) The inequality implies 𝑥 ∈ 𝑇1 By definition, set 𝑇1is convex. Remark:

• 1. Simultaneous linear constraints imply AND,

Intersection of the constraints

• 2. Linear constraints constitute a convex set.
• 1. Implicit vs Explicit Enumeration of Convex Set

• 1. Specification of Convex Set: Implicit Expression

Example:

10

𝑇 = 𝑦 ∈ 𝑆𝑛 𝑞𝑦(𝑢) ≤ 1 for 𝑢 ≤ 𝜌 3} 𝑥ℎ𝑓𝑠𝑓 𝑞𝑦 𝑢 = 𝑦1 cos 𝑢 + 𝑦2 cos 2𝑢 + ⋯ + 𝑦𝑛 cos 𝑛𝑢

• 1. Implicit vs Explicit Enumeration of Convex Set

Example:

11

𝑇2 = 𝑦 𝐵𝑦 ≥ 𝑐, 𝑦 ∈ 𝑆𝑜} 𝑇3 = 𝑦 𝐵𝑦 = 𝑐, 𝑦 ∈ 𝑆𝑜}

• 1. Specification of Set: Explicit Expression

Explicit Enumeration

12

𝑦 𝐵𝑦 ≤ 𝑐, 𝑦 ∈ 𝑆𝑜} 𝑤𝑡. 𝐵𝑦 𝑦 ∈ 𝑆𝑜} Example:

𝐵 = 1 1 1 −1 𝑐 = 2 1 1

• 1. Specification of Set: Explicit Expression

Implicit and Explicit Conversion

13

Example: x 𝐵𝑦 ≤ 𝑐, 𝑦 ∈ 𝑆𝑜} 𝑤𝑡 {𝑉𝜄| 1𝑈𝜄 = 1, 𝜄 ∈ 𝑆+

𝑛}

1 1 1 −1 −1 𝑦1 𝑦2 ≤ 2 1 1 1 1 1 −1 −1 1 −1 −1 3 𝜄1 𝜄2 𝜄3 𝜄4

• 1. Implicit vs Explicit Enumeration of Convex Set

14

Remark: Implicit Expression: Constraints of the problem formulation Explicit Enumeration: Formulation of the objective function The interchange may not be trivial. 𝑞 𝑓𝑟𝑣𝑏𝑢𝑗𝑝𝑜𝑡 𝑜 𝑤𝑏𝑠𝑗𝑏𝑐𝑚𝑓𝑡 𝑑𝑝𝑛𝑐 𝑞, 𝑜 𝑞𝑝𝑡𝑡𝑗𝑐𝑚𝑓 𝑤𝑓𝑠𝑢𝑓𝑦 𝑞𝑝𝑗𝑜𝑢𝑡. min 𝑔

0(𝑦)

𝑡. 𝑢. 𝐵𝑦 ≤ 𝑐 𝑦 ∈ 𝑆𝑜 min 𝑔

0(𝑉𝜄)

𝑡. 𝑢. 𝐽𝑈𝜄 ≤ 1 𝜄 ∈ 𝑆+

𝑛

Every vector 𝑣𝑗 in matrix 𝑉 is a solution of n equations in constraint 𝐵𝑦 ≤ 𝑐

• 1. Implicit vs Explicit Enumeration of Convex Set

Explicit Enumeration

15

𝑇4 = 𝐵𝑦 + 𝑐 𝑑𝑈𝑦 + 𝑒 𝑑𝑈𝑦 + 𝑒 > 0, 𝑦 ∈ 𝐷4} (Projective Function) 𝑇5 = 𝑨 𝑢 𝑨 ∈ 𝑆𝑜, 𝑢 > 0, 𝑨, 𝑢 ∈ 𝐷5} (Perspective Function) 𝑇4 is convex if 𝐷4 is convex 𝑇5 is convex if 𝐷5 is convex

• 1. Implicit vs Explicit Enumeration of Convex Set

Statement: 𝑇5 𝑗𝑡 𝑑𝑝𝑜𝑤𝑓𝑦. Proof:

16

Given

𝑨1 𝑢1

∈ 𝑇5,

𝑨2 𝑢2

∈ 𝑇5, Let us set 𝑨3 = 𝛽𝑨1 + β𝑨2, 𝑢3 = 𝛽𝑢1 + 𝛾𝑢2, ∀𝛽 + 𝛾 = 1, 𝛽, 𝛾 ≥ 0 𝑨3 𝑢3 = 𝛽𝑨1 + β𝑨2 𝛽𝑢1 + β𝑢2 = 𝛽𝑢1 𝛽𝑢1 + β𝑢2 𝑨1 𝑢1 + 𝛾𝑢2 𝛽𝑢1 + β𝑢2 𝑨2 𝑢2 Let 𝛽′ = 𝛽𝑢1 𝛽𝑢1 + β𝑢2 , 𝛾′ = 𝛾𝑢2 𝛽𝑢1 + β𝑢2 (Note that 𝛽′ + 𝛾′ = 1, 𝛽′, 𝛾′ ≥ 0), we have 𝑨3 𝑢3 = 𝛽′ 𝑨1 𝑢1 + 𝛾′ 𝑨2 𝑢2 ∈ 𝑇5 Therefore, by definition 𝑇5 is convex. We then have

• 2. Convex Set Terms and Definitions

17

Definitions: Affine Set, Cone, and Convex Hull Given 𝑣1, 𝑣2, ⋯ , 𝑣𝑙 ∈ 𝑆𝑜, function 𝑔 𝑣, 𝜄 = 𝜄1𝑣1 + 𝜄2𝑣2 + ⋯ + 𝜄𝑙𝑣𝑙 and two conditions

• 1. 𝜄1+𝜄2 + ⋯ + 𝜄𝑙 = 1
• 2. 𝜄𝑗≥ 0 ∀𝑗
• i. f u, θ

condition 1}: Affine set

• ii. f u, θ

condition 2}: Cone

• iii. f u, θ

conditions 1 and 2}: Convex hull 𝐹𝑦1: 𝜄1𝑣1 + 𝜄2𝑣2 = 𝑣1 + 𝜄2(𝑣2 − 𝑣1) 𝐹𝑦2: 𝜄1𝑣1 + 𝜄2𝑣2 + 𝜄3𝑣3

Definitions: Hyperplane and Half Spaces

18

Hyperplane 𝑦 𝑏𝑈𝑦 = 𝑐}, 𝑏 ∈ 𝑆𝑜, 𝑐 ∈ 𝑆

• r

𝑦 𝑏𝑈(𝑦 − 𝑦0) = 0}, for any 𝑦0 ∈ 𝑆𝑜, 𝑏 ∈ 𝑆𝑜, 𝑐 ∈ 𝑆 Half Space 𝑦 𝑏𝑈𝑦 ≤ 𝑐} 𝑏 ∈ 𝑆𝑜, 𝑐 ∈ 𝑆

• r

𝑦 𝑏𝑈(𝑦 − 𝑦0) ≤ 0} 𝐹𝑦: 𝑦 𝑦1 + 𝑦2 = 1} 𝑝𝑠 𝑦 [1,1] 𝑦1 𝑦2 − 0.5 0.5 = 0} 𝑦 𝑏𝑈(𝑦 − 𝑦0) ≤ 0}, 𝑏𝑈= [1,1], 𝑐 = 1, 𝑦0 = [2, −1] 𝑔 𝑦1, 𝑦2 = 𝑦1 + 𝑦2 − 1 normalize the expression:

𝑏𝑈 𝑏 2 𝑦= 𝑐 𝑏 2

• 2. Sets and Definitions

Ex : 3 variables 𝑦|𝑏𝑈𝑦 = 𝑐 , 𝑏𝑈 = 1,1,1 , 𝑐 = 6 Ex : 4 variables 𝑦|𝑏𝑈𝑦 = 𝑐 , 𝑏𝑈 = 1,1,1,1 , 𝑐 = 6 (1) degrees of freedom : 𝑜 − 1 𝑆𝑜 . (2) all vectors (𝑦 − 𝑧) are orthogonal to direction 𝑏, i.e. 𝑏𝑈 𝑦 − 𝑧 = 0, ∀𝑦, 𝑧 in the hyperplane Proof: Exercise: Conceptually (visually) construct hyperplane.

• 2. Sets and Definitions: Hyperplanes

19

Hyperplane : as an Equal potential of cost function min𝑔

0 𝑦 = 𝑑𝑈𝑦

𝑓. 𝑕. 1, 2 𝑦1 𝑦2

𝜖𝑔

0 𝑦

𝜖𝑦1 = 1 𝜖𝑔

0 𝑦

𝜖𝑦2 = 2

Vector 𝑑 is the sensitivity or cost of vector 𝑦1 𝑦2

• 2. Sets and Definitions: Hyperplanes

20

Hyperplane : as a linearized constraint 𝑏𝑈𝑦 ≤ 𝑐 𝑓. 𝑕. 1, 2 𝑦1 𝑦2 ≤ 10 Remark :

• Hyperplane is one key building block of convex optimization.

(theory, algorithms, applications for machine learning, deep learning, …)

• Each hyperplane separates the space into two half spaces.
• If 𝑜 ≥ 𝑞, 𝑞 hyperplanes can separate the space into 2𝑞

disjoint regions.

• 2. Sets and Definitions

21

Ⅴ. Polyhedra (plural) : Poly (many) Hedron (face) 𝑄 = 𝑦 𝐵𝑦 ≤ 𝑐, 𝐷𝑦 = 𝑒} 𝐵 = 𝑏1

𝑈

𝑏2

𝑈

… 𝑏𝑛

𝑈

C = 𝑑1

𝑈

𝑑2

𝑈

… 𝑑𝑞

𝑈

• 2. Sets and Definitions

22

ⅤI. Matrix Space : Positive Semidefinite Cone ① 𝑇𝑜 = 𝑌 ∈ 𝑆𝑜⨉𝑜 𝑌 = 𝑌𝑈} Symmetric Matrix ② 𝑇+

𝑜 = 𝑌 ∈ 𝑇𝑜 𝑌 ⪰ 0}

𝑗. 𝑓. 𝑧𝑈𝑌𝑧 ≥ 0, ∀𝑧 𝑇++

𝑜

= 𝑌 ∈ 𝑇𝑜 𝑌 ≻ 0} 𝑗. 𝑓. 𝑧𝑈𝑌𝑧 > 0, ∀𝑧 ≠ 0 Ex: 𝑌 = 𝑦 𝑧 𝑧 𝑨 ∈ 𝑇+

2 ⇔ 𝑦 ≥ 0, 𝑨 ≥ 0, 𝑦𝑨 ≥ 𝑧2

[𝑏 𝑐]𝑌 𝑏 𝑐 = 𝑏2𝑦 + 𝑐2𝑨 + 2𝑏𝑐𝑧 ≥ 0, ∀𝑏, 𝑐 ∈ ℝ

• 2. Sets and Definitions

23

Proof : 1 −𝑦−1𝑧 1 𝑦 𝑧 𝑧 𝑨 1 −𝑦−1𝑧 1 = 𝑦 𝑨 − 𝑦−1𝑧2 Let 𝑆 = 1 −𝑦−1𝑧 1 We have 𝑏 𝑐 𝑌 𝑏 𝑐 = [𝑏 𝑐]𝑆−𝑈𝑆𝑈𝑌𝑆𝑆−1 𝑏 𝑐 = 𝑏 𝑐 𝑆−𝑈 𝑦 𝑨 − 𝑦−1𝑧2 𝑆−1 𝑏 𝑐

• 2. Sets and Definitions

24

𝑦 𝑏𝑈𝑦 = 𝑐} (Classification, Optimization, Duality) Theorem : Given two convex sets 𝐷 ∩ 𝐸 = ∅ in 𝑆𝑜 ∃𝑏 ∈ 𝑆𝑜, 𝑐 ∈ 𝑆, 𝑡. 𝑢. 𝑏𝑈𝑦 ≤ 𝑐, ∀𝑦 ∈ 𝐷 𝑏𝑈𝑦 ≥ 𝑐, ∀𝑦 ∈ 𝐸 Actually 𝑏 = 𝑒 − 𝑑, 𝑐 =

𝑒 2

2− 𝑑 2 2

2

i.e. 𝑔 𝑦 = 𝑏𝑈𝑦 − 𝑐 = 𝑒 − 𝑑 𝑈(𝑦 −

𝑒+𝑑 2 )

For 𝑒𝑗𝑡𝑢 𝐷, 𝐸 = inf 𝑣 − 𝑤 2 𝑣 ∈ 𝐷, 𝑤 ∈ 𝐸}

• 3. Separating Hyperplane

25

Proof : ∀𝑤 ∈ 𝐸, 𝑏𝑈𝑤 ≥ 𝑏𝑈𝑒 should be true By contradiction, suppose that 𝑏𝑈𝑤 < 𝑏𝑈𝑒 Then we can show that 𝑒 + 𝑢(𝑤 − 𝑒) is close to 𝑑 for 𝑢 > 0 Because

𝑒 𝑒𝑢 𝑒 + 𝑢 𝑤 − 𝑒 − 𝑑 2 2 = 2 𝑒 − 𝑑 𝑈 𝑤 − 𝑒 < 0

We have 𝑒 + 𝑢 𝑤 − 𝑒 − 𝑑 2 < 𝑒 − 𝑑 2 for tiny 𝑢 > 0

• 3. Separating Hyperplane

26

Given set 𝐷 ∈ 𝑆𝑜, and a point 𝑦0 on the boundary of 𝐷, the hyperplan 𝑦 𝑏𝑈𝑦 = 𝑏𝑈𝑦0} is called supporting hyperplane of C if 𝑏𝑈𝑦 ≤ 𝑏𝑈𝑦0, ∀𝑦 ∈ 𝐷. Supporting Hyperplane Theorem: For any nonempty convex set 𝐷, and a point 𝑦0 on the boundary of 𝐷, There exists a support hyperplane to 𝐷 at 𝑦0. Proof: A separating hyperplane that separates interior 𝐷 and {𝑦0} is a supporting hyperplane.

• 3. Supporting Hyperplane

27

Given Cone 𝐿 (i.e. 𝐿 = {σ𝑗=1

𝑙

𝜄𝑗𝑣𝑗 |𝜄𝑗 > 0, 𝑣𝑗 ∈ 𝑆𝑜, ∀𝑗}) 𝐿∗ = 𝑧 𝑦𝑈𝑧 ≥ 0, ∀𝑦 ∈ 𝐿} Ex: 1. 𝐿 = 𝑆+

𝑜 ∶ 𝐿∗ = 𝑆+ 𝑜

• 2. 𝐿 = 𝑇+

𝑜 ∶ 𝐿∗ = 𝑇+ 𝑜

• 3. 𝐿 =

𝑦, 𝑢 𝑦 2 ≤ 𝑢 ∶ 𝐿∗ = {(𝑦, 𝑢)| 𝑦 2 ≤ 𝑢}

• 4. 𝐿 =

𝑦, 𝑢 𝑦 1 ≤ 𝑢 ∶ 𝐿∗ = {(𝑦, 𝑢)| 𝑦 ∞ ≤ 𝑢}

• 4. Dual Cones

28

Show that cone 𝐿 = 𝑦, 𝑢 𝑦 1 ≤ 𝑢 has its dual 𝐿∗ = 𝑦, 𝑢 𝑦 ∞ ≤ 𝑢 Proof : Statement 𝑦𝑈𝑣 + 𝑢𝑤 ≥ 0, ∀ 𝑦 1 ≤ 𝑢 ↔ 𝑣 ∞ ≤ 𝑤 L=>R By contradiction, suppose that 𝑣 ∞ > 𝑤 We can find ∃𝑦 𝑡. 𝑢 𝑦 1 ≤ 1, 𝑦𝑈𝑣 > 𝑤 Setting t=1, then we have 𝑣𝑈 −𝑦 + 𝑤 < 0. R=>L Given 𝑦 1 ≤ 𝑢, 𝑣 ∞ ≤ 𝑤 𝑣𝑈 −𝑦/𝑢 1 ≤ 𝑣 ∞ ≤ 𝑤 Thus, 𝑣𝑈 −𝑦 ≤ 𝑤𝑢

29

• 4. Dual Cones

Definition: 𝑦 ≤𝐿 𝑧 if 𝑧 − 𝑦 ∈ 𝐿 Theorem: 𝑦 ≤𝐿 𝑧 iff 𝜇𝑈𝑦 ≤ 𝜇𝑈𝑧, ∀𝜇 ∈ 𝐿∗ Examples

30

• 4. Dual Cones

The polyhedral cone 𝑊 = {𝑦|𝐵𝑦 ≥0} has its dual cone 𝑊∗ = {𝐵𝑈𝑤|𝑤 ≥ 0} Proof : By definition 𝑊∗ = {𝑧|𝑦𝑈𝑧 ≥ 0, ∀𝑦 ∈ 𝑊} Thus 𝑊∗ = {𝑧|𝑦𝑈𝑧 ≥ 0, ∀𝐵𝑦 ≥ 0} Let 𝑧 = 𝐵𝑈𝑤, we have 𝑦𝑈𝑧 = 𝑦𝑈𝐵𝑈𝑤 > 0 if 𝑤 ≥ 0 Ex: 𝐵 = 1 2 1 −1 i.e. 𝑦1 + 2𝑦2 ≥ 0, 𝑦1 − 𝑦2 ≥ 0 𝐵𝑈 = 1 1 2 −1 i.e. {𝜄1 1 2 + 𝜄2 1 −1 |𝜄1, 𝜄2 ≥ 0}

31

• 4. Dual Cones

Remark: 𝑦0 + ∆𝑦 ∆𝑦 ∈ 𝐿 (1) 𝐿 cone can be translated to 𝑦0 (2) If 𝑏 ∈ 𝐿∗ Then 𝑏𝑈𝑦 ≥ 0, ∀𝑦 ∈ 𝐿, i.e. −𝑏𝑦 is a supporting hyperplane

• f cone 𝐿

(3) Suppose that the current feasible search region is at corner 𝑦0 and 𝑦0 + ∆𝑦 ∆𝑦 ∈ 𝐿, ||∆𝑦|| < 𝑠 is a local feasible region

• f a convex set

If 𝛼𝑔

0 𝑦0 ∈ 𝐿∗, i.e. 𝛼𝑔 0 𝑦0 𝑈∆𝑦 ≥ 0, ∀∆𝑦 ∈ 𝐿

Then 𝑦0 is an optimal solution

32

• 4. Dual Cones