Duality (I) Lijun Zhang zlj@nju.edu.cn http://cs.nju.edu.cn/zlj - PowerPoint PPT Presentation

Duality (I) Lijun Zhang zlj@nju.edu.cn http://cs.nju.edu.cn/zlj

Outline  The Lagrange Dual Function  The Lagrange Dual Function  Lower Bound on Optimal Value  The Lagrange Dual Function and Conjugate Functions  The Lagrange Dual Problem  Making Dual Constraints Explicit  Weak Duality  Strong Duality and Slater’s Constraint Qualification

Optimization Problems  Standard Form min 𝑔 � 𝑦 s. t. 𝑔 � 𝑦 � 0, 𝑗 � 1, . . . , 𝑛 �1� ℎ � 𝑦 � 0 𝑗 � 1, . . . , 𝑞  Domain is nonempty � � 𝒠 � � dom 𝑔 � ∩ � dom ℎ � ��� ��� ∗  Denote the optimal value by  We do not assume the problem is convex

The Lagrangian � � �  The Lagrangian � � 𝑀 𝑦, 𝜇, 𝜉 � 𝑔 � 𝑦 � � 𝜇 � 𝑔 � 𝑦 � � 𝜉 � ℎ � 𝑦 ��� ��� � �   � : the Lagrange multiplier associated with the -th inequality constraint �  𝜉 � : the Lagrange multiplier associated with the -th equality constraint �  Vectors and : dual variables or Lagrange multiplier vectors

The Lagrange Dual Function � �  𝑕 𝜇, 𝜉 � inf �∈𝒠 𝑀 𝑦, 𝜇, 𝜉 � � � inf �∈𝒠 𝑔 � 𝑦 � � 𝜇 � 𝑔 � 𝑦 � � 𝜉 � ℎ � 𝑦 ��� ���  When is unbounded below in ,  is concave  𝑕 is the pointwise infimum of a family of affine functions of 𝜇, 𝑤  It is unconstrained

Outline  The Lagrange Dual Function  The Lagrange Dual Function  Lower Bound on Optimal Value  The Lagrange Dual Function and Conjugate Functions  The Lagrange Dual Problem  Making Dual Constraints Explicit  Weak Duality  Strong Duality and Slater’s Constraint Qualification

Lower Bounds on  For any and any 𝑕 𝜇, 𝜉 � 𝑞 ∗  Proof  is a feasible point for original problem 𝑔 � 𝑦 � � 0, ℎ � 𝑦 � � 0  Since � � � 𝜇 � 𝑔 � 𝑦 � � � 𝜉 � ℎ � 𝑦 � � 0 ��� ���  Therefore � � 𝑀 𝑦 �, 𝜇, 𝜉 � 𝑔 � 𝑦 � � � 𝜇 � 𝑔 � 𝑦 � � � 𝜉 � ℎ � 𝑦 � � 𝑔 � 𝑦 � ��� ���

Lower Bounds on  For any and any 𝑕 𝜇, 𝜉 � 𝑞 ∗  Proof  Hence 𝑕 𝜇, 𝜉 � inf �∈𝒠 𝑀 𝑦, 𝜇, 𝜉 � 𝑀 𝑦 �, 𝜇, 𝜉 � 𝑔 � 𝑦 �  Note that for any feasible �  Discussions  The lower bound is vacuous, when 𝑕 𝜇, 𝜉 � �∞  It is nontrivial only when 𝜇 ≽ 0, 𝜇, 𝜉 ∈ dom 𝑕  Dual feasible: �𝜇, 𝜉 � with 𝜇 ≽ 0, 𝜇, 𝜉 ∈ dom 𝑕

Example  A Simple Problem with  Lower bound from a dual feasible point  Solid curve: objective function 𝑔 �  Dashed curve: constraint function 𝑔 �  Feasible set: �0.46, 0.46 (indicated by the two dotted vertical lines)

Example  A Simple Problem with  Lower bound from a dual feasible point 𝑕 𝜇 � inf �∈𝒠 𝑀 𝑦, 𝜇 � 𝑀 𝑦, 𝜇 � 𝑔 � �𝑦�  Optimal point and value: 𝑦 ∗ � �0.46, 𝑞 ∗ � 1.54  Dotted curves: 𝑀 𝑦, 𝜇 for 𝜇 � 0.1, 0.2, … , 1.0 . Each has a minimum value smaller than 𝑞 ∗ as on • the feasible set (and for 𝜇 � 0 ), 𝑀 𝑦, 𝜇 � 𝑔 � 𝑦

Example  The dual function  Neither � nor � is convex, but the dual function is concave ∗ (the optimal  Horizontal dashed line: value of the problem)

Linear Approximation Interpretation  Rewrite (1) as unconstrained problem � � min 𝑔 � 𝑦 � � 𝐽 � 𝑔 � 𝑦 � � 𝐽 � ℎ � 𝑦 �2� ��� ���  is the indicator function for the � nonpositive reals 𝐽 � 𝑣 � �0 𝑣 � 0, ∞ 𝑣 � 0.  � is the indicator function of

Linear Approximation Interpretation  In the formulation (2)  expresses our irritation or displeasure � associated with a constraint function value : zero if � , infinite if � �  gives our displeasure for an equality � constraint value �  Our displeasure rises from zero to infinite as transitions from nonpositive to positive �

Linear Approximation Interpretation  In the formulation (2)  Suppose we replace � with linear function with 𝜉 � � , where , and � �  Objective becomes the Lagrangian � � 𝑀 𝑦, 𝜇, 𝜉 � 𝑔 � 𝑦 � � 𝜇 � 𝑔 � 𝑦 � � 𝜉 � ℎ � 𝑦 ��� ���  Dual function value is optimal value of � � min 𝑔 � 𝑦 � � 𝜇 � 𝑔 � 𝑦 � � 𝜉 � ℎ � 𝑦 �3� ��� ���

Linear Approximation Interpretation  In the formulation (3)  We replace � and � with linear or “soft” displeasure functions  For an inequality constraint, our displeasure is zero when � , and is positive when (assuming ) � �  In (2), any nonpositive value of � is acceptable  In (3), we actually derive pleasure from constraints that have margin, i.e., from �

Linear Approximation Interpretation  Interpretation of Lower Bound  The linear function is an underestimator of the indicator function 𝜇 � 𝑣 � 𝐽 � 𝑣 𝜉 � 𝑣 � 𝐽 � �𝑣�  Lower Bound Property � � 𝑔 � 𝑦 � � 𝐽 � 𝑔 � 𝑦 � � 𝐽 � ℎ � 𝑦 � ��� ��� � � 𝑀 𝑦, 𝜇, 𝜉 � 𝑔 � 𝑦 � � 𝜇 � 𝑔 � 𝑦 � � 𝜉 � ℎ � 𝑦 ��� ���

Example  Least-squares Solution of Linear Equations 𝑦 � 𝑦 min s. t. 𝐵𝑦 � 𝑐 ���   No inequality constraints  (linear) equality constraints  Lagrangian 𝑀 𝑦, 𝜉 � 𝑦 � 𝑦 � 𝜉 � �𝐵𝑦 � 𝑐� � �  Domain:

Example  Least-squares Solution of Linear Equations 𝑦 � 𝑦 min s. t. 𝐵𝑦 � 𝑐  Dual Function � 𝑦 � 𝑦 � 𝜉 � �𝐵𝑦 � 𝑐� 𝑕 𝜉 � inf � 𝑀 𝑦, 𝜉 � inf  Optimality condition � 𝑀 𝑦, 𝜉 � 2𝑦 � 𝐵 � 𝜉 � 0 ⇒ 𝑦 � � 1/2 𝐵 � 𝜉 𝛼

Example  Least-squares Solution of Linear Equations 𝑦 � 𝑦 min s. t. 𝐵𝑦 � 𝑐  Dual Function ⇒ 𝑕 𝜉 � 𝑀 � 1/2 𝐵 � 𝜉, 𝜉 � � 1/4 𝜉 � 𝐵𝐵 � 𝜉 � 𝑐 � 𝜉  Concave Function  Lower Bound Property � 1/4 𝜉 � 𝐵𝐵 � 𝜉 � 𝑐 � 𝜉 � inf 𝑦 � 𝑦 𝐵𝑦 � 𝑐

Example  Standard Form LP 𝑑 � 𝑦 min s. t. 𝐵𝑦 � 𝑐 𝑦 ≽ 0  Inequality constraints: � �  Lagrangian 𝜇 � 𝑦 � � 𝜉 � 𝐵𝑦 � 𝑐 � 𝑀 𝑦, 𝜇, 𝜉 � 𝑑 � 𝑦 � ∑ ��� � �𝑐 � 𝜉 � 𝑑 � 𝐵 � 𝜉 � 𝜇 � 𝑦  Dual Function 𝑕 𝜇, 𝜉 � inf � 𝑀 𝑦, 𝜇, 𝜉 � �𝑐 � 𝜉 � inf � 𝑑 � 𝐵 � 𝜉 � 𝜇 � 𝑦

Example  Standard Form LP 𝑑 � 𝑦 min s. t. 𝐵𝑦 � 𝑐 𝑦 ≽ 0  Inequality constraints: � �  Dual Function 𝑕 𝜇, 𝜉 � ��𝑐 � 𝜉 𝐵 � 𝜉 � 𝜇 � 𝑑 � 0, �∞ otherwise.  The lower bound is nontrivial only when � and satisfy and

Outline  The Lagrange Dual Function  The Lagrange Dual Function  Lower Bound on Optimal Value  The Lagrange Dual Function and Conjugate Functions  The Lagrange Dual Problem  Making Dual Constraints Explicit  Weak Duality  Strong Duality and Slater’s Constraint Qualification

Conjugate Function �  Its conjugate function is 𝑔 ∗ 𝑧 � 𝑧 � 𝑦 � 𝑔 𝑦 sup �∈��� �  dom 𝑔 ∗ � 𝑧|𝑔 ∗ 𝑧 � ∞  𝑔 ∗ is always convex

The Lagrange Dual Function and Conjugate Functions  A Simple Example min 𝑔 𝑦 s. t. 𝑦 � 0  Lagrangian 𝑀 𝑦, 𝜉 � 𝑔 𝑦 � 𝜉 � 𝑦  Dual Function � 𝑔 𝑦 � 𝜉 � 𝑦 𝑕 𝜉 � inf � � 𝑔 ∗ �𝜉 �𝜉 � 𝑦 � 𝑔 𝑦 � � sup �

The Lagrange Dual Function and Conjugate Functions  A More General Example min 𝑔 � 𝑦 s. t. 𝐵𝑦 ≼ 𝑐 𝐷𝑦 � 𝑒  Lagrangian � 𝑦 � 𝜇 � 𝐵𝑦 � 𝑐 � 𝜉 � 𝐷𝑦 � 𝑒 𝑀 𝑦, 𝜉 � 𝑔  Dual Function � 𝑦 � 𝜇 � 𝐵𝑦 � 𝑐 � 𝜉 � 𝐷𝑦 � 𝑒 𝑕 𝜉 � inf � 𝑔 � �𝑐 � 𝜇 � 𝑒 � 𝜉 � inf � 𝑦 � 𝐵 � 𝜇 � 𝐷 � 𝜉 � 𝑦� � �𝑔 ∗ ��𝐵 � 𝜇 � 𝐷 � 𝜉� � �𝑐 � 𝜇 � 𝑒 � 𝜉 � 𝑔 �

The Lagrange Dual Function and Conjugate Functions  A More General Example min 𝑔 � 𝑦 s. t. 𝐵𝑦 ≼ 𝑐 𝐷𝑦 � 𝑒  Lagrangian � 𝑦 � 𝜇 � 𝐵𝑦 � 𝑐 � 𝜉 � 𝐷𝑦 � 𝑒 𝑀 𝑦, 𝜉 � 𝑔  Dual Function 𝑕 𝜉 � �𝑐 � 𝜇 � 𝑒 � 𝜉 � 𝑔 ∗ ��𝐵 � 𝜇 � 𝐷 � 𝜉� � � � ∗  �