New primal-dual subgradient methods for Convex Problems with - - PowerPoint PPT Presentation

New primal-dual subgradient methods for Convex Problems with Functional Constraints

Yurii Nesterov, CORE/INMA (UCL) January 12, 2015 (Les Houches)

• Yu. Nesterov

New primal-dual methods for functional constraints 1/19

Outline

1 Constrained optimization problem 2 Lagrange multipliers 3 Dual function and dual problem 4 Augmented Lagrangian 5 Switching subgradient methods 6 Finding the dual multipliers 7 Complexity analysis

• Yu. Nesterov

New primal-dual methods for functional constraints 2/19

Optimization problem: simple constraints

• Yu. Nesterov

New primal-dual methods for functional constraints 3/19

Optimization problem: simple constraints

Consider the problem: min

x∈Q f (x),

• Yu. Nesterov

New primal-dual methods for functional constraints 3/19

Optimization problem: simple constraints

Consider the problem: min

x∈Q f (x),

where

• Yu. Nesterov

New primal-dual methods for functional constraints 3/19

Optimization problem: simple constraints

Consider the problem: min

x∈Q f (x),

where Q is a closed convex set:

• Yu. Nesterov

New primal-dual methods for functional constraints 3/19

Optimization problem: simple constraints

Consider the problem: min

x∈Q f (x),

where Q is a closed convex set: x, y ∈ Q ⇒ [x, y] ⊆ Q,

• Yu. Nesterov

New primal-dual methods for functional constraints 3/19

Optimization problem: simple constraints

Consider the problem: min

x∈Q f (x),

where Q is a closed convex set: x, y ∈ Q ⇒ [x, y] ⊆ Q, f is a subdifferentiable on Q convex function:

• Yu. Nesterov

New primal-dual methods for functional constraints 3/19

Optimization problem: simple constraints

Consider the problem: min

x∈Q f (x),

where Q is a closed convex set: x, y ∈ Q ⇒ [x, y] ⊆ Q, f is a subdifferentiable on Q convex function: f (y) ≥ f (x) + ∇f (x), y − x, x, y ∈ Q, ∇f (x) ∈ ∂f (x).

• Yu. Nesterov

New primal-dual methods for functional constraints 3/19

Optimization problem: simple constraints

Consider the problem: min

x∈Q f (x),

where Q is a closed convex set: x, y ∈ Q ⇒ [x, y] ⊆ Q, f is a subdifferentiable on Q convex function: f (y) ≥ f (x) + ∇f (x), y − x, x, y ∈ Q, ∇f (x) ∈ ∂f (x). Optimality condition:

• Yu. Nesterov

New primal-dual methods for functional constraints 3/19

Optimization problem: simple constraints

Consider the problem: min

x∈Q f (x),

where Q is a closed convex set: x, y ∈ Q ⇒ [x, y] ⊆ Q, f is a subdifferentiable on Q convex function: f (y) ≥ f (x) + ∇f (x), y − x, x, y ∈ Q, ∇f (x) ∈ ∂f (x). Optimality condition: point x∗ ∈ Q is optimal iff

• Yu. Nesterov

New primal-dual methods for functional constraints 3/19

Optimization problem: simple constraints

Consider the problem: min

x∈Q f (x),

where Q is a closed convex set: x, y ∈ Q ⇒ [x, y] ⊆ Q, f is a subdifferentiable on Q convex function: f (y) ≥ f (x) + ∇f (x), y − x, x, y ∈ Q, ∇f (x) ∈ ∂f (x). Optimality condition: point x∗ ∈ Q is optimal iff ∇f (x∗), x − x∗ ≥ 0, ∀x ∈ Q.

• Yu. Nesterov

New primal-dual methods for functional constraints 3/19

Optimization problem: simple constraints

Consider the problem: min

x∈Q f (x),

where Q is a closed convex set: x, y ∈ Q ⇒ [x, y] ⊆ Q, f is a subdifferentiable on Q convex function: f (y) ≥ f (x) + ∇f (x), y − x, x, y ∈ Q, ∇f (x) ∈ ∂f (x). Optimality condition: point x∗ ∈ Q is optimal iff ∇f (x∗), x − x∗ ≥ 0, ∀x ∈ Q. Interpretation:

• Yu. Nesterov

New primal-dual methods for functional constraints 3/19

Optimization problem: simple constraints

Consider the problem: min

x∈Q f (x),

where Q is a closed convex set: x, y ∈ Q ⇒ [x, y] ⊆ Q, f is a subdifferentiable on Q convex function: f (y) ≥ f (x) + ∇f (x), y − x, x, y ∈ Q, ∇f (x) ∈ ∂f (x). Optimality condition: point x∗ ∈ Q is optimal iff ∇f (x∗), x − x∗ ≥ 0, ∀x ∈ Q. Interpretation: Function increases along any feasible direction.

• Yu. Nesterov

New primal-dual methods for functional constraints 3/19

Optimization problem: functional constraints

• Yu. Nesterov

New primal-dual methods for functional constraints 4/19

Optimization problem: functional constraints

Problem:

• Yu. Nesterov

New primal-dual methods for functional constraints 4/19

Optimization problem: functional constraints

Problem: min

x∈Q{f0(x), fi(x) ≤ 0, i = 1, . . . , m},

• Yu. Nesterov

New primal-dual methods for functional constraints 4/19

Optimization problem: functional constraints

Problem: min

x∈Q{f0(x), fi(x) ≤ 0, i = 1, . . . , m},

where

• Yu. Nesterov

New primal-dual methods for functional constraints 4/19

Optimization problem: functional constraints

Problem: min

x∈Q{f0(x), fi(x) ≤ 0, i = 1, . . . , m},

where Q is a closed convex set,

• Yu. Nesterov

New primal-dual methods for functional constraints 4/19

Optimization problem: functional constraints

Problem: min

x∈Q{f0(x), fi(x) ≤ 0, i = 1, . . . , m},

where Q is a closed convex set, all fi are convex and subdifferentiable on Q, i = 0, . . . , m:

• Yu. Nesterov

New primal-dual methods for functional constraints 4/19

Optimization problem: functional constraints

Problem: min

x∈Q{f0(x), fi(x) ≤ 0, i = 1, . . . , m},

where Q is a closed convex set, all fi are convex and subdifferentiable on Q, i = 0, . . . , m: fi(y) ≥ fi(x) + ∇fi(x), y − x, x, y ∈ Q, ∇fi(x) ∈ ∂fi(x).

• Yu. Nesterov

New primal-dual methods for functional constraints 4/19

Optimization problem: functional constraints

Problem: min

x∈Q{f0(x), fi(x) ≤ 0, i = 1, . . . , m},

where Q is a closed convex set, all fi are convex and subdifferentiable on Q, i = 0, . . . , m: fi(y) ≥ fi(x) + ∇fi(x), y − x, x, y ∈ Q, ∇fi(x) ∈ ∂fi(x). Optimality condition (KKT, 1951):

• Yu. Nesterov

New primal-dual methods for functional constraints 4/19

Optimization problem: functional constraints

Problem: min

x∈Q{f0(x), fi(x) ≤ 0, i = 1, . . . , m},

where Q is a closed convex set, all fi are convex and subdifferentiable on Q, i = 0, . . . , m: fi(y) ≥ fi(x) + ∇fi(x), y − x, x, y ∈ Q, ∇fi(x) ∈ ∂fi(x). Optimality condition (KKT, 1951): point x∗ ∈ Q is optimal iff

• Yu. Nesterov

New primal-dual methods for functional constraints 4/19

Optimization problem: functional constraints

Problem: min

x∈Q{f0(x), fi(x) ≤ 0, i = 1, . . . , m},

where Q is a closed convex set, all fi are convex and subdifferentiable on Q, i = 0, . . . , m: fi(y) ≥ fi(x) + ∇fi(x), y − x, x, y ∈ Q, ∇fi(x) ∈ ∂fi(x). Optimality condition (KKT, 1951): point x∗ ∈ Q is optimal iff there exist Lagrange multipliers λ(i)

∗ ≥ 0, i = 1, . . . , m, such that

• Yu. Nesterov

New primal-dual methods for functional constraints 4/19

Optimization problem: functional constraints

Problem: min

x∈Q{f0(x), fi(x) ≤ 0, i = 1, . . . , m},

where Q is a closed convex set, all fi are convex and subdifferentiable on Q, i = 0, . . . , m: fi(y) ≥ fi(x) + ∇fi(x), y − x, x, y ∈ Q, ∇fi(x) ∈ ∂fi(x). Optimality condition (KKT, 1951): point x∗ ∈ Q is optimal iff there exist Lagrange multipliers λ(i)

∗ ≥ 0, i = 1, . . . , m, such that

(1) : ∇f0(x∗) +

m

• i=1

λ(i)

∗ ∇fi(x∗), x − x∗ ≥ 0,

∀x ∈ Q,

• Yu. Nesterov

New primal-dual methods for functional constraints 4/19

Optimization problem: functional constraints

Problem: min

x∈Q{f0(x), fi(x) ≤ 0, i = 1, . . . , m},

where Q is a closed convex set, all fi are convex and subdifferentiable on Q, i = 0, . . . , m: fi(y) ≥ fi(x) + ∇fi(x), y − x, x, y ∈ Q, ∇fi(x) ∈ ∂fi(x). Optimality condition (KKT, 1951): point x∗ ∈ Q is optimal iff there exist Lagrange multipliers λ(i)

∗ ≥ 0, i = 1, . . . , m, such that

(1) : ∇f0(x∗) +

m

• i=1

λ(i)

∗ ∇fi(x∗), x − x∗ ≥ 0,

∀x ∈ Q, (2) : fi(x∗) ≤ 0, i = 1, . . . , m, (feasibility)

• Yu. Nesterov

New primal-dual methods for functional constraints 4/19

Optimization problem: functional constraints

Problem: min

x∈Q{f0(x), fi(x) ≤ 0, i = 1, . . . , m},

where Q is a closed convex set, all fi are convex and subdifferentiable on Q, i = 0, . . . , m: fi(y) ≥ fi(x) + ∇fi(x), y − x, x, y ∈ Q, ∇fi(x) ∈ ∂fi(x). Optimality condition (KKT, 1951): point x∗ ∈ Q is optimal iff there exist Lagrange multipliers λ(i)

∗ ≥ 0, i = 1, . . . , m, such that

(1) : ∇f0(x∗) +

m

• i=1

λ(i)

∗ ∇fi(x∗), x − x∗ ≥ 0,

∀x ∈ Q, (2) : fi(x∗) ≤ 0, i = 1, . . . , m, (feasibility) (3) : λ(i)

∗ fi(x∗) = 0,

i = 1, . . . , m. (complementary slackness)

• Yu. Nesterov

New primal-dual methods for functional constraints 4/19

Lagrange multipliers: interpretation

• Yu. Nesterov

New primal-dual methods for functional constraints 5/19

Lagrange multipliers: interpretation

Let I ⊆ {1, . . . , m} be an arbitrary set of indexes.

• Yu. Nesterov

New primal-dual methods for functional constraints 5/19

Lagrange multipliers: interpretation

Let I ⊆ {1, . . . , m} be an arbitrary set of indexes. Denote fI(x) = f0(x) +

i∈I

λ(i)

∗ fi(x).

• Yu. Nesterov

New primal-dual methods for functional constraints 5/19

Lagrange multipliers: interpretation

Let I ⊆ {1, . . . , m} be an arbitrary set of indexes. Denote fI(x) = f0(x) +

i∈I

λ(i)

∗ fi(x). Consider the problem

• Yu. Nesterov

New primal-dual methods for functional constraints 5/19

Lagrange multipliers: interpretation

Let I ⊆ {1, . . . , m} be an arbitrary set of indexes. Denote fI(x) = f0(x) +

i∈I

λ(i)

∗ fi(x). Consider the problem

PI : min

x∈Q{fI(x) : fi(x) ≤ 0, i ∈ I}.

• Yu. Nesterov

New primal-dual methods for functional constraints 5/19

Lagrange multipliers: interpretation

Let I ⊆ {1, . . . , m} be an arbitrary set of indexes. Denote fI(x) = f0(x) +

i∈I

λ(i)

∗ fi(x). Consider the problem

PI : min

x∈Q{fI(x) : fi(x) ≤ 0, i ∈ I}.

Observation: in any case, x∗ is the optimal solution of problem PI.

• Yu. Nesterov

New primal-dual methods for functional constraints 5/19

Lagrange multipliers: interpretation

Let I ⊆ {1, . . . , m} be an arbitrary set of indexes. Denote fI(x) = f0(x) +

i∈I

λ(i)

∗ fi(x). Consider the problem

PI : min

x∈Q{fI(x) : fi(x) ≤ 0, i ∈ I}.

Observation: in any case, x∗ is the optimal solution of problem PI. Interpretation: λ(i)

∗

are the shadow prices for resources.

• Yu. Nesterov

New primal-dual methods for functional constraints 5/19

Lagrange multipliers: interpretation

Let I ⊆ {1, . . . , m} be an arbitrary set of indexes. Denote fI(x) = f0(x) +

i∈I

λ(i)

∗ fi(x). Consider the problem

PI : min

x∈Q{fI(x) : fi(x) ≤ 0, i ∈ I}.

Observation: in any case, x∗ is the optimal solution of problem PI. Interpretation: λ(i)

∗

are the shadow prices for resources. (Kantorovich, 1939)

• Yu. Nesterov

New primal-dual methods for functional constraints 5/19

Lagrange multipliers: interpretation

Let I ⊆ {1, . . . , m} be an arbitrary set of indexes. Denote fI(x) = f0(x) +

i∈I

λ(i)

∗ fi(x). Consider the problem

PI : min

x∈Q{fI(x) : fi(x) ≤ 0, i ∈ I}.

Observation: in any case, x∗ is the optimal solution of problem PI. Interpretation: λ(i)

∗

are the shadow prices for resources. (Kantorovich, 1939) Application examples:

• Yu. Nesterov

New primal-dual methods for functional constraints 5/19

Lagrange multipliers: interpretation

Let I ⊆ {1, . . . , m} be an arbitrary set of indexes. Denote fI(x) = f0(x) +

i∈I

λ(i)

∗ fi(x). Consider the problem

PI : min

x∈Q{fI(x) : fi(x) ≤ 0, i ∈ I}.

Observation: in any case, x∗ is the optimal solution of problem PI. Interpretation: λ(i)

∗

are the shadow prices for resources. (Kantorovich, 1939) Application examples: Traffic congestion: car flows on roads ⇔ size of queues.

• Yu. Nesterov

New primal-dual methods for functional constraints 5/19

Lagrange multipliers: interpretation

Let I ⊆ {1, . . . , m} be an arbitrary set of indexes. Denote fI(x) = f0(x) +

i∈I

λ(i)

∗ fi(x). Consider the problem

PI : min

x∈Q{fI(x) : fi(x) ≤ 0, i ∈ I}.

Observation: in any case, x∗ is the optimal solution of problem PI. Interpretation: λ(i)

∗

are the shadow prices for resources. (Kantorovich, 1939) Application examples: Traffic congestion: car flows on roads ⇔ size of queues. Electrical networks: currents in the wires ⇔ voltage potentials, etc.

• Yu. Nesterov

New primal-dual methods for functional constraints 5/19

Lagrange multipliers: interpretation

Let I ⊆ {1, . . . , m} be an arbitrary set of indexes. Denote fI(x) = f0(x) +

i∈I

λ(i)

∗ fi(x). Consider the problem

PI : min

x∈Q{fI(x) : fi(x) ≤ 0, i ∈ I}.

Observation: in any case, x∗ is the optimal solution of problem PI. Interpretation: λ(i)

∗

are the shadow prices for resources. (Kantorovich, 1939) Application examples: Traffic congestion: car flows on roads ⇔ size of queues. Electrical networks: currents in the wires ⇔ voltage potentials, etc. Main question: How to compute (x∗, λ∗)?

• Yu. Nesterov

New primal-dual methods for functional constraints 5/19

Algebraic interpretation

• Yu. Nesterov

New primal-dual methods for functional constraints 6/19

Algebraic interpretation

Consider the Lagrangian L(x, λ) = f0(x) +

m

• i=1

λ(i)fi(x).

• Yu. Nesterov

New primal-dual methods for functional constraints 6/19

Algebraic interpretation

Consider the Lagrangian L(x, λ) = f0(x) +

m

• i=1

λ(i)fi(x). Condition KKT(1): ∇f0(x∗) +

m

• i=1

λ(i)

∗ ∇fi(x∗), x − x∗ ≥ 0,

∀x ∈ Q,

• Yu. Nesterov

New primal-dual methods for functional constraints 6/19

Algebraic interpretation

Consider the Lagrangian L(x, λ) = f0(x) +

m

• i=1

λ(i)fi(x). Condition KKT(1): ∇f0(x∗) +

m

• i=1

λ(i)

∗ ∇fi(x∗), x − x∗ ≥ 0,

∀x ∈ Q, implies x∗ ∈ Arg min

x∈Q L(x, λ∗).

• Yu. Nesterov

New primal-dual methods for functional constraints 6/19

Algebraic interpretation

Consider the Lagrangian L(x, λ) = f0(x) +

m

• i=1

λ(i)fi(x). Condition KKT(1): ∇f0(x∗) +

m

• i=1

λ(i)

∗ ∇fi(x∗), x − x∗ ≥ 0,

∀x ∈ Q, implies x∗ ∈ Arg min

x∈Q L(x, λ∗).

Define the dual function φ(λ) = min

x∈Q L(x, λ), λ ≥ 0.

• Yu. Nesterov

New primal-dual methods for functional constraints 6/19

Algebraic interpretation

Consider the Lagrangian L(x, λ) = f0(x) +

m

• i=1

λ(i)fi(x). Condition KKT(1): ∇f0(x∗) +

m

• i=1

λ(i)

∗ ∇fi(x∗), x − x∗ ≥ 0,

∀x ∈ Q, implies x∗ ∈ Arg min

x∈Q L(x, λ∗).

Define the dual function φ(λ) = min

x∈Q L(x, λ), λ ≥ 0. It is concave!

• Yu. Nesterov

New primal-dual methods for functional constraints 6/19

Algebraic interpretation

Consider the Lagrangian L(x, λ) = f0(x) +

m

• i=1

λ(i)fi(x). Condition KKT(1): ∇f0(x∗) +

m

• i=1

λ(i)

∗ ∇fi(x∗), x − x∗ ≥ 0,

∀x ∈ Q, implies x∗ ∈ Arg min

x∈Q L(x, λ∗).

Define the dual function φ(λ) = min

x∈Q L(x, λ), λ ≥ 0. It is concave!

By Danskin’s Theorem, ∇φ(λ) = (f1(x(λ)), . . . , fm(x(λ)), with x(λ) ∈ Arg min

x∈Q L(x, λ).

• Yu. Nesterov

New primal-dual methods for functional constraints 6/19

Algebraic interpretation

Consider the Lagrangian L(x, λ) = f0(x) +

m

• i=1

λ(i)fi(x). Condition KKT(1): ∇f0(x∗) +

m

• i=1

λ(i)

∗ ∇fi(x∗), x − x∗ ≥ 0,

∀x ∈ Q, implies x∗ ∈ Arg min

x∈Q L(x, λ∗).

Define the dual function φ(λ) = min

x∈Q L(x, λ), λ ≥ 0. It is concave!

By Danskin’s Theorem, ∇φ(λ) = (f1(x(λ)), . . . , fm(x(λ)), with x(λ) ∈ Arg min

x∈Q L(x, λ).

Conditions KKT(2,3): fi(x∗) ≤ 0, λ(i)

∗ fi(x∗) = 0, i = 1, . . . , m,

• Yu. Nesterov

New primal-dual methods for functional constraints 6/19

Algebraic interpretation

Consider the Lagrangian L(x, λ) = f0(x) +

m

• i=1

λ(i)fi(x). Condition KKT(1): ∇f0(x∗) +

m

• i=1

λ(i)

∗ ∇fi(x∗), x − x∗ ≥ 0,

∀x ∈ Q, implies x∗ ∈ Arg min

x∈Q L(x, λ∗).

Define the dual function φ(λ) = min

x∈Q L(x, λ), λ ≥ 0. It is concave!

By Danskin’s Theorem, ∇φ(λ) = (f1(x(λ)), . . . , fm(x(λ)), with x(λ) ∈ Arg min

x∈Q L(x, λ).

Conditions KKT(2,3): fi(x∗) ≤ 0, λ(i)

∗ fi(x∗) = 0, i = 1, . . . , m,

imply (x∗ = x(λ∗)) λ∗ ∈ Arg max

λ≥0 φ(λ).

• Yu. Nesterov

New primal-dual methods for functional constraints 6/19

Algorithmic aspects

• Yu. Nesterov

New primal-dual methods for functional constraints 7/19

Algorithmic aspects

Main idea: solve the dual problem max

λ≥0 φ(λ)

• Yu. Nesterov

New primal-dual methods for functional constraints 7/19

Algorithmic aspects

Main idea: solve the dual problem max

λ≥0 φ(λ)

by the subgradient method:

• Yu. Nesterov

New primal-dual methods for functional constraints 7/19

Algorithmic aspects

Main idea: solve the dual problem max

λ≥0 φ(λ)

by the subgradient method: 1. Compute x(λk) and define ∇φ(λk) = (f1(x(λk)), . . . , fm(x(λk))).

• Yu. Nesterov

New primal-dual methods for functional constraints 7/19

Algorithmic aspects

Main idea: solve the dual problem max

λ≥0 φ(λ)

by the subgradient method: 1. Compute x(λk) and define ∇φ(λk) = (f1(x(λk)), . . . , fm(x(λk))). 2. Update λk+1 = ProjectRn

+ (λk + hk∇φ(λk)).

• Yu. Nesterov

New primal-dual methods for functional constraints 7/19

Algorithmic aspects

Main idea: solve the dual problem max

λ≥0 φ(λ)

by the subgradient method: 1. Compute x(λk) and define ∇φ(λk) = (f1(x(λk)), . . . , fm(x(λk))). 2. Update λk+1 = ProjectRn

+ (λk + hk∇φ(λk)).

Stepsizes hk > 0 are defined in the usual way.

• Yu. Nesterov

New primal-dual methods for functional constraints 7/19

Algorithmic aspects

Main idea: solve the dual problem max

λ≥0 φ(λ)

by the subgradient method: 1. Compute x(λk) and define ∇φ(λk) = (f1(x(λk)), . . . , fm(x(λk))). 2. Update λk+1 = ProjectRn

+ (λk + hk∇φ(λk)).

Stepsizes hk > 0 are defined in the usual way. Main difficulties:

• Yu. Nesterov

New primal-dual methods for functional constraints 7/19

Algorithmic aspects

Main idea: solve the dual problem max

λ≥0 φ(λ)

by the subgradient method: 1. Compute x(λk) and define ∇φ(λk) = (f1(x(λk)), . . . , fm(x(λk))). 2. Update λk+1 = ProjectRn

+ (λk + hk∇φ(λk)).

Stepsizes hk > 0 are defined in the usual way. Main difficulties: Each iteration is time consuming.

• Yu. Nesterov

New primal-dual methods for functional constraints 7/19

Algorithmic aspects

Main idea: solve the dual problem max

λ≥0 φ(λ)

by the subgradient method: 1. Compute x(λk) and define ∇φ(λk) = (f1(x(λk)), . . . , fm(x(λk))). 2. Update λk+1 = ProjectRn

+ (λk + hk∇φ(λk)).

Stepsizes hk > 0 are defined in the usual way. Main difficulties: Each iteration is time consuming. Unclear termination criterion.

• Yu. Nesterov

New primal-dual methods for functional constraints 7/19

Algorithmic aspects

Main idea: solve the dual problem max

λ≥0 φ(λ)

by the subgradient method: 1. Compute x(λk) and define ∇φ(λk) = (f1(x(λk)), . . . , fm(x(λk))). 2. Update λk+1 = ProjectRn

+ (λk + hk∇φ(λk)).

Stepsizes hk > 0 are defined in the usual way. Main difficulties: Each iteration is time consuming. Unclear termination criterion. Low rate of convergence

• Yu. Nesterov

New primal-dual methods for functional constraints 7/19

Algorithmic aspects

Main idea: solve the dual problem max

λ≥0 φ(λ)

by the subgradient method: 1. Compute x(λk) and define ∇φ(λk) = (f1(x(λk)), . . . , fm(x(λk))). 2. Update λk+1 = ProjectRn

+ (λk + hk∇φ(λk)).

Stepsizes hk > 0 are defined in the usual way. Main difficulties: Each iteration is time consuming. Unclear termination criterion. Low rate of convergence (O 1

ǫ2

• upper-level iterations).
• Yu. Nesterov

New primal-dual methods for functional constraints 7/19

Augmented Lagrangian (1970’s) [Hestenes, Powell, Rockafellar, Polyak, Bertsekas, . . .]

• Yu. Nesterov

New primal-dual methods for functional constraints 8/19

Augmented Lagrangian (1970’s) [Hestenes, Powell, Rockafellar, Polyak, Bertsekas, . . .]

Define the Augmented Lagrangian

• LK(x, λ) = f0(x) +

1 2K m

• i=1
• λ(i) + Kfi(x)

2

+ − 1 2K λ2 2,

λ ∈ Rm, where K > 0 is a penalty parameter.

• Yu. Nesterov

New primal-dual methods for functional constraints 8/19

Augmented Lagrangian (1970’s) [Hestenes, Powell, Rockafellar, Polyak, Bertsekas, . . .]

Define the Augmented Lagrangian

• LK(x, λ) = f0(x) +

1 2K m

• i=1
• λ(i) + Kfi(x)

2

+ − 1 2K λ2 2,

λ ∈ Rm, where K > 0 is a penalty parameter. Consider the dual function ˆ φ(λ) = min

x∈Q

• L(x, λ).
• Yu. Nesterov

New primal-dual methods for functional constraints 8/19

Augmented Lagrangian (1970’s) [Hestenes, Powell, Rockafellar, Polyak, Bertsekas, . . .]

Define the Augmented Lagrangian

• LK(x, λ) = f0(x) +

1 2K m

• i=1
• λ(i) + Kfi(x)

2

+ − 1 2K λ2 2,

λ ∈ Rm, where K > 0 is a penalty parameter. Consider the dual function ˆ φ(λ) = min

x∈Q

• L(x, λ).

Main properties. Function ˆ φ is concave.

• Yu. Nesterov

New primal-dual methods for functional constraints 8/19

Augmented Lagrangian (1970’s) [Hestenes, Powell, Rockafellar, Polyak, Bertsekas, . . .]

Define the Augmented Lagrangian

• LK(x, λ) = f0(x) +

1 2K m

• i=1
• λ(i) + Kfi(x)

2

+ − 1 2K λ2 2,

λ ∈ Rm, where K > 0 is a penalty parameter. Consider the dual function ˆ φ(λ) = min

x∈Q

• L(x, λ).

Main properties. Function ˆ φ is concave. Its gradient is Lipschitz continuous with constant 1

K .

• Yu. Nesterov

New primal-dual methods for functional constraints 8/19

Augmented Lagrangian (1970’s) [Hestenes, Powell, Rockafellar, Polyak, Bertsekas, . . .]

Define the Augmented Lagrangian

• LK(x, λ) = f0(x) +

1 2K m

• i=1
• λ(i) + Kfi(x)

2

+ − 1 2K λ2 2,

λ ∈ Rm, where K > 0 is a penalty parameter. Consider the dual function ˆ φ(λ) = min

x∈Q

• L(x, λ).

Main properties. Function ˆ φ is concave. Its gradient is Lipschitz continuous with constant 1

K .

Its unconstrained maximum is attained at the optimal dual solution.

• Yu. Nesterov

New primal-dual methods for functional constraints 8/19

Augmented Lagrangian (1970’s) [Hestenes, Powell, Rockafellar, Polyak, Bertsekas, . . .]

Define the Augmented Lagrangian

• LK(x, λ) = f0(x) +

1 2K m

• i=1
• λ(i) + Kfi(x)

2

+ − 1 2K λ2 2,

λ ∈ Rm, where K > 0 is a penalty parameter. Consider the dual function ˆ φ(λ) = min

x∈Q

• L(x, λ).

Main properties. Function ˆ φ is concave. Its gradient is Lipschitz continuous with constant 1

K .

Its unconstrained maximum is attained at the optimal dual solution. The corresponding point ˆ x(λ∗) is the optimal primal solution.

• Yu. Nesterov

New primal-dual methods for functional constraints 8/19

Augmented Lagrangian (1970’s) [Hestenes, Powell, Rockafellar, Polyak, Bertsekas, . . .]

Define the Augmented Lagrangian

• LK(x, λ) = f0(x) +

1 2K m

• i=1
• λ(i) + Kfi(x)

2

+ − 1 2K λ2 2,

λ ∈ Rm, where K > 0 is a penalty parameter. Consider the dual function ˆ φ(λ) = min

x∈Q

• L(x, λ).

Main properties. Function ˆ φ is concave. Its gradient is Lipschitz continuous with constant 1

K .

Its unconstrained maximum is attained at the optimal dual solution. The corresponding point ˆ x(λ∗) is the optimal primal solution. Hint: Check that the equation

• λ(i) + Kfi(x)
• + = λ(i)

is equivalent to KKT(2,3).

• Yu. Nesterov

New primal-dual methods for functional constraints 8/19

Method of Augmented Lagrangians

• Yu. Nesterov

New primal-dual methods for functional constraints 9/19

Method of Augmented Lagrangians

Note that ∇ˆ φ(λ) = 1

K

• λ(i) + Kfi(x)
• + − 1

K λ.

• Yu. Nesterov

New primal-dual methods for functional constraints 9/19

Method of Augmented Lagrangians

Note that ∇ˆ φ(λ) = 1

K

• λ(i) + Kfi(x)
• + − 1

K λ.

Therefore, the usual gradient method λk+1 = λk + K∇ˆ φ(λk) is exactly as follows:

• Yu. Nesterov

New primal-dual methods for functional constraints 9/19

Method of Augmented Lagrangians

Note that ∇ˆ φ(λ) = 1

K

• λ(i) + Kfi(x)
• + − 1

K λ.

Therefore, the usual gradient method λk+1 = λk + K∇ˆ φ(λk) is exactly as follows: Method: λk+1 = (λk + Kf (ˆ x(λk)))+.

• Yu. Nesterov

New primal-dual methods for functional constraints 9/19

Method of Augmented Lagrangians

Note that ∇ˆ φ(λ) = 1

K

• λ(i) + Kfi(x)
• + − 1

K λ.

Therefore, the usual gradient method λk+1 = λk + K∇ˆ φ(λk) is exactly as follows: Method: λk+1 = (λk + Kf (ˆ x(λk)))+. Advantage: Fast convergence of the dual process.

• Yu. Nesterov

New primal-dual methods for functional constraints 9/19

Method of Augmented Lagrangians

Note that ∇ˆ φ(λ) = 1

K

• λ(i) + Kfi(x)
• + − 1

K λ.

Therefore, the usual gradient method λk+1 = λk + K∇ˆ φ(λk) is exactly as follows: Method: λk+1 = (λk + Kf (ˆ x(λk)))+. Advantage: Fast convergence of the dual process. Disadvantages:

• Yu. Nesterov

New primal-dual methods for functional constraints 9/19

Method of Augmented Lagrangians

Note that ∇ˆ φ(λ) = 1

K

• λ(i) + Kfi(x)
• + − 1

K λ.

Therefore, the usual gradient method λk+1 = λk + K∇ˆ φ(λk) is exactly as follows: Method: λk+1 = (λk + Kf (ˆ x(λk)))+. Advantage: Fast convergence of the dual process. Disadvantages: Difficult iteration.

• Yu. Nesterov

New primal-dual methods for functional constraints 9/19

Method of Augmented Lagrangians

Note that ∇ˆ φ(λ) = 1

K

• λ(i) + Kfi(x)
• + − 1

K λ.

Therefore, the usual gradient method λk+1 = λk + K∇ˆ φ(λk) is exactly as follows: Method: λk+1 = (λk + Kf (ˆ x(λk)))+. Advantage: Fast convergence of the dual process. Disadvantages: Difficult iteration. Unclear termination.

• Yu. Nesterov

New primal-dual methods for functional constraints 9/19

Method of Augmented Lagrangians

Note that ∇ˆ φ(λ) = 1

K

• λ(i) + Kfi(x)
• + − 1

K λ.

Therefore, the usual gradient method λk+1 = λk + K∇ˆ φ(λk) is exactly as follows: Method: λk+1 = (λk + Kf (ˆ x(λk)))+. Advantage: Fast convergence of the dual process. Disadvantages: Difficult iteration. Unclear termination. No global complexity analysis.

• Yu. Nesterov

New primal-dual methods for functional constraints 9/19

Method of Augmented Lagrangians

Note that ∇ˆ φ(λ) = 1

K

• λ(i) + Kfi(x)
• + − 1

K λ.

Therefore, the usual gradient method λk+1 = λk + K∇ˆ φ(λk) is exactly as follows: Method: λk+1 = (λk + Kf (ˆ x(λk)))+. Advantage: Fast convergence of the dual process. Disadvantages: Difficult iteration. Unclear termination. No global complexity analysis. Do we have an alternative?

• Yu. Nesterov

New primal-dual methods for functional constraints 9/19

Problem formulation

• Yu. Nesterov

New primal-dual methods for functional constraints 10/19

Problem formulation

Problem: f ∗ = inf

x∈Q{f0(x) : fi(x) ≤ 0, i = 1, . . . , m},

• Yu. Nesterov

New primal-dual methods for functional constraints 10/19

Problem formulation

Problem: f ∗ = inf

x∈Q{f0(x) : fi(x) ≤ 0, i = 1, . . . , m}, where

fi(x), i = 0, . . . , m, are closed convex functions on Q endowed with a first-order black-box oracles,

• Yu. Nesterov

New primal-dual methods for functional constraints 10/19

Problem formulation

Problem: f ∗ = inf

x∈Q{f0(x) : fi(x) ≤ 0, i = 1, . . . , m}, where

fi(x), i = 0, . . . , m, are closed convex functions on Q endowed with a first-order black-box oracles, Q ⊂ E is a bounded simple closed convex set.

• Yu. Nesterov

New primal-dual methods for functional constraints 10/19

Problem formulation

Problem: f ∗ = inf

x∈Q{f0(x) : fi(x) ≤ 0, i = 1, . . . , m}, where

fi(x), i = 0, . . . , m, are closed convex functions on Q endowed with a first-order black-box oracles, Q ⊂ E is a bounded simple closed convex set. (We can solve some auxiliary optimization problems over Q.)

• Yu. Nesterov

New primal-dual methods for functional constraints 10/19

Problem formulation

Problem: f ∗ = inf

x∈Q{f0(x) : fi(x) ≤ 0, i = 1, . . . , m}, where

fi(x), i = 0, . . . , m, are closed convex functions on Q endowed with a first-order black-box oracles, Q ⊂ E is a bounded simple closed convex set. (We can solve some auxiliary optimization problems over Q.) Defining the Lagrangian L(x, λ) = f0(x) +

m

• i=1

λ(i)fi(x), x ∈ Q, λ ∈ Rm

+,

• Yu. Nesterov

New primal-dual methods for functional constraints 10/19

Problem formulation

Problem: f ∗ = inf

x∈Q{f0(x) : fi(x) ≤ 0, i = 1, . . . , m}, where

fi(x), i = 0, . . . , m, are closed convex functions on Q endowed with a first-order black-box oracles, Q ⊂ E is a bounded simple closed convex set. (We can solve some auxiliary optimization problems over Q.) Defining the Lagrangian L(x, λ) = f0(x) +

m

• i=1

λ(i)fi(x), x ∈ Q, λ ∈ Rm

+,

we can introduce the Lagrangian dual problem f∗

def

= sup

λ∈Rm

+

φ(λ),

• Yu. Nesterov

New primal-dual methods for functional constraints 10/19

Problem formulation

Problem: f ∗ = inf

x∈Q{f0(x) : fi(x) ≤ 0, i = 1, . . . , m}, where

fi(x), i = 0, . . . , m, are closed convex functions on Q endowed with a first-order black-box oracles, Q ⊂ E is a bounded simple closed convex set. (We can solve some auxiliary optimization problems over Q.) Defining the Lagrangian L(x, λ) = f0(x) +

m

• i=1

λ(i)fi(x), x ∈ Q, λ ∈ Rm

+,

we can introduce the Lagrangian dual problem f∗

def

= sup

λ∈Rm

+

φ(λ), where φ(λ) def = inf

x∈Q L(x, λ).

• Yu. Nesterov

New primal-dual methods for functional constraints 10/19

Problem formulation

Problem: f ∗ = inf

x∈Q{f0(x) : fi(x) ≤ 0, i = 1, . . . , m}, where

fi(x), i = 0, . . . , m, are closed convex functions on Q endowed with a first-order black-box oracles, Q ⊂ E is a bounded simple closed convex set. (We can solve some auxiliary optimization problems over Q.) Defining the Lagrangian L(x, λ) = f0(x) +

m

• i=1

λ(i)fi(x), x ∈ Q, λ ∈ Rm

+,

we can introduce the Lagrangian dual problem f∗

def

= sup

λ∈Rm

+

φ(λ), where φ(λ) def = inf

x∈Q L(x, λ).

Clearly, f ∗ ≥ f∗.

• Yu. Nesterov

New primal-dual methods for functional constraints 10/19

Problem formulation

Problem: f ∗ = inf

x∈Q{f0(x) : fi(x) ≤ 0, i = 1, . . . , m}, where

fi(x), i = 0, . . . , m, are closed convex functions on Q endowed with a first-order black-box oracles, Q ⊂ E is a bounded simple closed convex set. (We can solve some auxiliary optimization problems over Q.) Defining the Lagrangian L(x, λ) = f0(x) +

m

• i=1

λ(i)fi(x), x ∈ Q, λ ∈ Rm

+,

we can introduce the Lagrangian dual problem f∗

def

= sup

λ∈Rm

+

φ(λ), where φ(λ) def = inf

x∈Q L(x, λ).

Clearly, f ∗ ≥ f∗. Later, we will show f ∗ = f∗ algorithmically.

• Yu. Nesterov

New primal-dual methods for functional constraints 10/19

Bregman distances

• Yu. Nesterov

New primal-dual methods for functional constraints 11/19

Bregman distances

Prox-function: d(·) is strongly convex on Q with parameter one: d(y) ≥ d(x) + ∇d(x), y − x + 1

2y − x2,

x, y ∈ Q.

• Yu. Nesterov

New primal-dual methods for functional constraints 11/19

Bregman distances

Prox-function: d(·) is strongly convex on Q with parameter one: d(y) ≥ d(x) + ∇d(x), y − x + 1

2y − x2,

x, y ∈ Q. Denote by x0 the prox-center of the set Q: x0 = arg min

x∈Q d(x).

• Yu. Nesterov

New primal-dual methods for functional constraints 11/19

Bregman distances

Prox-function: d(·) is strongly convex on Q with parameter one: d(y) ≥ d(x) + ∇d(x), y − x + 1

2y − x2,

x, y ∈ Q. Denote by x0 the prox-center of the set Q: x0 = arg min

x∈Q d(x).

Assume d(x0) = 0.

• Yu. Nesterov

New primal-dual methods for functional constraints 11/19

Bregman distances

Prox-function: d(·) is strongly convex on Q with parameter one: d(y) ≥ d(x) + ∇d(x), y − x + 1

2y − x2,

x, y ∈ Q. Denote by x0 the prox-center of the set Q: x0 = arg min

x∈Q d(x).

Assume d(x0) = 0. Bregman distance: β(x, y) = d(y) − d(x) − ∇d(x), y − x, x, y ∈ Q.

• Yu. Nesterov

New primal-dual methods for functional constraints 11/19

Bregman distances

Prox-function: d(·) is strongly convex on Q with parameter one: d(y) ≥ d(x) + ∇d(x), y − x + 1

2y − x2,

x, y ∈ Q. Denote by x0 the prox-center of the set Q: x0 = arg min

x∈Q d(x).

Assume d(x0) = 0. Bregman distance: β(x, y) = d(y) − d(x) − ∇d(x), y − x, x, y ∈ Q. Clearly, β(x, y) ≥ 1

2x − y2 for all x, y ∈ Q.

• Yu. Nesterov

New primal-dual methods for functional constraints 11/19

Bregman distances

Prox-function: d(·) is strongly convex on Q with parameter one: d(y) ≥ d(x) + ∇d(x), y − x + 1

2y − x2,

x, y ∈ Q. Denote by x0 the prox-center of the set Q: x0 = arg min

x∈Q d(x).

Assume d(x0) = 0. Bregman distance: β(x, y) = d(y) − d(x) − ∇d(x), y − x, x, y ∈ Q. Clearly, β(x, y) ≥ 1

2x − y2 for all x, y ∈ Q.

Bregman mapping: for x ∈ Q, g ∈ E ∗ and h > 0

• Yu. Nesterov

New primal-dual methods for functional constraints 11/19

Bregman distances

Prox-function: d(·) is strongly convex on Q with parameter one: d(y) ≥ d(x) + ∇d(x), y − x + 1

2y − x2,

x, y ∈ Q. Denote by x0 the prox-center of the set Q: x0 = arg min

x∈Q d(x).

Assume d(x0) = 0. Bregman distance: β(x, y) = d(y) − d(x) − ∇d(x), y − x, x, y ∈ Q. Clearly, β(x, y) ≥ 1

2x − y2 for all x, y ∈ Q.

Bregman mapping: for x ∈ Q, g ∈ E ∗ and h > 0 define Bh(x, g) = arg min

y∈Q{hg, y − x + β(x, y)}.

• Yu. Nesterov

New primal-dual methods for functional constraints 11/19

Bregman distances

Prox-function: d(·) is strongly convex on Q with parameter one: d(y) ≥ d(x) + ∇d(x), y − x + 1

2y − x2,

x, y ∈ Q. Denote by x0 the prox-center of the set Q: x0 = arg min

x∈Q d(x).

Assume d(x0) = 0. Bregman distance: β(x, y) = d(y) − d(x) − ∇d(x), y − x, x, y ∈ Q. Clearly, β(x, y) ≥ 1

2x − y2 for all x, y ∈ Q.

Bregman mapping: for x ∈ Q, g ∈ E ∗ and h > 0 define Bh(x, g) = arg min

y∈Q{hg, y − x + β(x, y)}.

Examples: Euclidean distance, Entropy distance, etc.

• Yu. Nesterov

New primal-dual methods for functional constraints 11/19

Switching subgradient methods: Primal Method

• Yu. Nesterov

New primal-dual methods for functional constraints 12/19

Switching subgradient methods: Primal Method

Input parameter: the step size h > 0.

• Yu. Nesterov

New primal-dual methods for functional constraints 12/19

Switching subgradient methods: Primal Method

Input parameter: the step size h > 0. Initialization : Compute the prox-center x0.

• Yu. Nesterov

New primal-dual methods for functional constraints 12/19

Switching subgradient methods: Primal Method

Input parameter: the step size h > 0. Initialization : Compute the prox-center x0. Iteration k ≥ 0 :

• Yu. Nesterov

New primal-dual methods for functional constraints 12/19

Switching subgradient methods: Primal Method

Input parameter: the step size h > 0. Initialization : Compute the prox-center x0. Iteration k ≥ 0 : a) Define Ik = {i ∈ {1, . . . , m} : fi(xk) > h∇fi(xk)∗}.

• Yu. Nesterov

New primal-dual methods for functional constraints 12/19

Switching subgradient methods: Primal Method

Input parameter: the step size h > 0. Initialization : Compute the prox-center x0. Iteration k ≥ 0 : a) Define Ik = {i ∈ {1, . . . , m} : fi(xk) > h∇fi(xk)∗}. b) If Ik = ∅, then compute xk+1 = Bh

• xk,

∇f0(xk) ∇f0(xk)∗

• .
• Yu. Nesterov

New primal-dual methods for functional constraints 12/19

Switching subgradient methods: Primal Method

Input parameter: the step size h > 0. Initialization : Compute the prox-center x0. Iteration k ≥ 0 : a) Define Ik = {i ∈ {1, . . . , m} : fi(xk) > h∇fi(xk)∗}. b) If Ik = ∅, then compute xk+1 = Bh

• xk,

∇f0(xk) ∇f0(xk)∗

• .

c) If Ik = ∅, then choose arbitrary ik ∈ Ik and define hk =

fik (xk) ∇fik (xk)2

∗ .

• Yu. Nesterov

New primal-dual methods for functional constraints 12/19

Switching subgradient methods: Primal Method

Input parameter: the step size h > 0. Initialization : Compute the prox-center x0. Iteration k ≥ 0 : a) Define Ik = {i ∈ {1, . . . , m} : fi(xk) > h∇fi(xk)∗}. b) If Ik = ∅, then compute xk+1 = Bh

• xk,

∇f0(xk) ∇f0(xk)∗

• .

c) If Ik = ∅, then choose arbitrary ik ∈ Ik and define hk =

fik (xk) ∇fik (xk)2

∗ . Compute xk+1 = Bhk(xk, ∇fik(xk)).

• Yu. Nesterov

New primal-dual methods for functional constraints 12/19

Switching subgradient methods: Primal Method

Input parameter: the step size h > 0. Initialization : Compute the prox-center x0. Iteration k ≥ 0 : a) Define Ik = {i ∈ {1, . . . , m} : fi(xk) > h∇fi(xk)∗}. b) If Ik = ∅, then compute xk+1 = Bh

• xk,

∇f0(xk) ∇f0(xk)∗

• .

c) If Ik = ∅, then choose arbitrary ik ∈ Ik and define hk =

fik (xk) ∇fik (xk)2

∗ . Compute xk+1 = Bhk(xk, ∇fik(xk)).

After t ≥ 0 iterations, define Ft = {k ∈ {0, . . . , t} : Ik = ∅}.

• Yu. Nesterov

New primal-dual methods for functional constraints 12/19

Switching subgradient methods: Primal Method

Input parameter: the step size h > 0. Initialization : Compute the prox-center x0. Iteration k ≥ 0 : a) Define Ik = {i ∈ {1, . . . , m} : fi(xk) > h∇fi(xk)∗}. b) If Ik = ∅, then compute xk+1 = Bh

• xk,

∇f0(xk) ∇f0(xk)∗

• .

c) If Ik = ∅, then choose arbitrary ik ∈ Ik and define hk =

fik (xk) ∇fik (xk)2

∗ . Compute xk+1 = Bhk(xk, ∇fik(xk)).

After t ≥ 0 iterations, define Ft = {k ∈ {0, . . . , t} : Ik = ∅}. Denote N(t) = |F(t)|.

• Yu. Nesterov

New primal-dual methods for functional constraints 12/19

Switching subgradient methods: Primal Method

Input parameter: the step size h > 0. Initialization : Compute the prox-center x0. Iteration k ≥ 0 : a) Define Ik = {i ∈ {1, . . . , m} : fi(xk) > h∇fi(xk)∗}. b) If Ik = ∅, then compute xk+1 = Bh

• xk,

∇f0(xk) ∇f0(xk)∗

• .

c) If Ik = ∅, then choose arbitrary ik ∈ Ik and define hk =

fik (xk) ∇fik (xk)2

∗ . Compute xk+1 = Bhk(xk, ∇fik(xk)).

After t ≥ 0 iterations, define Ft = {k ∈ {0, . . . , t} : Ik = ∅}. Denote N(t) = |F(t)|. It is possible that N(t) = 0.

• Yu. Nesterov

New primal-dual methods for functional constraints 12/19

Finding the dual multipliers

• Yu. Nesterov

New primal-dual methods for functional constraints 13/19

Finding the dual multipliers

if N(t) > 0, define the dual multipliers as follows:

• Yu. Nesterov

New primal-dual methods for functional constraints 13/19

Finding the dual multipliers

if N(t) > 0, define the dual multipliers as follows: λ(0)

t

= h

k∈Ft 1 ∇f0(xk)∗ ,

• Yu. Nesterov

New primal-dual methods for functional constraints 13/19

Finding the dual multipliers

if N(t) > 0, define the dual multipliers as follows: λ(0)

t

= h

k∈Ft 1 ∇f0(xk)∗ ,

λ(i)

t

=

1 λ(0)

t

• k∈Ai(t)

hk, i = 1, . . . , m,

• Yu. Nesterov

New primal-dual methods for functional constraints 13/19

Finding the dual multipliers

if N(t) > 0, define the dual multipliers as follows: λ(0)

t

= h

k∈Ft 1 ∇f0(xk)∗ ,

λ(i)

t

=

1 λ(0)

t

• k∈Ai(t)

hk, i = 1, . . . , m, where Ai(t) = {k ∈ {0, . . . , t} : ik = i}, 0 ≤ i ≤ m.

• Yu. Nesterov

New primal-dual methods for functional constraints 13/19

Finding the dual multipliers

if N(t) > 0, define the dual multipliers as follows: λ(0)

t

= h

k∈Ft 1 ∇f0(xk)∗ ,

λ(i)

t

=

1 λ(0)

t

• k∈Ai(t)

hk, i = 1, . . . , m, where Ai(t) = {k ∈ {0, . . . , t} : ik = i}, 0 ≤ i ≤ m. Denote St =

k∈Ft 1 ∇f0(xk)∗ .

• Yu. Nesterov

New primal-dual methods for functional constraints 13/19

Finding the dual multipliers

if N(t) > 0, define the dual multipliers as follows: λ(0)

t

= h

k∈Ft 1 ∇f0(xk)∗ ,

λ(i)

t

=

1 λ(0)

t

• k∈Ai(t)

hk, i = 1, . . . , m, where Ai(t) = {k ∈ {0, . . . , t} : ik = i}, 0 ≤ i ≤ m. Denote St =

k∈Ft 1 ∇f0(xk)∗ . If Ft = ∅, then we define St = 0.

• Yu. Nesterov

New primal-dual methods for functional constraints 13/19

Finding the dual multipliers

if N(t) > 0, define the dual multipliers as follows: λ(0)

t

= h

k∈Ft 1 ∇f0(xk)∗ ,

λ(i)

t

=

1 λ(0)

t

• k∈Ai(t)

hk, i = 1, . . . , m, where Ai(t) = {k ∈ {0, . . . , t} : ik = i}, 0 ≤ i ≤ m. Denote St =

k∈Ft 1 ∇f0(xk)∗ . If Ft = ∅, then we define St = 0.

For proving convergence of the switching strategy,

• Yu. Nesterov

New primal-dual methods for functional constraints 13/19

Finding the dual multipliers

if N(t) > 0, define the dual multipliers as follows: λ(0)

t

= h

k∈Ft 1 ∇f0(xk)∗ ,

λ(i)

t

=

1 λ(0)

t

• k∈Ai(t)

hk, i = 1, . . . , m, where Ai(t) = {k ∈ {0, . . . , t} : ik = i}, 0 ≤ i ≤ m. Denote St =

k∈Ft 1 ∇f0(xk)∗ . If Ft = ∅, then we define St = 0.

For proving convergence of the switching strategy, we find an upper bound for the gap δt = 1

St

• k∈F(t)

f0(xk) ∇f0(xk)∗ − φ(λt),

• Yu. Nesterov

New primal-dual methods for functional constraints 13/19

Finding the dual multipliers

if N(t) > 0, define the dual multipliers as follows: λ(0)

t

= h

k∈Ft 1 ∇f0(xk)∗ ,

λ(i)

t

=

1 λ(0)

t

• k∈Ai(t)

hk, i = 1, . . . , m, where Ai(t) = {k ∈ {0, . . . , t} : ik = i}, 0 ≤ i ≤ m. Denote St =

k∈Ft 1 ∇f0(xk)∗ . If Ft = ∅, then we define St = 0.

For proving convergence of the switching strategy, we find an upper bound for the gap δt = 1

St

• k∈F(t)

f0(xk) ∇f0(xk)∗ − φ(λt),

assuming that N(t) > 0.

• Yu. Nesterov

New primal-dual methods for functional constraints 13/19

Convergence result

• Yu. Nesterov

New primal-dual methods for functional constraints 14/19

Convergence result

Main inequality: λ(0)

t δt

• Yu. Nesterov

New primal-dual methods for functional constraints 14/19

Convergence result

Main inequality: λ(0)

t δt ≤ r0(x) + 1 2N(t)h2 − 1 2(t − N(t))h2

• Yu. Nesterov

New primal-dual methods for functional constraints 14/19

Convergence result

Main inequality: λ(0)

t δt ≤ r0(x) + 1 2N(t)h2 − 1 2(t − N(t))h2 = r0(x) − 1 2th2 + N(t)h2.

• Yu. Nesterov

New primal-dual methods for functional constraints 14/19

Convergence result

Main inequality: λ(0)

t δt ≤ r0(x) + 1 2N(t)h2 − 1 2(t − N(t))h2 = r0(x) − 1 2th2 + N(t)h2.

Denote D = max

x∈Q r0(x).

• Yu. Nesterov

New primal-dual methods for functional constraints 14/19

Convergence result

Main inequality: λ(0)

t δt ≤ r0(x) + 1 2N(t)h2 − 1 2(t − N(t))h2 = r0(x) − 1 2th2 + N(t)h2.

Denote D = max

x∈Q r0(x).

• Theorem. If the number t ≥ 2

h2 D,

• Yu. Nesterov

New primal-dual methods for functional constraints 14/19

Convergence result

Main inequality: λ(0)

t δt ≤ r0(x) + 1 2N(t)h2 − 1 2(t − N(t))h2 = r0(x) − 1 2th2 + N(t)h2.

Denote D = max

x∈Q r0(x).

• Theorem. If the number t ≥ 2

h2 D, then F(t) = ∅.

• Yu. Nesterov

New primal-dual methods for functional constraints 14/19

Convergence result

Main inequality: λ(0)

t δt ≤ r0(x) + 1 2N(t)h2 − 1 2(t − N(t))h2 = r0(x) − 1 2th2 + N(t)h2.

Denote D = max

x∈Q r0(x).

• Theorem. If the number t ≥ 2

h2 D, then F(t) = ∅.

In this case δt ≤ Mh

• Yu. Nesterov

New primal-dual methods for functional constraints 14/19

Convergence result

Main inequality: λ(0)

t δt ≤ r0(x) + 1 2N(t)h2 − 1 2(t − N(t))h2 = r0(x) − 1 2th2 + N(t)h2.

Denote D = max

x∈Q r0(x).

• Theorem. If the number t ≥ 2

h2 D, then F(t) = ∅.

In this case δt ≤ Mh and max

1≤i≤m fi(xk) ≤ Mh, k ∈ F(t)

where M = max

0≤k≤t max 0≤i≤m ∇fi(xk)∗.

• Yu. Nesterov

New primal-dual methods for functional constraints 14/19

Convergence result

Main inequality: λ(0)

t δt ≤ r0(x) + 1 2N(t)h2 − 1 2(t − N(t))h2 = r0(x) − 1 2th2 + N(t)h2.

Denote D = max

x∈Q r0(x).

• Theorem. If the number t ≥ 2

h2 D, then F(t) = ∅.

In this case δt ≤ Mh and max

1≤i≤m fi(xk) ≤ Mh, k ∈ F(t)

where M = max

0≤k≤t max 0≤i≤m ∇fi(xk)∗.

Proof:

• Yu. Nesterov

New primal-dual methods for functional constraints 14/19

Convergence result

Main inequality: λ(0)

t δt ≤ r0(x) + 1 2N(t)h2 − 1 2(t − N(t))h2 = r0(x) − 1 2th2 + N(t)h2.

Denote D = max

x∈Q r0(x).

• Theorem. If the number t ≥ 2

h2 D, then F(t) = ∅.

In this case δt ≤ Mh and max

1≤i≤m fi(xk) ≤ Mh, k ∈ F(t)

where M = max

0≤k≤t max 0≤i≤m ∇fi(xk)∗.

Proof: If F(t) = ∅, then N(t) = 0.

• Yu. Nesterov

New primal-dual methods for functional constraints 14/19

Convergence result

Main inequality: λ(0)

t δt ≤ r0(x) + 1 2N(t)h2 − 1 2(t − N(t))h2 = r0(x) − 1 2th2 + N(t)h2.

Denote D = max

x∈Q r0(x).

• Theorem. If the number t ≥ 2

h2 D, then F(t) = ∅.

In this case δt ≤ Mh and max

1≤i≤m fi(xk) ≤ Mh, k ∈ F(t)

where M = max

0≤k≤t max 0≤i≤m ∇fi(xk)∗.

Proof: If F(t) = ∅, then N(t) = 0. Consequently, λ(0)

t

= 0.

• Yu. Nesterov

New primal-dual methods for functional constraints 14/19

Convergence result

Main inequality: λ(0)

t δt ≤ r0(x) + 1 2N(t)h2 − 1 2(t − N(t))h2 = r0(x) − 1 2th2 + N(t)h2.

Denote D = max

x∈Q r0(x).

• Theorem. If the number t ≥ 2

h2 D, then F(t) = ∅.

In this case δt ≤ Mh and max

1≤i≤m fi(xk) ≤ Mh, k ∈ F(t)

where M = max

0≤k≤t max 0≤i≤m ∇fi(xk)∗.

Proof: If F(t) = ∅, then N(t) = 0. Consequently, λ(0)

t

= 0. This is impossible for t big enough.

• Yu. Nesterov

New primal-dual methods for functional constraints 14/19

Convergence result

Main inequality: λ(0)

t δt ≤ r0(x) + 1 2N(t)h2 − 1 2(t − N(t))h2 = r0(x) − 1 2th2 + N(t)h2.

Denote D = max

x∈Q r0(x).

• Theorem. If the number t ≥ 2

h2 D, then F(t) = ∅.

In this case δt ≤ Mh and max

1≤i≤m fi(xk) ≤ Mh, k ∈ F(t)

where M = max

0≤k≤t max 0≤i≤m ∇fi(xk)∗.

Proof: If F(t) = ∅, then N(t) = 0. Consequently, λ(0)

t

= 0. This is impossible for t big enough. Finally, λ(0)

t

≥ h

M N(t).

• Yu. Nesterov

New primal-dual methods for functional constraints 14/19

Convergence result

Main inequality: λ(0)

t δt ≤ r0(x) + 1 2N(t)h2 − 1 2(t − N(t))h2 = r0(x) − 1 2th2 + N(t)h2.

Denote D = max

x∈Q r0(x).

• Theorem. If the number t ≥ 2

h2 D, then F(t) = ∅.

In this case δt ≤ Mh and max

1≤i≤m fi(xk) ≤ Mh, k ∈ F(t)

where M = max

0≤k≤t max 0≤i≤m ∇fi(xk)∗.

Proof: If F(t) = ∅, then N(t) = 0. Consequently, λ(0)

t

= 0. This is impossible for t big enough. Finally, λ(0)

t

≥ h

M N(t). Therefore, if t is big enough,

• Yu. Nesterov

New primal-dual methods for functional constraints 14/19

Convergence result

Main inequality: λ(0)

t δt ≤ r0(x) + 1 2N(t)h2 − 1 2(t − N(t))h2 = r0(x) − 1 2th2 + N(t)h2.

Denote D = max

x∈Q r0(x).

• Theorem. If the number t ≥ 2

h2 D, then F(t) = ∅.

In this case δt ≤ Mh and max

1≤i≤m fi(xk) ≤ Mh, k ∈ F(t)

where M = max

0≤k≤t max 0≤i≤m ∇fi(xk)∗.

Proof: If F(t) = ∅, then N(t) = 0. Consequently, λ(0)

t

= 0. This is impossible for t big enough. Finally, λ(0)

t

≥ h

M N(t). Therefore, if t is big enough, then

δt ≤ N(t)h2

λ(0)

t

• Yu. Nesterov

New primal-dual methods for functional constraints 14/19

Convergence result

Main inequality: λ(0)

t δt ≤ r0(x) + 1 2N(t)h2 − 1 2(t − N(t))h2 = r0(x) − 1 2th2 + N(t)h2.

Denote D = max

x∈Q r0(x).

• Theorem. If the number t ≥ 2

h2 D, then F(t) = ∅.

In this case δt ≤ Mh and max

1≤i≤m fi(xk) ≤ Mh, k ∈ F(t)

where M = max

0≤k≤t max 0≤i≤m ∇fi(xk)∗.

Proof: If F(t) = ∅, then N(t) = 0. Consequently, λ(0)

t

= 0. This is impossible for t big enough. Finally, λ(0)

t

≥ h

M N(t). Therefore, if t is big enough, then

δt ≤ N(t)h2

λ(0)

t

≤ Mh.

• Yu. Nesterov

New primal-dual methods for functional constraints 14/19

Dual subgradient method

• Yu. Nesterov

New primal-dual methods for functional constraints 15/19

Dual subgradient method

Define the averaging coefficients {ak}k≥0, and nondecreasing scaling coefficients {βk}k≥0. Denote At =

t

• k=0

ak.

• Yu. Nesterov

New primal-dual methods for functional constraints 15/19

Dual subgradient method

Define the averaging coefficients {ak}k≥0, and nondecreasing scaling coefficients {βk}k≥0. Denote At =

t

• k=0

ak. Initialization : Define ℓ0(x) ≡ 0, x ∈ Q.

• Yu. Nesterov

New primal-dual methods for functional constraints 15/19

Dual subgradient method

Define the averaging coefficients {ak}k≥0, and nondecreasing scaling coefficients {βk}k≥0. Denote At =

t

• k=0

ak. Initialization : Define ℓ0(x) ≡ 0, x ∈ Q. Iteration k ≥ 0 : a) Compute xk = arg min

x∈Q {ℓk(x) + βkd(x)}.

• Yu. Nesterov

New primal-dual methods for functional constraints 15/19

Dual subgradient method

Define the averaging coefficients {ak}k≥0, and nondecreasing scaling coefficients {βk}k≥0. Denote At =

t

• k=0

ak. Initialization : Define ℓ0(x) ≡ 0, x ∈ Q. Iteration k ≥ 0 : a) Compute xk = arg min

x∈Q {ℓk(x) + βkd(x)}.

b) Define Ik = {i ∈ [1 : m] : f (i)(xk) ≥ ǫ}.

• Yu. Nesterov

New primal-dual methods for functional constraints 15/19

Dual subgradient method

Define the averaging coefficients {ak}k≥0, and nondecreasing scaling coefficients {βk}k≥0. Denote At =

t

• k=0

ak. Initialization : Define ℓ0(x) ≡ 0, x ∈ Q. Iteration k ≥ 0 : a) Compute xk = arg min

x∈Q {ℓk(x) + βkd(x)}.

b) Define Ik = {i ∈ [1 : m] : f (i)(xk) ≥ ǫ}. c) If Ik = ∅, then ℓk+1(x) = ℓk(x) + ak[f (0)(xk) + ∇f (0)(xk), x − xk].

• Yu. Nesterov

New primal-dual methods for functional constraints 15/19

Dual subgradient method

Define the averaging coefficients {ak}k≥0, and nondecreasing scaling coefficients {βk}k≥0. Denote At =

t

• k=0

ak. Initialization : Define ℓ0(x) ≡ 0, x ∈ Q. Iteration k ≥ 0 : a) Compute xk = arg min

x∈Q {ℓk(x) + βkd(x)}.

b) Define Ik = {i ∈ [1 : m] : f (i)(xk) ≥ ǫ}. c) If Ik = ∅, then ℓk+1(x) = ℓk(x) + ak[f (0)(xk) + ∇f (0)(xk), x − xk]. d) If Ik = ∅, then choose arbitrary ik ∈ Ik and define ℓk+1(x) = ℓk(x) + ak[f (ik)(xk) + ∇f (ik)(xk), x − xk].

• Yu. Nesterov

New primal-dual methods for functional constraints 15/19

Convergence result

• Yu. Nesterov

New primal-dual methods for functional constraints 16/19

Convergence result

Define A0(t) = {k ∈ [0 : t] : Ik = ∅}, N(t) = |A0(t)|, and σt =

• k∈A0(t)

ak, λ(i)

t

=

1 σt

• k∈Ai(t)

ak, i = 1, . . . , m, where Ai(t) = {k ∈ [0 : t] : ik = i}, 1 ≤ i ≤ m.

• Yu. Nesterov

New primal-dual methods for functional constraints 16/19

Convergence result

Define A0(t) = {k ∈ [0 : t] : Ik = ∅}, N(t) = |A0(t)|, and σt =

• k∈A0(t)

ak, λ(i)

t

=

1 σt

• k∈Ai(t)

ak, i = 1, . . . , m, where Ai(t) = {k ∈ [0 : t] : ik = i}, 1 ≤ i ≤ m. If N(t) > 0, then define the gap δt = 1

σt

• k∈F(t)

akf (0)(xk) − φ(λt).

• Yu. Nesterov

New primal-dual methods for functional constraints 16/19

Convergence result

Define A0(t) = {k ∈ [0 : t] : Ik = ∅}, N(t) = |A0(t)|, and σt =

• k∈A0(t)

ak, λ(i)

t

=

1 σt

• k∈Ai(t)

ak, i = 1, . . . , m, where Ai(t) = {k ∈ [0 : t] : ik = i}, 1 ≤ i ≤ m. If N(t) > 0, then define the gap δt = 1

σt

• k∈F(t)

akf (0)(xk) − φ(λt). Denote D = max

x∈Q d(x).

• Yu. Nesterov

New primal-dual methods for functional constraints 16/19

Convergence result

Define A0(t) = {k ∈ [0 : t] : Ik = ∅}, N(t) = |A0(t)|, and σt =

• k∈A0(t)

ak, λ(i)

t

=

1 σt

• k∈Ai(t)

ak, i = 1, . . . , m, where Ai(t) = {k ∈ [0 : t] : ik = i}, 1 ≤ i ≤ m. If N(t) > 0, then define the gap δt = 1

σt

• k∈F(t)

akf (0)(xk) − φ(λt). Denote D = max

x∈Q d(x).

• T. Let all subgradients be bounded by M. Then for any t ≥ 0

σt · (δt − ǫ) + Atǫ ≤ βt+1D + 1

2M2 t

• k=0

a2

k

βk .

• Yu. Nesterov

New primal-dual methods for functional constraints 16/19

Convergence result

Define A0(t) = {k ∈ [0 : t] : Ik = ∅}, N(t) = |A0(t)|, and σt =

• k∈A0(t)

ak, λ(i)

t

=

1 σt

• k∈Ai(t)

ak, i = 1, . . . , m, where Ai(t) = {k ∈ [0 : t] : ik = i}, 1 ≤ i ≤ m. If N(t) > 0, then define the gap δt = 1

σt

• k∈F(t)

akf (0)(xk) − φ(λt). Denote D = max

x∈Q d(x).

• T. Let all subgradients be bounded by M. Then for any t ≥ 0

σt · (δt − ǫ) + Atǫ ≤ βt+1D + 1

2M2 t

• k=0

a2

k

βk .

If Atǫ > βt+1D + 1

2M2 t

• k=0

a2

k

βk , then λ(0) t

> 0 and δt ≤ ǫ.

• Yu. Nesterov

New primal-dual methods for functional constraints 16/19

Convergence result

Define A0(t) = {k ∈ [0 : t] : Ik = ∅}, N(t) = |A0(t)|, and σt =

• k∈A0(t)

ak, λ(i)

t

=

1 σt

• k∈Ai(t)

ak, i = 1, . . . , m, where Ai(t) = {k ∈ [0 : t] : ik = i}, 1 ≤ i ≤ m. If N(t) > 0, then define the gap δt = 1

σt

• k∈F(t)

akf (0)(xk) − φ(λt). Denote D = max

x∈Q d(x).

• T. Let all subgradients be bounded by M. Then for any t ≥ 0

σt · (δt − ǫ) + Atǫ ≤ βt+1D + 1

2M2 t

• k=0

a2

k

βk .

If Atǫ > βt+1D + 1

2M2 t

• k=0

a2

k

βk , then λ(0) t

> 0 and δt ≤ ǫ. Example: at ≡ 1, βt ≈ √t ⇒ t ≈ O 1

ǫ2

• .
• Yu. Nesterov

New primal-dual methods for functional constraints 16/19

Quasi-monotone method

• Yu. Nesterov

New primal-dual methods for functional constraints 17/19

Quasi-monotone method

Initialization : Define ℓ0(x) ≡ 0, x0 = ⊲ ⊳, and σ0 = 0. Iteration t ≥ 0 :

• Yu. Nesterov

New primal-dual methods for functional constraints 17/19

Quasi-monotone method

Initialization : Define ℓ0(x) ≡ 0, x0 = ⊲ ⊳, and σ0 = 0. Iteration t ≥ 0 : a) Set vt = arg min

x∈Q {ℓt(x) + βtd(x)}, It def

= {i ∈ [1 : m] : f (i)(vt) ≥ ǫ}.

• Yu. Nesterov

New primal-dual methods for functional constraints 17/19

Quasi-monotone method

Initialization : Define ℓ0(x) ≡ 0, x0 = ⊲ ⊳, and σ0 = 0. Iteration t ≥ 0 : a) Set vt = arg min

x∈Q {ℓt(x) + βtd(x)}, It def

= {i ∈ [1 : m] : f (i)(vt) ≥ ǫ}. b) If It = ∅, then set xt+1 = xt, σt+1 = σt, and choose arbitrary it ∈ It. Update ℓt+1(x) = ℓt(x) + at+1[f (it)(vt) + ∇f (it)(vt), x − vt].

• Yu. Nesterov

New primal-dual methods for functional constraints 17/19

Quasi-monotone method

Initialization : Define ℓ0(x) ≡ 0, x0 = ⊲ ⊳, and σ0 = 0. Iteration t ≥ 0 : a) Set vt = arg min

x∈Q {ℓt(x) + βtd(x)}, It def

= {i ∈ [1 : m] : f (i)(vt) ≥ ǫ}. b) If It = ∅, then set xt+1 = xt, σt+1 = σt, and choose arbitrary it ∈ It. Update ℓt+1(x) = ℓt(x) + at+1[f (it)(vt) + ∇f (it)(vt), x − vt]. c) Otherwise, σt+1 = σt + at+1, τt = at+1

σt+1 , xt+1 = (1 − τt)xt + τtvt.

Update ℓt+1(x) = ℓt(x) + at+1[f (0)(xt+1) + ∇f (0)(xt+1), x − xt+1].

• Yu. Nesterov

New primal-dual methods for functional constraints 17/19

Quasi-monotone method

Initialization : Define ℓ0(x) ≡ 0, x0 = ⊲ ⊳, and σ0 = 0. Iteration t ≥ 0 : a) Set vt = arg min

x∈Q {ℓt(x) + βtd(x)}, It def

= {i ∈ [1 : m] : f (i)(vt) ≥ ǫ}. b) If It = ∅, then set xt+1 = xt, σt+1 = σt, and choose arbitrary it ∈ It. Update ℓt+1(x) = ℓt(x) + at+1[f (it)(vt) + ∇f (it)(vt), x − vt]. c) Otherwise, σt+1 = σt + at+1, τt = at+1

σt+1 , xt+1 = (1 − τt)xt + τtvt.

Update ℓt+1(x) = ℓt(x) + at+1[f (0)(xt+1) + ∇f (0)(xt+1), x − xt+1]. Operation x0 = ⊲ ⊳ ∈ E indicates that x0 is not chosen yet.

• Yu. Nesterov

New primal-dual methods for functional constraints 17/19

Convergence result

• Yu. Nesterov

New primal-dual methods for functional constraints 18/19

Convergence result

• Theorem. 1. All points xt = x⊲

⊳ are ǫ-feasible.

• Yu. Nesterov

New primal-dual methods for functional constraints 18/19

Convergence result

• Theorem. 1. All points xt = x⊲

⊳ are ǫ-feasible.

• 2. If all subgradients are bounded by M, then ∀x ∈ Q, t ≥ 0,

σt(f (0)(xt) − ǫ) + Atǫ ≤ ℓt(x) + βtd(x) + 1

2M2 t

• k=1

a2

k

βk−1 .

• Yu. Nesterov

New primal-dual methods for functional constraints 18/19

Convergence result

• Theorem. 1. All points xt = x⊲

⊳ are ǫ-feasible.

• 2. If all subgradients are bounded by M, then ∀x ∈ Q, t ≥ 0,

σt(f (0)(xt) − ǫ) + Atǫ ≤ ℓt(x) + βtd(x) + 1

2M2 t

• k=1

a2

k

βk−1 .

• 3. No later than Atǫ > βtD + 1

2M2 t

• k=1

a2

k

βk−1 , we get σt > 0 and

• Yu. Nesterov

New primal-dual methods for functional constraints 18/19

Convergence result

• Theorem. 1. All points xt = x⊲

⊳ are ǫ-feasible.

• 2. If all subgradients are bounded by M, then ∀x ∈ Q, t ≥ 0,

σt(f (0)(xt) − ǫ) + Atǫ ≤ ℓt(x) + βtd(x) + 1

2M2 t

• k=1

a2

k

βk−1 .

• 3. No later than Atǫ > βtD + 1

2M2 t

• k=1

a2

k

βk−1 , we get σt > 0 and

f (xt) − φ(λt) ≤ f (xt) − 1

σt min x∈Q ℓt(x) ≤ ǫ.

• Yu. Nesterov

New primal-dual methods for functional constraints 18/19

Convergence result

• Theorem. 1. All points xt = x⊲

⊳ are ǫ-feasible.

• 2. If all subgradients are bounded by M, then ∀x ∈ Q, t ≥ 0,

σt(f (0)(xt) − ǫ) + Atǫ ≤ ℓt(x) + βtd(x) + 1

2M2 t

• k=1

a2

k

βk−1 .

• 3. No later than Atǫ > βtD + 1

2M2 t

• k=1

a2

k

βk−1 , we get σt > 0 and

f (xt) − φ(λt) ≤ f (xt) − 1

σt min x∈Q ℓt(x) ≤ ǫ.

Example: at ≡ 1, βt ≈ √t ⇒ t ≈ O 1

ǫ2

• .
• Yu. Nesterov

New primal-dual methods for functional constraints 18/19

Convergence result

• Theorem. 1. All points xt = x⊲

⊳ are ǫ-feasible.

• 2. If all subgradients are bounded by M, then ∀x ∈ Q, t ≥ 0,

σt(f (0)(xt) − ǫ) + Atǫ ≤ ℓt(x) + βtd(x) + 1

2M2 t

• k=1

a2

k

βk−1 .

• 3. No later than Atǫ > βtD + 1

2M2 t

• k=1

a2

k

βk−1 , we get σt > 0 and

f (xt) − φ(λt) ≤ f (xt) − 1

σt min x∈Q ℓt(x) ≤ ǫ.

Example: at ≡ 1, βt ≈ √t ⇒ t ≈ O 1

ǫ2

• .

NB: this is true for the whole sequence!

• Yu. Nesterov

New primal-dual methods for functional constraints 18/19

Conclusion

• Yu. Nesterov

New primal-dual methods for functional constraints 19/19

Conclusion

• 1. Optimal primal-dual solution can be approximated by simple

switching subgradient schemes.

• Yu. Nesterov

New primal-dual methods for functional constraints 19/19

Conclusion

• 1. Optimal primal-dual solution can be approximated by simple

switching subgradient schemes.

• 2. Approximations of dual multipliers have natural interpretation
• Yu. Nesterov

New primal-dual methods for functional constraints 19/19

Conclusion

• 1. Optimal primal-dual solution can be approximated by simple

switching subgradient schemes.

• 2. Approximations of dual multipliers have natural interpretation:

relative importance of corresponding constraints during the adjustments process.

• Yu. Nesterov

New primal-dual methods for functional constraints 19/19

Conclusion

• 1. Optimal primal-dual solution can be approximated by simple

switching subgradient schemes.

• 2. Approximations of dual multipliers have natural interpretation:

relative importance of corresponding constraints during the adjustments process.

• 3. However, it has optimal worst-case efficiency estimate
• Yu. Nesterov

New primal-dual methods for functional constraints 19/19

Conclusion

• 1. Optimal primal-dual solution can be approximated by simple

switching subgradient schemes.

• 2. Approximations of dual multipliers have natural interpretation:

relative importance of corresponding constraints during the adjustments process.

• 3. However, it has optimal worst-case efficiency estimate even if

the dual optimal solution does not exist.

• Yu. Nesterov

New primal-dual methods for functional constraints 19/19

Conclusion

• 1. Optimal primal-dual solution can be approximated by simple

switching subgradient schemes.

• 2. Approximations of dual multipliers have natural interpretation:

relative importance of corresponding constraints during the adjustments process.

• 3. However, it has optimal worst-case efficiency estimate even if

the dual optimal solution does not exist.

• 4. Many interesting questions (influence of smoothness, strong

convexity, etc.)

• Yu. Nesterov

New primal-dual methods for functional constraints 19/19

Conclusion

• 1. Optimal primal-dual solution can be approximated by simple

switching subgradient schemes.

• 2. Approximations of dual multipliers have natural interpretation:

relative importance of corresponding constraints during the adjustments process.

• 3. However, it has optimal worst-case efficiency estimate even if

the dual optimal solution does not exist.

• 4. Many interesting questions (influence of smoothness, strong

convexity, etc.) Thank you for your attention!

• Yu. Nesterov

New primal-dual methods for functional constraints 19/19