From Nesterovs Estimate Sequence To Riemannian Acceleration - - PowerPoint PPT Presentation

From Nesterov’s Estimate Sequence To Riemannian Acceleration

Kwangjun Ahn, Suvrit Sra COLT 2020 arXiv: https://arxiv.org/abs/2001.08876

Riemannian Optimization?

• (Euclidean) Optimization:
• Riemannian Optimization:

min

𝑦∈ℝ 𝑔(𝑦)

min

𝑦∈𝑁 𝑔(𝑦)

𝑁= a Riemannian manifold

𝑔: ℝ → ℝ 𝑔: 𝑁 → ℝ

• Accel. Gradient Method!
• Yurii Nesterov 80s
• Accel. Gradient Descent:

For 𝑢 0,1,2, … 𝑦1 𝑧 𝛽1 𝑨 𝑧 𝑧1 𝑦1 𝛿1𝛼𝑔 𝑦1 𝑨1 𝑦1 𝛾1 𝑨 𝑦1 𝜃1𝛼𝑔𝑦1

• Accel. Gradient Method: Theory
• Yurii Nesterov 80s

Nesterov showed: 𝑔 𝑧 𝑔 𝑦∗ 𝑃 1

• 𝑴
• For 𝜗-approx. solution,

We only need t 𝑃

𝑴 log

• .

Acceleration! For 𝜈 ≼ 𝛼𝑔 𝑦 ≼ 𝑀 (and indeed optimal for this class!) For 𝜈 ≼ 𝛼𝑔 𝑦 ≼ 𝑀 𝑔 𝑦 𝑔 𝑦∗ 𝑃 1

• 𝑴
• For 𝜗-approx. solution,

We need 𝑃

𝑴 log

• many iterations.

C.f. Gradient Descent:

Natural Question..

Could we develop such landmark result for curved spaces (Riem. manifolds)? Turns out to be challenging question:

Li et al.17 (NIPS) reduces the task to solving nonlinear equations.

Not clear whether whether these equations are even feasible or tractably solvable.

Alimisis et al.20 (AISTATS): Continuous dynamic approach

Not clear whether the discretization yields accel.

Most concrete result: Zhang-Sra18 (COLT) proposed an alg. guaranteed to accel. locally. Global accel? Open!

Challenge!

• Nesterovs analysis is called the Estimate Sequence

technique

• Nesterovs analysis relies on linear structure!

not clear if it generalizes to non-linear space like Riem. manifolds.

• Nesterovs analysis entails non-trivial algebraic tricks!

Hard to understand; its scope has puzzled researchers for years.

Riemannian Accel. GD

(Euclidean) Accel. Gradient Descent:

𝑦𝑢+1 𝑧𝑢 𝛽𝑢+1 𝑨𝑢 𝑧𝑢 𝑧𝑢+1 𝑦𝑢+1 𝛿𝑢+1𝛼𝑔 𝑦𝑢+1 𝑨𝑢+1 𝑦𝑢+1 𝛾𝑢+1 𝑨𝑢 𝑦𝑢+1 𝜃𝑢+1𝛼𝑔𝑦𝑢+1

Riemannian Accel. Gradient Descent:

𝑦𝑢+1 𝐹𝑦𝑞 𝛽𝑢+1 ⋅ 𝐹𝑦𝑞

−1 𝑨𝑢

𝑧𝑢+1 𝐹𝑦𝑞 𝛿𝑢+1 ⋅ 𝛼𝑔 𝑦𝑢+1 𝑨𝑢+1 𝐹𝑦𝑞 𝛾𝑢+1 ⋅ 𝐹𝑦𝑞

−1

𝑨𝑢 𝜃𝑢+1𝛼𝑔 𝑦𝑢+1

Space is curved, causes “distortion”

1.How does this affect the convergence rate?

• Non-linear recursive relation:

−/ 1−

• 𝟐

𝜺 𝜊 2

𝜈/𝑀 1

𝜐 𝑣 𝑣𝑣 𝜈/𝑀 1 𝑣 𝜄 𝑣 𝑣2

1 𝜈/𝑀

𝜄 𝑣 1 2 𝑣2 𝜄 𝑣 1 5 𝑣2 𝜀 5

Severer the distortion gets, Slower the convergence rate becomes! No matter how severe the distortion

• Riem. AGD always faster than RGD!

To achieve full accel. i.e. 𝜈/𝑀, we need bring 𝜀 down to 1! How do we control/estimate the distortion?

Global Accel for Riem. Case! Th Thm 2. . Given: 𝜊0 0

𝜊𝑢+1𝜊𝑢+1−2𝜈Δ 1−𝜊𝑢+1

• 𝟐

𝜺𝒖+𝟐 𝜊𝑢 2

Find 𝜊𝑢+1 ∈ 2𝜈Δ, 1 such that

𝑔 𝑧𝑢+1 𝑔 𝑦∗ 𝑃 1 𝜊1 1 𝜊2 ⋯ 1 𝜊𝑢+1

the magnitude of metric distortion at iteration t

where 𝜺𝒖+𝟐 𝑼𝒆𝒚𝒖, 𝒜𝒖 for some computable function 𝑈.

s.t. 1 𝜊𝑢 𝜈/𝑀 for all 𝑢. (2) 𝜊𝑢 quickly converges to 𝜈/𝑀. quickly acheives 𝐠𝐯𝐦𝐦 acceleartion! strictly 𝐠𝐛𝐭𝐮𝐟𝐬 than nonaccel GD!

Open problem

Obtaining acceleration the non-strongly convex case?

Remarks

★ Using strongly convex perturbation can be done ★ But, extra

factor

★ More crucially, our current proof needs to ensure all


iterates remain within a set of specific size to be able
 to ensure acceleration. Removing this limitation is valuable

O(log 1/ϵ)