Problem Definition Machine learning problems, such as image - - PowerPoint PPT Presentation

Zhize Li, Jian Li

IIIS, Tsinghua University https://zhizeli.github.io/ Dec 6th, NeurIPS 2018

A Simple Proximal Stochastic Gradient Method for Nonsmooth Nonconvex Optimization

Problem Definition

Machine learning problems, such as image classification or voice recognition, are usually modeled as a (nonconvex) optimization problem:

Zhize Li (Tsinghua) A Simple ProxSVRG+ Algorithm

Goal: find a good enough solution (parameters) , e.g.,

min

𝜄 𝑀 𝜄 .

෠ 𝜄

‖𝛼𝑀 ෠ 𝜄 ‖2 ≤ 𝜗

2/7

Problem Definition

We consider the more general nonsmooth nonconvex case:

Zhize Li (Tsinghua) A Simple ProxSVRG+ Algorithm

min𝑦𝛸 𝑦 : = 𝑔 𝑦 + ℎ 𝑦 = 1 𝑜 ෍

𝑗=1 𝑜

ሻ 𝑔

𝑗(𝑦 + ℎ 𝑦 ,

Where and all are possibly nonconvex (loss on data samples),

ሻ 𝑔

𝑗(𝑦

ሻ 𝑔(𝑦

and is nonsmooth but convex (e.g., regularizer

• r indicator

function for some convex set ).

ሻ ℎ(𝑦 𝑚1 ‖𝑦‖1 𝐷 ሻ 𝐽𝐷(𝑦

3/7

Problem Definition

We consider the more general nonsmooth nonconvex case:

Zhize Li (Tsinghua) A Simple ProxSVRG+ Algorithm

min𝑦𝛸 𝑦 : = 𝑔 𝑦 + ℎ 𝑦 = 1 𝑜 ෍

𝑗=1 𝑜

ሻ 𝑔

𝑗(𝑦 + ℎ 𝑦 ,

Where and all are possibly nonconvex (loss on data samples),

ሻ 𝑔

𝑗(𝑦

ሻ 𝑔(𝑦

Benefit of : try to deal with the nonsmooth and constrained problems.

ሻ ℎ(𝑦

and is nonsmooth but convex (e.g., regularizer

• r indicator

function for some convex set ).

ሻ ℎ(𝑦 𝑚1 ‖𝑦‖1 𝐷 ሻ 𝐽𝐷(𝑦

3/7

Our Results

We propose a simple ProxSVRG+ algorithm, which recovers/improves several previous results (e.g., ProxGD, ProxSVRG/SAGA, SCSG).

Zhize Li (Tsinghua) A Simple ProxSVRG+ Algorithm 4/7

Our Results

We propose a simple ProxSVRG+ algorithm, which recovers/improves several previous results (e.g., ProxGD, ProxSVRG/SAGA, SCSG).

Zhize Li (Tsinghua) A Simple ProxSVRG+ Algorithm

Benefits: simpler algorithm, simpler analysis, better theoretical results,

4/7

Our Results

We propose a simple ProxSVRG+ algorithm, which recovers/improves several previous results (e.g., ProxGD, ProxSVRG/SAGA, SCSG).

Zhize Li (Tsinghua) A Simple ProxSVRG+ Algorithm

Benefits: simpler algorithm, simpler analysis, better theoretical results,

4/7

more attractive in practice (prefers moderate minibatch size, auto-adapt to local curvature, i.e., auto-switch to faster linear convergence in that regions although the objective function is generally nonconvex).

ሻ 𝑃(⋅ log Τ 1 𝜗

Theoretical Results

Our ProxSVRG+ prefers moderate minibatch size (red box) which is not too small for parallelism or vectorization and not too large for better generalization,

Zhize Li (Tsinghua) A Simple ProxSVRG+ Algorithm 5/7

Theoretical Results

Our ProxSVRG+ prefers moderate minibatch size (red box) which is not too small for parallelism or vectorization and not too large for better generalization,

Zhize Li (Tsinghua) A Simple ProxSVRG+ Algorithm 5/7

and uses less PO calls than ProxSVRG.

Theoretical Results

Our ProxSVRG+ prefers moderate minibatch size (red box) which is not too small for parallelism or vectorization and not too large for better generalization,

Zhize Li (Tsinghua) A Simple ProxSVRG+ Algorithm

Recently, [Zhou et al., 2018] and [Fang et al., 2018] improve the SFO to in the smooth setting.

൯ 𝑃( Τ 𝑜 Τ

1 2 𝜗

5/7

and uses less PO calls than ProxSVRG.

Experimental Results

Our ProxSVRG+ prefers much smaller minibatch size than ProxSVRG [Reddi et al., 2016], and performs much better than ProxGD and ProxSGD [Ghadimi et al., 2016].

Zhize Li (Tsinghua) A Simple ProxSVRG+ Algorithm 6/7

Our Poster: 5:00-7:00 PM Room 210 #5

Zhize Li (Tsinghua) A Simple ProxSVRG+ Algorithm

Thanks!

7/7