Parallel Gradient Descent for Multilayer Feedforward Neural Networks - - PowerPoint PPT Presentation

Parallel Gradient Descent for Multilayer Feedforward Neural Networks

Palash Goyal1 Nitin Kamra1 Sungyong Seo1 Vasileios Zois1

1Department of Computer Science

University of Southern California

May 9, 2016

(University of Southern California) Parallel Gradient Descent for Multilayer Feedforward Neural Networks May 9, 2016 1 / 24

Outline

1

Introduction

2

Gradient Descent

3

Forward Propagation and Backpropagation

4

Parallel Gradient Descent

5

Experiments

6

Results and analysis

(University of Southern California) Parallel Gradient Descent for Multilayer Feedforward Neural Networks May 9, 2016 2 / 24

Outline

1

Introduction

2

Gradient Descent

3

Forward Propagation and Backpropagation

4

Parallel Gradient Descent

5

Experiments

6

Results and analysis

(University of Southern California) Parallel Gradient Descent for Multilayer Feedforward Neural Networks May 9, 2016 3 / 24

Introduction

How to learn to classify objects from images? What algorithms to use? How to scale up these algorithms?

(University of Southern California) Parallel Gradient Descent for Multilayer Feedforward Neural Networks May 9, 2016 4 / 24

Classification

Dataset D = {x(i), y(i)}i=1:N with x(i) ∈ RD and labels y(i) ∈ RP Make accurate prediction ˆ y on unseen data point x Classifier (parameters θ) approximates label as: y ≈ ˆ y = F(x; θ) Classifier learns parameters (θ) from data D to minimize a pre-specified loss function

(University of Southern California) Parallel Gradient Descent for Multilayer Feedforward Neural Networks May 9, 2016 5 / 24

Neuron

a = f (wTx + b) w ∈ Rn = Weight vector b ∈ R = Scalar bias

(University of Southern California) Parallel Gradient Descent for Multilayer Feedforward Neural Networks May 9, 2016 6 / 24

Classifier: Neural Network

For each layer, zl = (Wl)Txl + bl; al = f (zl) W l ∈ Rnl−1×nl = Weight vector bl ∈ Rnl = Scalar bias

(University of Southern California) Parallel Gradient Descent for Multilayer Feedforward Neural Networks May 9, 2016 7 / 24

Outline

1

Introduction

2

Gradient Descent

3

Forward Propagation and Backpropagation

4

Parallel Gradient Descent

5

Experiments

6

Results and analysis

(University of Southern California) Parallel Gradient Descent for Multilayer Feedforward Neural Networks May 9, 2016 8 / 24

Gradient Descent

Minimize the Mean-Squared Error loss: LMSE(θ) = 1 N

N

• i=1

(y(i) − f (x(i); θ))2 Algorithm: Gradient Descent

1 Initialize all weights (θ) randomly with small values close to 0. 2 Repeat until convergence {

θk := θk − α∂LMSE ∂θk ∀k ∈ {1, 2, ..., K} } Minibatch gradient descent considers a subset of examples

(University of Southern California) Parallel Gradient Descent for Multilayer Feedforward Neural Networks May 9, 2016 9 / 24

Outline

1

Introduction

2

Gradient Descent

3

Forward Propagation and Backpropagation

4

Parallel Gradient Descent

5

Experiments

6

Results and analysis

(University of Southern California) Parallel Gradient Descent for Multilayer Feedforward Neural Networks May 9, 2016 10 / 24

Forward Propagation

(University of Southern California) Parallel Gradient Descent for Multilayer Feedforward Neural Networks May 9, 2016 11 / 24

Backpropagation

(University of Southern California) Parallel Gradient Descent for Multilayer Feedforward Neural Networks May 9, 2016 12 / 24

Outline

1

Introduction

2

Gradient Descent

3

Forward Propagation and Backpropagation

4

Parallel Gradient Descent

5

Experiments

6

Results and analysis

(University of Southern California) Parallel Gradient Descent for Multilayer Feedforward Neural Networks May 9, 2016 13 / 24

Parallelizing Gradient Descent

Two ways to parallelize: Parallelize Gradient Descent: Derivative of the loss function has the following form: ∂LMSE ∂θk = 1 N

N

• i=1

(yi − f (xi; θ))∂f (xi; θ) ∂θk Distribute training examples, compute partial gradients, sum up partial gradients Parallelize Backpropagation: Parallelize matrix vector multiplications in forward propagation and backpropagation algorithms

(University of Southern California) Parallel Gradient Descent for Multilayer Feedforward Neural Networks May 9, 2016 14 / 24

Outline

1

Introduction

2

Gradient Descent

3

Forward Propagation and Backpropagation

4

Parallel Gradient Descent

5

Experiments

6

Results and analysis

(University of Southern California) Parallel Gradient Descent for Multilayer Feedforward Neural Networks May 9, 2016 15 / 24

MNIST dataset

28x28 images of handwritten digits 50,000 training examples, 10,000 test examples, 10,000 validation examples Labels: 0 to 9 (one-hot encoding)

(University of Southern California) Parallel Gradient Descent for Multilayer Feedforward Neural Networks May 9, 2016 16 / 24

Experiments

Network structures # Layers # Nodes # Num (In,Hidden,Out) (In,Hidden,Out) Params Network1 1,1,1 784,1024,10 800,000 Network2 1,2,1 784,1024,1024,10 1,860,000 Serial, Parallelize over examples (Pthreads, CUDA) Serial (BLAS), Parallelize matrix computations (BLAS) Serial (Keras:Theano), Parallel (Keras:Theano), GPU (Keras:Theano) Analyze time per epoch, gigaflops for each implementation Analyze speedup from parallelization over serial counterparts

(University of Southern California) Parallel Gradient Descent for Multilayer Feedforward Neural Networks May 9, 2016 17 / 24

Outline

1

Introduction

2

Gradient Descent

3

Forward Propagation and Backpropagation

4

Parallel Gradient Descent

5

Experiments

6

Results and analysis

(University of Southern California) Parallel Gradient Descent for Multilayer Feedforward Neural Networks May 9, 2016 18 / 24

Results - Time per Epoch

(University of Southern California) Parallel Gradient Descent for Multilayer Feedforward Neural Networks May 9, 2016 19 / 24

Results - Gigaflops

(University of Southern California) Parallel Gradient Descent for Multilayer Feedforward Neural Networks May 9, 2016 20 / 24

Results - Speedup

(University of Southern California) Parallel Gradient Descent for Multilayer Feedforward Neural Networks May 9, 2016 21 / 24

Analysis

Our implementation

Parallel computing average speedup ≈ 10 Training time decreases as minibatch size decreases

BLAS

Parallelizing each matrix vector product gives even faster results Speedup independent of batch size, but less than our implementation

CUDA

Our CUDA implementation gives about ≈ 20x speedup If # neurons per layer are not perfect multiple of 32 then some threads do not participate in computation

Theano

Apparently combines both types of parallelization Theano CUDA scales very fast with batch size

(University of Southern California) Parallel Gradient Descent for Multilayer Feedforward Neural Networks May 9, 2016 22 / 24

Future Work

Combine the two parallelization techniques: Split training examples amongst threads, further hierarchically parallelize matrix computations for each individual example.

(University of Southern California) Parallel Gradient Descent for Multilayer Feedforward Neural Networks May 9, 2016 23 / 24

Thank you Questions?

(University of Southern California) Parallel Gradient Descent for Multilayer Feedforward Neural Networks May 9, 2016 24 / 24