SnaPEA : Predictive Early Activation for Reducing Computation In - - PowerPoint PPT Presentation
SnaPEA : Predictive Early Activation for Reducing Computation In Deep Convolutional Neural Networks * * Vahideh Akhlaghi Amir Yazdanbakhsh Kambiz Samadi Rajesh K. Gupta Hadi Esmaeilzadeh * Equal Contribution University
Vahideh Akhlaghi Amir Yazdanbakhsh Kambiz Samadi Rajesh K. Gupta Hadi Esmaeilzadeh
University of California, San Diego †Georgia Institute of Technology
‡Qualcomm Technologies, Inc
Equal Contribution
ISCA ’18
* *† ‡ *
. . .
Dog
GoogLeNet
283,000,000,000 Ops
SqueezeNet
222,000,000,000 Ops
AlexNet
1,147,000,000,000 Ops
VGG-16
16,362,000,000,000 Ops CNN models
Operations for inference
2
. . .
Dog
GoogLeNet
283,000,000,000 Ops
SqueezeNet
222,000,000,000 Ops
AlexNet
1,147,000,000,000 Ops
VGG-16
16,362,000,000,000 Ops CNN models
Operations for inference
3
How to reduce CNN computation with minimal effect on accuracy? Our solution: SnaPEA
4
. . . . . .
Pooling
Conv
ReLU Normal.
Conv
ReLU
Conv
ReLU
Kernels
. . . 𝒙𝒍𝒌𝒋𝒚𝒍𝒌𝒋
𝒋 𝒌 𝒍
5
. . . . . .
Pooling
Conv
ReLU Normal.
Conv
ReLU
Conv
ReLU
Rectified Linear Unit (ReLU)
Kernels
. . . 𝒙𝒍𝒌𝒋𝒚𝒍𝒌𝒋
𝒋 𝒌 𝒍
6
Large number of negative convolution outputs (61% on average)
0% 20% 40% 60% 80% 100%
AlexNet GoogLeNet SqueezeNet VGGNet Average
Negative Inputs to the Activation Layers
. . .
Conv
. . .
ReLU
GoogLeNet
Black pixels are zero values
7
Blue boxes are the performed operations in two highlighted convolutions
Input Output
Convolution
8
GoogLeNet
. . . . . .
Pooling
Conv
ReLU Normal.
Conv
ReLU
Conv
ReLU
Rectified Linear Unit (ReLU)
9
SnaPEA: Leveraging algorithmic structure of CNNs and runtime information
Reduce computation without accuracy loss Trade accuracy for further computation reduction Add minimal hardware overhead
10
+ + + + + + + + + + + + X +
+ +
+
+ + + +
ReLU
11
+ + + + + + + + + + + + X +
+ +
+
+ + + +
ReLU + + + + + +
+ + + + + + + + + + + + X + + + + + + +
12
20 40 60 80 100 AlexNet GoogleNet SqueezeNet VGGNet % Negative Weights
On average, 54% of the weights are negative
13
w X
+ + +
PartialSum ≤ th
+ + + +
Partial Sum
+ + + + + + + + + + + +
X
+
+ + + +
+ + + + + + + +
Speculation operations
14
+ + + + + + + + + + + +
X
+
+ + + +
+ + + + + + + +
No
Group 1 Group 2 Group n
Large absolute value
Large Small n largest weights largest weights from each group
Small absolute value
15
16
All convolution kernels in a CNN
Layer 2 Layer L Layer 1
(Th,N)
Kernel Profiling Local Optimization Global Optimization
17
Threshold Value # of Operations
Layer 2 Layer L Layer 1
Dog
Per kernel sensitivity analysis
All convolution kernels in a CNN
Layer 2 Layer L Layer 1
(Th,N)
Kernel Profiling Local Optimization Global Optimization
18
Threshold Value # of Operations
Layer 2 Layer L Layer 1
Cat
Per kernel sensitivity analysis
All convolution kernels in a CNN
Layer 2 Layer L Layer 1
(Th,N)
Kernel Profiling Local Optimization Global Optimization
19
Threshold Value # of Operations
Per kernel sensitivity analysis
Layer 2 Layer L Layer 1
Dog
All convolution kernels in a CNN
Layer 2 Layer L Layer 1
(Th,N)
Kernel Profiling Local Optimization Global Optimization
20
Set of configurations per layer
Layer 2 Layer L Layer 1 Dog
Kernel m Kernel 2 Kernel 1
Kernel m Kernel 2 Kernel 1
Dog
All convolution kernels in a CNN
Layer 2 Layer L Layer 1
(Th,N)
Kernel Profiling Local Optimization Global Optimization
21
Layer 2 Layer L Layer 1
Kernel k Kernel 2 Kernel 1
Kernel k Kernel 2 Kernel 1
Set of configurations per layer Dog Dog
All convolution kernels in a CNN
Layer 2 Layer L Layer 1
(Th,N)
Kernel Profiling Local Optimization Global Optimization
22
Layer 2 Layer L Layer 1
Cat
Layer L Kernel k Layer 1 Kernel 2 Layer 1 Kernel 1
Adjust parameters regarding the cross-layer effect
All convolution kernels in a CNN
Layer 2 Layer L Layer 1
(Th,N)
Kernel Profiling Local Optimization Global Optimization
23
Layer 2 Layer L Layer 1
Bird
Layer L Kernel k Layer 1 Kernel 2 Layer 1 Kernel 1
Adjust parameters regarding the cross-layer effect
All convolution kernels in a CNN
Layer 2 Layer L Layer 1
(Th,N)
Kernel Profiling Local Optimization Global Optimization
24
Adjust parameters regarding the cross-layer effect
Layer 2 Layer L Layer 1
Dog
Layer L Kernel k Layer 1 Kernel 2 Layer 1 Kernel 1
All convolution kernels in a CNN
Layer 2 Layer L Layer 1
(Th,N)
Kernel Profiling Local Optimization Global Optimization
25
Add low-overhead sign checks and threshold checks to the hardware
… …
PE1,1 PE1,m PEn,1 PEn,m
≤
Partial result Terminate Terminate
Sign-bit
Exact mode Predictive mode
Threshold
…
PAU MAC PAU PAU
…
MAC MAC PAU MAC PAU PAU
…
MAC MAC
…
PAU MAC PAU PAU
…
MAC MAC PAU MAC PAU PAU
…
MAC MAC
Prediction Activation Unit (PAU)
26
Compute Lane
Weight and In/Out Buffer Index Buffer K Compute Lanes MAC Prediction Activation Unit (PAU)
Processing Engine (PE)
27
CNN Model AlexNet 2012 GoogLeNet 2015 SqueezeNet 2016 VGG-16 2014 Top-1 Accuracy 57.2% 68.7% 57.5% 70.5% Top-5 Accuracy 80.1% 89.0% 80.3% 89.9%
Benchmarks Optimization Hardware implementation
Optimization algorithm built on top of Caffe Simulation: Cycle accurate Power estimation: Design Compiler using TSMC 45 nm Baseline design: Eyeriss with the same number MAC units (256) SnaPEA area overhead compared to Eyeriss: 4.5%
28
1.27 1.34 1.29 1.24 1.28 1.37 1.45 1.44 1.26 1.38 1.54 2.02 1.52 1.51 1.63 1.89 2.08 1.83 1.81 1.81
0.5 1 1.5 2 2.5
AlexNet GoogleNet SqueezeNet VGGNet Geomean Exact Predictive (accuracy loss <= 1%) Predictive (accuracy loss <= 2%) Predictive (accuracy loss <= 3%)
Speedup over Eyeriss
29
Layers in the predictive mode for accuracy loss ≤ 3%
Network % of Conv Layers Speedup Energy Improvement AlexNet 60.0 2.11 1.97 GoogleNet 84.2 2.14 2.04 SqueezeNet 65.4 1.94 1.84 VGGNet 61.5 1.87 1.73
On average, 68% of layers operate in the predictive mode (3% accuracy drop).
30
Highest speedup (3.6x) in a layer in GoogLeNet
31
SnaPEA
Exploit algorithmic structure and runtime information Reduce computations in convolutional layers Control the accuracy with multi-variable optimization Add minimal hardware overhead
Future directions
Leverage runtime information (e.g., patterns in inputs and activations) Expand to other activation functions (e.g., sigmoid) Tune up the hardware for more parallelism
32