Learning Accurate Low-bit Deep Neural Networks with Stochastic - - PowerPoint PPT Presentation

Learning Accurate Low-bit Deep Neural Networks with Stochastic Quantization

Yinpeng Dong1, Renkun Ni2, Jianguo Li3, Yurong Chen3, Jun Zhu1, Hang Su1

1Department of CST, Tsinghua University 2University of Virginia 3Intel Labs China

2

Deep Learning is Everywhere

Self-Driving Alpha Go Dota Machine Translation

3

Limitations

n More data + deeper models à more FLOPs + lager

memory

n Computation Intensive n Memory Intensive n Hard to deploy on mobile devices

4

Low-bit DNNs for Efficient Inference

n High Redundancy in DNNs; n Quantize full-precision(32-bits) weights to binary(1 bit)

• r ternary(2 bits) weights;

n Replace multiplication(convolution) by addition and

subtraction;

5

Typical Low-bit DNNs

n BinaryConnect:

𝐶" = $+1 with probability 𝑞 = 𝜏(𝑋

")

−1 with probability 1 − 𝑞

n BWN: minimize 𝑋 − 𝛽𝐶

𝐶" = 𝑡𝑗𝑕𝑜 𝑋

" ,

𝛽 = ∑ 𝑋

" @ "AB

𝑒

n TWN: minimize 𝑋 − 𝛽𝑈

𝑈" = E +1 if 𝑋

" > ∆

0 if 𝑋

" < ∆

−1 if 𝑋

" < −∆

, 𝛽 = ∑ 𝑋

"

• "∈M∆

𝐽∆ 𝐽∆ = 𝑗 𝑋

" > ∆ ,

∆= 0.7 𝑒 Q 𝑋

" @ "AB

6

Training & Inference of Low-bit DNN

n Let 𝑋 be the full-precision weights, 𝑅 be the low-bit

weights (𝐶, 𝑈, α𝐶, α𝑈).

n Forward propagation: quantize 𝑋 to 𝑅 and perform

convolution or multiplication

n Backward propagation: use 𝑅 to calculate gradients n Parameter update: 𝑋TUB = 𝑋T − 𝜃T WX

WYZ

n Inference: only need to keep low-bit weights 𝑅

7

Motivations

n Quantize all weights simultaneously; n Quantization error 𝑋 − 𝑅 may be large for some

elements/filters;

n Induce inappropriate gradient directions. n Quantize a portion of weights n Stochastic selection n Could be applied to any low-bit settings

8

Roulette Selection Algorithm

1.3

• 1.1

0.75 0.85 0.95 1.4

• 1.2
• 0.9

1.05

• 1.0
• 0.9

0.8

• 0.8

0.9 1.0

• 1.0

0.2 0.05 0.2 0.1

Selection Point

C1 C2 C3 C4

1-st selection: v=0.58 C2 selected Rotation

Selection Point

2-nd selection: v=0.37 C3 selected Rotation

1.3

• 1.1

0.75 0.85 1 1

• 1.2
• 1

1

• 1
• 1

0.8

• 1

1 1.0

• 1.0

Weight Matrix Quantization Error Stochastic Partition with r = 50% Hybrid Weight Matrix

𝑓" = 𝑋

" − 𝑅" B

𝑋

" B

Quantization Error: Quantization Probability: Larger quantization error means smaller quantization probability, e.g. 𝑞" ∝ B

]^

Quantization Ratio r: Gradually increase to 100%

9

Training & Inference

n Hybrid weight matrix 𝑅

_ 𝑅 _" = $𝑅" if channel i being selected 𝑋

" else

n Parameter update

𝑋TUB = 𝑋T − 𝜃T 𝜖𝑀 𝜖𝑅 _T

n Inference: all weights are quantized; use 𝑅 to perform

inference

10

Ablation Studies

n Selection Granularity:

¨ Filter-level > Element-level

n Selection/partition algorithms

¨ Stochastic (roulette) > deterministic (sorting) ~ fixed

(selection only at first iteration)

n Quantization functions

¨ Linear > Sigmoid > Constant ~ Softmax

n 𝑞" = exp

(𝑔

") ∑ exp

(𝑔

")

• ⁄

, where 𝑔 = B

]

n Quantization Ratio Update Scheme

¨ Exponential > Fine-tune > Uniformly

n 50% à 75% à 87.5% à 100%

11

Results -- CIFAR

Bits CIFAR-10 CIFAR-100 VGG-9 ResNet-56 VGG-9 ResNet-56 FWN 32 9.00 6.69 30.68 29.49 BWN 1 10.67 16.42 37.68 35.01 SQ-BWN 1 9.40 7.15 35.25 31.56 TWN 2 9.87 7.64 34.80 32.09 SQ-TWN 2 8.37 6.20 34.24 28.90 error (%) of VGG-9 and ResNet-56 trained with 5 different methods on the CIFAR-10 and

20 40 60 80 64 128 192 256 Loss Iter.(k) FWN BWN SQ-BWN 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2 64 128 192 256 Loss Iter.(k) FWN TWN SQ-TWN

12

Results -- ImageNet

Bits AlexNet-BN ResNet-18 top-1 top-5 top-1 top-5 FWN 32 44.18 20.83 34.80 13.60 BWN 1 51.22 27.18 45.20 21.08 SQ-BWN 1 48.78 24.86 41.64 18.35 TWN 2 47.54 23.81 39.83 17.02 SQ-TWN 2 44.70 21.40 36.18 14.26 (%) of AlexNet-BN and ResNet-18 trained with 5 different methods on

13

Conclusions

n We propose a stochastic quantization algorithm for

Low-bit DNN training

n Our algorithm can be flexibly applied to all low-bit

settings;

n Our algorithm help to consistently improve the

performance;

n We release our codes to public for future development

¨ https://github.com/dongyp13/Stochastic-Quantization

Q & A