SLIDE 1
Towards Accurate Post-training Network Quantization via Bit-split and Stitching
Peisong Wang, Qiang Chen, Xiangyu He, Jian Cheng
Institute of Automation, Chinese Academy of Sciences
1
- Background
- Motivation
- Approach
- Experiments
2
Towards Accurate Post-training Network Quantization via Bit-split - - PowerPoint PPT Presentation
Towards Accurate Post-training Network Quantization via Bit-split - - PowerPoint PPT Presentation
Towards Accurate Post-training Network Quantization via Bit-split and Stitching Peisong Wang , Qiang Chen, Xiangyu He, Jian Cheng Institute of Automation, Chinese Academy of Sciences 1 Outline Background Motivation Approach
SLIDE 2
SLIDE 3
3
Background
- Low-bit quantization has emerged as a promising compression technique
- Robustness to network architectures
- Hardware friendly
- Problems: low-bit quantization relies on
- Training data
- Large computational resources (CPUs, GPUs)
- Quantization skills and expertise
SLIDE 4
4
Background
Training-aware quantization
Pre-trained Model Network Quantization Finetune using data/labels
Post-training Quantization
Pre-trained Model Network Quantization
Data-free BP-free Easy to use This work
Krishnamoorthi, Raghuraman. "Quantizing deep convolutional networks for efficient inference: A whitepaper." arXiv preprint arXiv:1806.08342 (2018).
SLIDE 5
5
Motivation
Post-training quantization
Pretrained model
SLIDE 6
6
Motivation
Post-training quantization
Pretrained model Low-bit model
SLIDE 7
7
Motivation
Post-training quantization
Minimize the Di Distance Pretrained model Low-bit model
SLIDE 8
8
Motivation
Post-training quantization
Minimize the Di Distance
I. Define the distance II. Minimize the distance
Pretrained model Low-bit model
SLIDE 9
9
Related works
I. Define the distance II. Minimize the distance
127
- 127
|Max|
- |Max|
Krishnamoorthi, Raghuraman. "Quantizing deep convolutional networks for efficient inference: A whitepaper." arXiv preprint arXiv:1806.08342 (2018).
TF-lite Map the maximum weighs (activations) to the maximum low-bit number
SLIDE 10
10
Related works
I. Define the distance II. Minimize the distance
127
- 127
TensorRT Map the clip value to the maximum low-bit number
- utliers
- utliers
Szymon Migacz. 8-bit Inference with TensorRT. GTC 2017
SLIDE 11
11
Method
I. Define the distance II. Minimize the distance
Minimize the Di Distance Pretrained model Low-bit model This work Learns a low-bit mapping from input to the
- utput of every convolution.
Objective Previous work
SLIDE 12
12
Method
I. Define the distance II. Minimize the distance (Bit-split)
𝑟" 𝑟# 𝑟$ … 𝑟$%"
2' 2" 2$%#
SLIDE 13
13
Method
Optimize 𝛽 Optimize m-th bit
Wang, P., Hu, Q., Zhang, Y., Zhang, C., Liu, Y. and Cheng, J., 2018. Two-step quantization for low-bit neural networks. In Proceedings of the IEEE Conference on computer vision and pattern recognition (pp. 4376-4384).
SLIDE 14
14
Bit-Split for Post-training Network Quantization
Problem: Optimization:
SLIDE 15
15
Bit-Split Results
Weight Quantization: Both Weight and Activation Quantization:
SLIDE 16
16
Comparison with State-of-the-arts
SLIDE 17
17
Results on Detection and Instance segmentation
SLIDE 18
Thanks for your attention.
18
Codes are available at https://github.com/wps712/BitSplit peisong.wang@nlpr.ia.ac.cn