xdn towards efficient inference of residual neural
play

XDN: Towards Efficient Inference of Residual Neural Networks on - PowerPoint PPT Presentation

Jun 08, 2023 •758 likes •934 views

(XiaoDianNao) XDN: Towards Efficient Inference of Residual Neural Networks on Cambricon Chips 2019 BenchCouncil International Artificial Intelligence System Challenges Guangli Li , Xueying Wang, Xiu Ma Advisor: Prof. Xiaobing Feng

  1. (XiaoDianNao) XDN: Towards Efficient Inference of Residual Neural Networks on Cambricon Chips — 2019 BenchCouncil International Artificial Intelligence System Challenges — Guangli Li , Xueying Wang, Xiu Ma Advisor: Prof. Xiaobing Feng State Key Laboratory of Computer Architecture, Institute of Computing Technology, Chinese Academy of Sciences

  2. XiaoDianNao (XDN) International Artificial Intelligence System Challenges XiaoDianNao Team Team Members: Advisor: Guangli Li is a Ph.D. student at ICT, CAS. His research interests include Programming Systems and Deep Learning. Xueying Wang is a Ph.D. student at ICT, CAS. Her research interests include Parallel Programming and Deep Learning. Prof. Xiaobing Feng Xiu Ma is a Ph.D. student at Jilin Institute of Computing Technology, University and a visiting student at Chinese Academy of Sciences ICT, CAS. Her research interests include Programming Language and Complier.

  3. XiaoDianNao (XDN) International Artificial Intelligence System Challenges Track 2 International AI System Challenge based on Cambricon Chip 18 Sept. ~ 14 Oct. (about 4 weeks) Subject ： The implementation and optimization of convolutional neural network based image classification task on Cambricon AI Chips (MLUs) . Metrics ： Maximize the execution performance and minimize the prediction time (wall clock time) on provided test data. Artificial Intelligence Provided model: ResNet-50 (ACC: 84.39%) Chips (MLUs) Dataset: CIFAR-10 Provided Development Tools: Cambricon Caffe Experiment Platform: BenchCouncil Testbed

  4. XiaoDianNao (XDN) International Artificial Intelligence System Challenges Contents I. Optimization Methodology II. Implementation of the XDN Engine III. Experimental Results IV. Conclusion and Future Work

  5. XiaoDianNao (XDN) International Artificial Intelligence System Challenges Methodology input input input building block 1 conv conv+ conv 2 conv conv conv conv+ conv conv bn 3 bn scale sum scale sum ReLU ReLU input input input building block conv conv conv 1 pre-fusion of layers conv conv conv 2 fusion-guided pruning sum sum sum 3 post-fusion of layers • Aggressive Fusion Strategy • Fusion-Guided Pruning Method • Traditional Fusion Strategy

  6. XiaoDianNao (XDN) International Artificial Intelligence System Challenges XDN (XiaoDianNao) ——An Efficient Optimization and Inference Engine I III IV II V I. Pruning Optimizer and Trainer (aggressive fusion and pruning) II. Fusion Optimizer (traditional fusion) III. Auto-Tuner IV. Data Preprocessor and Executor V. Evaluator

  7. XiaoDianNao (XDN) International Artificial Intelligence System Challenges Implementation of XDN Pruning and Fusion • Caffe Cambricon Prototxt Pruning Fusion Caffe Optimizer Optimizer Caffe (Trainer) Model Original Model

  8. XiaoDianNao (XDN) International Artificial Intelligence System Challenges Implementation of XDN Pruning and Fusion • Caffe Cambricon Prototxt Pruning Fusion Caffe Optimizer Optimizer (Trainer) Caffe Model Optimized Model

  9. XiaoDianNao (XDN) International Artificial Intelligence System Challenges Implementation of XDN Data Preprocessing • OpenCV Inference Preprocess CIFAR-10 Images Original Executer: On-line Data Preprocess

  10. XiaoDianNao (XDN) International Artificial Intelligence System Challenges Implementation of XDN Data Preprocessing • Very Fast mmap() OpenCV Memory- mapped Inference Off-line Binary File Preprocess CIFAR-10 Images Off-line Data Preprocess & Efficient Memory-mapped File

  11. XiaoDianNao (XDN) International Artificial Intelligence System Challenges Implementation of XDN Auto-Tuning of Hyper-Parameters: A Grid Search Approach • thread Executor Search Space data parallel Score Function Model parallel Evaluator Auto-Tuner … Finding a best option of hyper-parameters

  12. XiaoDianNao (XDN) International Artificial Intelligence System Challenges Final Results p u d Total Time Optimization Accuracy e (ms) e 2263 0.8439 Original Baseline p 505 0.8439 OPT-A-1 S Filter 449 0.8462 OPT-B-1 x Pruning 4 428 0.8191 OPT-C-1 4 485 0.8441 OPT-A-2 . Pruning 7 + OPT-B-2 304 0.8455 Conv&BN Fusion 297 0.8203 OPT-C-2 Performance of ResNet-50 with XDN Engine Best Result: OPT-B-2, TIME=304ms, ACC=84.55%

  13. XiaoDianNao (XDN) International Artificial Intelligence System Challenges Performance Analysis Optimizations of XDN Total Time (ms) Pruning Fusion Data Preprocess Auto-Tuning 2263 1291 833 746 304 Performance of OPT-B-2 Each part contributes the final performance.

  14. XiaoDianNao (XDN) International Artificial Intelligence System Challenges Artifact Description BenchHub: http://125.39.136.212:8090/liguangli/bench19_xiaodiannao Detailed XDN Documentations (English and Chinese) • The Codes of Reproducible Experiments • Step-by-Step

  15. XiaoDianNao (XDN) International Artificial Intelligence System Challenges Conclusion and Future Work Highlight Contributions: 1. We proposed a optimization methodology Characteristics of Cambricon Chips • Fusion-Guided Pruning Method • 2. We implemented the Efficient XDN engine Pruner, Trainer, Optimizer, Auto-Tuner, Executer… • 3. We evaluated the performance Achieves high speedup (7.44x) without accuracy loss • In the future, we plan to: test the method on large-scale datasets, such as ImageNet; • extend the XDN engine to support more DNN models; • test the method on other AI chips; • … •

  16. XDN: Towards Efficient Inference of Residual Neural Networks on Cambricon Chips Thank You liguangli@ict.ac.cn

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Efficient Layout Hotspot Detection via Binarized Residual Neural Network Yiyang Jiang 1 , Fan Yang

Efficient Layout Hotspot Detection via Binarized Residual Neural Network Yiyang Jiang 1 , Fan Yang

Efficient Layout Hotspot Detection via Binarized Residual Neural Network Yiyang Jiang 1 , Fan Yang 1 , Hengliang Zhu 1 , Bei Yu 3 , Dian Zhou 2 , Xuan Zeng 1 1 State Key Lab of ASIC & System, Microelectronics Department, Fudan

357 views • 21 slides
Deep Compression and EIE: Deep Neural Network Model Compression and Efficient Inference

Deep Compression and EIE: Deep Neural Network Model Compression and Efficient Inference

Deep Compression and EIE: Deep Neural Network Model Compression and Efficient Inference Engine Song Han CVA group, Stanford University Apr 7, 2016 1 Intro about me and my advisor Fourth year PhD with Prof. Bill Dally at Stanford.

2.06k views • 70 slides
Deep Compression and EIE: Deep Neural Network Model Compression and Efficient Inference

Deep Compression and EIE: Deep Neural Network Model Compression and Efficient Inference

Deep Compression and EIE: Deep Neural Network Model Compression and Efficient Inference Engine Song Han CVA group, Stanford University Jan 6, 2015 A few words about us Fourth year PhD with Prof. Bill Dally at Stanford.

966 views • 54 slides
Efficient Neural Networks for Image Restoration Yulun Zhang Supervisor: Prof. Yun Fu SMILE lab,

Efficient Neural Networks for Image Restoration Yulun Zhang Supervisor: Prof. Yun Fu SMILE lab,

Efficient Neural Networks for Image Restoration Yulun Zhang Supervisor: Prof. Yun Fu SMILE lab, Northeastern University, Boston, US Research summary Deep convolutional neural networks for Image Restoration 1.Residual dense network. [ CVPR-2018

375 views • 33 slides
Clinical Inference in the Assessment of Mental Residual Functional Capacity David J Schretlen,

Clinical Inference in the Assessment of Mental Residual Functional Capacity David J Schretlen,

Clinical Inference in the Assessment of Mental Residual Functional Capacity David J Schretlen, PhD, ABPP OIDAP Panel Meeting 10 June 2009 1 Methods of Inference 1. Pathognomonic sign approach 2. Pattern analysis 3. Level of performance or

761 views • 44 slides
Pipeline Strategies and conversations behind securing a Residual Bequest Agenda 1. Why Residual?

Pipeline Strategies and conversations behind securing a Residual Bequest Agenda 1. Why Residual?

The Perpetual Pipeline Strategies and conversations behind securing a Residual Bequest Agenda 1. Why Residual? 2. Structured for Residual 3. Residual Results 4. Residual Strategy 5. Residual Relationships 6. Residual Data 7.

1.17k views • 43 slides
Same, Same But Different Recovering Neural Network Quantization Error Through Weight

Same, Same But Different Recovering Neural Network Quantization Error Through Weight

Same, Same But Different Recovering Neural Network Quantization Error Through Weight Factorization Eldad Meller ICML 2019 Neural Network Quantization Quantization of Neural Networks is needed for efficient inference Quantization adds

482 views • 9 slides
Inference for the difference of two percentile residual life functions Alba M. Franco-Pereira

Inference for the difference of two percentile residual life functions Alba M. Franco-Pereira

Introduction The problem A simulation study A real data example References Inference for the difference of two percentile residual life functions Alba M. Franco-Pereira Department of Statistics Universidad Carlos III de Madrid August, 2010

507 views • 39 slides
GPU INFERENCE ENGINE Michael Andersch, 7 th April 2016 WHAT IS INFERENCE, ANYWAYS? Building a

GPU INFERENCE ENGINE Michael Andersch, 7 th April 2016 WHAT IS INFERENCE, ANYWAYS? Building a

April 4-7, 2016 | Silicon Valley NVIDIA GIE: HIGH-PERFORMANCE GPU INFERENCE ENGINE Michael Andersch, 7 th April 2016 WHAT IS INFERENCE, ANYWAYS? Building a deep neural network based application Step 1: Use data to train the neural network -

1.02k views • 17 slides
Deep Residual Output Layers for Neural Language Generation Nikolaos Pappas, James Henderson June

Deep Residual Output Layers for Neural Language Generation Nikolaos Pappas, James Henderson June

Deep Residual Output Layers for Neural Language Generation Nikolaos Pappas, James Henderson June 13, 2019 Neural language generation Cat sat on ? Probability distribution at time t given context vector h t R d , weights W R d |V|

474 views • 14 slides
Algorithms in Nature Pruning in neural networks Neural network development 1. Efficient signal

Algorithms in Nature Pruning in neural networks Neural network development 1. Efficient signal

Algorithms in Nature Pruning in neural networks Neural network development 1. Efficient signal propagation [e.g. information processing & integration] 2. Robust to noise and failures [e.g. cell or synapse failure] 3. Cost-aware design [e.g.

874 views • 33 slides
GENERIC CUTS: AN EFFICIENT ALGORITHM FOR OPTIMAL INFERENCE IN HIGHER ORDER MRF-MAP Chetan Arora,

GENERIC CUTS: AN EFFICIENT ALGORITHM FOR OPTIMAL INFERENCE IN HIGHER ORDER MRF-MAP Chetan Arora,

GENERIC CUTS: AN EFFICIENT ALGORITHM FOR OPTIMAL INFERENCE IN HIGHER ORDER MRF-MAP Chetan Arora, Subhashis Banerjee, Prem Kalra, S.N. Maheshwari Indian Institute of Technology Delhi, INDIA MRF-MAP INFERENCE MRF-MAP INFERENCE HIGHER

374 views • 19 slides
R RITSUMEIKAN Introduction 1 Methodology Content 2 Experiments 3 Conclusion 4

R RITSUMEIKAN Introduction 1 Methodology Content 2 Experiments 3 Conclusion 4

ICIP 2019 Residual Convolutional Neural Networks with Global and Local Pathways for Classification of Focal Liver A dual attention dilated residual network for liver lesion classification and localization on CT images Lesions Xiao Chen,

779 views • 19 slides
Redundancy-free Residual Dispatch Using Ordered Binary Decision Diagrams for Efficient Dispatch

Redundancy-free Residual Dispatch Using Ordered Binary Decision Diagrams for Efficient Dispatch

Formula-based Dispatch BDD-based Dispatch Assessment Redundancy-free Residual Dispatch Using Ordered Binary Decision Diagrams for Efficient Dispatch Andreas Sewe Christoph Bockisch Mira Mezini Seventh International Workshop on Foundations

408 views • 39 slides
Fast Inference for Neural Vocoders Welcome, everyone! Im excited to be here today and get the

Fast Inference for Neural Vocoders Welcome, everyone! Im excited to be here today and get the

Fast Inference for Neural Vocoders Welcome, everyone! Im excited to be here today and get the opportunity to tell you a little bit about a few incredibly interesting systems challenges my team has encountered recently. Id like to preface

595 views • 44 slides
4-connected shift residual networks ICCV 2019 Neural Architects Workshop Andrew Brown, Pascal

4-connected shift residual networks ICCV 2019 Neural Architects Workshop Andrew Brown, Pascal

4-connected shift residual networks ICCV 2019 Neural Architects Workshop Andrew Brown, Pascal Mettes, Marcel Worring, University of Amsterdam Network costs increasing! Increasing accuracy on ImageNet has come at increasing cost Popular

280 views • 15 slides
Near-Optimal Offline Cleaning for Flash-Based SSDs MANSOUR SHAFAEI & PETER DESNOYERS

Near-Optimal Offline Cleaning for Flash-Based SSDs MANSOUR SHAFAEI & PETER DESNOYERS

Near-Optimal Offline Cleaning for Flash-Based SSDs MANSOUR SHAFAEI & PETER DESNOYERS NORTHEASTERN UNIVERSITY Outline Background Problem definition Approach Evaluation Conclusion 2 Background Performance of

551 views • 22 slides
TVM at Facebook Lots of contributors at FB and elsewhere TVM at Facebook Why TVM? Examples from

TVM at Facebook Lots of contributors at FB and elsewhere TVM at Facebook Why TVM? Examples from

TVM at Facebook Lots of contributors at FB and elsewhere TVM at Facebook Why TVM? Examples from Speech Synthesis Sparsity PyTorch Why TVM for ML Systems? - Performance matters - Flexibility matters - Portability matters ML Systems at

679 views • 29 slides
1 - Pruning Pseudocode - Pruning Properties Pruning has no effect on final result

1 - Pruning Pseudocode - Pruning Properties Pruning has no effect on final result

Recap: Resource Limits Introduction to Artificial Intelligence Cannot search to leaves max 4 V22.0472-001 Fall 2009 -2 4 Depth-limited search min min Instead, search a limited depth of -1 -2 4 9 tree Replace

299 views • 6 slides
Introduction to Machine Learning CART: Stopping Criteria & Pruning

Introduction to Machine Learning CART: Stopping Criteria & Pruning

Introduction to Machine Learning CART: Stopping Criteria & Pruning compstat-lmu.github.io/lecture_i2ml OVERFITTING TREES The recursive partitioning procedure used to grow a CART would run until every leaf only contains a single observation.

515 views • 8 slides
A stable skeletonization for tabletop gesture recognition Andoni Beristain, Manuel Graa

A stable skeletonization for tabletop gesture recognition Andoni Beristain, Manuel Graa

A stable skeletonization for tabletop gesture recognition Andoni Beristain, Manuel Graa Computational Intelligence Group, Universidad del Pais Vasco ICCSA 2010, Fukuoka, Japan 1 Contents Introduction Skeletonization algorithm

488 views • 23 slides
More on games (Ch. 5.4-5.6) Announcements Midterm next Tuesday: covers weeks 1-4 (Chapters 1-4)

More on games (Ch. 5.4-5.6) Announcements Midterm next Tuesday: covers weeks 1-4 (Chapters 1-4)

More on games (Ch. 5.4-5.6) Announcements Midterm next Tuesday: covers weeks 1-4 (Chapters 1-4) Take the full class period Open book/notes (can use ebook) ^^ No programing/code, internet searches or friends Exam is in this room I will

826 views • 55 slides
Model Compression Seminar: Advanced Machine Learning, SS 2016 Markus Beuckelmann

Model Compression Seminar: Advanced Machine Learning, SS 2016 Markus Beuckelmann

Model Compression Seminar: Advanced Machine Learning, SS 2016 Markus Beuckelmann markus.beuckelmann@stud.uni-heidelberg.de July 19, 2016 Markus Beuckelmann Model Compression July 19, 2016 1 / 33 Introduction Outline Outline 1 Overview

844 views • 43 slides
- Pruning the SASLOG Digging into the Roots of NOTEs, WARNINGs, and ERRORs A

- Pruning the SASLOG Digging into the Roots of NOTEs, WARNINGs, and ERRORs A

- Pruning the SASLOG Digging into the Roots of NOTEs, WARNINGs, and ERRORs A clipping from the original, provided for the Toronto Area SAS Society, December 2007 Andrew T. Kuligowski The Nielsen Company Pruning the SASLOG

391 views • 18 slides

More recommend