PaddlePaddle B a i d u D e e p L e a r n i n g O p e n S o u r c e F r a m e w o r k 2019.01 PA rallel D istributed D eep L earning http://www.paddlepaddle.org
Agenda A . B r i e f I n t r o d u c t i o n B . E c o s y s t e m C . R e a l W o r l d U s e C a s e s D . D e v e l o p e r s C o m m u n i t y
History of Baidu Deep Learning Models in CV ， Reinforcement Learning won championships in several international competitions ； 2017 Release of PaddlePaddle Suite Baidu News Feed recommendation system migrated to PaddlePaddle ； 2017 PaddlePaddle Fluid Released 2016 PaddlePaddle went open source The first ever NMT online translation engine launched 2015 Launched the STM-CTC based acoustic model PaddlePaddle’s first commit 2013 Baidu search Product: Phoenix Nest’s CTR based DNN prediction model launched 2012 DNN NLP, OCR models used in practices.
PaddlePaddle 3.0 – Towards Maturity PaddlePaddle 1.0 PaddlePaddle 2.0 PaddlePaddle 3.0 July 2018 Released PaddlePaddle 3.0 Including functional components like Friendly python API ， Initial Open Source Edition, command EasyDL, AI Studio, AutoDL, VisualDL Released the core framework , model line interaction interface ， zoo and Paddle Book, Support common DL networks Nov. 2018 Improved the ease of use and flexibility Released PaddlePaddle Fluid 1.0 Release PaddlePaddle Suite --Full-featured Deep Learning development kit for businesses and developers
Widely Recognized by the Government and Industry PaddlePaddle has established a “Deep The Only Deep Learning Technology and Application Learning National Team” with a number of domestic research National Engineering Laboratory institutions and universities. Education and Training Open Data Set Engineering platform Development Tools
Open Sourced Several International Competition Winning Models The world's top technology level, leading the direction of deep learning technology A w a r d - W i n n i n g M o d e l A w a r d s WIDER FACE （ 3 test subsets ） PyramidBo Model First place ActivityNet2017/2018 kinetics Attention Clusters Network Model First place C V Several Model Based on Faster R- First place Google AI Open Images-Object Detection Track CNN PARL(Reinforcement Learning) NIPS AI for Prosthetics Challenge First place
Agenda A . B r i e f I n t r o d u c t i o n B . E c o s y s t e m C . R e a l W o r l d U s e C a s e s D . D e v e l o p e r s C o m m u n i t y
P a d d l e P a d d l e S u i t e Full-featured Deep Learning Suite with Comprehensive, Leading Technology Modules and Service Platform Components E a s y D L V i s u a l D L A u t o D L A I S t u d i o Zero-based customized training and Visualization Tool for Training Network structure automation One-stop development platform service platform design Core F ramework P A R L P a d d l e R e c P a d d l e C V P a d d l e N L P Deep Reinforcement Learning Intelligent Recommendation Intelligent vision Intelligent text processing E D L P a d d l e F l u i d P a d d l e S e r v i n g A u t o D L Elastic deep learning calculation
Features of PaddlePaddle Core framework P a r a l l e l M u l t i p l e M u l t i - E n d H e t e r o g e n e o u s T r a i n i n g A l g o r i t h m s D e p l o y m e n t C o m p u t i n g Personalized recommendation, Supports multi-machine Rapid Deployment Fully supports for large- image classification, semantic multi-thread Multiple mobile end scale heterogeneous segmentation, face detection, asynchronous training support computing clusters CPU machine translation, reading and synchronous training 、 GPU 、 DSP 、 FPGA comprehension, lexical analysis, mode sentiment analysis
Large-Scale Heterogeneous Computing Cluster B a i d u A I O p e n P l a t f o r m B a i d u U n i f i e d D e e p L e a r n i n g P l a t f o r m P a d d l e P a d d l e A F S N o r m a n d y k 8 s D i s t r i b u t e d f i l e R e s o u r c e S c h e d u l i n g s t o r a g e M a t r i x C o n t a i n e r D o c k e r R e s o u r c e M a n a g e m e n t H a r d w a r e （ C P U ， G P U , F P G A ， … ） O p e n s o u r c e o r o p e n m o d u l e
Supports Parallel Training of Dense Parameters and Sparse Parameters L a r g e - s c a l e d e n s e Ultra - Large - Scale p a r a m e t e r s p a r s e p a r a m e t e r Data0 Data1 Data2 Data3 GPU 0 GPU 1 GPU 2 GPU 3 Data0 Data1 Data2 Data3 Data4 CTR estimation, semantic matching, and other tasks Computationally intensive tasks such as image with large data throughput classification and machine translation Parameter synchronization mode: Parameter synchronization mode ： Synchronous Asynchronous large-scale sparse parameter server Collective operation C P U b a s e d u l t r a - l a r g e - s c a l e G P U p a r a l l e l t r a i n i n g a s y n c h r o n o u s t r a i n i n g i s u n i q u e , s p e e d s u r p a s s e s s i m i l a r s u p p o r t i n g 1 0 0 b i l l i o n s c a l e f r a m e w o r k s i n m a i n s t r e a m p a r a m e t e r s , h u n d r e d s o f n o d e s t a s k s p a r a l l e l t r a i n i n g
Keep Building the Most Complete Model Collection PaddleRec – Scenario PaddleRec - CV intelligence PaddleRec - NLP Recommendation Intelligent Video Autonomous Search Machine Intelligent Medical Industry public Feed marketing analysis driving engine translation dialogue imaging inspection sentiment Covers all the cv application Provision of many classic recall and Models set Fulfills mainstream NLP tasks scenarios ranking algorithms Image Object Face classificatio semantic Comprehen detection detection DeepCTR GRU4Rec Text label Chinese n matching sion Semantic GAN OCR Segmentati on Sequence Chinese semantic 机器翻 译 semantic Multi-view Simnet Metric Video segmentation recall learning classification Application examples Haokan Baidu Baidu Baidu feed Baidu Map Baidu OCR Baidu feed Baidu video Nuomi translation
Multi-platform Service Deployment P a d d l e S e r v i n g P a d d l e M o b i l e P a d d l e A n y w h e r e • • Flexible adaptation to multiple Multiple hardware platform inference engines support: ARM CPU, Mali GPU, • Compatible with mainstream Qualcomm DSP, FPGA engine TensorRT • Fixed point quantization • Inference API, lib library • Low precision and efficient • CPU, GPU performance deep optimization quantitative calculation • Forward pass specific optimization
Deep Learning Optimizations for today’s challenges Bigger the scale of data; More Limited Memory & Video The model is getting complicated the model Demanding industrial memory structure; more complicated Requirement of Calculation requirements Larger feature size Time is harsh D e e p L e a r n i n g E f f i c i e n t D e c o d i n g M e t h o d Feature Optimization Parameter sharing Quantification Pruning Log Domain Pruning and Binary network Hash Net quantification Retraining Pyramid DNN Product Low precision Multi-Seed Dynamic Network quantification operation Random Hash Surgery Memory & Speed Optimization Speed optimization Memory optimization
PaddlePaddle Assistive Tools and Platform V i s ual DL A utoDL Eas yDL A I Studi o PARL Zero skill required deep Deep Visualized Deep Learning Automatic Network One-stop development learning training and service Reinforcement learning Tool Structure Design platform platform
PARL Tools for Reinforced Learning Env1 Env2 Env3 Won NIPS 2018 AI Prosthetics Challenge Agent Agent Wrapper CPU1 CPU2 CPU3 Data Server/Experience Buffer Agent Algorithm Wrapper GPU2 GPU3 PARL parallel framwork Computation Task 1 Computation Task 2 Algorithm 1 Algorithm 2 _learn _learn Computation _predict _predict Task 3 Target Driven DDPG + Bootstrapping Target Critic Policy Critic Model Model One thousand of CPU + Single GPU Model … PARL Algorithm component
Visual DL Visualize the overall Process of Training and Inferring Scalar Six components on visualization Two SDK ： C++ ， Python Supports ONNX ONNX network graph Histogram
AutoDL Support the Design, Transfer and Adaption of DL A u t o D L D e s i g n Search for several neural networks with excellent performance and different structures Network design A u t o D L T r a n s f e r Transfer pretrained models to new applications Create transfer model with small amount of data A u t o D L E d g e Network complexity optimization based on classic model, suits better for mobile deployment Adapt to edge computing
AutoDL Design Better than manual design network structure search based on deep conv 3x3 conv 3x3 reinforcement learning conv 2x2 maxpool 3x3 conv 3x3 Training Data avgpool 2x2 conv 3x3 conv 1x2 2x1 Dataset conv 3x3 conv 3x3 dilated 2x2 + dilated 2x2 conv 2x2 conv 3x3 conv 3x3 Conv 1x3 3x1 conv 3x3 sample student model avgpool 2x2 conv 3x3 conv 2x2 Network Network maxpool 3x3 conv 1x3 3x1 conv 3x3 Designer Evaluator dilated 2x2 conv 2x2 maxpool 3x3 compute reward to update network designer conv 2x2 conv 1x1 conv 1x3 3x1 avgpool 3x3 3x3maxpool global average pooling The network designed by AutoDL has the precision of 98% on CIFAR10 image classification dataset Surpassing classic network designed manually
Recommend
More recommend
