putting deep learning models in production sahil dua
play

Putting Deep Learning Models in Production Sahil Dua @sahildua2305 - PowerPoint PPT Presentation

Jun 19, 2023 •504 likes •938 views

Putting Deep Learning Models in Production Sahil Dua @sahildua2305 @sahildua2305 Lets imagine! @sahildua2305 @sahildua2305 But ... @sahildua2305 @sahildua2305 whoami Software Developer @ Booking.com Previously - Deep Learning

  1. Putting Deep Learning Models in Production Sahil Dua @sahildua2305 @sahildua2305

  2. Let’s imagine! @sahildua2305 @sahildua2305

  3. But ... @sahildua2305 @sahildua2305

  4. whoami ➔ Software Developer @ Booking.com ➔ Previously - Deep Learning Infrastructure ➔ Open Source Contributor (Git, Pandas, Kinto, go-github, etc.) ➔ Tech Speaker @sahildua2305 @sahildua2305

  5. Agenda ➔ Deep Learning at Booking.com ➔ Life-cycle of a model ➔ Training Models ➔ Serving Predictions @sahildua2305 @sahildua2305

  6. Deep Learning at Booking.com @sahildua2305 @sahildua2305

  7. Scale highlights . 1,500,000 + 1.4 million + room nights active properties booked in 220+ countries every 24 hours @sahildua2305 @sahildua2305

  8. Deep Learning ➔ Image understanding ➔ Translations ➔ Ads bidding ➔ ... @sahildua2305 @sahildua2305

  9. Image Tagging @sahildua2305 @sahildua2305

  10. Image Tagging @sahildua2305 @sahildua2305

  11. Image Tagging Sea view: 6.38 Balcony/Terrace: 4.82 Photo of the whole room: 4.21 Bed: 3.47 Decorative details: 3.15 Seating area: 2.70 @sahildua2305 @sahildua2305

  12. @sahildua2305 @sahildua2305

  13. Image Tagging Using the image tag information in the right context Swimming pool, Breakfast Buffet, etc. @sahildua2305 @sahildua2305

  14. Lifecycle of a model @sahildua2305 @sahildua2305

  15. Lifecycle of a model Data Train Deploy Analysis @sahildua2305 @sahildua2305

  16. Training a Model - on laptop @sahildua2305 @sahildua2305

  17. Training a Model - on laptop @sahildua2305 @sahildua2305

  18. Machine Learning workload ➔ Computationally intensive workload ➔ Often not highly parallelizable algorithms ➔ 10 to 100 GBs of data @sahildua2305 @sahildua2305

  19. Why Kubernetes (k8s)? ➔ Isolation ➔ Elasticity ➔ Flexibility @sahildua2305 @sahildua2305

  20. Why k8s – GPUs? ➔ In alpha since 1.3 ➔ Speed up 20X-50X resources: limits: alpha.kubernetes.io/nvidia-gpu: 1 @sahildua2305 @sahildua2305

  21. Training with k8s ➔ Base images with ML frameworks ◆ TensorFlow, Torch, VowpalWabbit, etc. ➔ Training code is installed at start time ➔ Data access - Hadoop (or PVs) @sahildua2305 @sahildua2305

  22. Startup Training pod Code .. start.sh train.py evaluate.py @sahildua2305 @sahildua2305

  23. Startup Training pod Data .. start.sh PV train.py evaluate.py @sahildua2305 @sahildua2305

  24. Streaming logs back Training pod Logs .. start.sh PV train.py evaluate.py @sahildua2305 @sahildua2305

  25. Exports the model Training pod model .. start.sh PV train.py evaluate.py @sahildua2305 @sahildua2305

  26. Serving predictions @sahildua2305 @sahildua2305

  27. Serving Predictions Input Features Client Model Prediction @sahildua2305 @sahildua2305

  28. Serving Predictions Input Features Client Model 1 Prediction Input Features Client Model X Prediction @sahildua2305 @sahildua2305

  29. Serving Predictions Input Features Client Model 1 Prediction Input Features Client Model X Prediction @sahildua2305 @sahildua2305

  30. Serving Predictions ➔ Stateless app with common code ➔ Containerized ➔ No model in image ➔ REST API for predictions @sahildua2305 @sahildua2305

  31. Serving Predictions Input App Features Client model Prediction @sahildua2305 @sahildua2305

  32. Serving Predictions ➔ Get trained model from Hadoop ➔ Load model in memory ➔ Warm it up ➔ Expose HTTP API ➔ Respond to the probes @sahildua2305 @sahildua2305

  33. Serving Predictions Input Features Client Prediction @sahildua2305 @sahildua2305

  34. Serving Predictions Input Features Client Prediction Input Features Client Prediction @sahildua2305 @sahildua2305

  35. Deploying a new model ➔ Create new Deployment ➔ Create new HTTP Route ➔ Wait for liveness/readiness probe @sahildua2305 @sahildua2305

  36. Performance PredictionTime = RequestOverhead + N*ComputationTime N is the number of instances to predict on @sahildua2305 @sahildua2305

  37. Optimizing for Latency ➔ Do not predict if you can precompute ➔ Reduce Request Overhead ➔ Predict for one instance ➔ Quantization (float 32 => fixed 8) ➔ TensorFlow specific: freeze network & optimize for inference @sahildua2305 @sahildua2305

  38. Optimizing for Throughput ➔ Do not predict if you can precompute ➔ Batch requests ➔ Parallelize requests @sahildua2305 @sahildua2305

  39. Summary ➔ Training models in pods ➔ Serving models ➔ Optimizing serving for latency/throughput @sahildua2305 @sahildua2305

  40. Next steps ➔ Tooling to control hundred deployments ➔ Autoscale prediction service ➔ Hyper parameter tuning for training @sahildua2305 @sahildua2305

  41. Want to get in touch? LinkedIn / Twitter / GitHub @sahildua2305 Website www.sahildua.com @sahildua2305 @sahildua2305

  42. THANK YOU @sahildua2305 @sahildua2305 @sahildua2305

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

and Features in Deep Learning Interpretation Sahil Singla Joint work with Eric Wallace, Shi

and Features in Deep Learning Interpretation Sahil Singla Joint work with Eric Wallace, Shi

Understanding Impacts of High-Order Loss Approximations and Features in Deep Learning Interpretation Sahil Singla Joint work with Eric Wallace, Shi Feng, Soheil Feizi University of Maryland Pacific Ballroom # 69 , 6:30-9:00 PM June 13th 2019

650 views • 37 slides
Structured Probabilistic Models for Deep Learning Lecture slides for Chapter 16 of Deep Learning

Structured Probabilistic Models for Deep Learning Lecture slides for Chapter 16 of Deep Learning

Structured Probabilistic Models for Deep Learning Lecture slides for Chapter 16 of Deep Learning www.deeplearningbook.org Ian Goodfellow 2016-10-04 Roadmap Challenges of Unstructured Modeling Using Graphs to Describe Model Structure

974 views • 34 slides
DEEP LEARNING DEPLOYMENT WITH NVIDIA TENSORRT Shashank Prasanna Deep Learning in Production -

DEEP LEARNING DEPLOYMENT WITH NVIDIA TENSORRT Shashank Prasanna Deep Learning in Production -

DEEP LEARNING DEPLOYMENT WITH NVIDIA TENSORRT Shashank Prasanna Deep Learning in Production - Current Approaches - Deployment Challenges NVIDIA TensorRT - Programmable Inference Accelerator - Performance, Optimizations and Features

438 views • 28 slides
Lottery ticket hypothesis By : Grishma Gupta, Lokit Paras 1.Motivation Deep learning models

Lottery ticket hypothesis By : Grishma Gupta, Lokit Paras 1.Motivation Deep learning models

Lottery ticket hypothesis By : Grishma Gupta, Lokit Paras 1.Motivation Deep learning models have shown promising results in many domains. However, such models often have millions of parameters. The deep learning models face the following

314 views • 16 slides
Visual deep learning models, in particular for face recognition and models of invariant

Visual deep learning models, in particular for face recognition and models of invariant

Visual deep learning models, in particular for face recognition and models of invariant recognition in the ventral stream Center Name Presenter Name Towards a theory of the above tomaso poggio, CBMM, BCS, CSAIL, McGovern MIT Plan

1.14k views • 81 slides
Approach to practical application of Deep Learning in manufacturer's production line Masahiro

Approach to practical application of Deep Learning in manufacturer's production line Masahiro

Approach to practical application of Deep Learning in manufacturer's production line Masahiro Kashiwagi, Hiroyuki Kusaka, Kiminori Kurosawa and Kenji Nishide 1 1 Applications Service Agriculture Advertisement Education Security Deep

504 views • 32 slides
models to understand visual cortex 11-785 Introduction to Deep Learning Fall 2017 Michael Tarr

models to understand visual cortex 11-785 Introduction to Deep Learning Fall 2017 Michael Tarr

using deep learning models to understand visual cortex 11-785 Introduction to Deep Learning Fall 2017 Michael Tarr Department of Psychology Center for the Neural Basis of Cognition this lecture A bit out of order Oct 2: Models of

462 views • 42 slides
Evaluation of Deep Learning Evaluation of Deep Learning Models for Network Models for Network

Evaluation of Deep Learning Evaluation of Deep Learning Models for Network Models for Network

Evaluation of Deep Learning Evaluation of Deep Learning Models for Network Models for Network Performance Prediction for Performance Prediction for Scientific Facilities Scientific Facilities Makiya Nakashima: Texas A&M

438 views • 16 slides
Learning Deep Structured Models for Semantic Segmentation Guosheng Lin Semantic Segmentation

Learning Deep Structured Models for Semantic Segmentation Guosheng Lin Semantic Segmentation

Learning Deep Structured Models for Semantic Segmentation Guosheng Lin Semantic Segmentation Outline Exploring Context with Deep Structured Models Guosheng Lin, Chunhua Shen, Ian Reid, Anton van dan Hengel; Efficient Piecewise Training

992 views • 45 slides
Deep Learning and Hardware: Matching the Demands from the Machine Learning Community Ekapol

Deep Learning and Hardware: Matching the Demands from the Machine Learning Community Ekapol

Deep Learning and Hardware: Matching the Demands from the Machine Learning Community Ekapol Chuangsuwanich Department of Computer Engineering, Chulalongkorn University Deep learning Artificial Neural Networks rebranded Deeper models Bigger

808 views • 44 slides
Semi-supervised Learning with Deep Generative Models Diedrik P. Kingma, Danilo J. Rezende, Shakir

Semi-supervised Learning with Deep Generative Models Diedrik P. Kingma, Danilo J. Rezende, Shakir

Semi-supervised Learning with Deep Generative Models Diedrik P. Kingma, Danilo J. Rezende, Shakir Mohamed, Max Welling What is Deep Learning very good at? Classifying highly structured data -ImageNet -Part of Speech Tagging -MNIST

195 views • 18 slides
Learning Deep Generative Models Inference & Representation Lecture 12 Rahul G. Krishnan

Learning Deep Generative Models Inference & Representation Lecture 12 Rahul G. Krishnan

Introduction Variational Inference Deep Generative Models Summary Learning Deep Generative Models Inference & Representation Lecture 12 Rahul G. Krishnan Fall 2015 Rahul G. Krishnan Learning Deep Generative Models Introduction

1.22k views • 83 slides
Feature extraction from deep models Olgert Denas Synopsis Intro to deep models Applications

Feature extraction from deep models Olgert Denas Synopsis Intro to deep models Applications

Feature extraction from deep models Olgert Denas Synopsis Intro to deep models Applications Neurons & Nets dimer Learning & Depth G1E model Feature extraction Theory 1 Layer Nets Neural computation

1.32k views • 45 slides
FROM SCRATCH TO PRODUCTION Andrew Liu, Ryan Shen Deep Learning Solution Architect Defect

FROM SCRATCH TO PRODUCTION Andrew Liu, Ryan Shen Deep Learning Solution Architect Defect

DEFECT INSPECTION FROM SCRATCH TO PRODUCTION Andrew Liu, Ryan Shen Deep Learning Solution Architect Defect Inspection and its challenges NGC Docker images AGENDA Model set up - Unet Data preparation - DAGM Production - Speed up with

691 views • 33 slides
4 Deep Generative Models BVM 2018 Tutorial: Advanced Deep Learning Methods Jens Petersen Dept.

4 Deep Generative Models BVM 2018 Tutorial: Advanced Deep Learning Methods Jens Petersen Dept.

4 Deep Generative Models BVM 2018 Tutorial: Advanced Deep Learning Methods Jens Petersen Dept. of Neuroradiology, Heidelberg University Hospital Div. of Medical Image Computing, DKFZ Heidelberg Faculty of Physics & Astronomy, Heidelberg

1.48k views • 65 slides
Deep-Learning: Unsupervised Generative models Deep Belief Networks Deep Stacked AutoEncoders

Deep-Learning: Unsupervised Generative models Deep Belief Networks Deep Stacked AutoEncoders

Deep-Learning: Unsupervised Generative models Deep Belief Networks Deep Stacked AutoEncoders Generative Adversarial Networks Pr. Fabien MOUTARDE Center for Robotics MINES ParisTech PSL Universit Paris Fabien.Moutarde@mines-paristech.fr

777 views • 23 slides
Uni.lu HPC School 2019 PS12a: Machine / Deep learning I Keras/Tensorflow CPU/GPU Uni.lu High

Uni.lu HPC School 2019 PS12a: Machine / Deep learning I Keras/Tensorflow CPU/GPU Uni.lu High

Uni.lu HPC School 2019 PS12a: Machine / Deep learning I Keras/Tensorflow CPU/GPU Uni.lu High Performance Computing (HPC) Team Dr. F. Pinel University of Luxembourg (UL), Luxembourg http://hpc.uni.lu Dr. F. Pinel & Uni.lu HPC Team

1.13k views • 5 slides
Deep Learning to Evaluate Secure RSA Implementations Mathieu Carbone, Vincent Conin, Marie-Angela

Deep Learning to Evaluate Secure RSA Implementations Mathieu Carbone, Vincent Conin, Marie-Angela

Deep Learning to Evaluate Secure RSA Implementations Mathieu Carbone, Vincent Conin, Marie-Angela Cornlie, Franois Dassance, Guillaume Dufresne, Ccile Dumas, Emmanuel Prouff and Alexandre Venelli CEA LETI, France Thales ITSEF, France

677 views • 29 slides
Deep Learning on GPUs March 2016 What is Deep Learning? GPUs and DL AGENDA DL in practice

Deep Learning on GPUs March 2016 What is Deep Learning? GPUs and DL AGENDA DL in practice

Deep Learning on GPUs March 2016 What is Deep Learning? GPUs and DL AGENDA DL in practice Scaling up DL 2 What is Deep Learning? 3 DEEP LEARNING EVERYWHERE INTERNET & CLOUD MEDICINE & BIOLOGY SECURITY & DEFENSE MEDIA &

1.45k views • 39 slides
Deep Learning in Computer Vision Caner Hazrba Deep Learning in Action 24. June 15

Deep Learning in Computer Vision Caner Hazrba Deep Learning in Action 24. June 15

Deep Learning in Computer Vision Caner Hazrba Deep Learning in Action 24. June 15 Computer Vision Group 6 Postdocs, 16 PhD students Caner Hazrba | vision.in.tum.de Deep Learning in Computer Vision 2 Research in Computer Vision

960 views • 30 slides
Interpretable & Transparent Deep Learning Fraunhofer HHI, Machine Learning Group Wojciech

Interpretable & Transparent Deep Learning Fraunhofer HHI, Machine Learning Group Wojciech

Fraunhofer Image Processing Heinrich Hertz Institute Interpretable & Transparent Deep Learning Fraunhofer HHI, Machine Learning Group Wojciech Samek Northern Lights Deep Learning Workshop (NLDL19) Troms, Norway 8th January 2019

1.15k views • 51 slides
Deep learning for HEP/NP at NERSC Jlab Machine Learning Workshop November 6 th 2018 Wahid Bhimji

Deep learning for HEP/NP at NERSC Jlab Machine Learning Workshop November 6 th 2018 Wahid Bhimji

Deep learning for HEP/NP at NERSC Jlab Machine Learning Workshop November 6 th 2018 Wahid Bhimji (feat. Steve Farrell, Thorsten Kurth, Ben Nachman, Mustafa Mustafa, Michela Paganini, Prabhat, Evan Racah and others) and others) NERSC,

871 views • 45 slides
DeepMPLS: Fast Analysis of MPLS Configurations Using Deep Learning Fabien Geyer 1,2 and Stefan

DeepMPLS: Fast Analysis of MPLS Configurations Using Deep Learning Fabien Geyer 1,2 and Stefan

DeepMPLS: Fast Analysis of MPLS Configurations Using Deep Learning Fabien Geyer 1,2 and Stefan Schmid 3 IFIP Networking 2019 Tuesday 21 st May, 2019 3 Faculty of Computer Science, 1 Chair of Network Architectures and Services 2 Airbus Central

632 views • 25 slides
DEEP LEARNING FOR ACTIVITY RECOGNITION (A BRIEF AND INCOMPLETE SURVEY) GRAHAM TAYLOR VISION,

DEEP LEARNING FOR ACTIVITY RECOGNITION (A BRIEF AND INCOMPLETE SURVEY) GRAHAM TAYLOR VISION,

DEEP LEARNING FOR ACTIVITY RECOGNITION (A BRIEF AND INCOMPLETE SURVEY) GRAHAM TAYLOR VISION, LEARNING AND GRAPHICS GROUP & MOVEMENT GROUP COURANT INSTITUTE OF MATHEMATICAL SCIENCES NEW YORK UNIVERSITY NEW YORK, NY USA Papers and software

488 views • 33 slides

More recommend