machine learning and discovery with kubernetes
play

Machine learning and discovery with Kubernetes William Benton - PowerPoint PPT Presentation

Jan 31, 2024 •133 likes •907 views

Machine learning and discovery with Kubernetes William Benton @willb willb@redhat.com What do machine learning workflows look like? @willb #SEMLA19 @willb #SEMLA19 @willb #SEMLA19 @willb #SEMLA19 @willb #SEMLA19 @willb #SEMLA19

  1. Machine learning and discovery with Kubernetes William Benton • @willb • willb@redhat.com

  2. What do machine learning workflows look like?

  3. @willb #SEMLA19

  4. @willb #SEMLA19

  5. @willb #SEMLA19

  6. @willb #SEMLA19

  7. @willb #SEMLA19

  8. @willb #SEMLA19

  9. @willb #SEMLA19

  10. codifying problem 
 data collection and metrics and cleaning @willb #SEMLA19

  11. data collection feature model training and cleaning engineering and tuning @willb #SEMLA19

  12. data collection feature model training and cleaning engineering and tuning @willb #SEMLA19

  13. feature model training model engineering and tuning validation @willb #SEMLA19

  14. feature model training model engineering and tuning validation @willb #SEMLA19

  15. model model monitoring and validation deployment validation @willb #SEMLA19

  16. model model monitoring and validation deployment validation @willb #SEMLA19

  17. codifying problem 
 data collection feature model training model model monitoring and and metrics and cleaning engineering and tuning validation deployment validation @willb #SEMLA19

  18. codifying problem 
 data collection feature model training model model monitoring and and metrics and cleaning engineering and tuning validation deployment validation defining types prototyping and interfaces @willb #SEMLA19

  19. codifying problem 
 data collection feature model training model model monitoring and and metrics and cleaning engineering and tuning validation deployment validation unit, behavioral, and formal integration testing verification @willb #SEMLA19

  20. codifying problem 
 data collection feature model training model model monitoring and and metrics and cleaning engineering and tuning validation deployment validation deployment monitoring @willb #SEMLA19

  21. @willb #SEMLA19

  22. @willb #SEMLA19

  23. What’s a container?

  24. % @willb #SEMLA19

  25. % pip install numpy @willb #SEMLA19

  26. executable /usr/bin/pip arguments pip install numpy environment LANG=en_US USER=willb ... virtual memory file handles root filesystem / process table network routes @willb #SEMLA19

  27. executable /usr/bin/pip arguments pip install numpy environment LANG=en_US USER=willb ... virtual memory file handles root filesystem / process table network routes @willb #SEMLA19

  28. executable /usr/bin/pip arguments pip install numpy environment LANG=en_US USER=willb ... virtual memory file handles root filesystem / process table network routes @willb #SEMLA19

  29. executable /usr/bin/pip arguments pip install numpy environment LANG=en_US USER=willb ... virtual memory file handles root filesystem / process table network routes @willb #SEMLA19

  30. executable /usr/bin/pip arguments pip install numpy environment LANG=en_US USER=willb ... virtual memory file handles root filesystem / process table network routes @willb #SEMLA19

  31. executable /usr/bin/pip arguments pip install numpy environment LANG=en_US USER=willb ... virtual memory file handles root filesystem /var/lib/envs/main process table network routes @willb #SEMLA19

  32. executable /usr/bin/pip arguments pip install numpy MAXIMUM environment LANG=en_US USER=willb ... 100 virtual memory file handles root filesystem /var/lib/envs/main process table km / h network routes @willb #SEMLA19

  33. Immutable images a6afd91e user application code 6b8cad3e 33721112 configuration and e8cae4f6 2bb6ab16 installation recipes a8296f7e base image 979229b9 @willb #SEMLA19

  34. Stateless microservices @willb #SEMLA19

  35. Stateless microservices @willb #SEMLA19

  36. Stateless microservices @willb #SEMLA19

  37. Stateless microservices @willb #SEMLA19

  38. Stateless microservices @willb #SEMLA19

  39. Stateless microservices @willb #SEMLA19

  40. Stateless microservices @willb #SEMLA19

  41. Stateless microservices @willb #SEMLA19

  42. Declarative app configuration @willb #SEMLA19

  43. Integration and deployment OK! @willb #SEMLA19

  44. Integration and deployment application code OK! configuration and installation recipes base image @willb #SEMLA19

  45. Integration and deployment application code OK! configuration and installation recipes base image @willb #SEMLA19

  46. Integration and deployment application code configuration and installation recipes base image @willb #SEMLA19

  47. What containers offer 
 data scientists

  48. @willb #SEMLA19

  49. @willb #SEMLA19

  50. O K ! O K ! @willb #SEMLA19

  51. No friction: mybinder.org @willb #SEMLA19

  52. More flexible: source-to-image % @willb #SEMLA19

  53. More flexible: source-to-image % builder image application image https://github.com/openshift/source-to-image @willb #SEMLA19

  56. @willb #SEMLA19

  57. @willb #SEMLA19

  60. @willb #SEMLA19

  61. m A @willb #SEMLA19

  62. m A @willb #SEMLA19

  63. (joint) distribution of input data? distribution of predictions? m A distribution of acyclic paths taken through scoring code? @willb #SEMLA19

  64. @willb #SEMLA19

  65. Where from here?

  66. application developers data engineers web and transform developer UI events mobile transform federate databases archive file, object transform reporting storage train management models data scientists @willb #SEMLA19

  67. application developers data engineers web and transform developer UI events mobile transform federate databases archive file, object transform reporting storage train management models machine learning engineers data scientists @willb #SEMLA19

  68. radanalytics.io @willb #SEMLA19

  69. opendatahub.io @willb #SEMLA19

  70. Kubeflow @willb #SEMLA19

  71. What did we talk about today?

  72. @willb #SEMLA19

  73. @willb #SEMLA19

  74. @willb #SEMLA19

  75. @willb #SEMLA19

  76. THANKS willb@redhat.com • @willb https://chapeau.freevariable.com @willb #SEMLA19

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Machine Learning: Der KDD-Prozess Knowledge Discovery in Databases Machine Learning Data-Mining

Machine Learning: Der KDD-Prozess Knowledge Discovery in Databases Machine Learning Data-Mining

Machine Learning: Der KDD-Prozess Knowledge Discovery in Databases Machine Learning Data-Mining Aufgaben Aufgabe Von Daten zum Wissen Wissensextraktion durch We are drowing in automatisches Erkennen von information,

1.01k views • 47 slides
Machine-Learning & AI-based Approaches for GPCR Bioactive Ligand Discovery Sebastian Raschka,

Machine-Learning & AI-based Approaches for GPCR Bioactive Ligand Discovery Sebastian Raschka,

Cambridge Healthtech Institute's 14th Annual GPCR-Based Drug Discovery Discovery on Target Conference September 19, 2019 | Boston, MA https://www.discoveryontarget.com/GPCR-drug-discovery Machine-Learning & AI-based Approaches for GPCR

411 views • 38 slides
Kubernetes for machine learning: productivity over primitives Sophie Watson @sophwats

Kubernetes for machine learning: productivity over primitives Sophie Watson @sophwats

Kubernetes for machine learning: productivity over primitives Sophie Watson @sophwats sophie@redhat.com William Benton @willb willb@redhat.com Kubernetes for machine learning: productivity over primitives Sophie

1.24k views • 102 slides
Machine Learning for Information Discovery Thorsten Joachims Cornell University Department of

Machine Learning for Information Discovery Thorsten Joachims Cornell University Department of

Machine Learning for Information Discovery Thorsten Joachims Cornell University Department of Computer Science (Supervised) Machine Learning GENERAL: EXAMPLE: Text Retrieval Input: Input: training examples queries with relevance

700 views • 28 slides
D ata-driven drug discovery for a variety of diseases by machine learning

D ata-driven drug discovery for a variety of diseases by machine learning

D ata-driven drug discovery for a variety of diseases by machine learning Yoshihiro Yamanishi Kyushu University Medical Ins;tute of Bioregula;on

833 views • 25 slides
Edge-based Discovery of Training Data for Machine Learning CMU authors Deep Learning Recipe

Edge-based Discovery of Training Data for Machine Learning CMU authors Deep Learning Recipe

CSci 8980 Edge-based Discovery of Training Data for Machine Learning CMU authors Deep Learning Recipe Collect a large amount of data and label it Select a model and train a DNN Deploy the DNN for inference Labelled Data Some

409 views • 14 slides
Discovery of Latent Factors in High-dimensional Data via Spectral Methods Furong Huang

Discovery of Latent Factors in High-dimensional Data via Spectral Methods Furong Huang

Discovery of Latent Factors in High-dimensional Data via Spectral Methods Furong Huang University of Maryland Workshop on Quantum Machine Learning 1 / 39 Machine Learning - Excitements Success of Supervised Learning Image classification

1.24k views • 80 slides
Airflow on Kubernetes: Containerizing your Workflows By Michael Hewitt Agenda Kubernetes

Airflow on Kubernetes: Containerizing your Workflows By Michael Hewitt Agenda Kubernetes

Airflow on Kubernetes: Containerizing your Workflows By Michael Hewitt Agenda Kubernetes Overview 1 Airflows integration with Kubernetes 2 Deployment of Airflow on Kubernetes 3 Kubernetes Pod Operator and its benefits 4 DAG Development

2.79k views • 14 slides
An Exercise in An Exercise in Machine Learning Machine Learning

An Exercise in An Exercise in Machine Learning Machine Learning

An Exercise in An Exercise in Machine Learning Machine Learning http://www.cs.iastate.edu/~cs573x/bbsilab.html Machine Learning Software Preparing Data Building Classifiers Interpreting Results Machine Learning Software

496 views • 26 slides
Machine Learning By Alex Scarlatos What is Machine Learning? Machine Learning is the process by

Machine Learning By Alex Scarlatos What is Machine Learning? Machine Learning is the process by

Machine Learning By Alex Scarlatos What is Machine Learning? Machine Learning is the process by which computers can be trained through observation, rather than being explicitly programmed. Basically, a program is given input and adjusts its

556 views • 17 slides
Machine Learning: Study of algorithms that improve their performance P at some task T

Machine Learning: Study of algorithms that improve their performance P at some task T

Machine Learning 10-601 Tom M. Mitchell Machine Learning Department Carnegie Mellon University January 12, 2015 Today: Readings: What is machine learning? The Discipline of ML Decision tree learning Mitchell, Chapter 3

872 views • 29 slides
Traditional Machine Learning: Unsupervised Learning Juhan Nam Traditional Machine Learning

Traditional Machine Learning: Unsupervised Learning Juhan Nam Traditional Machine Learning

GCT634/AI613: Musical Applications of Machine Learning (Fall 2020) Traditional Machine Learning: Unsupervised Learning Juhan Nam Traditional Machine Learning Pipeline in Classification Tasks A set of hand-designed audio features are selected

398 views • 26 slides
with Kubernetes and Tensorflow Daniel van der Ende & Tim van Cann o IT Consultancy o 40

with Kubernetes and Tensorflow Daniel van der Ende & Tim van Cann o IT Consultancy o 40

Data Council NYC 2019 Reducing Flight Delays with Kubernetes and Tensorflow Daniel van der Ende & Tim van Cann o IT Consultancy o 40 Data Scientists, Machine Learning Engineers, and Data Engineers o Based in Amsterdam o We help

885 views • 56 slides
Discovery of Latent Factors in High-dimensional Data Using Tensor Methods Furong Huang

Discovery of Latent Factors in High-dimensional Data Using Tensor Methods Furong Huang

Discovery of Latent Factors in High-dimensional Data Using Tensor Methods Furong Huang University of California, Irvine Machine Learning Conference 2016 New York City 1 / 24 Machine Learning - Modern Challenges Big Data Challenging Tasks

686 views • 57 slides
CS 335 Machine Learning What is Machine Learning? Dan Sheldon Spring 2019 What is Machine

CS 335 Machine Learning What is Machine Learning? Dan Sheldon Spring 2019 What is Machine

1/22/19 CS 335 Machine Learning What is Machine Learning? Dan Sheldon Spring 2019 What is Machine Learning? A Simple Task: Recognize Obama How do you program a computer to Recognize faces? Recommend movies? Decide which web

581 views • 5 slides
Machine Learning Machine Learning: algorithms that use experience to improve their

Machine Learning Machine Learning: algorithms that use experience to improve their

Machine Learning Machine Learning: algorithms that use experience to improve their performance We use machine learning in situations where it is very challenging (or impossible) to define the rules by hand: e.g. face detection

1.04k views • 28 slides
C B + k i 1 / i i , 1 i n , i ( 2 1 ) can be used

C B + k i 1 / i i , 1 i n , i ( 2 1 ) can be used

Priority Inheritance Protocol Midterm Grades 96 100 Let the blocker task inherit the priority of 90 80 79 the task being blocked 80 70 critical section 60 Grade P(1) only blocked by 4 50 40 27 1 1 1 30 20 Inherit

275 views • 4 slides
PIP-III Options and Overview Valeri Lebedev Fermilab Workshop on Megawatt Rings & IOTA/FAST

PIP-III Options and Overview Valeri Lebedev Fermilab Workshop on Megawatt Rings & IOTA/FAST

PIP-III Options and Overview Valeri Lebedev Fermilab Workshop on Megawatt Rings & IOTA/FAST Collaboration May 7-10, 2018, Fermilab Objectives The only definition of PIP-III we know: PIP-III will follow PIP-II Choice of parameters

320 views • 31 slides
PIP-II: Powering Discoveries in High Energy Physics Lia Merminga In partnership with: Fermilab

PIP-II: Powering Discoveries in High Energy Physics Lia Merminga In partnership with: Fermilab

PIP-II: Powering Discoveries in High Energy Physics Lia Merminga In partnership with: Fermilab India/DAE Italy/INFN LLRF Workshop 2019 UK/STFC France/CEA/Irfu, CNRS/IN2P3 September 29 - October 3, 2019 Chicago, IL 2 9/30/2019 Outline

401 views • 39 slides
Scientific Programming in mpags-python.github.io Steven Bamford An introduction to scientific

Scientific Programming in mpags-python.github.io Steven Bamford An introduction to scientific

PHYS4038/MLiS and AS1/MPAGS Scientific Programming in mpags-python.github.io Steven Bamford An introduction to scientific programming with Session 3 : Staying organised Session 3 In this session: Organising your python installation

360 views • 22 slides
Facility Quality Forum Guidelines A Feedback and data PIP Tool for Residents, Families, and Staff

Facility Quality Forum Guidelines A Feedback and data PIP Tool for Residents, Families, and Staff

Facility Quality Forum Guidelines A Feedback and data PIP Tool for Residents, Families, and Staff Part of the Facility QAA/QI/QAPI Suggested participants: Appoint a facilitator that is a trusted resource to residents, families, and staff. A

39 views • 3 slides
UL HPC School 2017 PS9: [Advanced] Prototyping with Python UL High Performance Computing (HPC)

UL HPC School 2017 PS9: [Advanced] Prototyping with Python UL High Performance Computing (HPC)

UL HPC School 2017 PS9: [Advanced] Prototyping with Python UL High Performance Computing (HPC) Team C. Parisot University of Luxembourg (UL), Luxembourg http://hpc.uni.lu C. Parisot (University of Luxembourg) UL HPC School 2017 1 / 12

271 views • 14 slides
Tut#14: A4 Prerequisites CPSC 501 Dr. J. Hudson University of Calgary Arshia Hosseini T01/T02

Tut#14: A4 Prerequisites CPSC 501 Dr. J. Hudson University of Calgary Arshia Hosseini T01/T02

Tut#14: A4 Prerequisites CPSC 501 Dr. J. Hudson University of Calgary Arshia Hosseini T01/T02 Option 1: Virtual Environment Python Virtual Environments allow Python packages to be installed in an isolated location for a particular

685 views • 7 slides
Tensorflow 2.x Review Session CS330: Deep Multi-task and Meta Learning 9/17/2019 Rafael

Tensorflow 2.x Review Session CS330: Deep Multi-task and Meta Learning 9/17/2019 Rafael

Tensorflow 2.x Review Session CS330: Deep Multi-task and Meta Learning 9/17/2019 Rafael Rafailov Overview 1. Installation a. Installing on your machine b. Using GPUs c. Using Google Colab 2. Tensorflow Basics a. Data pipelines b.

892 views • 24 slides

More recommend