Building Efficient ML Pipelines and Responsible AI Solutions Adi - - PowerPoint PPT Presentation

▶
building efficient ml pipelines and responsible ai
SMART_READER_LITE
LIVE PREVIEW

Building Efficient ML Pipelines and Responsible AI Solutions Adi - - PowerPoint PPT Presentation

Jul 07, 2023 235 likes •722 views

Building Efficient ML Pipelines and Responsible AI Solutions Adi Polak Microsoft @adipolak Trust LETS START FROM THE BEGINNING. What happens when we get raw data? @adipolak @adipolak ML Process / Life Cycle 1 Gather Data 2

slide-1
SLIDE 1

Building Efficient ML Pipelines and Responsible AI Solutions

Adi Polak Microsoft

@adipolak

slide-2
SLIDE 2
slide-3
SLIDE 3

Trust

slide-4
SLIDE 4
  • LET’S START FROM THE BEGINNING.

What happens when we get raw data?

@adipolak

slide-5
SLIDE 5

@adipolak

slide-6
SLIDE 6

Gather Data

ML Process / Life Cycle

Feature Extract, Clean and Normalize Select algorithm Evaluate model Data/Insights visualization

4 5 1 2 3

Repeat!

@adipolak

slide-7
SLIDE 7

@adipolak

slide-8
SLIDE 8

But in real life: Accuracy < 0.5 ROC curve ☹

@adipolak

slide-9
SLIDE 9

Aim for high Accuracy

@adipolak

slide-10
SLIDE 10

What can you do?

Automate!

@adipolak

slide-11
SLIDE 11

HOW? Pipelines!

slide-12
SLIDE 12

What are pipelines?

slide-13
SLIDE 13

Visualize

Azure Machine Learning

Big Data/ ML Pipelines

@adipolak

slide-14
SLIDE 14

Demo

Apache Spark ML Pipelines

@adipolak

slide-15
SLIDE 15
slide-16
SLIDE 16
slide-17
SLIDE 17
slide-18
SLIDE 18
slide-19
SLIDE 19
slide-20
SLIDE 20
slide-21
SLIDE 21
slide-22
SLIDE 22
slide-23
SLIDE 23

Stepan Pushkarev, CTO, Hydrosphere.io

@adipolak

slide-24
SLIDE 24

Visualize

Azure Machine Learning Azure Machine Learning

Big Data/ ML Pipelines

@adipolak

slide-25
SLIDE 25

High accuracy! But, at what cost?

@adipolak

slide-26
SLIDE 26

false positives

@adipolak

slide-27
SLIDE 27

Human centric

@adipolak

Responsible AI

slide-28
SLIDE 28
slide-29
SLIDE 29

Pixabay

slide-30
SLIDE 30

Our updated goals:

Lawful Ethical Robust

@adipolak

slide-31
SLIDE 31

ML is a black box

ML algorithm Data Model Training: Testing/Prediction: Model Data Prediction

@adipolak

slide-32
SLIDE 32

Visualize

Azure Machine Learning Azure Machine Learning

Big Data/ ML Pipelines

@adipolak

slide-33
SLIDE 33

Visualize

Azure Machine Learning

Big Data/ ML Pipelines

Check and transform the data Visualize the model

@adipolak

E x p l a i n e r s Balance the data

slide-34
SLIDE 34

How Microsoft support Responsible AI

ONLINE FREE HIGH QUALITY COURSES INVESTED 1B$ IN OPEN AI 115M$ GRANT FOR AI FOR GOOD OPEN SOURCE

@adipolak

aka.ms/free-responsible-ai-cour se aka.ms/ml-interpretability-to

  • l
slide-35
SLIDE 35

@adipolak

aka.ms/ml-interpretability-to

  • l
slide-36
SLIDE 36
slide-37
SLIDE 37
slide-38
SLIDE 38

aka.ms/ml-interpretability-to

  • l
slide-39
SLIDE 39

Azure Cognitive Services

@adipolak

aka.ms/AA6kex s

slide-40
SLIDE 40

Tools

Spark Streaming Spark ML Spark SQL MLflow

@adipolak

slide-41
SLIDE 41

But in real life: Accuracy < 0.5 ROC curve ☹

slide-42
SLIDE 42

Demo

Apache Spark ML Pipelines with Cognitive Services

@adipolak

slide-43
SLIDE 43
slide-44
SLIDE 44

Demo

@adipolak

slide-45
SLIDE 45

You are only Good as your Data is

@adipolak

Use explainers Understand your data

slide-46
SLIDE 46

Thank you !

aka.ms/free-responsible-ai-course aka.ms/twitter_sentiment_analysis aka.ms/ml-interpretability-tool aka.ms/ai-for-good-grant

Learn more !

@adipolak

slide-47
SLIDE 47

What is Machine learning

  • Lifecycle:
  • Gather data
  • Data preparation – clean it
  • Data wrangling
  • Data analysis
  • Feature extraction
  • Train model
  • Test model
  • Deployment