Instrumentation, Observability, and Monitoring of Machine Learning - - PowerPoint PPT Presentation

▶

Jul 20, 2023 621 likes •937 views

Instrumentation, Observability, and Monitoring of Machine Learning Models 1 About Me Google Engineer (2007-11) Clouderas Director of Data Science (2011-15) Slacks Director of Data Engineering (2015-2017) Slack Engineer

SLIDE 1

Instrumentation, Observability, and Monitoring of Machine Learning Models

SLIDE 2

About Me

Google Engineer

(2007-11)

Cloudera’s Director of

Data Science (2011-15)

Slack’s Director of Data

Engineering (2015-2017)

Slack Engineer (now)

SLIDE 3

“”

SLIDE 4

“”

SLIDE 5

The Genesis of This Talk

SLIDE 6

Machine Learning In the Wild

SLIDE 7

Data Science Meets DevOps

SLIDE 8

Some History

SLIDE 9

Logs via the ELK Stack

SLIDE 10

Metrics with Prometheus

SLIDE 11

Prometheus Architecture

SLIDE 12

Traces

SLIDE 13

A Word About Cardinality

SLIDE 14

Make Good Decisions By Avoiding Bad Decisions

SLIDE 15

The ML Test Score

SLIDE 16

The Map Is Not The Territory

SLIDE 17

Monitor Model Decay

SLIDE 18

Build Lots of Models

SLIDE 19

Deploy Your Models Like They Are Code*

SLIDE 20

Stand On The Shoulders of Giants

Ensembles
Experiments
Dark Tests
Canary
Sanity Checks

SLIDE 21

Tag All The Things

SLIDE 22

Circle of Competence

SLIDE 23

Garbage In...

SLIDE 24

Linking Online and Offline Metrics

SLIDE 25

Handling Cross-Language Feature Engineering

SLIDE 26

Know Your Dependencies

SLIDE 27

Monitoring For Critical Slices

SLIDE 28

Second-Order Thinking

SLIDE 29

On Razors

SLIDE 30