Instrumentation, Observability, and Monitoring of Machine Learning - PowerPoint PPT Presentation

Jul 20, 2023 •621 likes •935 views

Instrumentation, Observability, and Monitoring of Machine Learning Models 1 About Me Google Engineer (2007-11) Clouderas Director of Data Science (2011-15) Slacks Director of Data Engineering (2015-2017) Slack Engineer

Instrumentation, Observability, and Monitoring of Machine Learning Models 1
About Me ● Google Engineer (2007-11) ● Cloudera’s Director of Data Science (2011-15) ● Slack’s Director of Data Engineering (2015-2017) ● Slack Engineer (now)
“”
“”
The Genesis of This Talk
Machine Learning In the Wild
Data Science Meets DevOps
Some History
Logs via the ELK Stack
Metrics with Prometheus
Prometheus Architecture
Traces
A Word About Cardinality
Make Good Decisions By Avoiding Bad Decisions
The ML Test Score
The Map Is Not The Territory
Monitor Model Decay
Build Lots of Models
Deploy Your Models Like They Are Code*
Stand On The Shoulders of Giants ● Ensembles ● Experiments ● Dark Tests ● Canary ● Sanity Checks
Tag All The Things
Circle of Competence
Garbage In...
Linking Online and Offline Metrics
Handling Cross-Language Feature Engineering
Know Your Dependencies
Monitoring For Critical Slices
Second-Order Thinking
On Razors
http://slack.com/careers 30

Download Presentation

Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

introduction to AspectJ CS 119 a program instrumentation and monitoring framework monitoring

introduction to AspectJ CS 119 a program instrumentation and monitoring framework monitoring event generation event evaluation events instrumentation behavior specification specification 2 why AspectJ? so, monitoring a

1.31k views • 79 slides

Cross-ISA Machine Instrumentation Cross-ISA Machine Instrumentation using Fast and Scalable

Cross-ISA Machine Instrumentation Cross-ISA Machine Instrumentation using Fast and Scalable using Fast and Scalable Dynamic Binary Translation Dynamic Binary Translation Emilio G. Cota Columbia University Luca P. Carloni VEE'19 April 14,

1.05k views • 45 slides

ISM Micropile Test Instrumentation Precision Monitoring Inc . ISM Micropile Our Scope 1.

Precision Monitoring Inc . ISM Micropile Test Instrumentation Precision Monitoring Inc . ISM Micropile Our Scope 1. Design two engineered fills with 10 kPa and 20 kPa shear. (Thanks - Wing) 2. Set-up instrumentation to monitor

685 views • 20 slides

The Table! How to tap into machine data for observability and business analytics Karun

Dont Leave Money On The Table! How to tap into machine data for observability and business analytics Karun Subramanian IT Operations Expert www.karunsubramanian.com (c) Karun Subramanian About the Presenter 20+ Years of experience in

544 views • 31 slides

On Observability Richard Hartmann, RichiH@ { freenode,OFTC,IRCnet } , richih@ {

Introduction Intro Monitoring Complexity Services Observability Outro On Observability Richard Hartmann, RichiH@ { freenode,OFTC,IRCnet } , richih@ { debian,fosdem,richih } .org, @TwitchiH 2019-02-03 Richard Hartmann, RichiH@ {

721 views • 37 slides

Using AOP for Detailed Runtime Monitoring Instrumentation Jonathan E Cook, joncook@nmsu.edu

Using AOP for Detailed Runtime Monitoring Instrumentation Jonathan E Cook, joncook@nmsu.edu Amjad Nusayr, anusayr@cs.nmsu.edu The 2009 Workshop on Dynamic Analysis New Mexico State University Runtime Monitoring The act of observing an

590 views • 23 slides

ALBERTA INFRASTRUCTURE AND TRANSPORTATION INSTRUMENTATION MONITORING RESULTS FALL 2006 SECTION

ALBERTA INFRASTRUCTURE AND TRANSPORTATION INSTRUMENTATION MONITORING RESULTS FALL 2006 SECTION C PEACE REGION (PEACE RIVER I HIGH LEVEL) SITE PH1 0: HWY 726:02, EUREKA RIVER OBSERVATIONS 1. Field Program and Instrumentation Status 1.1 Three

94 views • 5 slides

GEOTECHNICAL INSTRUMENTATION FOR LANDSLIDE MONITORING M. Melih Demirkan, Ph.D., P.E. RELIABLE

GEOTECHNICAL INSTRUMENTATION FOR LANDSLIDE MONITORING M. Melih Demirkan, Ph.D., P.E. RELIABLE INSTRUM RELIABLE INSTRUMEN ENTA TATI TION ON & & DATA DATA MONITORI MONITORING NG SOLUTIO SOLUTIONS NS GEOTECHNICAL | GEODETIC |

181 views • 14 slides

PRESENTATION ONLY: Instrumentation and Monitoring during Construction of the Ingula Powerhouse

See discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/267158596 PRESENTATION ONLY: Instrumentation and Monitoring during Construction of the Ingula Powerhouse Caverns Conference Paper

400 views • 28 slides

Integrated Instrumentation for Monitoring at High Flow Sites Brian Polagye University of Washington

Integrated Instrumentation for Monitoring at High Flow Sites Brian Polagye University of Washington Northwest National Marine Renewable Energy Center Andrea Copping Pacific Northwest National Laboratory Environmental Monitoring, Regulatory Needs

215 views • 17 slides

Testing Observability Amy Phillips Testing Observability | Amy Phillips | @amyjph Amy

Testing Observability Amy Phillips Testing Observability | Amy Phillips | @amyjph Amy Phillips Engineering Manager at Moo Previously manager and tester at Songkick, The Guardian, and Yahoo! Can be found @amyjph Testing

822 views • 36 slides

ME 460: Electromechanical Systems Design ME 560: Precision Machine Design and Instrumentation

ME 460: Electromechanical Systems Design ME 560: Precision Machine Design and Instrumentation Prof. Andre Sharon 3-8776 sharon@bu.edu Fraunhofer USA Center for Manufacturing Innovation ME 460/560 Course Goals: Equip students with the

234 views • 10 slides

ALBERTA TRANSPORTATION NORTH CENTRAL REGION ATHABASCA AREA INSTRUMENTATION MONITORING RESULTS

ALBERTA TRANSPORTATION NORTH CENTRAL REGION ATHABASCA AREA INSTRUMENTATION MONITORING RESULTS FALL 2017 SECTION C SITE NC017A-1: HWY 63:12 TWINNING PROJECT BETWEEN SW18-90-9-W4M AND NW31-90-9-W4M ON HWY 63 NEAR FORT McMURRAY 1.

531 views • 8 slides

Observability of Vortex Flows Arthur J. Krener ajkrener@nps.edu Research supported in part by

Observability of Vortex Flows Arthur J. Krener ajkrener@nps.edu Research supported in part by NSF and AFOSR Observability of Nonlinear Dynamics x = f ( x ) y = h ( x ) x 0 x (0) = R n , R p , x I y I p n Observability of

1.5k views • 124 slides

Instrumentation best practices in Brewing Slide 1 Ola Wesstrom Instrumentation best practices in

Products Solutions Services Instrumentation best practices in Brewing Slide 1 Ola Wesstrom Instrumentation best practices in Brewing What we will cover Endress+Hauser overview Best Practice application examples Instrumentation for

384 views • 35 slides

Integrated Instrumentation for Marine Energy Monitoring Brian Polagye, James Joslin, and Andy

Integrated Instrumentation for Marine Energy Monitoring Brian Polagye, James Joslin, and Andy Stewart University of Washington Northwest National Marine Renewable Energy Center Andrea Copping Pacific Northwest National Laboratory EIMR:

309 views • 14 slides

SLACK AND HIPCHAT ANALYSIS A S URVEY OF G ROUP C OLLABORATION T OOLS AGENDA W HAT IS S LACK ?

SLACK AND HIPCHAT ANALYSIS A S URVEY OF G ROUP C OLLABORATION T OOLS AGENDA W HAT IS S LACK ? S LACK V IDEO S LACK D EMONSTRATION W HAT IS H IP C HAT ? H IP C HAT V IDEO H IP C HAT D EMONSTRATION W RAP -U P D ISCUSSION

181 views • 15 slides

SLACK BASICS Introducing Slack Group messaging system with the persistence,

TAKE CONTROL OF SLACK BASICS Introducing Slack Group messaging system with the persistence, searchability, and chronology of email Extensible with bots and integrations that bring third-party services into Slack Both free and

301 views • 13 slides

Building a Skilled Workforce... Dr. Michael E. Harris Dean, College and Career Readiness

Building a Skilled Workforce... Dr. Michael E. Harris Dean, College and Career Readiness Economic &Workforce Development Forsyth Technical Community College Forsyth Tech Skilled Worker Model It is a comprehensive workforce and social

282 views • 17 slides

Vocational Rehabilitation Overview Keith Ozols, VR Director Melissa Glover, Salem-Keizer Public

Presentation to the House Business and Labor Committee Vocational Rehabilitation Overview Keith Ozols, VR Director Melissa Glover, Salem-Keizer Public Schools Anthony Davis, student Deb Smith, Employment Services Coordinator Iraiah Estrada,

547 views • 22 slides

Children & Young People Service Presentation to Barrow Local Committee 8 March 2019

Children & Young People Service Presentation to Barrow Local Committee 8 March 2019 Serving the people of Cumbria Children & Young People Service The service is responsible for the following areas: M ulti Agency Safeguarding

202 views • 19 slides

Webinar Series Advancing Equal Employment Opportunities and Creating Inclusive Workplaces Part

Advancing Equal Employment Opportunities and Creating Inclusive Workplaces Part 7 Webinar Series Advancing Equal Employment Opportunities and Creating Inclusive Workplaces Part 7: Sharing Your Story with Confidence in the Workplace Hosted by:

356 views • 13 slides

COMPARATIVE ONBOARDING STUDY Dropbox and the Dropbox logo are trademarks of Dropbox, Inc. Color

COMPARATIVE ONBOARDING STUDY Dropbox and the Dropbox logo are trademarks of Dropbox, Inc. Color scheme for presentation taken from Dropboxs branding. Agenda Background/ Methods Research Goals

706 views • 70 slides

Long-term SLOs for reclaimed cloud computing resources Carvalho et al. (2014) Christopher

Long-term SLOs for reclaimed cloud computing resources Carvalho et al. (2014) Christopher Little Motivation Available Capacity CPU Cores Reservation Allocation g Usage Memory Motivation Available Capacity Reservation Allocation CPU

785 views • 15 slides