Automating Operations with Machine Learning Matt Callanan Senior - - PowerPoint PPT Presentation

Matt Callanan Senior Software Development Engineer Expedia Group

Automating Operations with Machine Learning

Willie Wheeler Principal Applications Engineer Expedia Group

Too many alerts Too many false alerts Alerts not actionable Diagnosis takes too long Remediation takes too long

Can automation and machine learning help solve these problems?

Automating Ops with ML Machine Learning Overview Automating Operations

Overview

Automating Operations

System healthy System unhealthy Detect Diagnose Remediate

Develop Build Deploy Monitor Rollback

Infrastructure Queues Services Network Databases Cloud Firewalls

Feature Switch Config Change

Develop Build Deploy Monitor Rollback

Infrastructure Queues Services Network Databases Cloud Firewalls

Feature Switch Config Change

System healthy System unhealthy Detect Diagnose Remediate

How does it work?

Alert Manager Deployment Bot Treatment

Remediation

Diagnostics Anomaly Detection Logging Tracing

Apps

Metrics

Alert Manager Deployment Bot Treatment

Remediation

Diagnostics Anomaly Detection Logging Tracing

Apps

Metrics

Machine Learning Overview

Machine learning systems perform tasks by learning from data, instead

• f requiring explicit programming.

Traditional programming

Computer Program Inputs Outputs

Machine Learning

Computer Inputs Outputs Program

There are many approaches to ML.

Convolutional neural network

Dog

https://www.flickr.com/photos/a_peach/8631368705

Recurrent neural network

“Hay dos estudiantes en la cocina.” “There are two students in the kitchen.”

Model Training Executable Model ML-enabled Application Training Data Model Repo

Data Ingestion Pre-Processing Data Analysis

Automating Ops with ML

1. Anomalies and Anomaly Detection

2. ML Ops 3. Our Approach

• 4. Situation Diagnostics

Anomalies and Anomaly Detection

An anomaly is an unusual data point.

Anomalies signal a change. We want to detect them.

Time Series Anomaly Detection with Adaptive Alerting

AA minimises Mean Time To Detect (MTTD) by performing anomaly detection on streaming time series data. AA supports classical and ML-based time series anomaly detection algos.

Constant threshold STL regression Recurrent neural network Holt-Winters AWS random cut forest Custom regression

Normal Weak Anomaly Strong Anomaly Strong Anomaly Weak Anomaly

}

} }

}

}

prediction actual T-1

anomalies (Kafka) metrics (Kafka)

Anomaly detectors (Kafka streams) { "metric": "details-tp99", "timestamp": 1549932163, "value": 150 } { "metric": "details-tp99", "timestamp": 1549932163, "value": 150, "anomaly": "WEAK" }

anomalies (Kafka)

Anomaly detectors (Kafka streams)

metrics (Kafka)

ML detectors Training data (S3) ML training

ML Ops

• Data Scientists and Engineers tend to
• perate in siloes
• Models can take many months to deploy
• Training and retraining consistency issues

• Data Scientists:
• Experiment Tracking for Training and HPO
• Engineers:
• Scalability, Robustness, Repeatability
• Reliable Deployment of Models

Kubernetes ingest

Data Store (e.g. S3)

Non-Kubernetes Runtime preprocess analyze train deploy Kubeflow / Kubeflow Pipelines

Container Op Container Op Container Op Container Op Container Op

Web Service, Streaming

“aa-ct-trainer” aa-ml-training-core Python Library aa-ml-training-pipelines Python Library aa-ml-training-docker /aa-ct-trainer Python code “*_starter.py” scripts /pipeline Python code “create_pipeline.py” aa-ct-trainer Dockerfile /aa-ct-trainer à /app

aa-ml-training-app-base Dockerfile aa-ml-training-miniconda3 Dockerfile Kubeflow Pipelines SDK Python Library pandas, scipy, matplotlib, boto3, etc. Python Libraries pandas, scipy, matplotlib, boto3, etc. Python Libraries pandas, scipy, matplotlib, boto3, etc. Python Libraries

aa-ct-trainer Docker Image

aa-ml-training-app-base Docker Image aa-ml-training-miniconda3 Docker Image

Alpine Linux

• ingest_starter.py
• preprocess_starter.py
• analyze_starter.py
• train_starter.py
• deploy_starter.py

Adaptive Alerting ML Training Repositories

aa-ml-training-pipelines-base Docker Image

“aa-ml- training- core” Library “aa-ml- training- pipelines” Library

aa-ml-training-pipelines-base Dockerfile

Docker Image Stack

Our Approach

• Anomaly Detection
• Building high-volume system
• Several thousand auto-generated CT detectors for 5xx
• Fine-tuned ML models
• Exploring additional Deep Learning and LSTM models
• Diagnostics
• Automated runbook diagnostics
• Exploring ML for situation diagnostics
• Auto-Remediation
• Automated rollback hints
• Piloting full auto-rollback remediation

• Ingest at least 4 weeks of data
• Remove outliers (e.g. Hampel filtering)
• Interpolate missing data
• Analyse data set (EDA)
• Visualise with scatter plots, ACF
• Perform tests, e.g adfuller
• Record properties of data set (e.g. stationary

vs single/dual seasonality)

• Based on profile properties, select algorithms to

explore

• For each algorithm:
• Explore hyperparameter space (HPO)
• For each HP combination, find the best fit
• Select model with best set of HPs for algorithm
• Compare scores of each algorithm
• Select the best model from the best algorithm
• Ensemble detectors could serve multiple models

• When detector is created, a schedule is

created, e.g. daily/weekly

• Scheduler initiates Data Prep, Training, and

Deployment tasks

• Good for fast anomaly detection
• Detect mean shift over period of time
• Less volatile anomaly classification

• Treat first 3 weeks as “training data”
• Treat final 7 days as “test data”

Situation Diagnostics

Time series anomaly detection generates a large volume of individually unactionable anomalies. We need a small number of actionable anomalies.

Haystack is Expedia Group’s OSS distributed tracing product. Key features:

Distributed tracing Trace telemetry Dynamic service graph Anomaly detection on telemetry State snapshotting

ExpediaDotCom/haystack

Haystack service graph

ExpediaDotCom/haystack

Can machine learning help?

• Incident classification
• Incident diagnostics
• Incident prediction

Haystack data creates new automation opportunities:

ExpediaDotCom/haystack

A graph network works with graph- structured inputs and outputs. Similar to convolutional networks, but can learn graph topology (not just grid).

Use cases

Graph net + classifier

Incident classification:

Hotel bookings drop

Graph net

Incident diagnostics:

Geo Service: bad deployment

Graph net

Incident Prediction:

ALERT: Hotel bookings drop in 8m

Summary

• Open loops are costly and error prone

Close Your Loops

• Outage Detection, Diagnosis, and Remediation can be automated

to create closed loop systems Automate to Reduce MTTK, MTTD, MTTR

• ML helps automate detection and diagnosis by converting

historical observations into predictions and classifications Machine Learning

• Break down the siloes between Data Science and Engineering

MLOps

• What story is your call graph telling you?

Graph Networks

Ops automation

Closing Loops and Opening Minds (AWS re:Invent) https://www.youtube.com/watch?v=O8xLxNje30M KubeFlow http://kubeflow.org

Forecasting & anomaly detection

Forecasting: Principles and Practice https://otexts.com/fpp2/ Outlier Detection: A Survey https://web.cs.hacettepe.edu.tr/~aykut/classes/spri ng2013/bil682/supplemental/Outlier_Detection_A_S urvey.pdf

Graph networks

Relational inductive biases, deep learning, and graph networks https://arxiv.org/abs/1806.01261 A comprehensive survey on graph neural networks https://arxiv.org/abs/1901.00596 Graph Nets https://github.com/deepmind/graph_nets

Machine learning

Awesome Public Datasets https://github.com/awesomedata/awesome- public-datasets Awesome TensorFlow https://github.com/jtoy/awesome-tensorflow

Thanks!

Matt Callanan: @mcallana Willie Wheeler: @williewheeler ExpediaDotCom/adaptive-alerting ExpediaDotCom/haystack

Hiring: www.lifeatexpedia.com