using multi system monitoring time series to predict
play

Using Multi-System Monitoring Time Series to Predict Performance - PowerPoint PPT Presentation

Apr 04, 2023 •648 likes •1.09k views

Using Multi-System Monitoring Time Series to Predict Performance Events Andreas Schrgenhumer Mario Kahlhofer Peter Chalupar Hanspeter Mssenbck Paul Grnbacher 09.11.2018 Motivation t 2 Motivation t 2 Motivation t Train ML 2

  1. Using Multi-System Monitoring Time Series to Predict Performance Events Andreas Schörgenhumer Mario Kahlhofer Peter Chalupar Hanspeter Mössenböck Paul Grünbacher 09.11.2018

  2. Motivation t 2

  3. Motivation t 2

  4. Motivation t Train ML 2

  5. Motivation t Train ML t 2

  6. Motivation t Train ML Predict t 2

  7. Motivation t Train ML Predict t 2

  8. Motivation t Train Straightforward: ML Single system • Single component • Predict Univariate time series • t 2

  9. Motivation Multiple, interlinked components 3

  10. Motivation Multiple, interlinked components Multivariate time series 3

  11. Motivation Multiple, Event to data interlinked connection components Multivariate time series 3

  12. Motivation Multiple systems Multiple, Event to data interlinked connection components … Multivariate time series 3

  13. Motivation Multiple systems Multiple, Event to data interlinked connection components … Multivariate time series ML Train 3

  14. Approach Configs Multi- Preprocessing ML System CSVs Framework Data 4

  15. Approach Configs Multi- Preprocessing ML System CSVs Framework Data (1) Data 4

  16. Approach Configs Multi- Preprocessing ML System CSVs Framework Data (1) Data (2) Preprocessing 4

  17. Approach Configs Multi- Preprocessing ML System CSVs Framework Data (1) Data (2) Preprocessing (3) Prediction 4

  18. (1) Data System 1 * Service * * Host 1 1 * * Network Disk Interface 5

  19. (1) Data System 1 250 systems 20-day export * Service * * Host 1 1 * * Network Disk Interface 5

  20. (1) Data Service slowdowns System 1 250 systems 20-day export * Events Service * * Host 1 1 * * Network Disk Interface 5

  21. (1) Data Service slowdowns System 1 250 systems 20-day export CPU load * Events Memory available Service SWAP available … * 11 Time * Series Host 1 1 * * Network Disk Interface 5

  22. (1) Data Service slowdowns System 1 250 systems 20-day export CPU load * Events Memory available Service SWAP available … * 11 Time * Available Series Host Read time Write time 1 1 … * * Network Disk Interface 13 Time Series 5

  23. (1) Data Service slowdowns System 1 250 systems 20-day export CPU load * Events Memory available Service SWAP available … * 11 Time * Available Bytes received Series Host Read time Bytes sent Write time Packets dropped 1 1 … … * * Network Disk Interface 13 Time 10 Time Series Series ... 1-minute resolution 5

  24. (2) Preprocessing – Framework Preprocessing Framework 6

  25. (2) Preprocessing – Framework • Input: YAMLs (configurations/configs) • Contains all necessary data processing settings • Easily changeable due to YAML format systems: - “sys1” systems: - “sys2” - “sys1” systems: timeSeries: - “sys2” - “sys1” - CPU_LOAD timeSeries: - “sys2” from: “2018 -01- 19 09:00” - CPU_LOAD timeSeries: to: “2018 -02- 02 09:00” from: “2018 -01- 19 09:00” - CPU_LOAD ... to: “2018 -02- 02 09:00” from: “2018 -01- 19 09:00” leadTime: 0 ... to: “2018 -02- 02 09:00” observationWindowsBoxes: leadTime: 0 ... CPU_LOAD: observationWindowsBoxes: leadTime: 0 Preprocessing - size: 60 CPU_LOAD: observationWindowsBoxes: step: 1 - size: 60 CPU_LOAD: Framework aggregationFunctions: step: 1 - size: 60 - “AVG” aggregationFunctions: step: 1 combinationFunctions: - “AVG” aggregationFunctions: - “AVG” combinationFunctions: - “AVG” samplingMode: “PER_EVENT” - “AVG” combinationFunctions: missingDataPointMode: “NAN” samplingMode: “PER_EVENT” - “AVG” addAttributes: true missingDataPointMode: “NAN” samplingMode: “PER_EVENT” ... addAttributes: true missingDataPointMode: “NAN” ... addAttributes: true ... 6

  26. (2) Preprocessing – Framework • Input: YAMLs (configurations/configs) • Contains all necessary data processing settings • Easily changeable due to YAML format • Output: CSVs (feature vectors) • Portable format, directly useable for ML systems: - “sys1” systems: - “sys2” - “sys1” systems: timeSeries: - “sys2” - “sys1” - CPU_LOAD timeSeries: - “sys2” from: “2018 -01- 19 09:00” - CPU_LOAD timeSeries: to: “2018 -02- 02 09:00” from: “2018 -01- 19 09:00” - CPU_LOAD ... to: “2018 -02- 02 09:00” from: “2018 -01- 19 09:00” leadTime: 0 ... to: “2018 -02- 02 09:00” CPU_LOAD:AVG System Label observationWindowsBoxes: leadTime: 0 ... CPU_LOAD:AVG System Label 0.95 sys1 Event CPU_LOAD: observationWindowsBoxes: leadTime: 0 CPU_LOAD:AVG System Label Preprocessing 0.95 sys1 Event - size: 60 CPU_LOAD: observationWindowsBoxes: 0.71 sys2 No event 0.95 sys1 Event step: 1 - size: 60 CPU_LOAD: 0.71 sys2 No event Framework 0.90 sys2 Event aggregationFunctions: step: 1 - size: 60 0.71 sys2 No event 0.90 sys2 Event - “AVG” aggregationFunctions: step: 1 0.87 sys2 No event 0.90 sys2 Event combinationFunctions: - “AVG” aggregationFunctions: 0.87 sys2 No event - “AVG” combinationFunctions: - “AVG” 0.84 sys1 No event 0.87 sys2 No event samplingMode: “PER_EVENT” - “AVG” combinationFunctions: 0.84 sys1 No event missingDataPointMode: “NAN” samplingMode: “PER_EVENT” - “AVG” 0.84 sys1 No event addAttributes: true missingDataPointMode: “NAN” samplingMode: “PER_EVENT” ... addAttributes: true missingDataPointMode: “NAN” ... addAttributes: true ... 6

  27. (2) Preprocessing – Config Settings Setting Example Systems [sys1, sys2, ...] Time series [Host: CPU_LOAD, Disk: AVAILABLE, ...] From: 19-01-2018 09:00 Time frame To: 02-02-2018 09:00 Sampling mode PER_EVENT, SLIDE_THROUGH Negative sampling source NON_EVENT_SERVICES, EVENT_SERVICES, ... Lead time 10 min Observation windows 60 min, AVG aggregation, AVG combination Missing data mode DROP, NAN, LAST_VALUE, ... Metadata System, special attributes, ... ... ... 7

  28. (2) Preprocessing – Example ... Service samplingMode: “PER_EVENT” Disk 1 leadTime: 5 observationWindowsBoxes: Disk 2 Host CPU_LOAD: - size: 15 Disk 3 Network aggregationFunctions: - “MIN” - “MAX” ... DISK_WRITE: - size: 30 aggregationFunctions: - “AVG” - “MIN” - “MAX” - “STD_DEV” combinationFunctions: - “AVG” - “MIN” - “MAX” - “AVG” BYTES_SENT: - size: 5 aggregationFunctions: - “NONE” ... - size: 30 aggregationFunctions: - “AVG” ... ... 8

  29. (2) Preprocessing – Example ... Service samplingMode: “PER_EVENT” Disk 1 leadTime: 5 observationWindowsBoxes: Disk 2 Host CPU_LOAD: - size: 15 Disk 3 Network aggregationFunctions: - “MIN” - “MAX” ... DISK_WRITE: - size: 30 aggregationFunctions: - “AVG” - “MIN” - “MAX” - “STD_DEV” combinationFunctions: - “AVG” - “MIN” - “MAX” - “AVG” BYTES_SENT: - size: 5 aggregationFunctions: - “NONE” ... - size: 30 aggregationFunctions: - “AVG” ... ... 8

  30. (2) Preprocessing – Example ... Service samplingMode: “PER_EVENT” Disk 1 leadTime: 5 observationWindowsBoxes: Disk 2 Host CPU_LOAD: - size: 15 Disk 3 Network aggregationFunctions: - “MIN” - “MAX” ... DISK_WRITE: - size: 30 aggregationFunctions: - “AVG” - “MIN” - “MAX” - “STD_DEV” combinationFunctions: - “AVG” - “MIN” - “MAX” - “AVG” BYTES_SENT: - size: 5 aggregationFunctions: - “NONE” ... - size: 30 aggregationFunctions: - “AVG” ... ... 8

  31. (2) Preprocessing – Example ... Service samplingMode: “PER_EVENT” Disk 1 leadTime: 5 observationWindowsBoxes: Disk 2 Host CPU_LOAD: - size: 15 Disk 3 Network aggregationFunctions: - “MIN” - “MAX” ... DISK_WRITE: - size: 30 aggregationFunctions: - “AVG” - “MIN” - “MAX” - “STD_DEV” combinationFunctions: - “AVG” - “MIN” - “MAX” - “AVG” BYTES_SENT: - size: 5 aggregationFunctions: - “NONE” ... - size: 30 aggregationFunctions: - “AVG” ... ... 8

  32. (2) Preprocessing – Example ... Service samplingMode: “PER_EVENT” Disk 1 leadTime: 5 observationWindowsBoxes: Disk 2 Host CPU_LOAD: - size: 15 Disk 3 Network aggregationFunctions: - “MIN” - “MAX” ... DISK_WRITE: - size: 30 aggregationFunctions: - “AVG” - “MIN” - “MAX” - “STD_DEV” combinationFunctions: - “AVG” - “MIN” - “MAX” - “AVG” BYTES_SENT: - size: 5 aggregationFunctions: - “NONE” ... - size: 30 aggregationFunctions: - “AVG” ... ... 8

  33. (2) Preprocessing – Example ... Service samplingMode: “PER_EVENT” Disk 1 leadTime: 5 observationWindowsBoxes: Disk 2 Host CPU_LOAD: - size: 15 Disk 3 Network aggregationFunctions: - “MIN” - “MAX” ... DISK_WRITE: - size: 30 aggregationFunctions: - “AVG” - “MIN” - “MAX” - “STD_DEV” combinationFunctions: - “AVG” - “MIN” - “MAX” - “AVG” BYTES_SENT: - size: 5 aggregationFunctions: - “NONE” ... - size: 30 aggregationFunctions: - “AVG” ... ... 8

  34. (3) Prediction 20 days, 250 systems, 34 time series t Preprocessing Framework 9

  35. (3) Prediction 20 days, 250 systems, 34 time series t 14d 6d Train Test Preprocessing Framework 9

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Dimensional Time Series Release for Traffic Monitoring Liyue Fan , Li Xiong, Vaidy Sunderam

Dimensional Time Series Release for Traffic Monitoring Liyue Fan , Li Xiong, Vaidy Sunderam

DBSec13 Differentially Private Multi- Dimensional Time Series Release for Traffic Monitoring Liyue Fan , Li Xiong, Vaidy Sunderam Department of Math & Computer Science Emory University 9/4/2013 DBSec'13: Privacy Preserving Traffic

539 views • 25 slides
The Multi-Level Feedback Queue Operating System: Three Easy Pieces 1 Youjip Won Multi-Level

The Multi-Level Feedback Queue Operating System: Three Easy Pieces 1 Youjip Won Multi-Level

8: Scheduling: The Multi-Level Feedback Queue Operating System: Three Easy Pieces 1 Youjip Won Multi-Level Feedback Queue (MLFQ) A Scheduler that learns from the past to predict the future. Objective: Optimize turnaround time

180 views • 16 slides
Mining Approximate Top-K Subspace Anomalies in Multi-Dimensional Time-Series Data Xiaolei Li,

Mining Approximate Top-K Subspace Anomalies in Multi-Dimensional Time-Series Data Xiaolei Li,

Mining Approximate Top-K Subspace Anomalies in Multi-Dimensional Time-Series Data Xiaolei Li, Jiawei Han University of Illinois at Urbana-Champaign VLDB 2007 1 Time Series Data Many applications produce time series data Intel stock 2

888 views • 60 slides
Time Series Analysis and Mining with R Time Series Decomposi- tion Time Series Forecasting

Time Series Analysis and Mining with R Time Series Decomposi- tion Time Series Forecasting

R and Time Series Data Time Series Analysis and Mining with R Time Series Decomposi- tion Time Series Forecasting Yanchang Zhao Time Series Clustering Time Series RDataMining.com Classification http://www.rdatamining.com/ R Functions

1.91k views • 42 slides
Time series vs. point processes for space-time surveillance in public health ohle 1 , 2 Thais

Time series vs. point processes for space-time surveillance in public health ohle 1 , 2 Thais

Motivation Case Study Temporal Monitoring Spatio-temporal Monitoring Discussion References Time series vs. point processes for space-time surveillance in public health ohle 1 , 2 Thais Correa 3 Michael H 1 Department of Statistics,

1.07k views • 28 slides
A multi-platform system for understanding, monitoring and forecasting the impact of aerosol

A multi-platform system for understanding, monitoring and forecasting the impact of aerosol

A multi-platform system for understanding, monitoring and forecasting the impact of aerosol pollutants in South-East Asia Santo V. Salinas [a] Team members: Tan Li [a] and Daniel Kalbermater [a] Collaborators: Ellsworth J. Welton [b] , Brent N.

669 views • 27 slides
Time series data mining Outline Basic Knowledge Multi variate association States association

Time series data mining Outline Basic Knowledge Multi variate association States association

Time series data mining Outline Basic Knowledge Multi variate association States association What is time series data Formally, a time series data is defined as a sequence of pairs T = [(p1, t1), (p2, t2), . . . , (pi, ti), . . . , (pn, tn)]

475 views • 26 slides
PanoVu Series PanoVu Series PanoVu Family DS-2DP0818ZIX-D/236/250/836 Huge PanoVu

PanoVu Series PanoVu Series PanoVu Family DS-2DP0818ZIX-D/236/250/836 Huge PanoVu

Introduction to PanoVu Series PanoVu Series PanoVu Family DS-2DP0818ZIX-D/236/250/836 Huge PanoVu DS-2DP1636ZIX-D/236/250/836 PanoVu series DS-2PT5326IZ-DE Outdoor Panoramic Monitoring DS-2PT5306IZ-DE Multi-view PanoVu Series

335 views • 22 slides
Descriptin, Discivery, Access if Multi.Dimensiinal Data and Time Series in the Virtual

Descriptin, Discivery, Access if Multi.Dimensiinal Data and Time Series in the Virtual

Descriptin, Discivery, Access if Multi.Dimensiinal Data and Time Series in the Virtual Observatiry F.Binnarel (CDS) ackniwledges extensive cillabiratve wirk with DAL WG, DM WG, TDIG, DADI cillabiratirs, Strabiurg CDS and SVOM teams Summary

516 views • 40 slides
Efficient Multi-Instance Learning for Activity Recognition from Time Series Data Using an

Efficient Multi-Instance Learning for Activity Recognition from Time Series Data Using an

Efficient Multi-Instance Learning for Activity Recognition from Time Series Data Using an Auto-Regressive Hidden Markov Model Xinze Guan, Raviv Raich, Weng-Keen Wong School of EECS, Oregon State University Email:

629 views • 27 slides
W HAT IS TIME SERIES D ATA ? W HAT IS TIME SERIES D ATA ? A value over time W HAT IS TIME

W HAT IS TIME SERIES D ATA ? W HAT IS TIME SERIES D ATA ? A value over time W HAT IS TIME

T IME S ERIES D ATA P APERS C OVERED Interactive Visualization of Serial Periodic Data John V. Carlis and Joseph A. Konstan Visualizing and Discovering Non-Trivial Patterns in Large Time Series Databases Jessica Lin, Eamonn Keogh,

764 views • 32 slides
Outline Time series and forecasting Time series objects 1 in R Basic time series functionality

Outline Time series and forecasting Time series objects 1 in R Basic time series functionality

Time series and forecasting in R 1 Time series and forecasting in R 2 Outline Time series and forecasting Time series objects 1 in R Basic time series functionality 2 The forecast package 3 Rob J Hyndman Exponential smoothing 4 ARIMA

714 views • 8 slides
Change detection in multi-dimensional datasets and time series Andrea De Simone

Change detection in multi-dimensional datasets and time series Andrea De Simone

Change detection in multi-dimensional datasets and time series Andrea De Simone andrea.desimone@sissa.it Univ. Camerino, 2019-02-26 [DS, Jacques arXiv:1807.06038] Outline 1 Two-Sample Test: Intro & Motivation 2 Nearest

659 views • 26 slides
Towards Contact-less Vital Sign Monitoring using a COTS Resource- Constrained Multi-Core System

Towards Contact-less Vital Sign Monitoring using a COTS Resource- Constrained Multi-Core System

Towards Contact-less Vital Sign Monitoring using a COTS Resource- Constrained Multi-Core System - an experience report - Michael G. Kruger, Rink P.W. Springer, Ger M. Kersten, and Reinder J. Bril ISORC, Valencia, Spain, May 7 th , 2019

148 views • 3 slides
ThermalPlot: Visualizing Multi- Attribute Time-Series Data Using a Thermal Metaphor Holger

ThermalPlot: Visualizing Multi- Attribute Time-Series Data Using a Thermal Metaphor Holger

ThermalPlot: Visualizing Multi- Attribute Time-Series Data Using a Thermal Metaphor Holger Stitz, Samuel Gratzl, Wolfgang Aigner, Marc Streit. IEEE Transactions on Visualization and Computer Graphics ( Volume: 22, Issue: 12, Dec. 1 2016 )

322 views • 18 slides
A brief about online monitoring system By Ajit Joshi & Amol Malode Regulatory Regime PCB

A brief about online monitoring system By Ajit Joshi & Amol Malode Regulatory Regime PCB

A brief about online monitoring system By Ajit Joshi & Amol Malode Regulatory Regime PCB Guidelines WHY IS REAL TIME MONITORING SYSTEM REQUIRED? Self Monitoring mechanism within the industries Increased management responsibility for

548 views • 41 slides
Det Detect ecting ng the he 1% 1%: Gr Grow owing ng the he Sci Science ence of of Vul

Det Detect ecting ng the he 1% 1%: Gr Grow owing ng the he Sci Science ence of of Vul

Det Detect ecting ng the he 1% 1%: Gr Grow owing ng the he Sci Science ence of of Vul Vulner nerabi ability y Di Discover scovery Laurie Williams laurie_williams@ncsu.edu Real people Real Projects Real Impact 1 2 3

810 views • 68 slides
IOPin: Runtime Profiling of Parallel I/ O in HPC S ystems Seong Jo (Shawn) Kim * , S on + ,

IOPin: Runtime Profiling of Parallel I/ O in HPC S ystems Seong Jo (Shawn) Kim * , S on + ,

IOPin: Runtime Profiling of Parallel I/ O in HPC S ystems Seong Jo (Shawn) Kim * , S on + , Wei-keng Liao + , eung Woo S Mahmut Kandemir*, Raj eev Thakur # , and Alok Choudhary + * : Pennsylvania S tate University + : Northwestern University # :

451 views • 16 slides
EFFICIENT SYMBOL-LEVEL TRANSMISSION IN ERROR- PRONE WIRELESS NETWORKS Pouya Ostovari, Jie Wu,

EFFICIENT SYMBOL-LEVEL TRANSMISSION IN ERROR- PRONE WIRELESS NETWORKS Pouya Ostovari, Jie Wu,

EFFICIENT SYMBOL-LEVEL TRANSMISSION IN ERROR- PRONE WIRELESS NETWORKS Pouya Ostovari, Jie Wu, and Abdallah Khreishah Center for Networked Computing http://www.cnc.temple.edu Agenda 2 Introduction Motivation Setting Proposed

555 views • 21 slides
Best Practices for the Consolidated Plan and Action Plan May 2019 Housekeeping Logistics:

Best Practices for the Consolidated Plan and Action Plan May 2019 Housekeeping Logistics:

Best Practices for the Consolidated Plan and Action Plan May 2019 Housekeeping Logistics: 90-minute webinar All lines are muted Submit technical issues through Question function Asking questions: There will be periodic

914 views • 54 slides
CS 188: Artificial Intelligence Lecture 7: Utility Theory Pieter Abbeel UC Berkeley Many

CS 188: Artificial Intelligence Lecture 7: Utility Theory Pieter Abbeel UC Berkeley Many

CS 188: Artificial Intelligence Lecture 7: Utility Theory Pieter Abbeel UC Berkeley Many slides adapted from Dan Klein 1 Maximum Expected Utility Why should we average utilities? Why not minimax? Principle of maximum expected

303 views • 7 slides
Adversarial Examples Hanxiao Liu April 2, 2018 1 / 22 Adversarial Examples Inputs to ML

Adversarial Examples Hanxiao Liu April 2, 2018 1 / 22 Adversarial Examples Inputs to ML

Adversarial Examples Hanxiao Liu April 2, 2018 1 / 22 Adversarial Examples Inputs to ML models that an attacker has intentionally designed to cause the model to make a mistake 1 Why this is interesting: Safety. Interpretability.

881 views • 22 slides
a DSL for Configuring a Fieldbus Mathijs Schuts & Jozef Hooman Philips & ESI TNO

a DSL for Configuring a Fieldbus Mathijs Schuts & Jozef Hooman Philips & ESI TNO

Improving Maintenance by Creating a DSL for Configuring a Fieldbus Mathijs Schuts & Jozef Hooman Philips & ESI TNO October, 2016 1 Software Life Cycle Different component in a high-tech system have different life cycle

451 views • 16 slides
Increasing Automation in the Backporting of Linux Drivers Using Coccinelle Luis R. Rodriguez,

Increasing Automation in the Backporting of Linux Drivers Using Coccinelle Luis R. Rodriguez,

Increasing Automation in the Backporting of Linux Drivers Using Coccinelle Luis R. Rodriguez, (SUSE Labs) Julia Lawall (Inria/LIP6/UPMC/Sorbonne University-Whisper) January, 2015 (Unpublished work) What is backporting? port Linux v3.18

682 views • 40 slides

More recommend