smartpred unsupervised
play

SmartPred: Unsupervised Hard Disk Failure Detection MLHPCS 2020 - PowerPoint PPT Presentation

Apr 04, 2024 •264 likes •413 views

SmartPred: Unsupervised Hard Disk Failure Detection MLHPCS 2020 PHILIPP ROMBACH & JANIS KEUPER 2 Hard Disk Drive Isolation Forest SmartPred Dataset Agenda Feature Selection Setup Evaluation Replacement

  1. SmartPred: Unsupervised Hard Disk Failure Detection MLHPCS 2020 PHILIPP ROMBACH & JANIS KEUPER

  2. 2  Hard Disk Drive  Isolation Forest  SmartPred  Dataset Agenda  Feature Selection  Setup  Evaluation  Replacement Process

  3. 3 Hard Disk Drive Structure Failures  Disk  Mechanical  Electrical  Controller Firmware

  4. 4 Hard Disk Drive - SMART Attribute Definition Attribute Definition SMART 1 Read Error Rate SMART 190 Airflow Temperature SMART 3 Spin Up Time SMART 191 G-sense Error Rate SMART 4 Start/Stop Count SMART 192 Power-off Retract Count SMART 5 Reallocated Sectors Count SMART 193 Load Cycle Count SMART 7 Seek Error Rate SMART 194 Temperature SMART 9 Power-On Hours SMART 197 Current Pending Sector Count SMART 10 Spin Retry Count SMART 198 Uncorrectable Sector Count SMART 12 Power Cycle Count SMART 199 UltraDMA CRC Error Count SMART 184 End-to-End error / IOEDC SMART 240 Head Flying Hours SMART 187 Reported Uncorrectable Errors SMART 241 Total LBAs Written SMART 188 Command Timeout SMART 242 Total LBAs Read

  5. 5 Isolation Forest Anomaly X 0 Normal Instance X n

  6. 6 SmartPred - Dataset date serial_number model capacity_bytes failure smart_5_normalized smart_5_raw 01.01.2019Z305B2QN ST4000DM000 4,00079E+12 0 119.0 0 01.01.2019ZJV0XJQ4 ST12000NM0007 1,20001E+13 0 82.0 85 01.01.2019ZJV0XJQ3 ST12000NM0007 1,20001E+13 0 80.0 255 01.01.2019ZJV0XJQ0 ST12000NM0007 1,20001E+13 1 78.0 1034 01.01.2019PL1331LAHG1S4H HGSTHMS5C4040ALE64 4,00079E+12 0 100.0 0  Samples: 12614746  Disks: 37768  Failed Disks: 868

  7. 7 SmartPred - HDD Operating Time

  8. 8 SmartPred - Feature Selection Correlation coefficient to feature Failure  SMART_5_raw: 0.07  SMART_187_raw: 0.02  SMART_197_raw: 0.03  SMART_5_diff: 0.12  SMART_187_diff: 0.01  SMART_197_diff: 0.03

  9. 9 SmartPred - Hyperparameter

  10. 10 SmartPred - Evaluation Precision Recall

  11. 11 SmartPred - Evaluation FDR FAR Precision Computation Time IF_FDR 84.54% 0.0073% 44.21% 1.10 s ± 10 ms IF_FAR 28.37% 0.0006% 77.85% 1.10 s ± 10 ms OneClassSVM 95.39% 0.2925% 2.19% 15.9 s ± 780 ms LocalOutlierFactor 96.19% 1.1916% 0.55% 16.0 s ± 671 ms

  12. 12 SmartPred - Replacement Process Node 1 Node 1 Node 2 Node 2

  13. 13 Thank you for watching!

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

UNSUPERVISED LEARNING, CLUSTERING UNSUPERVISED LEARNING UNSUPERVISED LEARNING Supervised

UNSUPERVISED LEARNING, CLUSTERING UNSUPERVISED LEARNING UNSUPERVISED LEARNING Supervised

UNSUPERVISED LEARNING, CLUSTERING UNSUPERVISED LEARNING UNSUPERVISED LEARNING Supervised learning: X - y pairs, f(x) function approximation Unsupervised learning: only X, no y Exploring the space of X measurements,

440 views • 14 slides
Introduction to PCA Unsupervised Learning in R Unsupervised learning Two methods of

Introduction to PCA Unsupervised Learning in R Unsupervised learning Two methods of

UNSUPERVISED LEARNING IN R Introduction to PCA Unsupervised Learning in R Unsupervised learning Two methods of clustering finding groups of homogeneous items Next up, dimensionality reduction Find structure in features

904 views • 29 slides
On the Limitations of Unsupervised Bilingual Dictionary Induction Anders Sgaard Sebastian

On the Limitations of Unsupervised Bilingual Dictionary Induction Anders Sgaard Sebastian

On the Limitations of Unsupervised Bilingual Dictionary Induction Anders Sgaard Sebastian Ruder Ivan Vuli Background: Unsupervised MT 2 Background: Unsupervised MT Recently: Unsupervised neural machine translation (Artetxe

1.36k views • 98 slides
Unsupervised Learning and Clustering l In unsupervised learning you are given a data set with no

Unsupervised Learning and Clustering l In unsupervised learning you are given a data set with no

Unsupervised Learning and Clustering l In unsupervised learning you are given a data set with no output classifications (labels) l Clustering is an important type of unsupervised learning PCA was another type of unsupervised learning l The

858 views • 38 slides
Unsupervised Learning Unsupervised vs Supervised Learning: Most of this course focuses on

Unsupervised Learning Unsupervised vs Supervised Learning: Most of this course focuses on

Unsupervised Learning Unsupervised vs Supervised Learning: Most of this course focuses on supervised learning methods such as regression and classification. In that setting we observe both a set of features X 1 , X 2 , . . . , X p for each

934 views • 50 slides
Unsupervised Learning Unsupervised vs Supervised Learning: Most of this course focuses on

Unsupervised Learning Unsupervised vs Supervised Learning: Most of this course focuses on

Unsupervised Learning Unsupervised vs Supervised Learning: Most of this course focuses on supervised learning methods such as regression and classification. In that setting we observe both a set of features X 1 , X 2 , . . . , X p for each

721 views • 60 slides
Automatic Learning of a Morphological Model Theory and Unsupervised Approaches Unsupervised

Automatic Learning of a Morphological Model Theory and Unsupervised Approaches Unsupervised

Automatic Learning of a Morphological Model Theory and Unsupervised Approaches Unsupervised Learning - ToC Foundations Problem description General architecture Papers: Goldsmith 2001 and 2006 Goldsmith and Hu 2004

1.23k views • 43 slides
Unsupervised Learning Andrea Passerini passerini@disi.unitn.it Machine Learning Unsupervised

Unsupervised Learning Andrea Passerini passerini@disi.unitn.it Machine Learning Unsupervised

Unsupervised Learning Andrea Passerini passerini@disi.unitn.it Machine Learning Unsupervised Learning Unsupervised Learning Setting Supervised learning requires the availability of labelled examples Labelling examples can be an extremely

752 views • 27 slides
12. Unsupervised Deep Learning CS 535 Deep Learning, Winter 2018 Fuxin Li With materials from

12. Unsupervised Deep Learning CS 535 Deep Learning, Winter 2018 Fuxin Li With materials from

12. Unsupervised Deep Learning CS 535 Deep Learning, Winter 2018 Fuxin Li With materials from Wanli Ouyang, Zsolt Kira, Lawrence Neal, Raymond Yeh, Junting Lou and Teck-Yian Lim Unsupervised Learning in General Unsupervised learning is

1.48k views • 115 slides
UNSUPERVISED LEARNING AND CLUSTERING Jeff Robble, Brian Renzenbrink, Doug Roberts Unsupervised

UNSUPERVISED LEARNING AND CLUSTERING Jeff Robble, Brian Renzenbrink, Doug Roberts Unsupervised

UNSUPERVISED LEARNING AND CLUSTERING Jeff Robble, Brian Renzenbrink, Doug Roberts Unsupervised Procedures A procedure that uses unlabeled data in its classification process. Why would we use these? Collecting and labeling large data sets can

544 views • 28 slides
Practical Unsupervised Learning INFO/CS 4300, Spring 2016 Jack Hessel Unsupervised Learning is

Practical Unsupervised Learning INFO/CS 4300, Spring 2016 Jack Hessel Unsupervised Learning is

Practical Unsupervised Learning INFO/CS 4300, Spring 2016 Jack Hessel Unsupervised Learning is Cool! But how can we use this in our projects? 1. 2. But first, lets look at our dataset... Data Dimensionality! Probability of Lung Cancer

999 views • 55 slides
lti 1 (typically) Unsupervised learning in NLP non-convex optimization lti 2

lti 1 (typically) Unsupervised learning in NLP non-convex optimization lti 2

Concavity and Initialization for Unsupervised Dependency Parsing Kevin Gimpel Noah A. Smith lti 1 (typically) Unsupervised learning in NLP non-convex optimization lti 2 Dependency Model with Valence (Klein & Manning,

718 views • 45 slides
Unsupervised learning introduction October 7, 2019 Unsupervised learning introduction

Unsupervised learning introduction October 7, 2019 Unsupervised learning introduction

Unsupervised learning introduction October 7, 2019 Unsupervised learning introduction October 7, 2019 1 / 39 Intro General statement of the problem One has a set of N observations ( x 1 , x 2 , ..., x N ) of a random p -vector X having

495 views • 32 slides
Machine Learning for NLP Unsupervised Learning Aurlie Herbelot 2019 Centre for Mind/Brain

Machine Learning for NLP Unsupervised Learning Aurlie Herbelot 2019 Centre for Mind/Brain

Machine Learning for NLP Unsupervised Learning Aurlie Herbelot 2019 Centre for Mind/Brain Sciences University of Trento 1 Unsupervised learning In unsupervised learning, we learn without training data. The idea is to find a

998 views • 67 slides
Unsupervised learning General introduction to unsupervised learning PCA Special directions

Unsupervised learning General introduction to unsupervised learning PCA Special directions

Unsupervised learning General introduction to unsupervised learning PCA Special directions These are special directions we will try to find. Best direction u : |u| 2 = 1 2 1. Minimize : d i X i T u is the projection length x i u d i T

1.01k views • 29 slides
Unsupervised Learning Introduction Nakul Verma Unsupervised Learning What can we learn from

Unsupervised Learning Introduction Nakul Verma Unsupervised Learning What can we learn from

Unsupervised Learning Introduction Nakul Verma Unsupervised Learning What can we learn from data when label information is not available? Supervised learning framework Supervised learning Data: Assumption: there is a (relatively simple)

442 views • 16 slides
Remanence: The difficulty of deletion 4/5/2010 Administrative Announcements All discussion

Remanence: The difficulty of deletion 4/5/2010 Administrative Announcements All discussion

Remanence: The difficulty of deletion 4/5/2010 Administrative Announcements All discussion sections are cancelled tomorrow (4/6) Remanence on hard disk Deleting a file does not delete the file even if you empty the Trash

459 views • 29 slides
2000-02-25 - Part 1 Lecture 22: Virtual reality Immersive visualization environments

2000-02-25 - Part 1 Lecture 22: Virtual reality Immersive visualization environments

Mats Nyl en February 25, 2000 Slide 1 of 13 2000-02-25 - Part 1 Lecture 22: Virtual reality Immersive visualization environments Interaction Haptics Systems Steering Extra: CAVE article by Cruz-Neira, Sandin

518 views • 13 slides
AirWave Bundle Hole-Home Gesture Recognition and Non-Contact Haptic Feedback Talk held by Damian

AirWave Bundle Hole-Home Gesture Recognition and Non-Contact Haptic Feedback Talk held by Damian

AirWave Bundle Hole-Home Gesture Recognition and Non-Contact Haptic Feedback Talk held by Damian Scherrer on April 30 th 2014 New Means of Communicating with Electronic Devices Input Whole-home gestures (WiSee) Response Non-contact haptic

1.02k views • 23 slides
3D Interaction with Virtual Environments based on Body and Brain Anatole Lcuyer (Inria)

3D Interaction with Virtual Environments based on Body and Brain Anatole Lcuyer (Inria)

3D Interaction with Virtual Environments based on Body and Brain Anatole Lcuyer (Inria) anatole.lecuyer@inria.fr 2 Introduction http://team.inria.fr/hybrid Anatole Lcuyer Inria Senior Researcher PARIS Hybrid team RENNES

469 views • 35 slides
DRAFT Virtual Disk Integrity in Real Time JP Blake * Chris Rogers SUNY Binghamton Assured

DRAFT Virtual Disk Integrity in Real Time JP Blake * Chris Rogers SUNY Binghamton Assured

DRAFT Virtual Disk Integrity in Real Time JP Blake * Chris Rogers SUNY Binghamton Assured Information Security Binghamton, NY, USA Rome, NY, USA B. XenClient XT Abstract This paper introduces the Virtual Disk Integrity in Real Time

78 views • 4 slides
12 Mass-Storage structure

12 Mass-Storage structure

. Swap-Space Management . . . . . . . . Disk Management Disk Scheduling Disk Structure Overview of Mass Storage Structure . . . . 12 Mass-Storage structure

498 views • 29 slides
SIGDOC Program Review Robert Pierce SIGDOC Chair robertp@us.ibm.com IBM Watson core technology

SIGDOC Program Review Robert Pierce SIGDOC Chair robertp@us.ibm.com IBM Watson core technology

SIGDOC Program Review Robert Pierce SIGDOC Chair robertp@us.ibm.com IBM Watson core technology group March 14, 2013 A call to action Managed change for immediate and future improvements Effectively communicated with our members

65 views • 6 slides
COMP 110-001 Mid-Term Review Yi Hong May 27, 2015 Announcement Midterm on Friday, May

COMP 110-001 Mid-Term Review Yi Hong May 27, 2015 Announcement Midterm on Friday, May

COMP 110-001 Mid-Term Review Yi Hong May 27, 2015 Announcement Midterm on Friday, May 29 Closed books, no notes, no computer Today A whirlwind tour of almost everything we have covered so far You should start preparing

990 views • 62 slides

More recommend