Deep Learning for Predictive Maintenance Pawel Morkisz GTC 2017

Agenda • Problem • Introduction and notion of deep neural networks o Convolutional layers o Residual networks (ResNet) • One dimensional convolutional networks in failure prediction o Approach o Results and the best architecture www.relia-sol.pl

The problem Substantial cost and safety hazards caused by machinery failure Inefficient operations unexpected downtimes, repairs • lower productivity and safety • Delayed timeline costly delays, • missing critical deadlines, • damaged customer relationships • PdM aMarket $4.9B by 2021, at CAGR of 28.4% • www.relia-sol.pl

Problem setting Data collected Pattern recognition Clear insights related to through thousands of indicate oncoming failure, operations, services, sensors malfunction or anomalies logistics, design • Sensor data collected in predefined time intervals • Well specified failure records www.relia-sol.pl

Data format Timestamp Sensor 1 Sensor 2 Sensor 3 Sensor 4 Sensor 5 Sensor 6 …. Failure? 01.02.2011 00:03 999,7 5,300 5,547 0,087 3491,7 -0,942 ... 0 01.02.2011 00:04 744,6 6,053 20,665 0,178 1436,9 -0,820 ... 0 01.02.2011 00:05 4,7 9,111 3,116 0,226 6151,9 -0,410 ... 0 01.02.2011 00:06 840,9 4,413 7,863 0,059 7759,8 -0,065 ... 0 01.02.2011 00:07 756,7 0,606 22,314 0,131 4474,9 -0,429 ... 0 01.02.2011 00:08 750,9 6,303 4,633 0,092 3664,1 -0,318 ... 0 01.02.2011 00:09 639,8 3,826 5,382 0,206 3999,1 -0,271 ... 0 01.02.2011 00:10 274,2 9,073 16,963 0,066 2834,0 -0,514 ... 0 01.02.2011 00:11 551,6 4,383 16,822 0,183 1808,3 -0,334 ... 0 01.02.2011 00:12 983,7 3,497 22,169 0,087 9260,7 -0,632 ... 0 01.02.2011 00:13 742,7 3,012 23,503 0,042 7537,9 -0,481 ... 0 01.02.2011 00:14 24,7 1,394 2,590 0,085 163,9 -0,048 ... 0 01.02.2011 00:15 568,9 5,846 4,161 0,133 8403,1 -0,909 ... 1 01.02.2011 00:16 329,2 8,313 7,152 0,006 5390,7 -0,456 ... 1 01.02.2011 00:17 269,1 9,835 3,013 0,098 2576,4 -0,908 ... 1 www.relia-sol.pl

Problem setting …. Failure? ... 0 • Determine the time horizon 𝑼 ... 0 ... 0 for failure prediction ... 0 • Observations during pre-failure period ... 0 marked as the distinguished class ... 0 ... 0 • Failure records itself can be removed or not, ... 1 depending on how much they differ from the ... 1 𝑼 ... 1 rest of the set ... 1 • Binary classification - evaluation of probability ... 1 ... 1 that observation precedes failure ... 1 ... 1 www.relia-sol.pl

Predictive maintenance - interpretation Time 1 2 3 4 5 Sensors www.relia-sol.pl

Evaluation Data set is divided into three parts • Timestamp Sensor 1 …. Failure? 01.02.2011 00:03 999,7 ... 0 o model learning, 01.02.2011 00:04 744,6 ... 0 o validation of hiper-parameters Learning 01.02.2011 00:05 4,7 ... 0 01.02.2011 00:06 840,9 ... 0 o final evaluation 01.02.2011 00:07 756,7 ... 0 Model search criterion in selected • 01.02.2011 00:08 750,9 ... 0 class is the quality on the second set 01.02.2011 00:09 639,8 ... 0 Validation 01.02.2011 00:10 274,2 ... 0 The quality criterion between • 01.02.2011 00:11 551,6 ... 0 classes is quality on the third set 01.02.2011 00:12 983,7 ... 0 01.02.2011 00:13 742,7 ... 0 Chronological division • 01.02.2011 00:14 24,7 ... 0 o prevents ‘prediction of the past Test 01.02.2011 00:15 568,9 ... 1 01.02.2011 00:16 329,2 ... 1 using future’ 01.02.2011 00:17 269,1 ... 1 www.relia-sol.pl

Evaluation • 𝐶 – number of not predicted failures, 𝐷 – number of false alarms Real • industrial problem False True significantly different costs of 𝑪 , 𝑫 A 𝑪 False • Class cost coefficient 𝑦 Classified (included in model training) 𝑫 D True 𝑦 1 𝐹𝑠𝑠 = 𝑪 1 + 𝑦 + 𝑫 1 + 𝑦 www.relia-sol.pl

Independence of the observations • Sensor data o Collected cyclically, remainder trend seasonal data Actual Series o Multidimensional time series o Dependent! • Many machine learning methods Seasonal Component Breakdown require independence • Data transformations Trend Component o Decomposition (trend, periodicity, etc.) o A lot of additional variables Remainder time www.relia-sol.pl

Convolution approach Image Convolved Feature 1 1 1 0 0 0 1 1 x1 1 x0 0 x1 4 3 4 32 0 0 1 x0 1 x1 1 x0 2 4 3 0 0 1 x1 1 xo 0 x1 32 3 0 1 1 0 0 www.relia-sol.pl

Convolutional Neural Network (CNN) • Weights sharing - less parameters • Better understanding of inherent data structure source: www.relia-sol.pl

Residual networks x • Deeper architectures because the residual layers usually learn small, near zero values weight layer • The winning architecture in Identify relu F(x) x many competitions weight layer • Great stability improvement observed H(x)=F(x)+x + relu www.relia-sol.pl

Convolutional neural networks in failure prediction Sensor 1 Sensor 2 Sensor 3 …. 999,7 5,300 5,547 ... • One dimensional filters 744,6 6,053 20,665 ... 4,7 9,111 3,116 ... • Applied only to columns, 840,9 4,413 7,863 ... i. e. on subsequent 756,7 0,606 22,314 ... 750,9 6,303 4,633 ... measurements from one 639,8 3,826 5,382 ... 274,2 9,073 16,963 ... sensor 551,6 4,383 16,822 ... 983,7 3,497 22,169 ... 742,7 3,012 23,503 ... 24,7 1,394 2,590 ... 568,9 5,846 4,161 ... 329,2 8,313 7,152 ... 269,1 9,835 3,013 ... www.relia-sol.pl

Network architecture Input layer 4 x Conv (3 x 1 filter) 8 x Conv (3 x 1 filter) 16 x Conv (3 x 1 filter) Dense layer (256 neurons) Output layer (2 neurons) www.relia-sol.pl

Techniques used • Loss function taking into account large disproportions in the number of classes • Batch normalization • L2 regularization • PReLU \ ReLU activation functions www.relia-sol.pl

Results 𝑦 1 𝐹𝑠𝑠 = 𝑪 1 + 𝑦 + 𝑫 𝑦 = 950 1 + 𝑦 False True 𝐹𝑠𝑠 Class\Real False 50892 5 XGBoost 5.02 True 29 49 False 47553 1.06 DNN ResNet 4.10 Average True 2893 52.94 False 50391 0 DNN ResNet 0.06 - Best True 55 54 www.relia-sol.pl

Occupancy data set • Attempt to use the same architecture • Setting hyper-parameters on validation data • Only changes: o Weights of classes o L2 regularization coefficient www.relia-sol.pl

Occupancy data set www.relia-sol.pl

Our solutions The Mind The Eye Cloud, IIoT system Edge device • Cloud based IIoT solution • Small plug & play predictive maintenance • Fast and painless deployment and integration device • Unlimited possibilities in: • Predictive model adjusted for the machine o Adding new machines (with hierarchy) • Scalable o Quick generatingpredictive models • Onboard computations – no necessity of o On-the-fly monitoring of assets constant Internet connection o Identifying the causes of failures • Integrable with majority of industrial • Immediate access to information worldwide transmission protocols • Low cost of purchase and deployment www.relia-sol.pl

Our solutions www.relia-sol.pl

THANK YOU FOR YOUR ATTENTION! Reliability Solutions Sp. z o.o. Lublańska 34, 31 - 476 Kraków Head-office: +48 (12) 394-11-21 Sales : +48 (12) 394-11-23 ACC: +48 (12) 627-77-15 We invite you to our booth 1132! R&D: +48 (12) 394-11-31 IT: +48 (12) 394-11-29 office@relia-sol.pl

Deep Learning for Predictive Maintenance Pawel Morkisz GTC 2017 - PowerPoint PPT Presentation

Deep Learning for Predictive Maintenance Pawel Morkisz GTC 2017 Agenda Problem Introduction and notion of deep neural networks o Convolutional layers o Residual networks (ResNet) One dimensional convolutional networks in failure

Hao Su July 6, 2017 Outline Overview of 3D deep learning 3D deep learning algorithms

ACCELERATE DEEP LEARNING WITH NVIDIA'S DEEP LEARNING PLATFORM | STEPHEN JONES | GTC16 DEEP

Deep Learning on GPUs March 2016 What is Deep Learning? GPUs and DL AGENDA DL in practice

Reproducibility and Replicability in Deep Reinforcement Learning (and Other Deep Learning

Image Classification with DIGITS NVIDIA Deep Learning Institute 1 DEEP LEARNING INSTITUTE DLI

Deep Neural Networks and Deep Reinforcement Learning Deep Learning, Goodfellow, Bengio and

Natural Language Processing with Deep Learning CS224N The Future of Deep Learning + NLP Kevin

An Overview of Deep Residual Learning Semih Yagcioglu 01.03.2016 Deep Residual Learning

Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree

Mocha.jl Deep Learning in Julia Chiyuan Zhang (@pluskid) CSAIL, MIT Deep Learning Learning

Minjie Wang Deep Learning Deep Learning trend in the past 10 years Caffe State-of-art DL

Deep Learning: Theory and Practice Deep Learning - Practical 02-04-2020 Considerations

Deep Learning: State of the Art (2020) Deep Learning Lecture Series https://deeplearning.mit.edu

12. Unsupervised Deep Learning CS 535 Deep Learning, Winter 2018 Fuxin Li With materials from

Regularization for Deep Learning Lecture slides for Chapter 7 of Deep Learning

Introduction to Deep Learning Outline Deep Learning RNN CNN Attention

A Similarity Measure for Formal Ontologies with an application to ontologies of a geographic kind

DEVELOPMENT BUG PREVENTION AND ISOLATION Erika Dignam and Ross Cunniff 04 April 2016 ABOUT US

Density-Based Clustering over an Evolving Data Stream with Noise Feng Cao Martin Ester

This circular consists of two annexes: (1) Annex 1 is the guidelines for the presentation of

A Renewable E nergy Strategy for the Republic of Cyprus and the Potential Contribution from the

Brief Overview What Is PaveXpress? Overview of the AASHTO Design Guide Page-by-Page

More on the Reliability Function of the BSC Alexander Barg Andrew McGregor DIMACS, Rutgers

Reliability and Uncertainty in Structural Dynamics S. A DHIKARI In collaboration with Prof. R S.