scalable anomaly detection with spark and sos
play

Scalable Anomaly Detection with Spark and SOS Strata NYC - PowerPoint PPT Presentation

May 12, 2023 •203 likes •871 views

Scalable Anomaly Detection with Spark and SOS Strata NYC September 26, 2019 Hi there, my name is Jeroen Janssens Today SOS, World! Anomalies and outliers Evaluating outlier-selection algorithms Various approaches to outlier

  1. Scalable Anomaly Detection with Spark and SOS Strata NYC September 26, 2019

  2. Hi there, my name is Jeroen Janssens

  3. Today ● SOS, World! ● Anomalies and outliers ● Evaluating outlier-selection algorithms ● Various approaches to outlier selection ● Stochastic Outlier Selection ● Conclusion

  4. SOS, World! 01-sos-world.ipynb

  5. Implementations of SOS ● Python: http://bit.ly/sos-python ● Spark: http://bit.ly/sos-spark ● R: http://bit.ly/sos-r ● Flink: http://bit.ly/sos-flink

  6. Anomalies and outliers

  7. An anomaly is an observation or event that deviates qualitatively from what is considered to be normal, according to a domain expert.

  8. Detecting anomalies is important ● Expensive ● Dangerous Mess up your model ●

  9. Human anomaly detection may suffer from ● Fatigue ● Information overload Emotional bias ●

  10. Feature-vector representation

  11. Dissimilarity-matrix representation

  12. From anomaly to outlier

  13. An outlier is a data point that deviates quantitatively from the majority of the data points, according to an outlier-selection algorithm.

  14. The symbiotic relationship between the domain expert and the algorithm

  15. Data flow diagram Data flow diagram illustrating the relationship between the domain expert (square) and the outlier-selection algorithm (top circle).

  16. Six Euler diagrams (1/2)

  17. Six Euler diagrams (2/2)

  18. Evaluating outlier-selection algorithms

  19. Confusion matrix Computer says no.

  20. Four possible outcomes

  21. Evaluation Illustration of relabelling a multi-class data set into multiple one-class data sets.

  22. Anomalies are rare In order to evaluate the algorithm we simulate anomalies to be rare. Banana for scale.

  23. Outlier scores The dashed line indicates the threshold chosen by the domain expert.

  24. ROC curve An ROC curve plots the false alarm rate against the hit rate for all possible thresholds.

  25. Various approaches to outlier selection

  26. Distribution-based outlier selection

  27. Distance-based outlier selection Size does matter

  28. Density-based outlier-selection

  29. Stochastic Outlier Selection

  30. Stochastic Outlier Selection ● Unsupervised outlier selection algorithm ● Employs concept of affinity (inspired by t-SNE) One parameter: perplexity ● Computes outlier probabilities ●

  31. t-Distributed Neighbor Embedding (t-SNE; Van der Maaten, Hinton) employs affinity to perform dimensionality reduction

  32. A data point is selected as an outlier when all the other data points have insufficient affinity with it.

  33. From input to output

  34. From feature matrix to dissimilarity matrix

  35. From input to output

  36. Smooth neighborhoods

  37. Affinity between data points

  38. From input to output

  39. From affinity to binding probability The binding matrix B is obtained by normalising each row in the affinity matrix A.

  40. Binding probabilities form a graph

  41. Binding probabilities form a graph

  42. Stochastic Neighbor Graph A data point belongs to the outlier class when no it is not selected by any other data points.

  43. Three SNGs The three SNGs Ga, Gb, and Gc are sampled from the discrete probability distribution P(G).

  44. Set of all SNGs

  45. Approximating outlier probabilities by sampling SNGs

  46. Demo: Sampling SNGs in CoffeeScript and D3 http://bit.ly/sos-d3

  47. Computing outlier probabilities through marginalisation

  48. Computing outlier probabilities in closed form

  49. Proof!

  50. Selecting outliers

  51. Adaptive variances via the perplexity parameter

  52. Continuous binary search

  53. Perplexity influences outlier probabilities

  54. Evaluation and comparison

  55. Putlier-score plots

  56. Real-world datasets

  57. Synthetic datasets

  58. Synthetic datasets

  59. Synthetic datasets

  60. Synthetic datasets

  61. SOS performs significantly better

  62. Spark implementation of SOS

  63. Spark implementation of SOS ● Developed by Fokko Driesprong ● Works with DataFrame API ● Available on GitHub ● Plan is to make it part of MLLib

  64. SOS on PySpark 92-pyspark-sos.ipynb

  65. Summary Outlier selection can support the detection of anomalies ● SOS is an intuitive and probabilistic algorithm to select outliers ● SOS has a very good performance ● No free lunch ●

  66. Thank you! Here are some links ● Blog: http://bit.ly/sos-blog ● D3 Demo: http://bit.ly/sos-d3 ● Python implementation: http://bit.ly/sos-python ● Spark implementation: http://bit.ly/sos-spark ● R implementation: http://bit.ly/sos-r ● Flink implementation: http://bit.ly/sos-flink

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Scalable Architecture for Anomaly Detection and Visualization in Power Generating Assets

Scalable Architecture for Anomaly Detection and Visualization in Power Generating Assets

Scalable Architecture for Anomaly Detection and Visualization in Power Generating Assets Paras Jain, Chirag Tailor , Sam Ford, Liexiao (Richard) Ding, Michael Phillips, Fang (Cherry) Liu, Nagi Gabraeel, Polo Chau 1/13 Background

325 views • 13 slides
Anomaly Detection of Trajectories Junier B. Oliva Anomaly Detection An anomaly (or outlier)

Anomaly Detection of Trajectories Junier B. Oliva Anomaly Detection An anomaly (or outlier)

Anomaly Detection of Trajectories Junier B. Oliva Anomaly Detection An anomaly (or outlier) in a dataset is an instance that is abnormal or unlikely based on the rest of the dataset If the instances in the dataset are labeled as

760 views • 19 slides
What is an anomaly? Alastair Rushworth Data Scientist DataCamp Anomaly Detection in R Defining

What is an anomaly? Alastair Rushworth Data Scientist DataCamp Anomaly Detection in R Defining

DataCamp Anomaly Detection in R ANOMALY DETECTION IN R What is an anomaly? Alastair Rushworth Data Scientist DataCamp Anomaly Detection in R Defining the term anomaly Anomaly: a data point or collection of data points that do not follow the

870 views • 21 slides
Anomaly Detection Jia-Bin Huang Virginia Tech Spring 2019 ECE-5424G / CS-5824 Administrative

Anomaly Detection Jia-Bin Huang Virginia Tech Spring 2019 ECE-5424G / CS-5824 Administrative

Anomaly Detection Jia-Bin Huang Virginia Tech Spring 2019 ECE-5424G / CS-5824 Administrative Anomaly Detection Motivation Developing an anomaly detection system Anomaly detection vs. supervised learning Choosing what features

514 views • 34 slides
Data Mining II Anomaly Detection Heiko Paulheim Anomaly Detection Also known as Outlier

Data Mining II Anomaly Detection Heiko Paulheim Anomaly Detection Also known as Outlier

Data Mining II Anomaly Detection Heiko Paulheim Anomaly Detection Also known as Outlier Detection Automatically identify data points that are somehow different from the rest Working assumption: There are considerably

984 views • 72 slides
Data Mining II Anomaly Detection Heiko Paulheim Anomaly Detection Also known as Outlier

Data Mining II Anomaly Detection Heiko Paulheim Anomaly Detection Also known as Outlier

Data Mining II Anomaly Detection Heiko Paulheim Anomaly Detection Also known as Outlier Detection Automatically identify data points that are somehow different from the rest Working assumption: There are considerably

905 views • 74 slides
Learning Rules for Anomaly Detection (LERAD) of Hostile Network Traffic Matt Mahoney Overview

Learning Rules for Anomaly Detection (LERAD) of Hostile Network Traffic Matt Mahoney Overview

Learning Rules for Anomaly Detection (LERAD) of Hostile Network Traffic Matt Mahoney Overview Prior Work in Network Anomaly Detection The 1999 DARPA Intrusion Detection Evaluation Packet Header Anomaly Detection (PHAD) Application

575 views • 18 slides
<Title> Yiqun Hu, SP Group Agenda Condition monitoring & anomaly detection

<Title> Yiqun Hu, SP Group Agenda Condition monitoring & anomaly detection

Data Council Singapre 2019 Autoencoder Forest for Anomaly Detection from IoT Time Series <Title> Yiqun Hu, SP Group Agenda Condition monitoring & anomaly detection Autoencoder for anomaly detection Autoencoder

540 views • 27 slides
In Incorporating Feedback in into Tree-based Anomaly Detection Shubhomoy Das, Weng-Keen Wong,

In Incorporating Feedback in into Tree-based Anomaly Detection Shubhomoy Das, Weng-Keen Wong,

In Incorporating Feedback in into Tree-based Anomaly Detection Shubhomoy Das, Weng-Keen Wong, Alan Fern, Thomas G. Dietterich and Md Amran Siddiqui School of EECS Anomaly Detection Goal: Identify rare or strange objects 2 Anomaly

756 views • 51 slides
Dataflow Anomaly Detection Presented By Archana Viswanath Computer Science and Engineering The

Dataflow Anomaly Detection Presented By Archana Viswanath Computer Science and Engineering The

Dataflow Anomaly Detection Presented By Archana Viswanath Computer Science and Engineering The Pennsylvania State University Anomaly Intrusion Detection Systems Anomaly Intrusion Detection Systems Model normal behavior. Attack - Any

417 views • 20 slides
Anomaly Based Network Intrusion Detection with Unsupervised Outlier Detection Jiong Zhang and

Anomaly Based Network Intrusion Detection with Unsupervised Outlier Detection Jiong Zhang and

Anomaly Based Network Intrusion Detection with Unsupervised Outlier Detection Jiong Zhang and Mohammad Zulkernine School of Computing Queens University, Kingston Ontario, Canada K7L 3N6 {zhang, mzulker} @cs.queensu.ca Abstract- Anomaly

284 views • 6 slides
Efficient Anomaly Detection by Isolation using Nearest Neighbour Ensemble Tharindu

Efficient Anomaly Detection by Isolation using Nearest Neighbour Ensemble Tharindu

Information Technology Efficient Anomaly Detection by Isolation using Nearest Neighbour Ensemble Tharindu Rukshan Bandaragoda Kai Ming Ting David Albrecht Fei Tony Liu Jonathan R. Wells Outline Overview of anomaly detection

657 views • 41 slides
Isolation trees Alastair Rushworth Data Scientist DataCamp Anomaly Detection in R Isolation

Isolation trees Alastair Rushworth Data Scientist DataCamp Anomaly Detection in R Isolation

DataCamp Anomaly Detection in R ANOMALY DETECTION IN R Isolation trees Alastair Rushworth Data Scientist DataCamp Anomaly Detection in R Isolation tree DataCamp Anomaly Detection in R Isolation tree plots DataCamp Anomaly Detection in R

631 views • 19 slides
On Entropy in Network Traffic Anomaly Detection Jayro Santiago-Paz, Deni Torres-Roman.

On Entropy in Network Traffic Anomaly Detection Jayro Santiago-Paz, Deni Torres-Roman.

Introduction Feature Extraction Entropy Calculation Anomaly detection Classification Open Issues On Entropy in Network Traffic Anomaly Detection Jayro Santiago-Paz, Deni Torres-Roman. Cinvestav, Campus Guadalajara, Mexico November 2015

690 views • 24 slides
ACCT 420: Topic modeling and anomaly detection Session 8 Dr. Richard M. Crowley 1 Front matter

ACCT 420: Topic modeling and anomaly detection Session 8 Dr. Richard M. Crowley 1 Front matter

ACCT 420: Topic modeling and anomaly detection Session 8 Dr. Richard M. Crowley 1 Front matter 2 . 1 Learning objectives Theory: NLP Anomaly detection Application: Understand annual report readability Examine the

1.12k views • 85 slides
An Evaluation of Effect of Packet Sampling on Anomaly Detection Method Takuya Motodate

An Evaluation of Effect of Packet Sampling on Anomaly Detection Method Takuya Motodate

An Evaluation of Effect of Packet Sampling on Anomaly Detection Method Takuya Motodate April 25, 2010 The 3rd CAIDA-WIDE-CASFI Joint Measurement Workshop @Osaka 1 2010 4 25 Background Anomaly Detection:

331 views • 20 slides
Conference on Seasonality, Seasonal Adjustment and their implications for Short-Term Analysis and

Conference on Seasonality, Seasonal Adjustment and their implications for Short-Term Analysis and

Conference on Seasonality, Seasonal Adjustment and their implications for Short-Term Analysis and Forecasting 10-12 May 2006 Comparison of Methods Accounting for Outliers during Seasonal Adjustment J. Aston Outliers SA Outliers Simulations

619 views • 25 slides
Outlier detection in Wireless Sensor Networks By: Johan Becker & Fredrik Nilsson Purpose of

Outlier detection in Wireless Sensor Networks By: Johan Becker & Fredrik Nilsson Purpose of

Outlier detection in Wireless Sensor Networks By: Johan Becker & Fredrik Nilsson Purpose of this presentation Aims: Introduce Wireless Sensor Networks (WSN) Discuss challenges in WSNs Demonstrate with a case study how a distributed system

321 views • 29 slides
Outlier Detection Outlier detection is both easy and difficult. It is easy since there are

Outlier Detection Outlier detection is both easy and difficult. It is easy since there are

Outlier Detection Outlier detection is both easy and difficult. It is easy since there are several relatively straightforward tests for the presence of outliers. It is difficult since there are no firm rules as to when outlier removal is

210 views • 20 slides
Outlier Detection Motivation: Fraud Detection http://i.imgur.com/ckkoAOp.gif Jian Pei: CMPT

Outlier Detection Motivation: Fraud Detection http://i.imgur.com/ckkoAOp.gif Jian Pei: CMPT

Outlier Detection Motivation: Fraud Detection http://i.imgur.com/ckkoAOp.gif Jian Pei: CMPT 741/459 Data Mining -- Outlier Detection (1) 2 Techniques: Fraud Detection Features Dissimilarity Groups and noise

878 views • 30 slides
Year / Level Analysis on Horse Age Number Of Starters Completion Rate 2014 One Star Two Star

Year / Level Analysis on Horse Age Number Of Starters Completion Rate 2014 One Star Two Star

Year / Level Analysis on Horse Age Number Of Starters Completion Rate 2014 One Star Two Star Three Star One Star Two Star Three Star 1 1774 2238 1035 72.6% 57.3% 40.2% 2 345 390 106 68.4% 54.4% 45.3% 3 78 62 20 62.8%

844 views • 51 slides
Keys to Success for Gifted Kids Virginia Bateman FCUSD If you could give your bright, curious,

Keys to Success for Gifted Kids Virginia Bateman FCUSD If you could give your bright, curious,

Keys to Success for Gifted Kids Virginia Bateman FCUSD If you could give your bright, curious, preconscious, possibly gift ft ed , seven year old the keys to a successful life what would they be? We all define success differently, but

472 views • 24 slides
Integrated Regional Resource Plan Local Advisory Committee Meeting #4 October 24, 2016

Integrated Regional Resource Plan Local Advisory Committee Meeting #4 October 24, 2016

GREENSTONE-MARATHON Integrated Regional Resource Plan Local Advisory Committee Meeting #4 October 24, 2016 Discussion Outline Background/Refresher LAC members present findings of socio-economic report Discussion of updates for the

270 views • 24 slides
Southern Alliance for Clean Energy comments on 2020 Ten Year Site Plans August 18, 2020 MAGGIE

Southern Alliance for Clean Energy comments on 2020 Ten Year Site Plans August 18, 2020 MAGGIE

Southern Alliance for Clean Energy comments on 2020 Ten Year Site Plans August 18, 2020 MAGGIE SHOBER Director of Utility Reform maggie@cleanenergy.org Southern Alliance for Clean Energy P.O. Box 1842 | Knoxville, TN 37901 2 S A C E M I S

159 views • 15 slides

More recommend