compression and similarity indexing for time series
play

Compression and Similarity Indexing for Time Series Masters Thesis - PowerPoint PPT Presentation

Jan 05, 2023 •510 likes •983 views

CHAIR PROF. BHM Compression and Similarity Indexing for Time Series Masters Thesis Marco Neumann | 19th of August 2016 KIT University of the State of Baden-Wuerttemberg and National Laboratory of the Helmholtz Association www.kit.edu

  1. CHAIR PROF. BÖHM Compression and Similarity Indexing for Time Series Master’s Thesis Marco Neumann | 19th of August 2016 KIT – University of the State of Baden-Wuerttemberg and National Laboratory of the Helmholtz Association www.kit.edu

  2. Outline Google 𝑜 -gram data Clean-up Similarity Baseline CASINO TIMES Final Words Marco Neumann – CASINO TIMES 19th of August 2016 2/34 1 Google 𝑜 -gram data 2 Clean-up 3 Similarity 4 Baseline 5 CASINO TIMES 6 Final Words

  3. Google 𝑜 -gram data Google 𝑜 -gram data Clean-up Similarity Baseline CASINO TIMES Final Words Marco Neumann – CASINO TIMES 19th of August 2016 3/34

  4. Public Data Set Google 𝑜 -gram data Clean-up Similarity Baseline CASINO TIMES Final Words Marco Neumann – CASINO TIMES 19th of August 2016 4/34

  5. Information Provided by the Data Set Similarities = hints for common cause Warning similarity ≠ causality Google 𝑜 -gram data Clean-up Similarity Baseline CASINO TIMES Final Words Marco Neumann – CASINO TIMES 19th of August 2016 5/34

  6. Current problems Clean-up 19th of August 2016 Marco Neumann – CASINO TIMES Final Words CASINO TIMES Baseline Similarity Google 𝑜 -gram data „similarity“ is not precisely defined 1 for interactive analysis data is „big“ 1 choosing possible candidates is subject to frame confirmation bias slow manual analysis 6/34

  7. Goals exact description of „similarity“ allowing of interactive nearest neighbor queries design & evaluation of baseline design & evaluation of an own approach Google 𝑜 -gram data Clean-up Similarity Baseline CASINO TIMES Final Words Marco Neumann – CASINO TIMES 19th of August 2016 7/34

  8. Clean-up Google 𝑜 -gram data Clean-up Similarity Baseline CASINO TIMES Final Words Marco Neumann – CASINO TIMES 19th of August 2016 8/34

  9. Steps OCR errors 19th of August 2016 Marco Neumann – CASINO TIMES Final Words CASINO TIMES Baseline Similarity Clean-up Google 𝑜 -gram data only last 256 years rare words lemmatisation stemming lowercase NFKC Unicode normalization word classes numbers 9/34 1 string filtering: 2 string normalization: 3 word normalization: 4 pruning:

  10. Results 1 -grams: ≈ 800 000 2 -grams: ≈ 6 400 000 Google 𝑜 -gram data Clean-up Similarity Baseline CASINO TIMES Final Words Marco Neumann – CASINO TIMES 19th of August 2016 10/34

  11. Similarity Google 𝑜 -gram data Clean-up Similarity Baseline CASINO TIMES Final Words Marco Neumann – CASINO TIMES 19th of August 2016 11/34

  12. Input Data Google 𝑜 -gram data Clean-up Similarity Baseline CASINO TIMES Final Words Marco Neumann – CASINO TIMES 19th of August 2016 12/34

  13. Normalization Google 𝑜 -gram data Clean-up Similarity Baseline CASINO TIMES Final Words Marco Neumann – CASINO TIMES 19th of August 2016 13/34

  14. (Smooth) Gradients Google 𝑜 -gram data Clean-up Similarity Baseline CASINO TIMES Final Words Marco Neumann – CASINO TIMES 19th of August 2016 14/34

  15. DTW Similar structure, but sometimes slightly off ⇒ use Dynamic Time Warping (DTW) (limited by a Sakoe-Chiba Band of radius 𝑠 ) VLDB, 2002, Exact Indexing of Dynamic Time Warping; copying is by permission of the Very Large Data Base Endowment. Google 𝑜 -gram data Clean-up Similarity Baseline CASINO TIMES Final Words Marco Neumann – CASINO TIMES 19th of August 2016 15/34

  16. Final Order Clean-up 19th of August 2016 Marco Neumann – CASINO TIMES Final Words CASINO TIMES Baseline Similarity Google 𝑜 -gram data on demand pre-calculation radius of 𝑠 using 𝜏 16/34 1 log (𝑦 + 1) 2 Gauss-smoothing 3 gradient calculation 4 DTW with warping

  17. Sanity Check Google 𝑜 -gram data Clean-up Similarity Baseline CASINO TIMES Final Words Marco Neumann – CASINO TIMES 19th of August 2016 17/34

  18. Examples of Philosophic Institute Google 𝑜 -gram data Clean-up Similarity Baseline CASINO TIMES Final Words Marco Neumann – CASINO TIMES 19th of August 2016 18/34

  19. Baseline Google 𝑜 -gram data Clean-up Similarity Baseline CASINO TIMES Final Words Marco Neumann – CASINO TIMES 19th of August 2016 19/34

  20. R-tree-based index VLDB, 2002, Exact Indexing of Dynamic Time Warping; copying is by permission of the Very Large Data Base Endowment. Google 𝑜 -gram data Clean-up Similarity Baseline CASINO TIMES Final Words Marco Neumann – CASINO TIMES 19th of August 2016 20/34

  21. Index Inefficiency Google 𝑜 -gram data Clean-up Similarity Baseline CASINO TIMES Final Words Marco Neumann – CASINO TIMES 19th of August 2016 21/34

  22. Performance Google 𝑜 -gram data Clean-up Similarity Baseline CASINO TIMES Final Words Marco Neumann – CASINO TIMES 19th of August 2016 22/34

  23. CASINO TIMES Google 𝑜 -gram data Clean-up Similarity Baseline CASINO TIMES Final Words Marco Neumann – CASINO TIMES 19th of August 2016 23/34

  24. Goals Clean-up 19th of August 2016 Marco Neumann – CASINO TIMES Final Words CASINO TIMES Baseline Similarity Google 𝑜 -gram data primary: use normal hardware slow pre-processing, fast search enable subrange queries w/o re-indexing secondary: compress data speed up nn queries using an index 24/34

  25. Wavelet decomposition Google 𝑜 -gram data Clean-up Similarity Baseline CASINO TIMES Final Words Marco Neumann – CASINO TIMES 19th of August 2016 25/34

  26. Information Merging Clean-up 19th of August 2016 Marco Neumann – CASINO TIMES Final Words CASINO TIMES Baseline Similarity Google 𝑜 -gram data search similar subtrees (of different time series) ( = compression error is below threshold) difference of coefficients is small same children merge node if: node-by-node greedy method process one whole tree at the time 26/34

  27. Example Google 𝑜 -gram data Clean-up Similarity Baseline CASINO TIMES Final Words Marco Neumann – CASINO TIMES 19th of August 2016 27/34

  28. Example (zoomed) Google 𝑜 -gram data Clean-up Similarity Baseline CASINO TIMES Final Words Marco Neumann – CASINO TIMES 19th of August 2016 28/34

  29. Weakness Google 𝑜 -gram data Clean-up Similarity Baseline CASINO TIMES Final Words Marco Neumann – CASINO TIMES 19th of August 2016 29/34

  30. Failed Improvements Clean-up 19th of August 2016 Marco Neumann – CASINO TIMES Final Words CASINO TIMES Baseline Similarity Google 𝑜 -gram data merge entire subtrees (same index structure) DB seeding information / subtree pruning drop time constraint for leaves DTW for leaves random boosting merge entire subtrees (FLANN) 30/34

  31. Final Words Google 𝑜 -gram data Clean-up Similarity Baseline CASINO TIMES Final Words Marco Neumann – CASINO TIMES 19th of August 2016 31/34

  32. Conclusion foundation for future research 2 : definition of similarity fast baseline algorithm knowledge about tree-like methods ⇒ not promising 2 starting collaboration with Prof. Dr. Sanders Google 𝑜 -gram data Clean-up Similarity Baseline CASINO TIMES Final Words Marco Neumann – CASINO TIMES 19th of August 2016 32/34

  33. Possible Ideas Google 𝑜 -gram data 19th of August 2016 Marco Neumann – CASINO TIMES Final Words CASINO TIMES Baseline Similarity Clean-up + patching compression using: time series encoding using functions (e.g. cubic splines) locality-preserving hashing static/dynamic downsampling (e.g. snappy, lz4, gzip, xz, brotli) general purpose compression of chunks non-IEEE data types (e.g. A-law and 𝜈 -law) IEEE-half floating point 33/34

  34. Thanks Dr.-Ing. Martin Schäler Prof. Dr.-Ing. Klemens Böhm IPD IT team Philosophic Friends Miguel Angel Meza Martínez Google 𝑜 -gram data Clean-up Similarity Baseline CASINO TIMES Final Words Marco Neumann – CASINO TIMES 19th of August 2016 34/34

  35. References I Title picture: 19th of August 2016 Marco Neumann – CASINO TIMES Kaushik Chakrabarti et al. „Locally Adaptive Dimensionality Reduction for Indexing Large Time Series Databases“. In: ACM Trans. [5] Lutz Bornmann and Rüdiger Mutz. „Growth rates of modern science: A bibliometric analysis“. In: CoRR abs/1402.4578 (2014). URL: [4] Ioannina . 1999, pp. 27–29. R. J. Alcock et al. „Time-series similarity queries employing a feature-based approach“. In: In 7 th Hellenic Conference on Informatics, [3] N. Ahmed, T. Natarajan, and K. R. Rao. „Discrete Cosine Transform“. In: IEEE Transactions on Computers C-23.1 (Jan. 1974), pp. 90–93. ISSN: [2] pp. 490–501. ISBN: 1-55860-379-4. of the 21th International Conference on Very Large Data Bases . VLDB ’95. San Francisco, CA, USA: Morgan Kaufmann Publishers Inc., 1995, [1] 35/34 cb 2013 „Casino Royale“ by Rebecca Siegel https://www.flickr.com/photos/grongar/8704148177/ Rakesh Agrawal et al. „Fast Similarity Search in the Presence of Noise, Scaling, and Translation in Time-Series Databases“. In: Proceedings 0018-9340. DOI: 10.1109/T-C.1974.223784 . http://arxiv.org/abs/1402.4578 . Database Syst. 27.2 (June 2002), pp. 188–228. ISSN: 0362-5915. DOI: 10.1145/568518.568520 .

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Quality of Similarity Rankings in Time Series T. Bernecker, in Time Series M. E. Houle, H.-P.

Quality of Similarity Rankings in Time Series T. Bernecker, in Time Series M. E. Houle, H.-P.

Quality of Similarity Rankings Quality of Similarity Rankings in Time Series T. Bernecker, in Time Series M. E. Houle, H.-P. Kriegel, P. Krger, 12th International Symposium on Spatial and Temporal M. Renz, E. Schubert, Databases (SSTD

1.14k views • 18 slides
Uncertain Time-Series Similarity: Return to the Basics Dallachiesa et al., VLDB 2012 Li Xiong,

Uncertain Time-Series Similarity: Return to the Basics Dallachiesa et al., VLDB 2012 Li Xiong,

Uncertain Time-Series Similarity: Return to the Basics Dallachiesa et al., VLDB 2012 Li Xiong, CS730 Problem Problem: uncertain time-series similarity Applications: location tracking of moving objects; traffic monitoring; remote

721 views • 34 slides
Indexing and Cla In lassify fying Gig igabytes of Time Series under Tim ime Warping C.W. Tan

Indexing and Cla In lassify fying Gig igabytes of Time Series under Tim ime Warping C.W. Tan

Indexing and Cla In lassify fying Gig igabytes of Time Series under Tim ime Warping C.W. Tan G.I. Webb F. Petitjean 20 2017 17 SIA SIAM In International l Con onference on on DATA MIN INING 27 27 April il 20 2017 17 1 2

824 views • 46 slides
We need a better perceptual similarity metric Lubomir Bourdev WaveOne, Inc. CVPR Workshop

We need a better perceptual similarity metric Lubomir Bourdev WaveOne, Inc. CVPR Workshop

We need a better perceptual similarity metric Lubomir Bourdev WaveOne, Inc. CVPR Workshop and Challenge on Learned Compression June 18th 2018 Challenges in benchmarking compression Measurement of perceptual similarity Consideration

563 views • 40 slides
Criticality hidden in acoustic emission time series from concrete specimen under compression

Criticality hidden in acoustic emission time series from concrete specimen under compression

1st International Electronic Conference on Applied Sciences 10/11/2020 - 30/11/2020 Criticality hidden in acoustic emission time series from concrete specimen under compression Gianni Niccolini 1 , Giuseppe Lacidogna 1 , Boris Rojo Tanzi 2 ,

241 views • 20 slides
NPFL103: Information Retrieval (3) Index construction, Distributed and dynamic indexing, Index

NPFL103: Information Retrieval (3) Index construction, Distributed and dynamic indexing, Index

Index construction Distributed indexing Dynamic indexing Index compression NPFL103: Information Retrieval (3) Index construction, Distributed and dynamic indexing, Index compression Pavel Pecina Institute of Formal and Applied Linguistics

923 views • 66 slides
CS6220: DATA MINING TECHNIQUES Mining Time Series Data Instructor: Yizhou Sun yzsun@ccs.neu.edu

CS6220: DATA MINING TECHNIQUES Mining Time Series Data Instructor: Yizhou Sun yzsun@ccs.neu.edu

CS6220: DATA MINING TECHNIQUES Mining Time Series Data Instructor: Yizhou Sun yzsun@ccs.neu.edu November 12, 2013 Mining Time Series Data Basic Concepts Time Series Prediction and Forecasting Time Series Similarity Search

649 views • 45 slides
LFZip: Lossy compression of multivariate time series data via improved prediction Shubham

LFZip: Lossy compression of multivariate time series data via improved prediction Shubham

LFZip: Lossy compression of multivariate time series data via improved prediction Shubham Chandak Stanford University DCC 2020 Paper ID: 111 Joint work with Kedar Tatwawadi, Stanford Tsachy Weissman, Stanford Chengtao Wen, Siemens

256 views • 25 slides
Triples compression and Indexing Antonio Faria, Javier D. Fernndez and Miguel A.

Triples compression and Indexing Antonio Faria, Javier D. Fernndez and Miguel A.

Triples compression and Indexing Antonio Faria, Javier D. Fernndez and Miguel A. Martinez-Prieto 3rd KEYSTONE Training School Keyword search in Big Linked Data 23 TH AUGUST 2017 Agenda RDF management overview K 2 -Tree data

695 views • 50 slides
Time Series Analysis and Mining with R Time Series Decomposi- tion Time Series Forecasting

Time Series Analysis and Mining with R Time Series Decomposi- tion Time Series Forecasting

R and Time Series Data Time Series Analysis and Mining with R Time Series Decomposi- tion Time Series Forecasting Yanchang Zhao Time Series Clustering Time Series RDataMining.com Classification http://www.rdatamining.com/ R Functions

1.91k views • 42 slides
An Analysis of the Feasibility of Graph Compression Techniques for Indexing Regular Path Queries

An Analysis of the Feasibility of Graph Compression Techniques for Indexing Regular Path Queries

An Analysis of the Feasibility of Graph Compression Techniques for Indexing Regular Path Queries Frank Tetzel, Hannes Voigt, Marcus Paradies, Wolfgang Lehner May 19, 2017 1 Regular Path Queries (RPQs) Matching paths conforming to regular

755 views • 15 slides
FTW: Fast Similarity Search under the Time Warping Distance Yasushi Sakurai (NTT Cyber Space

FTW: Fast Similarity Search under the Time Warping Distance Yasushi Sakurai (NTT Cyber Space

FTW: Fast Similarity Search under the Time Warping Distance Yasushi Sakurai (NTT Cyber Space Labs) Masatoshi Yoshikawa (Nagoya Univ.) Christos Faloutsos (Carnegie Mellon Univ.) Motivation n Time-series data q many applications n computational

509 views • 47 slides
Analysis of hierarchical metric-tree indexing schemes for similarity search in high-dimensional

Analysis of hierarchical metric-tree indexing schemes for similarity search in high-dimensional

Analysis of hierarchical metric-tree indexing schemes for similarity search in high-dimensional datasets Vladimir Pestov vpest283@uottawa.ca http://aix1.uottawa.ca/ vpest283 Department of Mathematics and Statistics University of Ottawa

1.22k views • 106 slides
N = * log tfidf f i , k i , k df Similarity

N = * log tfidf f i , k i , k df Similarity

Outline An Overview of Limitations of Database systems (Motivation for Information Retrieval IR systems) Information Retrieval Indexing Nov. 10, 2009 Similarity Measures Evaluation Maryam Karimzadehgan Other IR

329 views • 16 slides
W HAT IS TIME SERIES D ATA ? W HAT IS TIME SERIES D ATA ? A value over time W HAT IS TIME

W HAT IS TIME SERIES D ATA ? W HAT IS TIME SERIES D ATA ? A value over time W HAT IS TIME

T IME S ERIES D ATA P APERS C OVERED Interactive Visualization of Serial Periodic Data John V. Carlis and Joseph A. Konstan Visualizing and Discovering Non-Trivial Patterns in Large Time Series Databases Jessica Lin, Eamonn Keogh,

764 views • 32 slides
Outline Time series and forecasting Time series objects 1 in R Basic time series functionality

Outline Time series and forecasting Time series objects 1 in R Basic time series functionality

Time series and forecasting in R 1 Time series and forecasting in R 2 Outline Time series and forecasting Time series objects 1 in R Basic time series functionality 2 The forecast package 3 Rob J Hyndman Exponential smoothing 4 ARIMA

714 views • 8 slides
Development of high rate MWPC and data compression function with FADC (II) Nguyen Minh Truong

Development of high rate MWPC and data compression function with FADC (II) Nguyen Minh Truong

Development of high rate MWPC and data compression function with FADC (II) Nguyen Minh Truong (Osaka U.), Coworker: Masaharu Aoki(Osaka U.), Youichi Igarashi(KEK), Masatoshi Saito(KEK), Hiroaki Natori (KEK), Nguyen Duy Thong (Osaka U.) Open-It,

310 views • 27 slides
Short Cambrex Corporation NYSE:CBM Stephen Saroki OBrien Greene & Co.

Short Cambrex Corporation NYSE:CBM Stephen Saroki OBrien Greene & Co.

Short Cambrex Corporation NYSE:CBM Stephen Saroki OBrien Greene & Co. ssaroki@obriengreene.com Sohn Investment Conference April 2018 Introduction Investment Thesis Snapshot 1. Cambrex is a smaller player in the highly

711 views • 26 slides
Guidelines for Preparation of Technical Paper Presentation at WATMAN International Conference 2020

Guidelines for Preparation of Technical Paper Presentation at WATMAN International Conference 2020

Guidelines for Preparation of Technical Paper Presentation at WATMAN International Conference 2020 ABSTRACT Each paper should be provided with an Abstract of about 150- 200 words, reporting concisely on the purpose and results of the paper.

250 views • 4 slides
ICLR PERC Fire webinar - PERC Introduction Michael Sznyi Flood Resilience Program Lead,

ICLR PERC Fire webinar - PERC Introduction Michael Sznyi Flood Resilience Program Lead,

ICLR PERC Fire webinar - PERC Introduction Michael Sznyi Flood Resilience Program Lead, Zurich Insurance Group ICLR PERC Fire Webinar, November 2019 INTERNAL USE ONLY The Alliance in words The Zurich Flood Resilience Alliance is a

774 views • 28 slides
Attribute-Value Reordering for Efficient Hybrid OLAP O WEN K ASER Dept. of Computer Science and

Attribute-Value Reordering for Efficient Hybrid OLAP O WEN K ASER Dept. of Computer Science and

DOLAP03, November 7, 2003. Attribute-Value Reordering for Efficient Hybrid OLAP O WEN K ASER Dept. of Computer Science and Applied Statistics University of New Brunswick, Saint John, NB Canada D ANIEL L EMIRE National Research Council of

713 views • 37 slides
JPEG Compression Ian Snyder December 11, 2009 Ian Snyder JPEG Compression Outline

JPEG Compression Ian Snyder December 11, 2009 Ian Snyder JPEG Compression Outline

Outline Introduction Images and Compression Walkthrough of JPEG Compression Steps Complete Compression Process Results and Conclusion JPEG Compression Ian Snyder December 11, 2009 Ian Snyder JPEG Compression Outline Introduction Images

594 views • 45 slides
Agenda Technical background ! Same-Origin Policy ! Compression-based attacks !

Agenda Technical background ! Same-Origin Policy ! Compression-based attacks !

H TTP ! E ncrypted ! I nformation can be ! S tolen through ! T CP-windows by ! Mathy Vanhoef & Tom Van Goethem Agenda Technical background ! Same-Origin Policy ! Compression-based attacks ! SSL/TLS & TCP ! Nitty gritty

804 views • 51 slides
A Novel Scalable IPv6 Lookup Scheme Using Compressed Pipelined Tries Michel Hanna, Sangyeun Cho,

A Novel Scalable IPv6 Lookup Scheme Using Compressed Pipelined Tries Michel Hanna, Sangyeun Cho,

A Novel Scalable IPv6 Lookup Scheme Using Compressed Pipelined Tries Michel Hanna, Sangyeun Cho, and Rami Melhem Computer Science Department, University of Pittsburgh, Pittsburgh, PA, 15260, USA { mhanna,cho,melhem } @cs.pitt.edu Abstract. An

482 views • 14 slides

More recommend