cassandra for time series data
play

- Cassandra for Time Series Data - Joris Gillis, June 28, 2017 1 - PowerPoint PPT Presentation

Dec 08, 2022 •130 likes •597 views

- Cassandra for Time Series Data - Joris Gillis, June 28, 2017 1 Joris Gillis I am a software engineer at TrendMiner and focus on the enterprise scalability of our industrial analytics platform. I studied at the University of Hasselt and the

  1. - Cassandra for Time Series Data - Joris Gillis, June 28, 2017 1

  2. Joris Gillis I am a software engineer at TrendMiner and focus on the enterprise scalability of our industrial analytics platform. I studied at the University of Hasselt and the University of Antwerp in the field of Database theory and Data Mining. My interests are: • Big Data technology • Functional Programming • Athletics 2

  3. Agenda 1. Introduction 2. Why Cassandra? 3. How to model time series in Cassandra? 4. How to configure Cassandra for Time Series Data? 5. Q&A 3

  4. About TrendMiner TrendMiner is the Leading Modelling Free Industrial Analytics Platform to Analyze, Monitor and Predict Asset and Process Performance. With a proven track record in the (Petro-) Chemical and Oil & Gas industry to increase overall profitability, by improving production yield, lower costs, avoid unplanned process downtime, increase overall equipment efficiency and reduce safety risks. 4

  5. About TrendMiner About our company • Started in 2008 as Spin-off K.U.Leuven • > 70 Man Year research behind TrendMiner • Spin-out Idea from Bayer MaterialScience (Covestro) • Patented several core technologies for US/EU • Headquarter EMEA, Hasselt, Belgium • Headquarter US, Houston, TX • 60+ Employees and growing • Global OSIsoft PI ISV & OEM partner • Platform Agnostic vendor • Front runner in both Process & Asset Analytics 5

  6. Industry 4.0 Internet of Things Augmented Reality Wearables CONNECTIVITY Cyber Security Machine Learning Optimization & Prediction BIG DATA & ANALYTICS Additive Manufacturing Advanced Materials Autonomous Robotics ADVANCED MANUFACTURING Technologies that enable new ways of working and of doing business 6

  7. About TrendMiner About our software 7

  8. About TrendMiner Analyze 8

  9. About TrendMiner Monitor 9

  10. About TrendMiner Predict 10

  11. Problem statement Time series Complex analyses Plants across the • From thousands to millions globe • Resolution between 5 minutes and 1 second • E.g., 10 year history 11

  12. What is a Time Series? • A time series is a series of timestamped data points • Sometimes data points are spaced equidistantly • List<Tuple<Long, Float �>? 12

  13. Agenda 1. Introduction 2. Why Cassandra? 3. How to model time series in Cassandra? 4. How to configure Cassandra for Time Series Data? 5. Q&A 13

  14. Why Cassandra New technology Horizontal scaling Big Data => Big Index In-store analytics too limited for our needs Only HTTP interface

  15. Why Cassandra • DISADVANTAGES • ADVANTAGES • Overhead vs custom optimised format • Proven technology • No time series specific optimisations • Connectivity • E.g., Gorilla/Beringei • JDBC connector (also to Spark) • Delta-delta encoding for semi- • Edge locations equidistant points • Geographic distribution of data • Delta encoding for stable values • Support for storing and querying time series data • E.g., KairosDB uses Cassandra as underlying store

  16. Agenda 1. Introduction 2. Why Cassandra? 3. How to model time series in Cassandra? 4. How to configure Cassandra for Time Series Data? 5. Q&A 16

  17. How to model time series in Cassandra? Keys • Primary key • One or more columns identifying a row • PRIMARY KEY (A) • PRIMARY KEY (A, B) • Compound primary key • Partition key • First column(s) of primary key • E.g., PRIMARY KEY ((A, B), C) • A & B are composite partition key 17

  18. How to model time series in Cassandra? Partitioning & Clustering • A partition is mapped to a Cassandra node • All rows with same partition key on same node(s) • Clustering columns • Part of compound primary key • Define sorting inside partition Map<byte[], SortedMap<Clustering, Row  Partition Key Clustering columns Other columns 18

  19. How to model time series in Cassandra? Modelling: Simple CREATE TABLE temperature ( weatherstation_id uuid, event_time timestamp, temperature float, PRIMARY KEY (weatherstation_id, event_time) ); https://academy.datastax.com/resources/getting-started-time-series-data-modeling 19

  20. How to model time series in Cassandra? Modelling: Simple • Advantage SELECT temperature FROM temperature • Easy to understand WHERE weatherstation_id='1234ABCD' AND event_time > '2013-04-03 07:01:00' AND event_time < '2013-04-03 07:04:00'; • Simple to query • Disadvantage • All data for one time series in one partition • Max 2 billion rows per partition 20

  21. How to model time series in Cassandra? Modelling: Partitioned CREATE TABLE temperature_by_day ( weatherstation_id uuid, day date, event_time timestamp, temperature float, PRIMARY KEY ((weatherstation_id, date), event_time) ); https://academy.datastax.com/resources/getting-started-time-series-data-modeling 21

  22. How to model time series in Cassandra? Modelling: Partitioned • Advantage • Virtually no storage limitation • Disadvantage • Crossing bucket boundary => multiple queries • Need to specify id and day; otherwise unpredictable performance • If data comes in burst => uneven partition sizes 22

  23. Agenda 1. Introduction 2. Why Cassandra? 3. How to model time series in Cassandra? 4. How to configure Cassandra for Time Series Data? 5. Q&A 23

  24. How to configure Cassandra for Time Series Data? • Cassandra v3 • Storage engine refactored compared to v2 • Options to influence read and write performance • Compression • Compaction 24

  25. How to configure Cassandra for Time Series Data? How Cassandra Writes Data MemTable • Commit log row1 Write row2 Data row3 • Durability Memory Flush • Memtable Disk • Cache writes in memory Lorem ipsum dolor sit row1 amet, consectetur row1 adipiscing elit, sed do row1 Index row2 eiusmod tempor • Regularly flushed to disc Index row2 incididunt ut labore et Index row2 dolore magna aliqua. row3 Ut enim ad minim row3 veniam, quis nostrud row3 • Sorted Strings Table (SSTable) Commit Log SSTables • Compaction Compaction • Re-organise http://docs.datastax.com/en/cassandra/3.0/cassandra/dml/dmlHowDataWritten.html • Cleanup 25

  26. How to configure Cassandra for Time Series Data? How Cassandra Maintains Data • SSTable = immutable • Updates => Timestamped version of row • Deletes => Tombstones • Clean up old versions and tombstones • Compaction 26

  27. How to configure Cassandra for Time Series Data? How Cassandra Reads Data (Simplified) • How to get relevant data • SSTables • Partition key => node(s) • Bloom filter Partition Key • Returns list of SSTables that might contain rows for query • Partition index on SSTable • Keeps o ff set for each partition key row1 row1 Bloom row1 Index row2 • Memtable Index row2 Index row2 row3 Filter row3 row3 • Extract qualifying rows • Resolve • Timestamped rows • Tombstones 27

  28. How to configure Cassandra for Time Series Data? Compaction options • Size tiered compaction (default) • Levelled compaction • Time window compaction 28

  29. How to configure Cassandra for Time Series Data? Size tiered Compaction • Default strategy • Optimised for write heavy workloads • Compaction • When # similarly sized SSTables • Merge into one new file 29

  30. How to configure Cassandra for Time Series Data? Size tiered Compaction Example Total size: T4 T1 T3 T2 625MB 150MB 155MB 155MB 165MB Compaction T5 600MB 30

  31. How to configure Cassandra for Time Series Data? Size tiered Compaction Example T6 T7 155MB 165MB T5 600MB 31

  32. How to configure Cassandra for Time Series Data? Size tiered Compaction • Advantage • Write optimised • Disadvantage • Rows of a partition are spread across multiple SSTables • Holds on to stale data for a long time • A lot of memory needed as SSTables grow in size 32

  33. How to configure Cassandra for Time Series Data? Levelled Compaction • L(0) • Flushes from memtable • L(N > 0) • Fixed size SSTables (default: 160MB) • Each SSTable has range of partitions => NO OVERLAP! • L(1) holds at most 10 SSTables • L(N+1) can hold 10x more SSTables than L(N) 33

  34. How to configure Cassandra for Time Series Data? Levelled Compaction Example 2 rows per SSTable 2 files in L(1) 20 files in L(2) Compaction Row Partition 0 2 -1 1 r1 #0 r2 #2 L(0) r3 r4 r1 r2 r3 #-1 r4 #1 1 2 -1 0 r5 #-2 L(1) r6 #6 r4 r2 r3 r1 r7 #3 r8 #6 r9 #-1 L(2) r10 #-2 34

  35. How to configure Cassandra for Time Series Data? Example ctd. 2 rows per SSTable 2 files in L(1) 20 files in L(2) Compaction Row Partition 3 6 -2 6 r1 #0 r2 #2 L(0) r5 r6 r7 r8 r3 #-1 r4 #1 1 2 -2 -1 0 1 2 3 6 6 -1 0 r5 #-2 L(1) r6 #6 r4 r2 r5 r3 r1 r4 r2 r7 r6 r8 r3 r1 r7 #3 r8 #6 r9 #-1 L(2) r10 #-2 35

  36. How to configure Cassandra for Time Series Data? Example ctd. 2 rows per SSTable 2 files in L(1) 20 files in L(2) Compaction Row Partition r1 #0 L(0) r2 #2 r3 #-1 -2 -1 0 1 r4 #1 r5 r3 r1 r4 r5 #-2 r6 #6 L(1) 2 3 6 6 2 3 6 6 r7 #3 r2 r7 r6 r8 r2 r7 r6 r8 r8 #6 r9 #-1 -2 -1 0 1 r10 #-2 L(2) r5 r3 r1 r4 36

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Time Series Representations for Better Data Mining What can we do with time series data?

Time Series Representations for Better Data Mining What can we do with time series data?

Time Series Representations for Better Data Mining What can we do with time series data? Clustering Anomaly (outlier) detection Forecasting What are the problems with time series data? Noise Concept-drift (trend-shift

122 views • 11 slides
Why do you care? Time-series data is all over the place. Time-Series Data Kaitlin Duck

Why do you care? Time-series data is all over the place. Time-Series Data Kaitlin Duck

Why do you care? Time-series data is all over the place. Time-Series Data Kaitlin Duck Sherwood CS 533c What is Time-Series Data? What is Time-Series Data? Lines. Usually periodic Source: van Wijk and van Selow, Cluster and

85 views • 4 slides
Introduction to Time Series Heino Bohn Nielsen 1 of 15 Outline (1) What is a time series? (2)

Introduction to Time Series Heino Bohn Nielsen 1 of 15 Outline (1) What is a time series? (2)

Econometrics 2 Fall 2005 Introduction to Time Series Heino Bohn Nielsen 1 of 15 Outline (1) What is a time series? (2) Important characteristics of time series data. (3) Data sampling and stochastic processes. Cross-sectional data vs. time

374 views • 8 slides
Time Series Data Visualization Class 2, Part A 2 1 Time Series Data Fundamental

Time Series Data Visualization Class 2, Part A 2 1 Time Series Data Fundamental

Large Scale Information Visualization Jing Yang Fall 2007 1 Time Series Data Visualization Class 2, Part A 2 1 Time Series Data Fundamental chronological component to the data set Random sample of 4000 graphics from 15 of

763 views • 31 slides
Plotting time - series data IN TR OD U C TION TO DATA VISU AL IZATION W ITH MATP L OTL IB

Plotting time - series data IN TR OD U C TION TO DATA VISU AL IZATION W ITH MATP L OTL IB

Plotting time - series data IN TR OD U C TION TO DATA VISU AL IZATION W ITH MATP L OTL IB Ariel Rokem Data Scientist Time - series data INTRODUCTION TO DATA VISUALIZATION WITH MATPLOTLIB Climate change time - series date,co2,relative_temp

627 views • 27 slides
Time Series Data Visualization 2 1 Time Series Data Fundamental chronological component to

Time Series Data Visualization 2 1 Time Series Data Fundamental chronological component to

Information Visualization Jing Yang Spring 2007 1 Time Series Data Visualization 2 1 Time Series Data Fundamental chronological component to the data set Random sample of 4000 graphics from 15 of worlds newspapers and magazines

315 views • 31 slides
Time Series Analysis and Mining with R Time Series Decomposi- tion Time Series Forecasting

Time Series Analysis and Mining with R Time Series Decomposi- tion Time Series Forecasting

R and Time Series Data Time Series Analysis and Mining with R Time Series Decomposi- tion Time Series Forecasting Yanchang Zhao Time Series Clustering Time Series RDataMining.com Classification http://www.rdatamining.com/ R Functions

1.91k views • 42 slides
Data at the Speed of your Users Apache Cassandra and Spark for simple, distributed, near real-time

Data at the Speed of your Users Apache Cassandra and Spark for simple, distributed, near real-time

Data at the Speed of your Users Apache Cassandra and Spark for simple, distributed, near real-time stream processing. GOTO Copenhagen 2014 Rustam Aliyev Solution Architect at . @rstml Big Data? Photo:

675 views • 49 slides
Timeseries kinds and applications MAC H IN E L E AR N IN G FOR TIME SE R IE S DATA IN P YTH

Timeseries kinds and applications MAC H IN E L E AR N IN G FOR TIME SE R IE S DATA IN P YTH

Timeseries kinds and applications MAC H IN E L E AR N IN G FOR TIME SE R IE S DATA IN P YTH ON Chris Holdgraf Fello w, Berkele y Instit u te for Data Science Time Series MACHINE LEARNING FOR TIME SERIES DATA IN PYTHON Time Series

802 views • 37 slides
What the heck is time-series data (and why do I need a time-series database?) Ajay Kulkarni |

What the heck is time-series data (and why do I need a time-series database?) Ajay Kulkarni |

What the heck is time-series data (and why do I need a time-series database?) Ajay Kulkarni | Co-founder/CEO | ajay@timescale.com Fastest growing database category Source: DB Engines In this talk 1. What is time-series data? (hint: its

540 views • 36 slides
W HAT IS TIME SERIES D ATA ? W HAT IS TIME SERIES D ATA ? A value over time W HAT IS TIME

W HAT IS TIME SERIES D ATA ? W HAT IS TIME SERIES D ATA ? A value over time W HAT IS TIME

T IME S ERIES D ATA P APERS C OVERED Interactive Visualization of Serial Periodic Data John V. Carlis and Joseph A. Konstan Visualizing and Discovering Non-Trivial Patterns in Large Time Series Databases Jessica Lin, Eamonn Keogh,

764 views • 32 slides
CS6220: DATA MINING TECHNIQUES Mining Time Series Data Instructor: Yizhou Sun yzsun@ccs.neu.edu

CS6220: DATA MINING TECHNIQUES Mining Time Series Data Instructor: Yizhou Sun yzsun@ccs.neu.edu

CS6220: DATA MINING TECHNIQUES Mining Time Series Data Instructor: Yizhou Sun yzsun@ccs.neu.edu November 12, 2013 Mining Time Series Data Basic Concepts Time Series Prediction and Forecasting Time Series Similarity Search

649 views • 45 slides
Working w ith more than one time series VISU AL IZIN G TIME SE R IE S DATA IN P YTH ON Thomas

Working w ith more than one time series VISU AL IZIN G TIME SE R IE S DATA IN P YTH ON Thomas

Working w ith more than one time series VISU AL IZIN G TIME SE R IE S DATA IN P YTH ON Thomas Vincent Head of Data Science , Ge y Images Working w ith m u ltiple time series An isolated time series A le w ith m u ltiple time series

505 views • 29 slides
Mining Approximate Top-K Subspace Anomalies in Multi-Dimensional Time-Series Data Xiaolei Li,

Mining Approximate Top-K Subspace Anomalies in Multi-Dimensional Time-Series Data Xiaolei Li,

Mining Approximate Top-K Subspace Anomalies in Multi-Dimensional Time-Series Data Xiaolei Li, Jiawei Han University of Illinois at Urbana-Champaign VLDB 2007 1 Time Series Data Many applications produce time series data Intel stock 2

888 views • 60 slides
Apache Cassandra for Big Data Applications Christof Roduner Java User Group Switzerland COO and

Apache Cassandra for Big Data Applications Christof Roduner Java User Group Switzerland COO and

Apache Cassandra for Big Data Applications Christof Roduner Java User Group Switzerland COO and co-founder January 7, 2014 christof@scandit.com AGENDA 2 Cassandra origins and use How we use Cassandra Data model and query language

535 views • 49 slides
Introduction to Time Series Data and Analysis Simon Taylor Department of Mathematics and

Introduction to Time Series Data and Analysis Simon Taylor Department of Mathematics and

Introduction to Time Series Data and Analysis Simon Taylor Department of Mathematics and Statistics 20 April 2016 Contents What is Time Series Data? Analysis Tools Trace Plot Auto-Correlation Function Spectrum Time Series Models Moving

876 views • 18 slides
The Dark Matter density MW Components Global density Data: inner Data: outer Data: masers

The Dark Matter density MW Components Global density Data: inner Data: outer Data: masers

The Dark Matter density F. Nesti Problem The Dark Matter density MW Components Global density Data: inner Data: outer Data: masers Fabrizio Nesti Fits Annihilation Local density Universit` a dellAquila, Italy Method Data: Sun

428 views • 41 slides
OPEN-MIDPLANE DIPOLES FOR A MUON COLLIDER * R. Weggel # , J. Kolonko &amp; R. Scanlan, Particle

OPEN-MIDPLANE DIPOLES FOR A MUON COLLIDER * R. Weggel # , J. Kolonko &amp; R. Scanlan, Particle

Proceedings of 2011 Particle Accelerator Conference, New York, NY, USA TUP177 OPEN-MIDPLANE DIPOLES FOR A MUON COLLIDER * R. Weggel # , J. Kolonko &amp; R. Scanlan, Particle Beam Lasers, Northridge, CA D. Cline &amp; X. Ding, UCLA, Los Angeles, CA

247 views • 3 slides
Big Data Processing with Apache Spark Jay Urbain, PhD Credits: Resilient Distributed Datasets

Big Data Processing with Apache Spark Jay Urbain, PhD Credits: Resilient Distributed Datasets

Big Data Processing with Apache Spark Jay Urbain, PhD Credits: Resilient Distributed Datasets Resilient Distributed Datasets A Fault-T A Fault-Tolerant Abstraction for In-Memory Cluster Computing olerant Abstraction for In-Memory Cluster

1.32k views • 67 slides
Scripted Components Massimo Felici and Conrad Hughes mfelici@inf.ed.ac.uk conrad.hughes@ed.ac.uk

Scripted Components Massimo Felici and Conrad Hughes mfelici@inf.ed.ac.uk conrad.hughes@ed.ac.uk

Scripted Components Massimo Felici and Conrad Hughes mfelici@inf.ed.ac.uk conrad.hughes@ed.ac.uk http://www.inf.ed.ac.uk/teaching/courses/sapm/ Original slides: Dr James A. Bednar SAPM Spring 2008: Scripted Components 1 Spot the connection

416 views • 25 slides
Community Forums Receiving Input from Parents, Families and Staff School Location: Natomas

Community Forums Receiving Input from Parents, Families and Staff School Location: Natomas

Community Forums Receiving Input from Parents, Families and Staff School Location: Natomas Gateways Middle School 1 Objectives for this Forum Further share with our community the potential for facilities overcrowding (as early as the 2015-

407 views • 19 slides
CITIES, HEALTH AND WELL-BEING NOVEMBER 2011 Urban density, overcrowding and health Promoting High

CITIES, HEALTH AND WELL-BEING NOVEMBER 2011 Urban density, overcrowding and health Promoting High

CITIES, HEALTH AND WELL-BEING NOVEMBER 2011 Urban density, overcrowding and health Promoting High density while avoiding overcrowding Urban Age Conference, Hong Kong 16-17 November 2011 Cities, Health and Well-being Gora Mboup, Ph.D. Sr.

450 views • 22 slides
2 Corinthians 10:3, For though we walk in the flesh, we do not war according to the flesh. 2

2 Corinthians 10:3, For though we walk in the flesh, we do not war according to the flesh. 2

2 Corinthians 10:3, For though we walk in the flesh, we do not war according to the flesh. 2 Corinthians 10:4, For the weapons of our warfare are not carnal but mighty in God for pulling down strongholds, 2 Corinthians 10:5, casting

309 views • 26 slides
Inform Decision-Making in Business Polly Moseley PhD Researcher &amp; Producer Liverpool John

Inform Decision-Making in Business Polly Moseley PhD Researcher &amp; Producer Liverpool John

Operationalising Natural Capital to Inform Decision-Making in Business Polly Moseley PhD Researcher &amp; Producer Liverpool John Moores University &amp; Cambridge Institute of Sustainability Leadership Rese search arch Ar Area ea:

48 views • 4 slides

More recommend