introduction
play

Introduction Harsh realities of network analytics netbeam - PowerPoint PPT Presentation

Sep 29, 2023 •227 likes •568 views

Introduction Harsh realities of network analytics netbeam Demo Technology Stack Alternative Approaches Lessons Learned 2 ESnet Data, Analytics and Visualization Architecture 3 The Harsh Realities of Network

  2. Introduction ● Harsh realities of network analytics netbeam ● ● Demo ● Technology Stack ● Alternative Approaches Lessons Learned ● 2

  3. ESnet Data, Analytics and Visualization Architecture 3

  4. The Harsh Realities of Network Analytics 1. It’s a mess 3. There’s always more ● Your data isn’t neat and tidy ● More devices & more telemetry 2. Things change 4. It’s never really done ● Time and money are limited ● What you need today may not be what you need tomorrow. 4

  5. Coping strategies 1. It’s a mess 3. There’s always more ● Design knowing things won’t ● Rely on the cloud for scaling be tidy 2. Things change 4. It’s never really done ● “What” not “How” ● Keep raw data to keep your options open 5

  6. netbeam Network Analytics in Google Cloud Three Pillars 1. Real time analytics ○ Low latency, incomplete 2. Offline analytics ○ High latency, complete 3. Flexible data model ○ Changing needs? Recompute from raw data! Secret sauce: Apache Beam 6

  7. What is Apache Beam? 1. The Beam Programming Model SDKs for writing Beam pipelines 2. 3. Runners for existing distributed processing backends Apache Apex ○ ○ Apache Flink Apache Spark ○ ○ Google Cloud Dataflow Local runner for testing ○ 7 Slide courtesy of the Apache Beam Project

  8. The Evolution of Apache Beam Colossus BigTable PubSub Dremel Google Cloud Dataflow Spanner Megastore Millwheel Flume Apache Beam MapReduce 8 Slide courtesy of the Apache Beam Project

  9. SNMP collection Old SNMP system system Architecture Diagram Apache Beam (Stream Processing) avro Bigtable Apache Beam BigQuery BigQuery (realtime) (Batch Processing) (immutable) (historical) API Client 9

  10. SNMP collection Old SNMP system system Architecture Diagram Apache Beam (Stream) avro ● Google Pubsub ● Uses Python outside Align/rates of Google Cloud to poll devices and write Bigtable Rollups BigQuery BigQuery (realtime) to Pubsub topic 5m, 1h, 1d avg (immutable) (historical) ● Code within Google Cloud subscribes to Percentiles topic to process data ... API Client 10

  11. SNMP collection Old SNMP system system Architecture Diagram Apache Beam (Stream) avro ● Apache Beam / Google Dataflow Align/rates ● Stream processing ● Subscribes to Bigtable Rollups BigQuery BigQuery (realtime) Pubsub topic 5m, 1h, 1d avg (immutable) (historical) Percentiles ... API Client 11

  12. SNMP collection Old SNMP system system Architecture Diagram Apache Beam (Stream) avro ● Apache Beam / Google Dataflow Align/rates ● Stream processing ● Subscribes to Bigtable Rollups BigQuery BigQuery (realtime) Pubsub topic 5m, 1h, 1d avg (immutable) (historical) ● Raw data is written to BigQuery Percentiles ... API Client 12

  13. SNMP collection Old SNMP system system Architecture Diagram Apache Beam (Stream) avro ● Apache Beam / Google Dataflow Align/rates ● Stream processing ● Subscribes to Bigtable Rollups BigQuery BigQuery (realtime) Pubsub topic 5m, 1h, 1d avg (immutable) (historical) ● Raw data is written to BigQuery Percentiles ● Real time ... transformed data (e.g. aligned data rates) written to Bigtable API ● Writes and makes use of meta data in BigTable (not shown) Client 13

  14. SNMP collection Old SNMP system system Architecture Diagram Apache Beam (Stream) avro ● Cloud Bigtable ● Like HBase Align/rates ● Write to cells in rows, indexed by keys Bigtable Rollups BigQuery BigQuery (realtime) ● We write 1 day of 5m, 1h, 1d avg (immutable) (historical) data to a single row (columns are the time Percentiles of day, key is metric ... and day) ● Fast access to row by key, can serve data from here API ● Store one year Client 14

  15. SNMP collection Old SNMP system system Architecture Diagram Apache Beam (Stream) avro ● BigQuery ● Data warehousing Align/rates solution ● Cheap storage, SQL Bigtable Rollups BigQuery BigQuery (realtime) access, but not 5m, 1h, 1d avg (immutable) (historical) suitable for real-time access Percentiles ● Allows SQL queries ... for ad hoc investigation ● We store our source of truth here API Client 15

  16. SNMP collection Old SNMP system system Architecture Diagram Apache Beam (Stream) avro ● BigQuery ● Data warehousing Align/rates solution ● Cheap storage, SQL Bigtable Rollups BigQuery BigQuery (realtime) access, but not 5m, 1h, 1d avg (immutable) (historical) suitable for real-time access Percentiles ● Allows SQL queries ... for ad hoc investigation ● We store our source of truth here API ● Also store historical data (7 years), imported via avro files Client 16

  17. SNMP collection Old SNMP system system Architecture Diagram Apache Beam (Stream) avro ● Apache Beam / Google Dataflow Align/rates ● Batch processing ● Run with cron job Bigtable Rollups BigQuery BigQuery (realtime) 5m, 1h, 1d avg (immutable) (historical) Percentiles ... API Client 17

  18. SNMP collection Old SNMP system system Architecture Diagram Apache Beam (Stream) avro ● Apache Beam / Google Dataflow Align/rates ● Batch processing ● Run with cron job Bigtable Rollups BigQuery BigQuery (realtime) ● Recalculate Bigtable 5m, 1h, 1d avg (immutable) (historical) data each night from source of truth in Percentiles BigQuery ... API Client 18

  19. SNMP collection Old SNMP system system Architecture Diagram Apache Beam (Stream) avro ● Apache Beam / Google Dataflow Align/rates ● Batch processing ● Run with cron job Bigtable Rollups BigQuery BigQuery (realtime) ● Recalculate Bigtable 5m, 1h, 1d avg (immutable) (historical) data each night from source of truth in Percentiles BigQuery ... ● Process Bigtable rows into new rows of 5min, 1 hr and 1 day aggregations API Client 19

  20. SNMP collection Old SNMP system system Architecture Diagram Apache Beam (Stream) avro ● Apache Beam / Google Dataflow Align/rates ● Batch processing ● Run with cron job Bigtable Rollups BigQuery BigQuery (realtime) ● Recalculate Bigtable 5m, 1h, 1d avg (immutable) (historical) data each night from source of truth in Percentiles BigQuery ... ● Process Bigtable rows into new rows of 5min, 1 hr and 1 day aggregations API ● Additional pre-computed views e.g. percentiles for traffic distribution over a month Client 20

  21. SNMP collection Old SNMP system system Architecture Diagram Apache Beam (Stream) avro ● API ● Currently runs on Align/rates App Engine ● Node.js Bigtable Rollups BigQuery BigQuery (realtime) ● Serves data out of 5m, 1h, 1d avg (immutable) (historical) Bigtable ● Timeseries data is Percentiles served as ‘tiles’, each ... tile is one row ● Would like to use Cloud Endpoints Dataserver API and provide a gRPC (node.js) service ● Looking forward to grpc-web solution Client 21

  22. Use case example: Historical Trends 22

  23. Use case example: Historical Trends Per-day SNMP collection Interface totals BigQuery Stream to BQ system Bigtable Old SNMP system avro BigQuery (historical) Per-month totals Bigtable rows Dataserver API (node.js) Jan 1 Jan 2 ... Dec31 snmp-daily::2017-08::$interface 1.8 Pb 1.9 Pb ... 3.1 Pb Client Jan 1991 Feb 1991 ... Sep 2017 snmp-monthly-totals 23 28 Gb 29 Gb ... 56 Pb

  24. Use case: real time anomaly detection Generates avg for each interface over the past 3 months for that hour/day SNMP collection Baseline BigQuery Stream to BQ system generation Bigtable Anomaly detection Compares baseline to real time values to generate current deviation from normal Dataserver API (node.js) Mon Mon Mon Sun ... 12am 1am 2am 11pm baseline::5m::avg::$interface 2.1 1.9 0.3 ... 0.5 Client iface-1 iface-2 ... iface-n anomaly::5m::avg +0.1 +2.0 ... -1.5 24

  25. Use case example: Percentiles 25

  26. Use case example: Percentiles Daily rollups 5m avg SNMP collection Stream to Bigtable Bigtable system Percentiles Bigtable rows Dataserver API 1 2 ... 8640 (node.js) rollup-month-5m::2017-08::$interface::in 6Gbps 5Gbps ... 2Gbps Client 1 pct 2 pct ... 99 pct 26 percentiles::2017-08::$interface::in 0.1 Gbps 0.3 Gbps ... 22.1Gbps

  27. Example: Computing Total Traffic # Python Beam SDK pipeline = beam.Pipeline('DirectRunner') (pipeline | 'read' >> ReadFromText('./example.csv') | 'csv' >> beam.ParDo(FormatCSVDoFn()) | 'ifName key' >> beam.Map(group_by_device_interface) | 'group by iface' >> beam.GroupByKey() | 'compute rate' >> beam.FlatMap(compute_rate) | 'timestamp key' >> beam.Map(lambda row: (row['timestamp'], row['rateIn'])) | 'group by timestamp' >> beam.GroupByKey() | 'sum by timestamp' >> beam.Map(lambda rates: (rates[0], sum(rates[1]))) | 'format' >> beam.Map(lambda row: '{},{}'.format(row[0], row[1])) | 'save' >> beam.io.WriteToText('./total_by_timestamp')) pipeline.run() Full code available at: http://x1024.net/blog/2017/05/chinog-flexible-network-analytics-in-the-cloud/ 27

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION

INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION

TRAINING COURSE ON SOCIAL PROTECTION & FORMALIZATION TRINIDAD AND TOBAGO MARCH 15, 2017 INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION INTRODUCTION Design of the NIS Assistance from the

547 views • 23 slides
INTRODUCTION Introduction 2/42 INTRODUCTION Alternations I am giving bread to the

INTRODUCTION Introduction 2/42 INTRODUCTION Alternations I am giving bread to the

Patterns of variation in the expression of case and agreement Andrs Brny Leiden University Centre for Linguistics 9 November 2019, BaSIS Workshop a.barany@hum.leidenuniv.nl INTRODUCTION Introduction 2/42 INTRODUCTION

1.37k views • 42 slides
Introduction Introduction Introduction Introduction Outline Motivation Failures

Introduction Introduction Introduction Introduction Outline Motivation Failures

Introduction Introduction Introduction Introduction Outline Motivation Failures Definition and concepts Process and product properties Principles SE Approaches Motivation Software and the economy The economies of

1.03k views • 49 slides
Introduction Introduction Introduction Nationwide Cause for Concern 1

Introduction Introduction Introduction Nationwide Cause for Concern 1

Introduction Introduction Introduction Nationwide Cause for Concern 1 https://www.cdc.gov/std/stats17/infographic.htm Introduction Epidemiology in Western Colorado CPHE Surveillance Data Report, May 2018 Chlamydia Epidemiology in Western

977 views • 39 slides
DAQ introduction Purpose of this talk : (1) Introduction for those who have not been in every

DAQ introduction Purpose of this talk : (1) Introduction for those who have not been in every

DAQ introduction Purpose of this talk : (1) Introduction for those who have not been in every meeting (2) Some ideas of diagrams for section 7.1 (Introduction) of TP. Giles Barr 25 Jan 2018, Pembroke College On DUNE, the reality is there is a

474 views • 10 slides
INTRODUCTION cf. Schneider, Chapter 1 INTRODUCTION THEY ARE OUT TO GET YOU INTRODUCTION WHAT

INTRODUCTION cf. Schneider, Chapter 1 INTRODUCTION THEY ARE OUT TO GET YOU INTRODUCTION WHAT

ADVANCED COMPUTER SECURITY INTRODUCTION cf. Schneider, Chapter 1 INTRODUCTION THEY ARE OUT TO GET YOU INTRODUCTION WHAT IS SECURITY? A systems security policies describe What the system is supposed to do Store and provide access

479 views • 16 slides
Overview Introduction to SMIL Introduction to W3C and XML Introduction to SMIL

Overview Introduction to SMIL Introduction to W3C and XML Introduction to SMIL

Overview Introduction to SMIL Introduction to W3C and XML Introduction to SMIL Writing a SMIL file Adding Media Objects Martin Halvey Clipping media files January 2008 Timing and Synchronising Layout World Wide

387 views • 14 slides
14 Introduction Introduction Bad guys can put malware into Example: hosts

14 Introduction Introduction Bad guys can put malware into Example: hosts

Introduction Introduction source Encapsulation ISO/OSI reference model message M application segment H t H t M transport network datagram H n H n H t M presentation: allow applications to frame link H l H n H t M interpret

583 views • 4 slides
Previous Lecture Introduction to SMIL 2 Introduction to W3C and XML Introduction to SMIL

Previous Lecture Introduction to SMIL 2 Introduction to W3C and XML Introduction to SMIL

Previous Lecture Introduction to SMIL 2 Introduction to W3C and XML Introduction to SMIL Writing a SMIL file Adding Media Objects Martin Halvey Clipping media files January 2008 Timing and Synchronising Layout

322 views • 8 slides
Lecture 1 Andreas Habegger Introduction Zynq Introduction Zynq Introduction Zynq PS vs. PL

Lecture 1 Andreas Habegger Introduction Zynq Introduction Zynq Introduction Zynq PS vs. PL

Introduction Zynq Lecture 1 Andreas Habegger Introduction Zynq Introduction Zynq Introduction Zynq PS vs. PL Processing System Processor Peripherals Data Buses AXI Bus Conclusion BTE5380 - Embedded Systems Octobre 2014 Andreas Habegger

1.81k views • 30 slides
2018.06 01 SMILE5 Introduction S E 5 02 Alpha Cloud M I L 03 Company Introduction 04

2018.06 01 SMILE5 Introduction S E 5 02 Alpha Cloud M I L 03 Company Introduction 04

2018.06 01 SMILE5 Introduction S E 5 02 Alpha Cloud M I L 03 Company Introduction 04 Project References S E 5 PART.1 SMILE5 Introduction M I L Introduction STORION-SMILE5 Residential Energy Storage System Inverter 5kW Output Input

1.06k views • 25 slides
Introduction to CICS Course introduction Course introduction What is CICS? What is an

Introduction to CICS Course introduction Course introduction What is CICS? What is an

Introduction to CICS Course introduction Course introduction What is CICS? What is an application server? Why use an application server? Course introduction Services provided by CICS How CICS applications are defined Scaling CICS to meet

1.84k views • 146 slides
Network Visualization Introduction Presented by Shahed Introduction Introduction Basic

Network Visualization Introduction Presented by Shahed Introduction Introduction Basic

Network Visualization Introduction Presented by Shahed Introduction Introduction Basic building blocks Node Links (relationship between nodes) Spatial information Network data

584 views • 9 slides
Introduction Introduction (1) Main argument of the paper: universities are not only

Introduction Introduction (1) Main argument of the paper: universities are not only

Le jugement des pairs comme outil de management des universits Peer-review as a management tool Christine Musselin (CSO, Sciences Po et CNRS) Avril 2013, Lirrsistible ascension du capitalisme acadmique Introduction Introduction (1)

590 views • 28 slides
Introduction Introduction to storage and to storage and filesystems filesystems Introduction

Introduction Introduction to storage and to storage and filesystems filesystems Introduction

Moreno Baricevic Gilberto Daz Axel Kohlmeyer Stefano Cozzini ULA ICTP CNR-IOM DEMOCRITOS Merida, VENEZUELA Trieste, ITALY Trieste, ITALY Introduction Introduction to storage and to storage and filesystems filesystems Introduction

1.19k views • 51 slides
JABLOTRON JA-100 introduction November 2011 Introduction Todays goals First introduction of

JABLOTRON JA-100 introduction November 2011 Introduction Todays goals First introduction of

JABLOTRON JA-100 introduction November 2011 Introduction Todays goals First introduction of the brand new JA-100 alarm system Visions Basic architecture System components Setting Help us with final validations You are the FIRST ones

1.69k views • 111 slides
Learning: Nearest Neighbor, Perceptrons & Neural Nets Artificial Intelligence CSPP 56553

Learning: Nearest Neighbor, Perceptrons & Neural Nets Artificial Intelligence CSPP 56553

Learning: Nearest Neighbor, Perceptrons & Neural Nets Artificial Intelligence CSPP 56553 February 4, 2004 Nearest Neighbor Example II Credit Rating: Name L R G/P Classifier: Good / A 0 1.2 G Poor B 25

847 views • 49 slides
Pricing of Accreting Swaptions using QuantLib Dr. Andr Miemiec, 13./14. Nov. 2013 Agenda 1.

Pricing of Accreting Swaptions using QuantLib Dr. Andr Miemiec, 13./14. Nov. 2013 Agenda 1.

Pricing of Accreting Swaptions using QuantLib Dr. Andr Miemiec, 13./14. Nov. 2013 Agenda 1. Introduction 2. Model Description 3. Implementation in QuantLib 4. Pricing Quality Introduction Origin of the problem: Valuation of

761 views • 18 slides
Predicting Chroma from Luma using Frequency Domain Intra Prediction in Codecs Based on Lapped

Predicting Chroma from Luma using Frequency Domain Intra Prediction in Codecs Based on Lapped

Predicting Chroma from Luma using Frequency Domain Intra Prediction in Codecs Based on Lapped Transforms Nathan E. Egge Jean-Marc Valin Mozilla & The Xiph.Org Foundation Intra-Prediction of Chroma In 4:2:0 image data, chroma is 50% of

525 views • 21 slides
Reflecting on the Goal and Baseline of Exascale Computing Thomas C. Schulthess | T. Schulthess

Reflecting on the Goal and Baseline of Exascale Computing Thomas C. Schulthess | T. Schulthess

Reflecting on the Goal and Baseline of Exascale Computing Thomas C. Schulthess | T. Schulthess 1 Tracking supercomputer performance over time? Linpack benchmark solves: Ax = b | T. Schulthess 2 Tracking supercomputer performance

937 views • 78 slides
Pretty Good Democracy Peter Y A Ryan University of Luxembourg Vanessa Teague University of

Pretty Good Democracy Peter Y A Ryan University of Luxembourg Vanessa Teague University of

Pretty Good Democracy Peter Y A Ryan University of Luxembourg Vanessa Teague University of Melbourne Fribourg 6 September 2010 P Y A Ryan Pretty Good Democracy 1 Outline The challenge Pretty Good Democracy Threats

780 views • 26 slides
Software Testing Testing 1 Background Main objectives of a project: High Quality & High

Software Testing Testing 1 Background Main objectives of a project: High Quality & High

Software Testing Testing 1 Background Main objectives of a project: High Quality & High Productivity (Q&P) Quality has many dimensions reliability, maintainability, interoperability etc. Reliability is perhaps the most

1.46k views • 100 slides
NEPR208 - Adaptation properties and mechanisms Functional advantages in properties of a neural

NEPR208 - Adaptation properties and mechanisms Functional advantages in properties of a neural

NEPR208 - Adaptation properties and mechanisms Functional advantages in properties of a neural code and changes in those properties What is adaptation? Why do neural systems have a particular nonlinear and filter? Why does the nonlinearity and

931 views • 38 slides
Beba BEhavioural BAsed forwarding Giuseppe Bianchi OpenFlows platform The SDN/OpenFlow Model

Beba BEhavioural BAsed forwarding Giuseppe Bianchi OpenFlows platform The SDN/OpenFlow Model

Data Plane Programmability the next step in SDN Giuseppe Bianchi CNIT / University of Roma Tor Vergata Credits to: M. Bonola, A. Capone, C. Cascone, S. Pontarelli, D. Sanvito, M. Spaziani Brunella, V. Bruschi EU Support: Beba BEhavioural

797 views • 34 slides

More recommend