In-house management tools TF-NOC George Kargiotakis - - PowerPoint PPT Presentation

▶
in house management tools
SMART_READER_LITE
LIVE PREVIEW

In-house management tools TF-NOC George Kargiotakis - - PowerPoint PPT Presentation

Dec 05, 2023 195 likes •511 views

http://www.grnet.gr GRNET NOC In-house management tools TF-NOC George Kargiotakis (kargig@noc.grnet.gr) Andreas Polyrakis (apolyr@noc.grnet.gr) Leonidas Poulopoulos (leopoul@noc.grnet.gr) TF-NOC 2 GRNET NOC in-house management tools Why???

slide-1
SLIDE 1

http://www.grnet.gr

GRNET NOC In-house management tools

TF-NOC

George Kargiotakis (kargig@noc.grnet.gr) Andreas Polyrakis (apolyr@noc.grnet.gr) Leonidas Poulopoulos (leopoul@noc.grnet.gr)

slide-2
SLIDE 2

TF-NOC 2 GRNET NOC in-house management tools

slide-3
SLIDE 3

Why???

  • Automation
  • Time
  • PEBKACs
  • Share
  • “Where there an API there is a way”
  • No API ???
  • Try to “create” one

TF-NOC 3 GRNET NOC in-house management tools

slide-4
SLIDE 4

Tools

TF-NOC 4

  • MuPy
  • DCmon
  • Optical Network Mon
  • Servermon

GRNET NOC in-house management tools

slide-5
SLIDE 5

MUPY

TF-NOC 5 GRNET NOC in-house management tools

slide-6
SLIDE 6

TF-NOC 6 GRNET NOC in-house management tools

Why?

  • Munin: POWERFUL
  • Deployed in GRNET
  • Server monitoring

BUT

  • No AAI scheme
  • No info aggregation
  • No groupings
  • No saved queries

“Show all disk IOs graphs” (http://munin.ping.uio.no/) “Show disk graphs from gud and knuth”

slide-7
SLIDE 7

What?

TF-NOC 7 GRNET NOC in-house management tools

  • Parser
  • BeautifulSoup lib
  • Wrapper
  • Python Django
  • Saved queries
  • Open source https://code.grnet.gr/projects/mupy
  • Demo: http://mupy-demo.grnet.gr/
slide-8
SLIDE 8

DATACENTER POWER MONITORING

TF-NOC 8 GRNET NOC in-house management tools

slide-9
SLIDE 9

Datacenter Power-monitoring

TF-NOC 9 GRNET NOC in-house management tools

  • Datacenter Ministry of Education
  • Appliance
  • Collects data
  • Graphs
  • Weekly
  • FTP
  • CSV
slide-10
SLIDE 10

Why?

TF-NOC 10 GRNET NOC in-house management tools

  • Appliance web server is slooooowwww
  • Live queries cause load
  • No combination of metrics
  • No PUE calculation
  • No public web interface
  • But
  • Provides data in html table/csv
  • Every 5 mins
slide-11
SLIDE 11

How?

TF-NOC 11 GRNET NOC in-house management tools

  • HTML results parser
  • Store nodes/metrics in db
  • Store data in rrd files
  • Combine data to calculate PUE
  • Provide web interface
  • Beta internal
  • Source code soon available
slide-12
SLIDE 12

Sneak Peek-Dashboard

TF-NOC 12 GRNET NOC in-house management tools

slide-13
SLIDE 13

Sneak Peek-Nodes

TF-NOC 13 GRNET NOC in-house management tools

slide-14
SLIDE 14

Sneak Peek-Graphs

TF-NOC 14 GRNET NOC in-house management tools

slide-15
SLIDE 15

OPTICAL NETWORK MONITORING

TF-NOC 15 GRNET NOC in-house management tools

slide-16
SLIDE 16

Optical Network Monitoring

TF-NOC 16 GRNET NOC in-house management tools

  • Alcatel 1626LM, 1696MS
  • 3 months ago:
  • Alcatel NMS (PM module)
  • No attenuation (manual)
  • No triggers
  • Fiber Cuts
  • Manual filters(Alarm Surveillance)
  • Manual triggers

TIME TIME TIME TIME

slide-17
SLIDE 17

Alcatel tools

TF-NOC 17 GRNET NOC in-house management tools

  • Veeeryyy old UI
  • No graphing
slide-18
SLIDE 18

Optical NetAdmin… 3 months ago

TF-NOC 18 GRNET NOC in-house management tools

slide-19
SLIDE 19

Requirements

TF-NOC 19 GRNET NOC in-house management tools

  • Harvest
  • Element info
  • Performance data
  • Topology map
  • Graphs
  • Alerting
  • Visual, Mail
  • Open data
  • Web
slide-20
SLIDE 20

Architecture

TF-NOC 20 GRNET NOC in-house management tools

Optical Network Element Graphs IOO client RRD NMS GENOS IOO API Django Memcached Bootstrap UI RRD Analyser Topology Attenuation Attenuation Diff Fiber Cuts Element Info

slide-21
SLIDE 21

Optical Mon

TF-NOC 21 GRNET NOC in-house management tools

  • http://mon.grnet.gr (Optical Network Mon)
  • IOO Client: http://code.grnet.gr
  • Source code soon open
  • Open access
  • Frequent feature updates
slide-22
SLIDE 22

Anomaly detection

TF-NOC 22 GRNET NOC in-house management tools

  • On RRD files
  • Attenuation (diff of counters)
  • Iterate measurements
  • Pattern matching
  • If att==0 for x minutes then fiber_cut
  • If |att_diff|>a for x minutes then anomaly
  • More simple yet smart checks
  • Show visual alerts
  • Send mail to optnetadmins
slide-23
SLIDE 23

Anomaly detection improvements

TF-NOC 23 GRNET NOC in-house management tools

  • RRD Holt-Winters Time Series Forecasting
slide-24
SLIDE 24

Optical NetAdmin… now

TF-NOC 24 GRNET NOC in-house management tools

slide-25
SLIDE 25

SERVERMON

TF-NOC 25 GRNET NOC in-house management tools

slide-26
SLIDE 26

Sneak Peek

TF-NOC 26 GRNET NOC in-house management tools

slide-27
SLIDE 27

Sneak Peek

TF-NOC 27 GRNET NOC in-house management tools

slide-28
SLIDE 28

Sneak Peek

TF-NOC 28 GRNET NOC in-house management tools

slide-29
SLIDE 29

Sneak Peek

TF-NOC 29 GRNET NOC in-house management tools

slide-30
SLIDE 30

Source Code

  • https://code.grnet.gr/

TF-NOC 30 GRNET NOC in-house management tools

slide-31
SLIDE 31

Thank you

TF-NOC 31 GRNET NOC in-house management tools

Questions ?

https://twitter.com/grnetnoc https://www.facebook.com/noc.grnet.gr