Work Item C update: NOC tools, Work Item C update: NOC tools, p p - - PowerPoint PPT Presentation

Work Item C update: NOC tools, Work Item C update: NOC tools, p , p , interworking/interfacing issues, and interworking/interfacing issues, and automation automation

Maria Isabel Gandía Carriedo Communications Service Manager, CESCA 2nd TF-NOC meeting Technology Park Ljubljana, 16-02-2011

Agenda Agenda  Summary of the 1st TF-NOC meeting for WI C  Interesting topics: Answers related to tools  About multidomain tools: 3rd E2E provisioning workshop  Ideas for the survey

Summary of the 1st TF Summary of the 1st TF-

• NOC meeting for WI C

NOC meeting for WI C

C NetMRI h Ad-hoc scripts ScriptLogic Perspective

NOC tools, interworking/interfacing issues and NOC tools, interworking/interfacing issues and automation automation

Nagios MRTG Cacti ping IM MySQL dopplerVUe Ne NetMRI NetPrefect runch ScriptLogic Perspective

 A NOC uses a wide variety of tools to manage different

Nagios MRTG NDT tracero te Snort Twitter Sscan Just For Fun Network Monitoring System etGon NetPrefect NetC ass

types of information.  Th l f thi k it i t i ti t h t t l

NDT dbeacon Netflow traceroute Wireshark Radius Cyclops David system Just For Fun Network Monitoring System ng Ne NetQoS Performance Center SpyGla

 The goal of this work item is to investigate what tools are used and how they have been integrated into the NOC environment, focusing on the research and education

Wiki Ganglia Netflow MGEN Arbor Peakflow Zenoss Cyclops Axence nVision David system ipMonitor etInfo ion m S Orb

g community.

Cricket Request Tracker Zebra Arbor Peakflow FacebookCastle Rock N-visi arts ectrum SysO Cricket GoogleHobbit monitor BGPlay Nmap N Sma Spe

NOC tools NOC tools  http://www.slac.stanford.edu/xorg/nmtf/nmtf-tools.html Volunteer, unfunded effort built from a survey made by the ESnet Network Monitoring Task Force (NMTF) in 1995. New additions are from reader suggestions and recommendations Updated by Les Cottrell It contains

• recommendations. Updated by Les Cottrell. It contains

hundreds of tools.  http://globalnoc.iu.edu/grnoc-tool-set.html  http://www.debianhelp.co.uk/monitortools.htm  http://nms.gdd.net/index.php/Main_Page#NMS_Tools

The suggested workgroup (from ToR) The suggested workgroup (from ToR)  CESCA  DANTE  GRNET  IUCC  PIONIER  CARNet

NOC tools: Sorted topics (ToR number) NOC tools: Sorted topics (ToR number)

• 1. Survey of tools (1)

2 Monitoring tools (4)

• 2. Monitoring tools (4)
• 3. Reporting and statistics tools (5)

4 Performance management (6)

• 4. Performance management (6)
• 5. Ticketing systems (2)

6 Knowledge management (3)

• 6. Knowledge management (3)
• 7. Tool development (7)

Prioritisation, deliverables & timeline for Work Prioritisation, deliverables & timeline for Work Item C Item C

Sub‐ item Prio‐ rity 2nd TF‐NOC Jan‐2011 3rd TF‐NOC May‐2011 4th TF‐NOC Sep‐2011 5th TF‐NOC Jan‐2012 6th TF‐NOC May‐2012 7th TF‐NOC Sep‐2012 8th TF‐NOC Jan‐2013 9th TF‐NOC TNC 2013 1.a.1 1.1 Survey 1 a 2 1 1 Results (global) 1.a.2 1.1 Results (global) 1.b 1.2 Tools&functions 1.c 1.3 Taxonomy 4.a 2.1 Results (monitoring) 4.b 2.2 Deployment 4 c 2 3 SLA 4.c 2.3 SLA 5.a 3.1 Results (reporting) 5.b 3.2 What&who 5.c 3.3 Sensitive info 6.a 4.1 Results (performance) 6 b 4 2 Performance 6.b 4.2 Performance 2.a 5.1 Results (ticketing) 2.b 5.2 Ticket systems 3.a 6.1 Results (knowledge) 3.b 6.2 Sharing info 7 1 7 1 S d h 7.c.1 7.1 Survey ad‐hoc 7.c.2 7.1 Develop&acquire 7.d 7.2 DB integration 7.b 7.3 Report ad‐hoc 7.a 7.4 OS development T t l 1 6 2 3 2 3 2 2 Total: 1 6 2 3 2 3 2 2

Topics: 1 Survey of tools; 2 Ticketing systems; 3 Knowledge management; 4 Monitoring tools; 5 Reporting and statistics tools; 6 Performance management; 7 Tool development

Ideas for the survey about NOC tools Ideas for the survey about NOC tools a) NOC tools currently used by the TF-NOC participants b) Tools that have been tried out but did not work (for any reason) c) Tools that are planned to be inspected by NOCs d) T l d f ti th t i i d l d t t l d) Tools and functions that are missing and may lead to tool developments

Interesting topics Interesting topics  ACTION 4 on all (TF-NOC): to think about 5 questions that you want to know about other NOCs and send them to St f (NORDU t) Stefan (NORDUnet).  ACTION 5 on all (TF NOC): to think about how to introduce  ACTION 5 on all (TF-NOC): to think about how to introduce their NOC and send 5 important bullet points to Stefan (NORDUnet).

Interesting topics Interesting topics

Interesting Topics: Some Statistics Interesting Topics: Some Statistics  17 institutions gave an answer to action points 4 & 5 There were attendants from 16 different institutions in B l  Barcelona…  but there are more than 100 members on the list   82.35% of the institutions are explicitly interested in tools: 82.35% of the institutions are explicitly interested in tools:

• 70.59% are willing to learn
• 58.82% are willing to explain

 According to the answers we prefer learning than  According to the answers, we prefer learning than explaining what we do, but the amount of people willing to share information is quite good.  F tl i tit ti th t t t h i f ti  Frequently, an institution that wants to share information about a topic also wants to learn about the same topic  Some topics are quite generic.

The statistics graph The statistics graph

5 2 3 4 al s Learn 1 2 Explain Learn Tools in general Monitoring tools Integration of tools ledge manag... Ticketing systems

• rting and m...

ain tools: Dis... munication... Explain Learn In Knowle Tic SLA repor Multidomain Chat/commu

3 4 5 1 2

• 1. Tools in general

2 Monitoring tools

• ls in general

nitoring tools tion of tools manag... systems nd m... s: Dis... tion... Explain Learn

• 2. Monitoring tools
• 3. Multidomain tools

4 Ti k ti t

Tool Monit Integratio Knowledge ma Ticketing sy SLA reporting and Multidomain tools: Chat/communicatio

• 4. Ticketing systems
• 5. Knowledge management

S

• 6. SLA reporting and monitoring tools
• 7. Chat/communication/coordination tools
• 8. Integration of tools

• 1. Tools in general (I)
• 1. Tools in general (I)

Participants are interested in sharing information about… Participants are interested in getting input on… ARNES: NOC tools and automation ARNES: NOC tools and automation ARNES: NOC tools and automation (Integration of different tools on top of a common network database, Tools for SLA monitoring, SLA reporting, ARNES: NOC tools and automation (Integration of different tools on top

• f a common network database,

Tools for SLA monitoring, SLA for SLA monitoring, SLA reporting, Device authentication / authorisation management) Tools for SLA monitoring, SLA reporting, Device authentication / authorisation management) SWITCH: tools SWITCH: tools SWITCH: tools SWITCH: tools

• 1. Tools in general (II)
• 1. Tools in general (II)

Participants are interested in sharing information about… Participants are interested in getting input on… CESCA: NOC tools and how to adapt CESCA: NOC tools and how to adapt them to our needs (development). GNOC IU: Internally developed tools such our network database such our network database

[http://globalnoc.iu.edu/grnoc-tool-set.html]

PIONIER: NOC tools - tools used in th NOC i i f ti t l i i f ti the NOC, missing functions, tools developed and maintained by PSNC missing functions

• 2. Monitoring tools (I)
• 2. Monitoring tools (I)

Participants are interested in sharing information about… Participants are interested in getting input on… CARNET: NMS

• ur tool

CARNET: NMS tools CARNET: NMS - our tool functionalities / development CARNET: NMS - tools (functionalities) GRNET Vi li ti t l GRNET N t k it i t l GRNET: Visualisation tools GRNET: Network monitoring tools (may be different from SLA monitoring tools)

• 2. Monitoring tools(II)
• 2. Monitoring tools(II)

Participants are interested in sharing information about… Participants are interested in getting input on… DANTE: Main Monitoring systems and alarm thresholds RedIRIS: Monitoring tools. g FCCN: Which monitoring tools are you currently using. Which ones have you already used in the past and you already used in the past and you already used in the past and were abandoned. How do you keep network/topology maps updated? you already used in the past and were abandoned

• 3. Multidomain tools
• 3. Multidomain tools

Participants are interested in sharing information about… Participants are interested in getting input on… G OC GNOC IU: What methodologies and tools do you use to understand complex cross-network relationships and dependencies? and dependencies? SWITCH: tools to support the distributed handling of incidents

• 4. Ticketing systems
• 4. Ticketing systems

P ti i t i t t d i h i P ti i t i t t d i tti Participants are interested in sharing information about… Participants are interested in getting input on… NORDUNET: Troubleticketing system, SURFNET: ticketing systems g y experiences from building our own. g y CESCA: Ticketing Systems, how to

• ptimize their use.
• ptimize

experiences p

# 7823445

p

# 7823445

• 5. Knowledge management: Configuration
• 5. Knowledge management: Configuration

Management Database/Network Database Management Database/Network Database

Participants are interested in sharing information about Participants are interested in getting input on information about… input on… GNOC IU: Internally developed tools such our network database // / RedIRIS: Network database and integration with other monitoring [http://globalnoc.iu.edu/grnoc-tool- set.html] tools. RESTENA: Organization of inventory (tools, DB, naming conventions,etc.) and change management (technically and d i i t ti l ) administratively) SURFNET: CMDB experiences experiences

• 6. SLA reporting and monitoring tools
• 6. SLA reporting and monitoring tools

Participants are interested in sharing information about Participants are interested in getting input on information about… input on… ARNES: NOC tools and automation (Integration of different tools on top of ARNES: NOC tools and automation (Integration of different tools on top of a common network database, Tools for SLA monitoring, SLA reporting, Device authentication/authorisation management) a common network database, Tools for SLA monitoring, SLA reporting, Device authentication/authorisation management) management) management) GRNET: Automated SLA monitoring tools

• 7. Chat/communication/coordination tools (WI D)
• 7. Chat/communication/coordination tools (WI D)

Participants are interested in sharing information about Participants are interested in getting input on information about… input on… NORDUNET: Using external community tools (letting the letting the community answer questions instead

• f the NOC).

community answer questions

• 8. Integration of tools
• 8. Integration of tools

Participants are interested in sharing information about… Participants are interested in getting input on… ARNES: NOC tools and automation ARNES: NOC tools and automation ARNES: NOC tools and automation (Integration of different tools on top of a common network database, Tools for SLA monitoring, SLA reporting, ARNES: NOC tools and automation (Integration of different tools on top of a common network database, Tools for SLA monitoring, SLA reporting, g, p g, Device authentication/authorisation management). g, p g, Device authentication/authorisation management). CESCA: NOC tools and how to adapt CESCA: Integration of different tools. CESCA: NOC tools and how to adapt them to our needs (development). CESCA: Integration of different tools. BELNET: approach & tools: how the different departments & the NOC's PIONIER: data and tools integration - how? recommended tools? recommended different departments & the NOC s work together. how? recommended tools? RESTENA: An “all-in-one” NOC: t t / i ti /t l RedIRIS: Network database and i t ti ith th it i recommended structure/organization/tools. integration with other monitoring tools.

Ranking of interests Ranking of interests

Topics we’d like to learn about Monitoring tools (5) Topics we’d like to share information about Tools in general (5) g ( ) Integration of tools (4) Knowledge management (3) g ( ) Integration of tools (4) Monitoring tools (2) g g ( ) Tools in general (2) Ti k ti t (2) Monitoring tools (2) Knowledge management (1) Ti k ti t (1) Ticketing systems (2) SLA reporting and monitoring tools (2) Ticketing systems (1) SLA reporting and monitoring tools (1) (2) Multidomain tools (2) Chat/communication/coordination (1) Chat/communication/coordination tools(1) Chat/communication/coordination tools(0) Multidomain tools (0)

About multidomain tools: 3rd E2E About multidomain tools: 3rd E2E provisioning workshop provisioning workshop

About multidomain tools: 3rd E2E provisioning About multidomain tools: 3rd E2E provisioning workshop (Prague, December 2010) (I) workshop (Prague, December 2010) (I)  Some applications require high-capacity, low-latency and low-jitter end-to-end connection services. The major h ll i t bli hi th ti l t t th challenges in establishing these connections relate to the service architecture, including the need for standardized protocols and interfaces, on-demand tools, management p , , g solutions for accurate troubleshooting, as well as clear service provider roles and responsibilities.

About multidomain tools: 3rd E2E provisioning About multidomain tools: 3rd E2E provisioning workshop (II) workshop (II)

 Updates were given on:  Updates were given on:

• GÉANT tools and service architectures, including the

Bandwidth on Demand service based on tools like Bandwidth on Demand service based on tools like AutoBAHN and perfSONAR:

– Autobahn: Automatic Bandwidth Allocation across Heterogeneous Networks Heterogeneous Networks – perfSonar: Performance Service-Oriented Network Monitoring ARchitecture

I t t2’ ION i b d OSCARS/DRAGON

• Internet2’s ION service based on OSCARS/DRAGON

and perfSONAR:

– ION: Internet2 instantiation of a Dynamic Circuit Network y – OSCARS: On-demand Secure Circuits and Advance Reservation System – DRAGON: Dynamic Resource Allocation via GMPLS Optical DRAGON: Dynamic Resource Allocation via GMPLS Optical Networks

About multidomain tools: 3rd E2E provisioning About multidomain tools: 3rd E2E provisioning workshop (III) workshop (III)  The Mantychore and DYNES projects reported on the latest achievements in the area of provisioning virtual IP d d i t k t f d it and dynamic network systems for end sites.

• DYNES: Dynamic Network System
• Mantychore: from Manticore -> Making Articulated Private Networks

y g (APN) Network Topologies on Internet COREs

 Th t ti d l d ith th t ti f i  The presentations dealed with the automation of services, (network as a service for the users), but not with the distributed handling of incidents from the point of view of the NOC.

http://www.terena.org/activities/e2e/ws3/programme.html

About multidomain tools: 3rd E2E provisioning About multidomain tools: 3rd E2E provisioning workshop (IV) workshop (IV)  The lively discussions concluded with an agreement that it would be more productive to organise Birds of a Feather (B F) ti i ll l ith j t i d t (BoF) meetings in parallel with major events in order to intensively consult with various communities in 2011, instead of continuing the current workshop model of broad g p informational gatherings. -> TNC 2011  Ideally, application designers/developers and network administrators/operators i e the ‘user’ and the ‘service administrators/operators, i.e. the ‘user’ and the ‘service provider’, should come together to discuss and verify the E2E service architecture.  The results of these consultations will determine how TERENA should support the E2E community in the future.

Ideas for the survey Ideas for the survey

Prioritisation, deliverables & timeline for Work Prioritisation, deliverables & timeline for Work Item C Item C

Sub‐ item Prio‐ rity 2nd TF‐NOC Jan‐2011 3rd TF‐NOC May‐2011 4th TF‐NOC Sep‐2011 5th TF‐NOC Jan‐2012 6th TF‐NOC May‐2012 7th TF‐NOC Sep‐2012 8th TF‐NOC Jan‐2013 9th TF‐NOC TNC 2013 1.a.1 1.1 Survey 1 a 2 1 1 Results (global) 1.a.2 1.1 Results (global) 1.b 1.2 Tools&functions 1.c 1.3 Taxonomy 4.a 2.1 Results (monitoring) 4.b 2.2 Deployment 4 c 2 3 SLA 4.c 2.3 SLA 5.a 3.1 Results (reporting) 5.b 3.2 What&who 5.c 3.3 Sensitive info 6.a 4.1 Results (performance) 6 b 4 2 Performance 6.b 4.2 Performance 2.a 5.1 Results (ticketing) 2.b 5.2 Ticket systems 3.a 6.1 Results (knowledge) 3.b 6.2 Sharing info 7 1 7 1 S d h 7.c.1 7.1 Survey ad‐hoc 7.c.2 7.1 Develop&acquire 7.d 7.2 DB integration 7.b 7.3 Report ad‐hoc 7.a 7.4 OS development T t l 1 6 2 3 2 3 2 2 Total: 1 6 2 3 2 3 2 2

Topics: 1 Survey of tools; 2 Ticketing systems; 3 Knowledge management; 4 Monitoring tools; 5 Reporting and statistics tools; 6 Performance management; 7 Tool development

2nd TF 2nd TF-

• NOC, WI C, Priority 1: The survey

NOC, WI C, Priority 1: The survey  Topic: Survey of tools  Priority: 1  Subtopic: Collect information about which tools are used by the NOCs participating in TF-NOC  S t d d li bl  Suggested deliverables:

• Survey
• Report and statistics with the survey results

p y

Ideas for the survey about NOC tools Ideas for the survey about NOC tools a) NOC tools currently used by the TF-NOC participants b) Tools that have been tried out but did not work (for any reason) c) Tools that are planned to be inspected by NOCs d) T l /f ti th t i i d l d t t l d) Tools/functions that are missing and may lead to tool developments

you already used in the past and were abandoned letting the

• ptimize

experiences community answer questions

The aim is to have information about advantages or The aim is to have information about advantages or disadvantages of the tools that have been already tested disadvantages of the tools that have been already tested

missing functions recommended

disadvantages of the tools that have been already tested disadvantages of the tools that have been already tested by the community by the community

Ideas for the survey: please introduce the name of Ideas for the survey: please introduce the name of a tool and then answers the questions a tool and then answers the questions

Name Experience Rate* Improvements? Name I used to use it I use it I plan to use it 1 2 3 4 5 Desirable improvements

Tool Tool

Name Substitution? Comments Name I substituted it with… Why? The best The worst Other comments

Tool

* From 1(I wouldn’t recommend it) to 5 (It’s extremely useful)

Example Example

Name Experience Rate* Improvements? Name I used to use it I use it I plan to use it 1 2 3 4 5 Desirable improvements

MRTG

 

Threshold alarms MRTG

 

Threshold alarms

Name Substitution? Comments Name I substituted it with… Why? The best The worst Other comments

MRTG Cacti

• Different views

Easy to maintain & create scripts ith Not zoomable Easy to graph an 2

• Threshold alarms
• Able to integrate

different views create scripts with the text data files graph any 2 given values

* From 1(I wouldn’t recommend it) to 5 (It’s extremely useful)

A reference (I): A reference (I):

• 1. Monitoring tools:

a) Network Monitoring platforms (HP Openview, Pandora,…) b) General monitoring (MRTG, Cacti, Nagios, … with or without plugins)

a) Weather maps b) Thresholds and alarms

c) Infrastructure monitoring (Nagios, Ganglia, Zabbix,…) d) Diagnostic tools (ping, traceroute, mtr,..) ) g (p g, , , ) e) Flow monitoring (Netflow, cflow, sflow,… ) f) Syslog (logfile scanner, JFFNMS,…) g) Routing: route servers (zebra ) BGPmon looking glass g) Routing: route servers (zebra,…), BGPmon, looking-glass, …

A reference (II): A reference (II):

• 1. Monitoring tools(II):

h) Multicast monitoring tools: (dBeacon,…) i) Out-of-band access tools j) Network security k) Sniffing & analyzing (tcpdump, wireshark,…) ) g y g ( p p, , ) l) Topology documentation (Visio,…) m) Changes in network configurations (rancid,…) n) Control of configuration files (subversion ) n) Control of configuration files (subversion, …)

A reference (III): A reference (III):

• 2. Multidomain tools (perfSonar, Autobahn,…)

a) Do you use any tool to support the distributed handling of i id t ? Whi h ? incidents? Which one?

• 3. Reporting and statistics tools (Nagios, Zabbix,..)

4 Ticketing tools (RT Trac Buzilla )

• 4. Ticketing tools (RT, Trac, Buzilla,...)

For the comments:  Do the users have access to the tickets?  Are the tickets publicly available?

• 5. Performance management

a) Performance testing (IPerf, NDT,…) a) Performance testing (IPerf, NDT,…) b) Traffic generators (Bulk, Mgen,…)

A reference (IV): A reference (IV):

• 6. Chat/communication/coordination tools: IM, mailing lists,

skype,...

• 7. Databases: (Mysql, exchange,...)

For the comments:  Usage: What information do you store on them?  Usage: What information do you store on them?  Integration: How do you connect them to your management tools?

• 8. Knowledge management/documentation: (Plone, Wiki, ...)

a) What kind of information do you store? b) How do you structure it?

Thanks for your attention! Thanks for your attention! Questions? Suggestions? Questions? Suggestions? Questions? Suggestions? Questions? Suggestions?

i di @ t igandia@cesca.cat

L t’ k it! L t’ k it! Let’s work on it! Let’s work on it!