NOC Flash Introduction Terena TF NOC , Ljubjana Slovenia february - PowerPoint PPT Presentation

NOC Flash Introduction Terena TF NOC , Ljubjana Slovenia february 15- 16 2011 havard.kusslid@uninett.no NOC Manager

Network infrastructure Uninett operates its infrastructure on a 15 year long IRU (irrevokable right of use) agreement with Norwegian telco provider Ventelo on fibre or capacity. Upgrading at the cost of equipment after initial investment. New stretches often collaboration Ventelo are running the active DWDM equipment and servicing the fibre infrastructure. Uninett part-owning some local fibre infrastructure. 1 - 2,5, 10 Gbps network • 2 + 70 PoPs, 40 universities and university colleges • 200+ customer institutions, 250.000+ users. • +8000km of installed DWDM path (Amsterdam- • Rome is 1650km..) 3 separate paths north-south, latest addition lightpath • through Sweden- 10Gbps Narvik - Stockholm being delivered/ tested Planning for two 20Gbps (2X10Gbp) towards • NORDUnet from Oslo in the near future. Some backup paths still on SDH/155mbit. • Some smaller institutions still on xDSL solutions. • Challenges still on capacity on some stretches, and • with securing genuinely redundant access connections.

Hybrid network Lambda / sub lambda capabillities towards and between major universities  1Gbps, 2.5Gbps and 10Gbits per channel  40Gbps pr channel enabled.  100Gbps on the roadmap. 3 Equipment: Cisco & Juniper routers, high diversity in models. Nokia Siemens DWDM, Transmode DWDM/ CWDM equipment. Passive CWDM systems on some fibre acesses.

Services offered and maintained A quick search in our service catalogue shows that we have registered 126 services operated by UNINETT. This of cause includes internal services like internal backup- services, jabber, cfengine for configuration management etc. And not all our registered services are being monitored or «run» by the Uninett NOC. Our daughter company (UNINETT FAS) delivering and supporting common college administrative system from their own separate service NOC. Might end up under one "umbrella" (depending on an internal reorganizing process).. The UNINETT Cert is separate from the NOC, doing incident responce, IT security policy work / security revisions for our customers. 4 (Some of the) Services we support from the NOC: We act as «Help-desk» for the it-departments of our customers (the colleges mainly). We have little end-user contact, as they are mostly handled by local it organizations. But allso servicing some smaller govermental customers with less inhouse competence. IPv4/6 uni/multicast, delivery, troubleshooting  DNS/Hostmaster services  Registry services for .no  Inbound Email greylisting/ antivirus processing on central servers (Dike) before  delivering on to our customers. Running 30+ distributed masuring beacons, offering end-user speed-tests with  Network Diagnostic Tool -NDT, multicast connectivity, measuring one way delay, loss and jitter. Maintaining distributed "tool-boxes" delivering network tools for campus network  use : NAV (Proactive network management), Stager (netflow/ Qflow visualization), nfsen (netflow traffic analysis), hobit (service monitoring), tftp-server, syslog-server, radius-server

Tools to manage/ monitor the network: Main management tool is our home grown CMS system / inventory system (KIND): Contains Inventory data, circuits, services catalogue with (or without) SLA, customers, suppliers, operator instructions etc. Web frontend to database. 5

Tools to manage/ monitor the network, (cont.): Home grown/ open source monitoring tools : Zino is software package with functionality regarding network monitoring. The network status  monitor monitors status on links and routers, storing data used by the Ritz and Nemo clients. Ritz is a TCL/TK based application which present an overview of network failures. Ritz also works  as a ticket system, holding the administrative status of network failure events. 6 Web-based Network Maps: showing geographic and logic topology for UNINETT and the networks  of our customers. The maps contain information on router load, link capacity and more. The maps are interactive, and clicking on the wanted region, a more detailed map appear. Clicking on a router or a link shows menues with statistics. Pymetric - Homegrown internal routing visualizing tool. Traffic engineering and simulation tool  (metric adjustments, simulate effects of changes, outages, router failures). It was developed due to the fact that internal routing was dificult to follow, specially 2nd and 3rd paths. Hobbit/ Munin for service/ server monitoring.  http://software.uninett.no/

NOC organizing , Staffing, roles and responsibilities Centralized 24/7/365 NOC located at UNINETT in Trondheim The NOC operators are our network and systems experts, we have no 1.line helpdesk.  Most of our network (and systems of our systems) engineers participate as noc-operators  We have a rotating daytime staff (ca 20 persons) , 2 per shift.  Technichian-on-duty for outside office hours (weekly rotating)  Keep agreements - IE pay the four original Universities (oslo, bergen, trondheim, tromsø) to assist  hands-on our equipment in their locations. 7 How does the NOC work with different kinds of tools? (e.g. tool integration) We connect (link from) service monitor (Hobbit) to Customer DB to connect service with  customer data and service instructions. We need to be able to go from an event/ alarm to all service information / operator instructions and service history. Integration with knowledge-data / history log for service/ circuit from ticket-system into  CRM/Inventory is started. Early in development still We collect system and circuit alarms into a daily running log on an internal irc channel, and allso  collects the calls to the NOC, comments from operators and log messages from change management system. The irc log is used as a chat channel for the NOC and personell on other duties/ at their offices, and generates processed output for a daily event-log that is distributed internally. Tailored monitoring on NOC displays integrating different views in a combined view. Use  predefinedweb browser window

NOC front end What types of users are using your network and services? Our member institutions, Other NRENS and partners of our customers, The «public»  We need to/ try to document who the users of a service are in our service catalogue.  How does the SLAs or agreements you have with your customers look like? We have a simple "best effort" SLA for circuits/ networc connectivity. It refers to the «historic»  uptime statistics, typicly 99.9%, and refers to vendors SLA for the circuit. Makes no explicit guarantee and offers no penalties other than what can be deferred from the vendors SLA. 8 We have service agreement within UNINETT beetween the «owner» of the service and the NOC,  for NOC responsibillities . This agreement deals with procedural issues: Contact points, trouble shooting steps, responsible expert-level for escalation, service windows etc. We have a generic service statement for the NOC dealing with reachability/ escalation.  So far as SLA-statements exists between the owner of the service and a customer it needs to be  stated and operationalized in the NOC agreement for the service. Traditionally we have seen ourselves as a membership-organisation where best-effort have sufficed  Which tools are used to communicate with and keep track of users? RT for trouble ticketing, Email, Jabber, phone for case resolving Customer web portal (home grown/ integrated in CRM /KIND) for collecting information and displaying (a subset of) our information about the customer

Noc front end (cont.) How do your NOC communicate internally and with other groups in your organization? IM (Jabber, IRC), Email. Coordinating through department leader-team meetings. How does your NOC communicate with NOCs outside your organization? 9 (e.g. provider and/or customer NOCs) Mainly phone/ email Which tools are used to facilitate inter-NOC communication? Simple parsing of tickets (emails) from upstream (NORDUnet) and main infrastructure suppliers (Ventelo) about planned work/ issues - importing issue into calendar of planned work.

Documenting.. What information does your NOC document? Trouble ticketing  Event logs/ daily reports  Notifications of planned work/ outages to involved parties (internal, customer, supplier)  Updating information in CRM (KIND) to reflect day-to-day changes  Service documentation maintaining (the hands-on tips/ how-to's)  Which tools are used to create and update documentation? Template information for service documents in CRM/ Inventory 10 Wiki (Dokuwiki) and homemade CVS replication used to create the documentation. Exploring wsiwyg editors/ web forms for easier production/ maintainance of documentation Built-in functionallity forenforcing documentation updating in CRM (yearly revision/ signing off on status). Best Practice Documents available to share? Some «non-NOC-spesific» Practice Documents are available through Uninett's 'Campus Best Practices' work: physical infrastructure, audio-visual (AV), lightpath service, LAN infrastructure (including IPv6), wireless, Network monitoring, SIP and IP telephony, security and procurements. http://www.terena.org/activities/campus-bp/bpd.html Thank you havard.kusslid@uninett.no