DANTE Visits to NOCs - 2012 TF-NOC, Poznan 13 December 2012 - - PowerPoint PPT Presentation

Creating the global research village

Serving European NREN’s

DANTE Visits to NOCs - 2012

TF-NOC, Poznan 13 December 2012

Creating the global research village 2 2 2

Serving European NREN’s

Agenda

Purpose Scope Key areas of interest Completed Changes Future Plans

Creating the global research village 3 3 3

Serving European NREN’s

Why?

DANTE NOC 3 years old Challenge assumptions and avoid stagnation Funding under increasing pressure Find new efficiencies and cost savings Increase NOC’s usefulness Identify and offer new services

Creating the global research village 4 4 4

Serving European NREN’s

Clarification

Benchmarking “… used to measure performance using a specific indicator (cost per unit of measure, productivity per unit of measure, cycle time of x per unit of measure or defects per unit of measure) resulting in a metric of performance that is then compared to others.” Our work not so specific or quantitative More “Fact Finding”

Creating the global research village 5 5 5

Serving European NREN’s

Scope

NOCs Visited NORDUnet (Dec 2011) SWITCH (Jan 2012) PSNC (Jan 2012) Indiana University GNOC (Jul 2012) Selection Criteria R&E Networks International/continental footprint Optical transmission and IP network

Creating the global research village 6 6 6

Serving European NREN’s

Key Areas of Interest

NOC (Operations) Organization Out of Hours Activity Percentage of incidents Level of Coverage Processes and Standards Key Performance Indicators (KPIs) & Service Level Agreements (SLAs) Additional Services

Creating the global research village 7 7 7

Serving European NREN’s

NOC Organization

Numbers and types of operations staff

Org Teams Office Hours DANTE Service Desk, NCC (8), Specialists (12) SD: 24/7 NCC: 0600-2100 Mon-Fri Specialists: “9 to 5” Indiana U NOC Techs (23), Engineering (25) NOC: 24/7 Engineering: “9 to 5” NORDUnet NOC (7), Specialists (11) NOC: 0700-1900 Specialists: “9 to 5” SWITCH Global LAN (9) “9 to 5” PSNC Operators (6) NOC (14) Specialist (20) Operators: Overnight & weekends NOC: 0800-1700 Specialists: “9 to 5”

Creating the global research village 8 8 8

Serving European NREN’s

Out of Hours Activity

Organization Out of Hours Cover Percentage of Critical Incidents Out of Hours DANTE SD & On call NCC engineer 40% Indiana Uni NOC & On call engineer 33% NORDUnet On call NOC engineer 4% SWITCH Monitoring service & Call list [1 or 2 times per year] PSNC NOC Operators & Call list (engineers) 10%

Creating the global research village 9 9 9

Serving European NREN’s

Processes and Standards

Organization Processes and Standards DANTE Specific Work Instructions for SD; Procedures and Guides for

• NCC. ITIL based.

Indiana Uni GNOC All documentation kept in WebGUI wiki – both GNOC-wide and network specific (in separate areas) NORDUnet All documentation in Confluence. SWITCH Some documentation in Wiki; good use of annotation in configs; comprehensive fibre documentation PSNC No formal adoption of standards but NOC generally follow ITIL principles

Creating the global research village 10 10 10

Serving European NREN’s

KPIs and SLAs

Organization KPIs SLA DANTE Ticket open – 15 mins First update – 60/90 mins Multi-homed: 99.999 / 99.9% Single homed: ~99.4% Time to fix: 5 hours Indiana University [Ticket open <20 mins] Bespoke, based on information flow NORDUnet [Ticket open 15/30 mins] TBC SWITCH Considered but not implemented Multi-homed: 99.99% (inc PM) Single-homed: 99.8% (exc PM) PSNC Ticket queue length (but not formal) Leased fibres – MTTR 10 hrs

Creating the global research village 11 11 11

Serving European NREN’s

Additional Services

Organization Services DANTE NSHaRP; E2ECU; MDSD Indiana Uni GNOC “Software as a Service” NORDUnet Adobe Connect; WAYF; VM servers; IP peering SWITCH AAI, PKI, eduroam; VC; e-collaboration PSNC Eduroam; VC; DNS; TERENA certificates; archiving

Creating the global research village 12 12 12

Serving European NREN’s

Follow Up – Completed Changes

Planned work conducted after not before core working day 90 minute shift overlap Ad hoc briefings, training DANTE Operations team divided into specialisations IP, Transport and Security specialists NOC Tier 3 specialists moved to respective teams

Creating the global research village 13 13 13

Serving European NREN’s

Future Changes

Magdalene contract expires March 2014. Options Re-tender for fully outsourced, off-site Service Desk OOH off-site Service Desk & in-hours in-house Service Desk Fully in-house Service Desk In-house Ticket System Greater control Minimise risk and cost of selection Commercial Off the Shelf Focus on popular systems (JIRA, RT, OTRS)

Creating the global research village 14 14 14

Serving European NREN’s

Special Thanks to

NORDUnet Jorgen Qvist, Freisa Perez SWITCH Willi Huber, Felix Kugler, Ernst Heiri PSNC Szymon Trocha and all in PSNC NOC and Engineering Indiana University Global NOC Jim Williams, Marianne Chitwood, Brandon Beale, Luke Fowler, Jon Paul Herron, Stephen Peck, David Jent

Creating the global research village

Serving European NREN’s

Questions / Discussion