I get NOC'd down... ...but I get up again T owards excellent NOC - - PowerPoint PPT Presentation

...but I get up again T

• wards excellent NOC customer service

even when things go wrong

Gareth Eason (HEAnet)

TF-NOC Meeting, Barcelona, 2nd Sep 2010

“I get NOC'd down...”

Who am I?

• Gareth Eason

­ Network Operations Manager, HEAnet

John Boland (CEO) Gareth Eason

(Network Operations Manager)

Brian Nisbet

(NOC Manager)

NOC (Virtual T eam) NetOps team

Remember Copenhagen?

This is only presentation #1 (of at least 2) ... because this one has no answers,

• nly questions...

Barcelona

This is presentation #2 (of at least 2) ... where we realise there are no answers but there are some good ideas ...

Overview

I want to change:

• 1. how you think about your NOC
• 2. how your NOC thinks about its customers

Overview

• TF­NOC Work item
• Planning for failure
• When a supplier's service fails
• When a client's service fails
• Lessons learned

Overview

• TF­NOC Work item
• Planning for failure
• When a supplier's service fails
• When a client's service fails
• Lessons learned

TF-NOC: Work Items

• NOC Users
• Who are they?
• Agreements
• What agreements should exist?
• What should they contain?
• User Relations
• How do they relate to a NOC?
• Escalation & Priority
• When? How? How do we ensure resolution?

Overview

• TF­NOC Work item
• Planning for failure
• When a supplier's service fails
• When a client's service fails
• Lessons learned

“Expect success, but plan for failure” -- Unknown

Who you gonna call?

• NOC only gets called when a failure occurs.
• Be proactive – avoid failures!

­ Resilient solutions ­ Monitoring and alerting ­ Alerting relevant to client

• Consider alerting direct to the client

­ The NOC calls the client on detecting a problem ­ Web portal displaying monitoring for client

Who you gonna call?

• Make sure NOC is available

­ Designated call handler ­ Designated escalation paths ­ Consider a recorded message saying there is a problem

• Make sure problem is resolved

­ Designated call handler ­ Designated problem handler

• Make sure customer stays informed!

Overview

• TF­NOC Work item
• Planning for failure
• When a supplier's service fails
• When a client's service fails
• Lessons learned

Failure of a supplier

• Measure

­ how do we define (there is) a problem?

• Inform

­ Make sure the supplier knows ­ Make sure the rest of your organisation knows

• Plan to resolve

­ Agree what you want (from supplier)

• Agree timeline and metrics

­ Agree how you will both know it's resolved

Crisis Meeting

• Second last resort
• Collect data and metrics before the meeting
• Invite account manager
• Invite account manager's manager
• Avoid covering old ground
• Avoid assigning blame
• ...

Crisis Meeting

• 1. Agree there is a problem
• 2. Agree the scope of the problem
• 3. Agree a resolution to the problem
• Might be a resolution
• Might be periodic updates if long­running problem
• Define what you want to see from supplier
• 4. Agree how you'll know the problem is resolved
• 5. Agree the next meeting date

Next steps

• Supplier failure:

­ Escalate further (if useful) ­ Contractual penalties ­ Use alternate suppliers, if any

• Last resort: Material breach of contract

­ Material breach of contract / cessation of contract ­ ...might mean procurement again. ­ Dependant on legal jurisdiction

Overview

• TF­NOC Work item
• Planning for failure
• When a supplier's service fails
• When a client's service fails
• Lessons learned

Client Problem

• 1. Understand their problem
• 2. Triage
• 3. Define action plan (with client)
• 4. Implement Plan
• 5. Report on success (or otherwise) of plan
• 6. GOTO step 1.

Understand their problem

• Often biggest hurdle
• Understand direct and indirect implications

­ Loss of revenue? ­ Loss of face? ­ Media? Press? Publicity?

• Knowing your client well will help

­ You do have regular meetings with your clients, right?

• Find out what the client is actually trying to do!

Triage

• Can we solve the whole problem?
• Busy website

­ Are some parts of it busy? Can we isolate them? ­ Can we replicate the service

• Loss of connectivity

­ Are there vital services for which a few 3G modems will do?

• Figure out what the customer actually needs

Action plan

• 3 hour (emergency)
• 24 hour
• 3­day
• 7­day
• 1 month
• 3 month
• Long term (1 year, etc.)

Emergency Reactive Proactive Strategic

Implement Plan

• Put everything in place.
• Monitor towards success criteria

Report

• Discuss measurements with client
• Did things work as expected?
• Were the success criteria met?
• Were they the right success criteria?
• Go back to step 1 : Understand their problem

Business As Usual

• Agree (with customer) criteria for returning to

Business As Usual

• Learn from failure

­ Make the business as usual for NOC better business!

Overview

• TF­NOC Work item
• Planning for failure
• When a supplier's service fails
• When a client's service fails
• Lessons learned

Understanding

• Understand the problem

­ Educate suppliers about your pain points ­ Educate supplier on implications of problems (e.g. Delivering a router late means a client cannot register students at a new campus.) ­ Learn from customers – what are their pain points?

Communication is key

• Deal with the 'Mars' and 'Venus' compexes

­ Allow NOC to listen to problems – some customers just want to let you know. (Recorded message!) ­ Enable NOC to triage, work around and fix

• Make sure communication doesn't slow down

fault resolution

­ Designated person on point for comms. ­ Designated person on point for escalations.

Work around problems

• Don't be afraid to work around problems

­ Root cause repair might not be quick enough ­ Understand the customer's view of the problem – maybe something less than a complete fix is good enough.

• Agree and stick to action plans

­ Agree plan with customers ­ Ensure your NOC knows the plan. ­ Ensure your boss knows the plan!

Don't fear change!

If the action plan is not working... ... CHANGE IT!

Do one thing today to change how one customer thinks about your NOC in a positive way.