Network design and data collection in an academic & research - - PowerPoint PPT Presentation

aarnet

Australia's Academic and Research Network

Glen Turner

2010-02-08 10th workshop on data networks as formal objects (WODNAFO 10) The University of Adelaide

Network design and data collection in an academic & research network

Our network's design goals, 1

• Fast in the real world

– Large number of interactive users – Small number of massive data transfers

• Available in the real world

– 99.999% is a five minute outage per year. If that

five minutes falls at the wrong time, we have failed

– The definition of 'available' quickly becomes

difficult

Our network's design goals, 2

• Advanced

– We want to discourage the waste of “test bed”

networks which use commercially-available equipment

– We want to encourage research networking

equipment

– IPv6, multicast, QoS, dynamic light paths

• Enabling

– To campus networks – To related activities: authentication, video

conferencing, storage

– Increasingly to primary and secondary schools

AARNet is not a “typical” ISP

• Small team of experts

– Historically 5, about 40 people now

• Bias to insourcing

– If it is understood well enough to be in an

• utsourcing contract, then it isn't research
• Use our customers' rather odd combination of

strengths

– Large, accepting of risk, smart, cooperative,

focus on the long-run result

• Transparent

– Public outage notices, investment plans, etc

AARNet is not a typical NREN

• Funded from revenue, not from grant
• Commodity traffic for all researchers, teachers

and students – not research into networks – was the reason for our creation

• Founded by the universities, not the higher

education or industrial development bureaucracy

Net assets we own, sell and buy

• Rights of way and spectrum
• Ducts
• Fibres and radios
• Cores and transmitter channels
• WDM channels
• SDH circuits
• MPLS paths
• VLANs
• IP network

Example: inter-campus link

• Project management services for a fiber build

totally paid for and owned by the customer

• Duct and hut access, customer owns fiber
• Customer leases fiber pair and hut access
• Customer leases WDM channel
• Managed ethernet service, aka MPLS or VLAN
• IP access, customer uses IPSec tunnel

Typically, design modelling tools are confined to

• ne of these alternatives

Customers often make a poor initial choice

Goal: fast

• High bitrate, rather obviously
• Low latency is actually harder
• Low errors

– Good engineering practices – Some alternatives are poor: wireless and

undersea cable

• Big packets

– Complicates provisioning: one more thing to

agree on

• Traffic engineering

Fast → Security complications

• Fast for script kiddies too

– A worst-effort class and classification of

suspicious traffic

– Avoid NOC overhead by using customer-facing

BGP to signal miscreant traffic

• Fast for the receiver

– SYN flooding

Fast → Customer engineering

• Worst-performing link sets performance for

entire path

• Customers buy routers based on “sticker

speeds”

• Customer's technicians often poor practices

– No fiber cleaning, no inspection – No power budget

• Customers don't measure their networks

– Eg: Ethernet late collisions

Fast → Computer security politics

• Too fast for a lot of corporate firewalls

– The very idea of “one firewall to rule all traffic”

sounds wrong to network engineers, but it sounds right to security policy makers

• Difficult to change security policy until a project

at risk of failing

• Too much computer security focus is at the

network layer, since this is easy to solve, and too little in applications

– Result is applications that cannot be exposed to

the Internet, and thus VPNs as a substitute for application authentication

Goal: Availability, 1

• Diversity

– Of path

• “As built” maps of fiber paths, ours and of all

leased fiber and capacity

• Finding diversity can be difficult, especially at

some pits and building ingress and risers

– Of routing

• Ladder network topology

– Of management

• Change management

Goal: Availability, 2

• Configuration control is key

– Misconfiguration avoidance

• Test facilities
• Strong preference for routers with virtualisation
• Approval of change

– Misconfiguration recovery

• Automated source code configuration control

– Source repository – Who, when, where, what logging

• Approval of change time

– Database-driven networks are the aim

• That's my hope for algebra
• Vendors hate it

Goal, Availability, 3

• Monitoring of outages and hazards

– Real-time monitoring

• Out-of-band event notification
• Out-of-band notification of traffic flow changes

– Notification of changes

• aka, “wtf just happened?”
• Planned outages and hazards calendar
• Fault ticked opened for all changes, closed when

acceptance testing OK

• Capacity planning

– And failure scenarios

Goal, Availability, 4

• Fault handling

– Clear responsibility, AARNet's approach

• Changes: You broke it, you tell the NOC, you fix

it, you are honest about the cause

• Unexpected faults are dealt with or brokered by
• n-call person
• If the on-call person gives you the fault, it's yours

to keep

• Proactive assistance from more experienced

engineering staff

– Communication with customers

• Their essential processes will have manual

measures

Availability assists more availability

• If you have diverse paths, you can move traffic

around to avoid planned maintenance

• If the customer can select the path, they can

increase their availability too

• Avail. →Dynamic interior routing
• Often isn't

– Hard-coded interior default route

• Firewall failover and OSPF don't cooperate
• Significant cost educating customer

engineering staff

– In the ways of BGP – In OSPF – In network engineering, generally

Goal: advanced

• Compatible equipment

– IPv6

• Moving goalposts

– Inter-domain multicast, we're now at the 4th

architecture

• Customer engineering

– Why do I need that at all? – Researcher wants the feature, central networks

doesn't want the risk

– Light paths and OpenFlow switch are responses

to this problem

Advanced → …

• “Advanced” means there is no commercial

training available

• “Advanced” means we don't know the best

practice

• “Advanced” usually means doing it twice

– A Linux box running research software

• Don't care if paths are long, non-diverse, etc

– Supported code on commercial router

• Want production quality
• “Advanced” means no monitoring solution

Advanced → Grant funding

• If we support a feature, then there is no grant

funding for the researcher to use that feature

– It's now “infrastructure”, which is to be totally

funded by the university's operational budget

– But IT at a university isn't going to fund an

experimental activity

• But if the researcher builds a testbed VPN over
• ur network…

– So we have a product to solve that political

funding problem

Enabling

• The network designer and operator can help

solve other issues

• Federated authentication
• Experience exchange, aka conferences
• Purchasing

– If you are purchasing switches by the pallet-

load, we can't do better. But we can stop you paying retail price for a single unit

– Allows “best of breed” purchasing

• Eg: switches from Vendor A, firewall from

Vendor B

Enabling → Consortia

• In these areas the network is not the only

important entity, nor does if have the depth of experience to have the answers

• Bring to the table

– An operational environment

• Hosting, configuration control, monitoring,

restoration

– Economies of scale

• One more highly-available Linux service, costs

about $450pa

– Project management expertise

Enabling → Demo–production gap

• Takes about one elapsed year and three staff

year to

– Describe – Document – Training materials – Policy negotiation – Configuration control – Monitoring – Capacity planning

• Usually research funding ends at the demo :-(

A word about billing

• Billing can get very complex very quickly

– The highest cost of a national phone call is the

billing

• Simple billing is attractive
• Complexity from

– Users, particularly those where charges are

much greater than costs

– Competition, particularly those aiming a a

subset of your customers

– Changes in costs, such as the cost of

submarine capacity. Introduces a temporal element

Billing and data collection

• Accounting systems usually duplicate

monitoring systems

– Reconcile charges from others – Contestable charges for customers

• Accounting system then loads into invoicing

system

– Invoicing system usually holds some low-level

aggregate, such as hourly traffic quantities

– Invoicing system does heavy lifting for charging

plans

It's not what your network can do…

• We want to give you access to our data, on the

same terms that medical researchers get access to medical histories or social scientists get access to Bureau of Statistics unit records

• But the Telecommunications Act 1997 (Cth)

prevents this

• There may be sneaky ways around that
• But what we really need is

– E-mails from you explaining what research this

lack of access prevents

– You raising this with Federal politicians and staff

aarnet

Australia's Academic and Research Network

Glen Turner Network engineer

Network design and data collection in an NREN

www.gdt.id.au/~gdt/presentations/2010-02-08-wodnafo-aarnet/