Addressing(the(challenges of federation in the Nectar(Research(Cloud - PowerPoint PPT Presentation

Addressing(the(challenges of federation in the Nectar(Research(Cloud The(Nectar(Research(Cloud(is(part(of(the(Australian(Research(Data(Commons,(which(is( supported(by(the(Australian(Government(through(the(National(Collaborative(Research( Infrastructure(Strategy(to(establish(eResearch infrastructure(in(partnership(with(Australian( research(institutions,(organisations and(research(communities. PRESENTED(BY Paul(Coddington,(Associate(Director(Research(Cloud(and(Storage

NCRIS,'Nectar,'and'ARDC Nectar'was'founded'in'2011'through'the'federal'government’s' • National'Collaborative'Research'Infrastructure'Strategy'(NCRIS) Funded'the'Research'Cloud and'Virtual Laboratories • The 2016'NCRIS'Roadmap'called'for'a'merging'of'3'NCRIS'facilities' • (Nectar,'Research'Data'Storage,'Australian'National'Data'Service)'to' form'the'Australian'National'Data'Cloud/Commons'(ARDC) ARDC'is''part of a broader Digital'Data'and'eResearch Platforms' • theme'including'national'HPC'centres,'AARNet NREN,'and'AAF The'research'infrastructure'roadmap'also states that: • “The%Australian%Research%Data%Cloud%should%broadly%align%with%the%European% Open%Science%Cloud%and%other%global%initiatives.” 2

The$Nectar$Research$Cloud A world first… The Nectar Research Cloud is a partnership between 7 institutions and research A single integrated cloud organisations who are operating a operated by 7 national federated Openstack research cloud. partners and supporting over 15000 University of Melbourne • research users. National Computation Infrastructure (NCI) • Monash University • Queensland CyberInfrastructure Foundation • (QCIF) eResearch SA (eRSA) • University of Tasmania • Intersect, NSW •

Nectar'Cloud'Federation A"single"national"cloud"interface OpenStack*cells*to*support*7*regional*sites • Other*institutions*have*joined*with*their*own resources (Swinburne ," Auckland ," UTS,*RMIT ) • Users*can*request*a*specific*site*– or* deploy'anywhere. • National"services Cloud*dashboard*(Horizon)*and*API • Authentication*(Keystone,*AAF) • Image*repository*(Glance) • Federated"services Object*store*(Swift)* • Compute*(Nova)*and*volume*storage*(Cinder) • User*support,*help*desk,*user*guides,*documentation • 4

Nectar'Research'Cloud'Ops'Team …a#federation#of#7#operators#(~#16#FTE#total)# Node Operators Core Services User Support Team 5

Activities(to(coordinate(federated(ops Six$Monthly$Tech$and$Operations$Workshops Physically)bringing)Core)Services,)Node)Operators)and)DHD)together)for)2)days • Fortnightly$RC;ops$meetings • Video)Conference)of)Core)Services,)Node)Operators)and)some)DHD Weekly$Core$Services$meetings • Video)Conference)of)Core)Services)staff)from)UoM,)Monash)and)QCIF Fortnightly$DHD$meetings • Video)Conference)of)6)sites)participating)in)the)DHD Monthly$Control$Group$meetings • Video)Conference)– steering)group)with)ARDC)and)Node)ops)managers 6

Usage ~7500&VMs&in&use& ● ~40,000&cores&and&2&Pbytes of&cloud&data&storage&in&use& ● ~2000&users&running&VMs& ● ~4000&users&ran&VMs&last&year ● >15,000&users&since&2012&and&200&new&registered&users&per&month ● Tens&of&thousands&of&users&of&VMs&and&services&that&run&on&the&cloud& ● >2500&project&allocations ● >700&allocations&are&for&multiFinstitutional&collaboration ● >300&projects&with&national&research&grants&using&the&cloud ●

http://status.rc.nectar.org.au

Nectar'Cloud'Federation Mix$of$nationally$(NCRIS)$and$locally$funded$cloud$resources • Located$at$NCRIS$funded$Nodes$: and$other$organisations$wanting$ • to$federate$their$own$infrastructure$into$the$national$cloud Leveraging$co:investment$from$many$organisations • National$and/or$local$resource$allocations • National$service$and$standards$with$local$customisations$ • Standard$flavours$and$specialised$instance$configurations$(GPUs$etc) • Nectar$Official$Images$and$community$contributed$images • 9

Federation*Challenges Collaboration*is*great*but*has*overheads • Federated*organisations*in*Nectar*are*all*very*different • Federation is expanding and now international with NZ • Resource*allocation*and*charging • National merit=based*scheme*plus*local*allocation*processes • Federal*funding*plus*co=investment • Costs*currently*covered*by*institutions*not*end*users*(pros*and*cons) • Capacity*management*across*multiple*sites • Monitoring*and reporting*(internal*and*to*sites*and*funders) • Security*and*incident*response • 10

Federation*Challenges Governance • Authentication.and.authorisation • AAF.(and.Tuakiri).great.but.limited,.looking.to.support.Edugain and.OID • Currently.group.authorisation.only.within.Openstack • Provide.a.standard,.coherent.national.service.while.allowing.for. • innovation.and.local.customisations A.set.of.underpinning.services.and.standards.and.processes • With additional.site.or organisationFspecific.additions.or.variations • Some.sites.have.private/local.cells.and.local.dashboard • International.alignment.to.enable.easier.international.collaboration • 11

Future&Directions Continue(existing(services(while(refreshing(infrastructure( • Provide(more(higher7level(services,(not(just(IaaS • Jupyter, Rstudio,(HTC,(cluster(in(cloud,(containers,(Binder,(ML,(…( ▪ Better(handle(trade7off(between(local(customisations and(standard( • national(service Align(with(international(cloud(initiatives(e.g.(EGI(and(EOSC,(ORCA • How(to(best(take(advantage(of(public(cloud(and(multi7cloud • Work(with(NCRIS(capabilities(to(provide(cloud(services(customised • to(their(requirements,(e.g.(Biocommons Pathfinder(project Find(a(sustainable(long7term(approach • 12