BigData Express: Toward Predictable, Schedulable, and - PowerPoint PPT Presentation

FERMILAB-SLIDES-18-060-CD BigData Express: Toward Predictable, Schedulable, and High-performance Data Transfer Wenji Wu wenj@fnal.gov Internet2 Global Summit May 8, 2018 This manuscript has been authored by Fermi Research Alliance, LLC under Contract No. DE-AC02-07CH11359 with the U.S. Department of Energy, Office of Science, Office of High Energy Physics

BigData Express • Funded by DOE’s office of Advanced Scientific Computing Research (ASCR) • Collaborative effort by Fermilab and Oak Ridge National Laboratory – KISTI joined as a unfunded partner at 2017 – ESnet provides WAN service • A three-year research project – Start: Oct 1, 2015 – End: Sep 30, 2018 • http://bigdataexpress.fnal.gov

BigData Express Research Team • FNAL • ORNL – Wenji Wu (PI) – Nageswara Rao – Qiming Lu – Gary Liu – Liang Zhang – Amy Jin • KISTI – Sajith Sasidharan – Syed Asif Shah – Phil DeMar – Seo-Young Noh – Jin Kim Note: KISTI and ESnet are unfunded project partners

DOE Leadership Computing facilities offer computing and Big data enables scientific discoveries storage resources needed to process and analyze science data BigData Express Goal The efficient movement of science data from their sources into processing and storage facilities and ultimately on to user A distributed middleware system that provides a analysis is critical to the success of any such endeavor. schedulable, predictable, and high-performance data transfer service for the DOE’s large-scale Data transfer is now an essential function for science science facilities and their collaborators. discoveries, particularly within big data environments.

Why BigData Express? • Targeted at optimizing data transfers in high-speed networks – Large-scale data movement of Big Data Science – High-speed network environments (40/100GE+) • Builds on Multicore-Aware Data Transfer Middleware (MDTM) – mdtmFTP: a high-performance data transfer tool • Pipelined I/O-centric design to streamline data transfer • MDTM optimizes use of underlying multicore system • Extremely efficient in transferring of Lots Of Small Files (LOSF) – http://mdtm.fnal.gov • Orchestrates system (DTN), storage, & network (SDN) resources – To provide full end-to-end performance optimization

BigData Express versus SENSE • BigData Express is data transfer middleware – Uses SENSE for WAN SDN services • SENSE is a network service – Provides higher-level applications with SDN-type services – BigData Express is an application to SENSE • BigData Express and SENSE are each stand-alone services in their own right – BigData Express works fine without SENSE • WAN component is simply Best Effort – SENSE is agnostic to higher-level applications using its services

BigData Express Major Components BigData Express Major Components • BDE Web Portal • BDE Web Portal – Allow users to access BigData Express data – Allow users to access BigData Express data transfer services transfer services • BDE Scheduler • BDE Scheduler – DTN as a service – DTN as a service – Co-scheduling of DTN, storage, and network – Co-scheduling of DTN, storage, and network • BDE AmoebaNet • BDE AmoebaNet – Network as a service – Network as a service • mdtmFTP • mdtmFTP – a high-performance data transfer engine – a high-performance data transfer engine – http://mdtm.fnal.gov – http://mdtm.fnal.gov

BigData Express Major Components (cont.) • DTN Agents – Manage and configure DTNs – Collect and report the DTN configuration and status • Storage Agents – Manage and configure storage systems • Data Transfer Launching Agent – Launch data transfer jobs – Support different data transfer protocols

BigData Express -- Distributed A Peer-to-Peer model

BigData Express -- Flexible Data Transfer Federation … DTNs • Flexible to set up data DTNs transfer federations Data Transfer Federation Data Transfer Federation • Providing inherent support Networks for incremental deployment DTNs DTNs … … DTNs

BigData Express -- Scalable BDE Web Portal SDN Agent SDN Agent SDN Agent BDE Scheduler BDE Scheduler (AmoebaNet) Message Queue Data Transfer DTN Agent Data Transfer DTN Agent Launching Agent Data Transfer DTN Agent Launching Agent Launching Agent Storage Agent Storage Agent Storage Agent • BigData Express scheduler manages site resources through agents • Use RabbitMQ as message bus

BigData Express -- Extensible BDE Web Portal BDE Scheduler SDN Agent BDE Scheduler SDN Agent SDN Agent (AmoebaNet) Message Queue DTN Agent Data Transfer Launching Agent DTN Agent Data Transfer Launching Agent DTN Agent Data Transfer Launching Agent Storage Agent mdtmFTP GridFTP SRM XrootD … Storage Agent Plugin Plugin Plugin Plugin Storage Agent • Extensible Plugin framework to support various data transfer protocols • mdtmFTP, GridFTP, SRM, XrootD, …

BigData Express -- End-to-End Data Transfer Model Site A - Smart E2E Site B - Smart E2E Data Transfer Orchestrator Data Transfer Orchestrator Resource negotiation & brokering • Application-aware network service Web Portal Web Portal Scheduler Scheduler o On-demand programming Data Transfer Data Transfer n g e n Launching Agent Launching Agent g i R e r o e c e t r k i s u a o o t o r i u s b o r e n c & e R & n • Fast-provisioning of end-to-end o b i r t o a k i t e o r g i n e g n mdtmFTP mdtmFTP network paths with guaranteed QoS DTN Agent AmoebaNet AmoebaNet DTN Agent SENSE Storage Agent Storage Agent • Distributed resource negotiation & brokering LAN LAN WAN Storage Edge DTN Edge DTN Storage A End-to-End Data Transfer Loop with Guaranteed QoS

BigData Express -- Three Types of Data Transfer • Real-time data transfer • Deadline-bound data transfer • Best-effort data transfer

BigData Express vs. Globus Online Features BigData Express Globus Online • Distributed service • Architecture Centralized service • Flexible to set up data transfer federations • Extensible plugin framework to support multiple protocols: • Supported Protocols GridFTP o mdtmFTP o GridFTP, XrootD, SRM (coming soon) • Yes, Network as a service • • SDN Support Fast-provisioning end-to-end network paths with No guaranteed QoS • Real-time data transfer • • Supported Data Transfers Deadline-bound data transfer Best-effort data transfer • Best-effort data transfer • • Checksum Checksum Error Handling • • Retransmit Retransmit

BigData Express SC’17 DEMO • BigData Express: a schedulable, predictable, and high-performance data transfer service – QoS-guaranteed data transfer – DTN as a service – Network as a service – Distributed resource brokering/matching A DOE/SC/ASCR-sponsored research project Software is available at: http://bigdataexpress.fnal.gov

A Cross-Pacific SDN Testbed HP Z91000 AmoebaNet KREONET BDE BDE 40GE Web Protal Scheduler STP 192.2.2.7 134.75.125.77 DTN2 10GE STP 10GE 134.75.125.78 StarLight 192.2.2.9 10GE 192.2.2.8 DTN3 STP DTN4 134.75.125.76 134.75.125.80 ESNET OSCARS KISTI SW KISTI, South Korea To Internet 600W Chicago FNAL Border router STP bde-hp1.fnal.gov 40GE 40GE Pica8 P5101 Pica8 P3930 ESnet NSI AmoebaNet Circuit Service 4/1 4/3 4/5 4/7 47 49 51 65 73 74 48 66 4/2 4/4 4/6 4/8 50 52 yosemite.fnal.gov 40GE BDE BDE 40GE 40GE 1GE 40GE Web Protal Scheduler 192.2.2.1 192.2.2.2 192.2.2.3 BDE1 BDE2 BDE3 BDE-hp5 BDE4 Lustre file system wwwld1 Infiniband Switch wwwld5 (mgt) (oss) wwwld2 wwwld6 wwwld3 wwwld4 FNAL, US (oss) (oss) (oss) (oss)

BigData Express Deployment • Completed deployment: KISTI, UMD, StarLight, FNAL • Ongoing deployment: KSTAR, ESnet • Work with StarLight to deploy BDE at XRPs – Pacific Research Platform (PRP) – National Research Platform (NRP) – Global Research Platform (GRP) – The European Research Platform (ERP) – Asia Research Platform (ARP) • Collaborate with SENSE for BDE+SENSE deployment • Work with US CMS to deploy BDE at US CMS sites

Support Science • Fusion community – Work with KSTAR, KISTI, PPPL, and ORNL to transfer/stream data from KSTAR to US research institutions • XRPs (PRP, NRP, GRP, ERP, ARP) – Work with StarLight to deploy BDE at XRPs to support various science • HEP community – Work with US CMS to deploy BDE at US CMS sites • PI has been invited to give a BDE demo for US CMS • Tentatively scheduled for the last week of May, 2018

More information about BigData Express http://bigdataexpress.fnal.gov PI: Wenji Wu, Fermilab wenji@fnal.gov