BigData Express: Toward Predictable, Schedulable, and - - PowerPoint PPT Presentation

▶
bigdata express toward predictable schedulable and high
SMART_READER_LITE
LIVE PREVIEW

BigData Express: Toward Predictable, Schedulable, and - - PowerPoint PPT Presentation

Apr 24, 2023 292 likes •500 views

FERMILAB-SLIDES-18-060-CD BigData Express: Toward Predictable, Schedulable, and High-performance Data Transfer Wenji Wu wenj@fnal.gov Internet2 Global Summit May 8, 2018 This manuscript has been authored by Fermi Research Alliance, LLC under

slide-1
SLIDE 1

BigData Express: Toward Predictable, Schedulable, and High-performance Data Transfer

Wenji Wu wenj@fnal.gov Internet2 Global Summit May 8, 2018

FERMILAB-SLIDES-18-060-CD This manuscript has been authored by Fermi Research Alliance, LLC under Contract No. DE-AC02-07CH11359 with the U.S. Department

  • f Energy, Office of Science, Office of High Energy Physics
slide-2
SLIDE 2

BigData Express

  • Funded by DOE’s office of Advanced Scientific

Computing Research (ASCR)

  • Collaborative effort by Fermilab and Oak Ridge

National Laboratory

– KISTI joined as a unfunded partner at 2017 – ESnet provides WAN service

  • A three-year research project

– Start: Oct 1, 2015 – End: Sep 30, 2018

  • http://bigdataexpress.fnal.gov
slide-3
SLIDE 3

BigData Express Research Team

  • FNAL

– Wenji Wu (PI) – Qiming Lu – Liang Zhang – Amy Jin – Sajith Sasidharan – Phil DeMar

  • ORNL

– Nageswara Rao – Gary Liu

Note: KISTI and ESnet are unfunded project partners

  • KISTI

– Syed Asif Shah – Seo-Young Noh – Jin Kim

slide-4
SLIDE 4

BigData Express Goal A distributed middleware system that provides a schedulable, predictable, and high-performance data transfer service for the DOE’s large-scale science facilities and their collaborators.

Big data enables scientific discoveries

DOE Leadership Computing facilities offer computing and storage resources needed to process and analyze science data The efficient movement of science data from their sources into processing and storage facilities and ultimately on to user analysis is critical to the success of any such endeavor. Data transfer is now an essential function for science discoveries, particularly within big data environments.

slide-5
SLIDE 5

Why BigData Express?

  • Targeted at optimizing data transfers in high-speed networks

– Large-scale data movement of Big Data Science – High-speed network environments (40/100GE+)

  • Builds on Multicore-Aware Data Transfer Middleware (MDTM)

– mdtmFTP: a high-performance data transfer tool

  • Pipelined I/O-centric design to streamline data transfer
  • MDTM optimizes use of underlying multicore system
  • Extremely efficient in transferring of Lots Of Small Files (LOSF)

– http://mdtm.fnal.gov

  • Orchestrates system (DTN), storage, & network (SDN) resources

– To provide full end-to-end performance optimization

slide-6
SLIDE 6

BigData Express versus SENSE

  • BigData Express is data transfer middleware

– Uses SENSE for WAN SDN services

  • SENSE is a network service

– Provides higher-level applications with SDN-type services – BigData Express is an application to SENSE

  • BigData Express and SENSE are each stand-alone services in their
  • wn right

– BigData Express works fine without SENSE

  • WAN component is simply Best Effort

– SENSE is agnostic to higher-level applications using its services

slide-7
SLIDE 7

BigData Express Major Components

  • BDE Web Portal

– Allow users to access BigData Express data transfer services

  • BDE Scheduler

– DTN as a service – Co-scheduling of DTN, storage, and network

  • BDE AmoebaNet

– Network as a service

  • mdtmFTP

– a high-performance data transfer engine – http://mdtm.fnal.gov

BigData Express Major Components

  • BDE Web Portal

– Allow users to access BigData Express data transfer services

  • BDE Scheduler

– DTN as a service – Co-scheduling of DTN, storage, and network

  • BDE AmoebaNet

– Network as a service

  • mdtmFTP

– a high-performance data transfer engine – http://mdtm.fnal.gov

slide-8
SLIDE 8

BigData Express Major Components (cont.)

  • DTN Agents

– Manage and configure DTNs – Collect and report the DTN configuration and status

  • Storage Agents

– Manage and configure storage systems

  • Data Transfer Launching Agent

– Launch data transfer jobs – Support different data transfer protocols

slide-9
SLIDE 9

BigData Express -- Distributed

A Peer-to-Peer model

slide-10
SLIDE 10

DTNs

Networks

DTNs DTNs DTNs DTNs

…

Data Transfer Federation Data Transfer Federation

… …

Data Transfer Federation

BigData Express -- Flexible

  • Flexible to set up data

transfer federations

  • Providing inherent support

for incremental deployment

slide-11
SLIDE 11

BDE Scheduler SDN Agent Storage Agent DTN Agent DTN Agent SDN Agent BDE Web Portal Data Transfer Launching Agent Data Transfer Launching Agent Data Transfer Launching Agent Message Queue SDN Agent (AmoebaNet) DTN Agent Storage Agent Storage Agent BDE Scheduler

BigData Express -- Scalable

  • BigData Express scheduler manages site resources through agents
  • Use RabbitMQ as message bus
slide-12
SLIDE 12

BDE Scheduler SDN Agent Storage Agent DTN Agent DTN Agent SDN Agent BDE Web Portal Data Transfer Launching Agent Data Transfer Launching Agent Data Transfer Launching Agent mdtmFTP Plugin GridFTP Plugin XrootD Plugin … Message Queue SDN Agent (AmoebaNet) DTN Agent Storage Agent Storage Agent SRM Plugin BDE Scheduler

BigData Express -- Extensible

  • Extensible Plugin framework to support various data transfer protocols
  • mdtmFTP, GridFTP, SRM, XrootD, …
slide-13
SLIDE 13

BigData Express -- End-to-End Data Transfer Model

  • Application-aware network service
  • On-demand programming
  • Fast-provisioning of end-to-end

network paths with guaranteed QoS

  • Distributed resource negotiation &

brokering

LAN WAN LAN

mdtmFTP AmoebaNet AmoebaNet SENSE Edge DTN Storage Edge DTN Storage DTN Agent Storage Agent mdtmFTP DTN Agent Storage Agent Web Portal Scheduler Data Transfer Launching Agent Web Portal Scheduler Data Transfer Launching Agent

Site A - Smart E2E Data Transfer Orchestrator Site B - Smart E2E Data Transfer Orchestrator A End-to-End Data Transfer Loop with Guaranteed QoS Resource negotiation & brokering R e s

  • u

r c e n e g

  • t

i a t i

  • n

& b r

  • k

e r i n g R e s

  • u

r c e n e g

  • t

i a t i

  • n

& b r

  • k

e r i n g

slide-14
SLIDE 14

BigData Express -- Three Types of Data Transfer

  • Real-time data transfer
  • Deadline-bound data transfer
  • Best-effort data transfer
slide-15
SLIDE 15

BigData Express vs. Globus Online

Features

BigData Express Globus Online

Architecture

  • Distributed service
  • Flexible to set up data transfer federations
  • Centralized service

Supported Protocols

  • Extensible plugin framework to support multiple

protocols:

  • mdtmFTP
  • GridFTP, XrootD, SRM (coming soon)
  • GridFTP

SDN Support

  • Yes, Network as a service
  • Fast-provisioning end-to-end network paths with

guaranteed QoS

  • No

Supported Data Transfers

  • Real-time data transfer
  • Deadline-bound data transfer
  • Best-effort data transfer
  • Best-effort data transfer

Error Handling

  • Checksum
  • Retransmit
  • Checksum
  • Retransmit
slide-16
SLIDE 16

BigData Express SC’17 DEMO

  • BigData Express: a schedulable, predictable, and high-performance

data transfer service

– QoS-guaranteed data transfer – DTN as a service – Network as a service – Distributed resource brokering/matching

A DOE/SC/ASCR-sponsored research project Software is available at: http://bigdataexpress.fnal.gov

slide-17
SLIDE 17

A Cross-Pacific SDN Testbed

4/1 4/2 4/3 4/4 4/5 4/6 4/7 4/8 BDE1 BDE2 BDE3 Pica8 P5101

ESNET

40GE 40GE 40GE 40GE 40GE OSCARS ESnet NSI Circuit Service AmoebaNet

KISTI SW

DTN3 HP Z91000

FNAL, US KISTI, South Korea

bde-hp1.fnal.gov yosemite.fnal.gov AmoebaNet BDE Web Protal BDE Scheduler BDE Web Protal BDE Scheduler DTN4 Infiniband Switch wwwld1 (mgt) wwwld3 (oss) wwwld4 (oss) wwwld2 (oss) wwwld5 (oss) wwwld6 (oss) Lustre file system 73 74

KREONET

600W Chicago StarLight STP STP 49 50 51 52 BDE4 Pica8 P3930 40GE 47 48 BDE-hp5 1GE 65 66 STP FNAL Border router DTN2 40GE STP

134.75.125.77 134.75.125.78 134.75.125.76 134.75.125.80 192.2.2.7 192.2.2.8 192.2.2.9

To Internet

192.2.2.1 192.2.2.2 192.2.2.3

10GE 10GE 10GE

slide-18
SLIDE 18

BigData Express Deployment

  • Completed deployment: KISTI, UMD, StarLight, FNAL
  • Ongoing deployment: KSTAR, ESnet
  • Work with StarLight to deploy BDE at XRPs

– Pacific Research Platform (PRP) – National Research Platform (NRP) – Global Research Platform (GRP) – The European Research Platform (ERP) – Asia Research Platform (ARP)

  • Collaborate with SENSE for BDE+SENSE deployment
  • Work with US CMS to deploy BDE at US CMS sites
slide-19
SLIDE 19

Support Science

  • Fusion community

– Work with KSTAR, KISTI, PPPL, and ORNL to transfer/stream data from KSTAR to US research institutions

  • XRPs (PRP, NRP, GRP, ERP, ARP)

– Work with StarLight to deploy BDE at XRPs to support various science

  • HEP community

– Work with US CMS to deploy BDE at US CMS sites

  • PI has been invited to give a BDE demo for US CMS
  • Tentatively scheduled for the last week of May, 2018
slide-20
SLIDE 20

More information about BigData Express

http://bigdataexpress.fnal.gov

PI: Wenji Wu, Fermilab wenji@fnal.gov