FTS3 and GFAL2 Michal Simon, CERN Manchester, 10/04/2013 EMI is - - PowerPoint PPT Presentation

▶
fts3 and gfal2
SMART_READER_LITE
LIVE PREVIEW

FTS3 and GFAL2 Michal Simon, CERN Manchester, 10/04/2013 EMI is - - PowerPoint PPT Presentation

Mar 24, 2024 449 likes •633 views

FTS3 and GFAL2 Michal Simon, CERN Manchester, 10/04/2013 EMI is partially funded by the European Commission under Grant Agreement RI-261611 Overview FTS3 Status Features Directions GFAL2 Introduction Features

slide-1
SLIDE 1

EMI is partially funded by the European Commission under Grant Agreement RI-261611

FTS3 and GFAL2

Michal Simon, CERN

Manchester, 10/04/2013

slide-2
SLIDE 2

EMI INFSO-RI-261611

  • FTS3

– Status – Features – Directions

  • GFAL2

– Introduction – Features – Summary

Overview

10/04/2013 CERN, IT-GT-DMS 2

slide-3
SLIDE 3

EMI INFSO-RI-261611

  • Stable and functionally comparable to FTS2
  • Installed at CERN (pilot), RAL, ASGC, PIC and

BNL

  • Used by CERN experiments (Atlas and CMS)

for production and debug transfers

  • Stress and scalability testing (vs. FTS2):

– 26.6% less number of VMs – 14.2% less memory – 2.2% more transfers

FTS3: Status

10/04/2013 CERN, IT-GT-DMS 3

slide-4
SLIDE 4

EMI INFSO-RI-261611

  • Alternative Database back-ends

(Oracle or MySQL)

  • Additional protocols: xroot and http

– FTS3 plugins are under development – Infrastructure requires an upgrade

  • Session reuse

(improved performance for the ‘many small files’ use case)

FTS3: New features

10/04/2013 CERN, IT-GT-DMS 4

slide-5
SLIDE 5

EMI INFSO-RI-261611

  • Smart retry logic
  • Improved UI:

– FTS3 is backwards compatible with glite clients – There is a new, improved CLI that gives access to the new features (e.g. JSON formatted output) – Transfer submission and status retrieval APIs available:

  • in Python
  • through REST interface

FTS3: Event newer features

10/04/2013 CERN, IT-GT-DMS 5

slide-6
SLIDE 6

EMI INFSO-RI-261611

– Works out of the box (no configuration required):

  • Good defaults
  • Auto-tuned transfers

– Endpoint-centric manual configuration

  • VO specific shares

FTS3: Configuration

10/04/2013 CERN, IT-GT-DMS 6

Turns quadratic scaling behaviour into linear

slide-7
SLIDE 7

EMI INFSO-RI-261611

  • Dynamic adjustment (based on recent

history) of:

– Number of active transfers – Number of streams – The timeout

  • FTS3 has been running

for 8 months relying on the auto-tuner

  • Planned for the future:

– VO specific shares on top of the auto-tuner

FTS3: Transfer-job auto-tuner

10/04/2013 CERN, IT-GT-DMS 7

slide-8
SLIDE 8

EMI INFSO-RI-261611

  • FTS 3 publishes monitoring messages

– per every state transition – periodically (state of the queue, failure rate, etc.)

  • Web interface for monitoring:

– transfer-jobs – queue – statistics – auto-tuner

  • Dashboard monitoring

FTS3: Advanced monitoring

10/04/2013 CERN, IT-GT-DMS 8

slide-9
SLIDE 9

EMI INFSO-RI-261611

– Source: alternative replicas (or protocol) – Destination: alternative SEs (or protocol) – FTS3 will choose the best SEs and the best protocol to carry out the transfer job

FTS3: Multiple replica utilization

10/04/2013 CERN, IT-GT-DMS 9

slide-10
SLIDE 10

EMI INFSO-RI-261611

– Released in EMI3 – Forthcoming releases in EPEL – Deployment plan: to be decided

  • Either central or distributed
  • Can be kept open until mid 2014

– FTS3 pilot: https://fts3-pilot.cern.ch:8443 (feel free to try it out!!!) – Wiki: https://svnweb.cern.ch/trac/fts3/wiki – Stress testing:

https://svnweb.cern.ch/trac/fts3/wiki/StressTesting

FTS3: Directions

10/04/2013 CERN, IT-GT-DMS 10

slide-11
SLIDE 11

EMI INFSO-RI-261611

  • One toolkit for all the GRID and Cloud data
  • perations:

– Client data access

  • support for all common protocols and storage systems

– Data management

  • unified API for all data management

– Data transfer

  • 3rd party transfer support

GFAL2: The workhorse of FTS3

10/04/2013 CERN, IT-GT-DMS 11

slide-12
SLIDE 12

EMI INFSO-RI-261611

  • Carries out all the transfer-jobs submitted

to FTS3 (3rd party transfers)

  • Hides the GRID complexity
  • Provides common API for

all the access and transfer protocols

  • Thanks to the adopted plug-in mechanism

reduces the number of dependencies

GFAL2: The workhorse of FTS3

10/04/2013 CERN, IT-GT-DMS 12

slide-13
SLIDE 13

EMI INFSO-RI-261611

  • Take a and b in:
  • {srm, guid, rfio, dcap, gsidcap, gsiftp, file, xrootd,

http, dav}

  • Automatic protocol resolution:
  • gfal_copyfile(a://src/file, b://dest/file)
  • GFAL2 does all the work for you!

GFAL2: Protocol generic filecopy

10/04/2013 CERN, IT-GT-DMS 13

slide-14
SLIDE 14

EMI INFSO-RI-261611

GFAL2: Thread-safe session reuse

10/04/2013 CERN, IT-GT-DMS 14

slide-15
SLIDE 15

EMI INFSO-RI-261611

  • Supports all common storage elements
  • Supports lots of protocols:

– SRM, GRIDFTP, GUID, XROOTD, HTTP(S) and more

  • Supports POSIX extended attributes
  • No environment variables

– although, it is compatible with the environment variables from GFAL 1.0 era for an easy transition

GFAL2: Other features

10/04/2013 CERN, IT-GT-DMS 15

slide-16
SLIDE 16

EMI INFSO-RI-261611

  • libgfal2:
  • a C library
  • A set of independent plugins
  • gfal2-python:
  • simple to use python-bindings
  • gfalFS

(fuse module for gfal2):

  • Allows to mount any GRID endpoint in a local folder
  • gfal-tools:
  • experimental CLI: gfal-ls, gfal-copy, gfal-mkdir etc.

GFAL2: How to use it?

10/04/2013 CERN, IT-GT-DMS 16

slide-17
SLIDE 17

EMI INFSO-RI-261611

  • Core component of FTS3

– will be supported in any case

  • Released in EMI2 and EMI3 (www.eu-emi.eu)

– will be supported after the end of EMI

  • Released in EPEL
  • There are plans to do packaging for Debian

GFAL2: Summary

10/04/2013 CERN, IT-GT-DMS 17