CESNET storage activities overview Ji Hork (jiri.horky@cesnet.cz) - - PowerPoint PPT Presentation

▶
cesnet storage activities overview
SMART_READER_LITE
LIVE PREVIEW

CESNET storage activities overview Ji Hork (jiri.horky@cesnet.cz) - - PowerPoint PPT Presentation

Jun 08, 2023 213 likes •441 views

12 th TF-Storage Meeting - Berlin CESNET storage activities overview Ji Hork (jiri.horky@cesnet.cz) 6th of March, 2013 History Relatively new activity to CESNET started in autumn 2010 with 2FTEs preparation phase continued until

slide-1
SLIDE 1

CESNET storage activities overview

Jiří Horký (jiri.horky@cesnet.cz)

12th TF-Storage Meeting - Berlin 6th of March, 2013

slide-2
SLIDE 2
  • Relatively new activity to CESNET
  • started in autumn 2010 with 2FTEs
  • preparation phase continued until the end of

2011 with no HW  no services

  • Some childhood problems appeared afterwards
  • the department restructured in April 2012
  • 6.5 FTEs in total right now

6/3/2013 CESNET storage activities overview 2

History

slide-3
SLIDE 3

EU structural funds + projects of Ministry of Education, Youth and Sports

  • Research and Development for Innovations OP

(“R&DI OP”) - project “eIGeR”

  • 100 mil. CZK ~ 4 mil. EUR (infrastructure only)
  • May 2011 – October 2013 (end of pilot phase)
  • Project “Large infrastructure”
  • 2011 – 2015, operational costs
  • commitment to sustain operation at least to 2018

6/3/2013 CESNET storage activities overview 3

Financing and scale

slide-4
SLIDE 4
  • Three geographically separated storage locations

(Pilsen, PardubiceJihlava, Brno) to support research and science community

  • Large research projects
  • Public universities
  • Academy of Sciences
  • Public Libraries
  • Digitalization of books
  • ...
  • Total capacity 16.6 PB + another tender in

preparation

6/3/2013 CESNET storage activities overview 4

Data storage facility

slide-5
SLIDE 5

6/3/2013 CESNET storage activities overview 5

Distributed storage locations

slide-6
SLIDE 6

6/3/2013 CESNET storage activities overview 6

  • Assumptions:
  • Emphasis on economical aspects + transparent

behavior to users -> HSM system

  • Backup and archival demands foreseen ->

requirement for a tape library

  • Difference in usage patterns will be covered by

migration policies, e.g.:

  • archival – move everything from disk to tape

almost immediately

  • input files for batch jobs – migrate only files

touched > one month ago

Technical assumptions

slide-7
SLIDE 7

6/3/2013 CESNET storage activities overview 7

  • Hosted by University of Western Bohemia, delivered in the

end of 2011, in pilot operation since May 2012

  • Multi-tier, managed by DMF (Data Migration Facility), SLES

OS, CXFS file system (SGI)

  • Disk systems in two IS4600 arrays:
  • Tier1: 50 TB FC drives
  • Tier2: 450 TB SATA drives
  • Tier3: Spectra T-Finity tape library with dual robotics
  • 2200 slots with LTO5 (3.3PB in total), 2300 free slots
  • 8Gbit redundant FC SAN, 10 Gbit Ethernet
  • 2x10 Gbit connection to CESNET2 network
  • 6 frontends managed by Pacemaker in HA cluster
  • 2 HSM servers in HA mode, one system for administration

The Pilsen storage site

slide-8
SLIDE 8

6/3/2013 CESNET storage activities overview 8

The Pilsen storage site

slide-9
SLIDE 9

6/3/2013 CESNET storage activities overview 9

  • Tender finished in the beginning of 2013
  • Hosted by Vysočina Region Authority
  • Multi-tier, GPFS filesystem, Tivoli Storage Manager (IBM)
  • Disk systems in two DSC3700 arrays (+JBODs):
  • Tier1: 800 TB SATA drives
  • Tier2:
  • 2.5 PB MAID (Promise)
  • 3.8 PB Tape – IBM TS3500 with TS1140 drives, dual robotics
  • 16Gbit redundant FC SAN, 10 Gbit Ethernet
  • 2x10 Gbit connection to CESNET2 network
  • 5 frontends, 2 HSM servers in HA mode

The Jihlava storage site

slide-10
SLIDE 10

6/3/2013 CESNET storage activities overview 10

The Jihlava storage site

Only being delivered during this week!

slide-11
SLIDE 11

6/3/2013 CESNET storage activities overview 11

  • Tender finished in the very beggining of 2013
  • Hosted by Brno University of Technology
  • Multi-tier, GPFS filesystem, Tivoli Storage Manager (IBM)
  • Disk systems in two DSC3700 arrays (+JBODs):
  • Tier1: 400 TB SATA drives
  • Tier2:
  • 1.8 PB MAID (Proware)
  • 3.5 PB Tape – IBM TS3500 with TS1140 tape drives, dual

robotics

  • 16Gbit redundant FC SAN, 10 Gbit Ethernet
  • 2x10 Gbit connection to CESNET2 network
  • 5 frontends, 2 HSM servers in HA mode

The Brno storage site

slide-12
SLIDE 12

6/3/2013 CESNET storage activities overview 12

The Brno storage site

Server room still under construction!

slide-13
SLIDE 13

6/3/2013 CESNET storage activities overview 13

Use cases

  • Long term storage with high redundancy

(geographical replicas)

– but no data curing

  • Institutional backups and/or archive

– individual backups supported as well

  • Input/output data for batch jobs
  • Exchange point of scientific data among

collaborations using standard file protocols

  • not suitable for

– on-line data processing – critical services

slide-14
SLIDE 14

6/3/2013 CESNET storage activities overview 14

Access to storage services

  • File access – access to the same namespace
  • our “core bussiness” – taking advantage of file-

level migration to lower tiers – NFSv4 with strong Kerberos authentication – SCP, SFTP (sshfs) – FTPS – rsync

slide-15
SLIDE 15

6/3/2013 CESNET storage activities overview 15

Access to storage services

  • Filesender
  • our first storage service – but not heavily advertised yet
  • up to 400 GB files (images of systems)
  • still stable, can resume! 
  • upload speed limit of ~100Mbit 
  • command line API?
slide-16
SLIDE 16

6/3/2013 CESNET storage activities overview 16

Access to storage services

  • Grid Storage Element

– dCache for ATLAS (LHC at CERN) and AUGER experiments – coupled with HSM to enable efficient storage of not heavily used data – 30TB on disks + 2x100TB on tapes – data are actually read here, not only written 

slide-17
SLIDE 17

6/3/2013 CESNET storage activities overview 17

Access to storage services

  • Block access

– iSCSI directly from arrays – no migration possible using HSM – speed (latency) issues – only in special and well justified cases (7TB so far) – but very stable service 

slide-18
SLIDE 18

6/3/2013 CESNET storage activities overview 18

User management

  • One user identity in the whole CESNET

e-Infrastructure (data storage group, NGI_CZ, video conferencing services...)

  • Divide and conquer in practice:

– users arranged in groups with an admin from the group – resource owners negotiate on service quantity and quality (quotas, migration policies...) with the group admin – the group admin decides who can join the group and what quality of service each particular user receives

  • Managed by Perun v3 system developed in collaboration

with other projects

slide-19
SLIDE 19

6/3/2013 CESNET storage activities overview 19

  • We require users to be well verified

– “somebody we trust has seen user’s ID” – managed by eduID.cz federation – then we create a Kerberos principal & password which forms his e-Infrastructure identity – decoupled from federation after registration

  • Grid Storage Element and iSCSI are special

User management

slide-20
SLIDE 20

6/3/2013 CESNET storage activities overview 20

Open questions

  • Inter site replication

– FTP backend for each HSM in the beginning

  • each site will migrate to another site as to lower tier
  • Long term goal: One namespace across all

three sites accessible by standard file protocols

– suggestions?

slide-21
SLIDE 21
  • 3.8 PB in pilot operation
  • mainly backups and archive data
  • 16.6 PB available soon
  • Broad range of user demands
  • broad range of technical challenges (aka

problems :-)

  • surprisingly many bugs & patches of

commonly used tools/apps!

6/3/2013 CESNET storage activities overview 21

Conclusion

slide-22
SLIDE 22

6/3/2013 CESNET storage activities overview 22

Thank you for your attention. Questions? Jiří Horký (jiri.horky@cesnet.cz)