DSS Data & Storage Services CERN Lustre Evaluation and Storage - - PowerPoint PPT Presentation
DSS Data & Storage Services CERN Lustre Evaluation and Storage Outlook Tim Bell Arne Wiebalck HEPiX, Lisbon 20 th April 2010 CERN IT Department CH-1211 Genve 23 Switzerland www.cern.ch/i t DSS Agenda Lustre Evaluation Summary
CERN IT Department CH-1211 Genève 23 Switzerland
www.cern.ch/it
CERN IT Department CH-1211 Genève 23 Switzerland
www.cern.ch/it
Internet Services
CERN Lustre Evaluation and Storage Outlook - 2
CERN IT Department CH-1211 Genève 23 Switzerland
www.cern.ch/it
Internet Services
– CERN Advanced STORage Manager (CASTOR) – 23 PB, 120 million files, 1’352 servers
– Analysis of the experiments’ data – 1 PB access with XRootD
– >150 projects – Experiments’ code (build infrastructure) – CVS/SVN, Indico, Twiki, …
– 20’000 users on AFS – 50’000 volumes, 25 TB, 1.5 billion acc/day, 50 servers – 400 million files
3
CERN IT Department CH-1211 Genève 23 Switzerland
www.cern.ch/it
Internet Services
– Life Cycle Management – Backup – Strong Authentication – Fault-tolerance – Acceptable performance for small files and random I/O – HSM interface
– Replication – Privilege delegation – WAN access – Strong administrative control
– See the results of the HEPiX FSWG
4
CERN IT Department CH-1211 Genève 23 Switzerland
www.cern.ch/it
Internet Services
– Not OK: no support for live data migration, Lustre or kernel upgrades, monitoring, version compatibility
– OK: LVM snaphots for MDS plus TSM for files worked w/o problems
– Almost OK: Incomplete code in v2.0, full implementation expected Q4/2010 or Q1/2011
5
CERN IT Department CH-1211 Genève 23 Switzerland
www.cern.ch/it
Internet Services
– OK: MDS and OSS failover (we used a fully redundant multipath iSCSI setup)
– Almost OK: Problems when mixing small and big files (striping)
– Not OK: Not supported yet, but under active development
6
CERN IT Department CH-1211 Genève 23 Switzerland
www.cern.ch/it
Internet Services
– Not OK: not supported (would help with data migration and availability)
– Not OK: not supported
– Not OK: may become possible once Kerberos is fully implemented (cross-realm setups)
– Not OK: pools not mandatory, striping settings cannot be enforced
7
CERN IT Department CH-1211 Genève 23 Switzerland
www.cern.ch/it
Internet Services
coupling
– Recovery case
– Some of the requested features are on the roadmap since years, some are simply dropped
purpose file system
– Most of our requested features are not needed in the primary customers‘ environment
8
CERN IT Department CH-1211 Genève 23 Switzerland
www.cern.ch/it
Internet Services
based storage consolidation at CERN
roadmap, so it’s worthwhile to keep an eye on it
https://twiki.cern.ch/twiki/pub/DSSGroup/LustreEvaluation/CERN_Lustre_Evaluation.pdf
9
CERN IT Department CH-1211 Genève 23 Switzerland
www.cern.ch/it
Internet Services
CERN Lustre Evaluation and Storage Outlook - 10
CERN IT Department CH-1211 Genève 23 Switzerland
www.cern.ch/it
Internet Services
– 28 PB tape used currently at CERN – 20 PB/year expected
– Warranty expiry on disk servers – Tape drive repacking to new densities
– TOTAL space, not new data volume recorded – Interconnect between source and target – Metadata handling overheads per file
– Conflicts between user data serving and refresh
Storage Outlook - 11
CERN IT Department CH-1211 Genève 23 Switzerland
www.cern.ch/it
Internet Services
Storage Outlook - 12
5000 10000 15000 20000 25000 30000 35000 Nov-08 Dec-08 Jan-09 Feb-09 Mar-09 Apr-09 May-09 Jun-09 Jul-09 Aug-09 Sep-09 Tapes Repacked
Total Tapes Repacked
resources as LHC data recording
planning for sites with large archives
CERN IT Department CH-1211 Genève 23 Switzerland
www.cern.ch/it
Internet Services
Storage Outlook - 13
2 4 6 8 10 12 14 16 2009 2010 2011 2012 2013 2014 2015 Maximum capacity (TB)
Disk and Tape Capacity Projections
Pessimistic disk projection Optimistic disk projection Pessimistic tape projection Optimistic tape projection
a tape based solution?
CERN IT Department CH-1211 Genève 23 Switzerland
www.cern.ch/it
Internet Services
Storage Outlook - 14
CERN IT Department CH-1211 Genève 23 Switzerland
www.cern.ch/it
Internet Services
Storage Outlook - 15
Head Node Disk Array Disk Array Disk Array Disk Array Disk Array Disk Array Disk Array Disk Array Head Node
CERN IT Department CH-1211 Genève 23 Switzerland
www.cern.ch/it
Internet Services
Storage Outlook - 16
20 40 60 80 100 120 140 160 180 Tape with just in time repack Storage in a box Storage in a rack Storage in a rack with backup Normalised Cost (Current=100)
5 year archive cost
CERN IT Department CH-1211 Genève 23 Switzerland
www.cern.ch/it
Internet Services
Storage Outlook - 17
20 40 60 80 100 120 140 160 2011 2012 2013 2014 2015 kWatts
Annual Power Consumption
Nearline Repack Just in Time Storage in a rack Storage in a rack with backup
CERN IT Department CH-1211 Genève 23 Switzerland
www.cern.ch/it
Internet Services
– Corruptions – Scrubbing
– Fail-over testing
– Disk spin down / up
– 40 days to drain at gigabit ethernet speeds
– Monitoring, Repair, Install
– How much effort is it to run
Storage Outlook - 18
CERN IT Department CH-1211 Genève 23 Switzerland
www.cern.ch/it
Internet Services
CERN Lustre Evaluation and Storage Outlook - 19
CERN IT Department CH-1211 Genève 23 Switzerland
www.cern.ch/it
CERN IT Department CH-1211 Genève 23 Switzerland
www.cern.ch/it
Internet Services
Presentation Title - 21