High Performance Experiment Data Archiving with gStore Chep 2012, - - PowerPoint PPT Presentation

▶
high performance
SMART_READER_LITE
LIVE PREVIEW

High Performance Experiment Data Archiving with gStore Chep 2012, - - PowerPoint PPT Presentation

Apr 07, 2024 263 likes •635 views

High Performance Experiment Data Archiving with gStore Chep 2012, New York May 21, 2012 Horst Goeringer, Matthias Feyerabend, Sergei Sedykh H.Goeringer@gsi.de Overview 1. Introduction GSI and FAIR 2. How gStore works 3. gStore SW and HW

slide-1
SLIDE 1

High Performance Experiment Data Archiving with gStore

Chep 2012, New York May 21, 2012

Horst Goeringer, Matthias Feyerabend, Sergei Sedykh

H.Goeringer@gsi.de

slide-2
SLIDE 2

Chep12 New York gStore - GSI Data Archive System 2

Overview

1. Introduction GSI and FAIR 2. How gStore works 3. gStore SW and HW components 4. some features 5. gStore usage

  • gStore and lustre
  • nline data storage from running experiments

6. Outlook

slide-3
SLIDE 3

Chep12 New York gStore - GSI Data Archive System 3

Budget: 106 Mio. € (90% Bund,10% Hessen) External Scientific Users: 1200 Employees: 1100 Large Scale Facilities: Accelerators and Experiments

GSI Helmholtzzentrum für Schwerionenforschung – Center for Heavy Ion Research

slide-4
SLIDE 4

Chep12 New York gStore - GSI Data Archive System 4

Research Areas at GSI

Plasma Physics (5%)

Hot dense plasma Ion-plasma-interaction

Materials Research (5%)

Ion-Solid-Interactions Structuring of materials with ion beams

Accelerator Technology (10%)

Linear accelerator Synchrotrons and storage rings

Biophysics and radiation medicine (15%)

Radiobiological effect of ions Cancer therapy with ion beams

Nuclear Physics (50%)

Nuclear reactions up to highest energies Superheavy elements Hot dense nuclear matter

Atomic Physics (15%)

Atomic Reactions Precision spectroscopy of highly charged ions

slide-5
SLIDE 5

Chep12 New York gStore - GSI Data Archive System 5

GSI-today

  • all kinds of ions
  • max. 90% speed of light

GSI-tomorrow / FAIR

  • new isotopes
  • Anti-Protons
  • 10.000 times more sensitive
  • higher speed

FAIR – Facility for Antiproton and Ion Research

100 m

slide-6
SLIDE 6

Chep12 New York gStore - GSI Data Archive System 6

FAIR – Facility for Antiproton and Ion Research

slide-7
SLIDE 7

Chep12 New York gStore - GSI Data Archive System 7

gStore: storage view

slide-8
SLIDE 8

Chep12 New York gStore - GSI Data Archive System 8

gStore: software view two main parts:

  • 1. TSM: Tivoli Storage manager (IBM)

– handles automatic tape libraries (ATL)

and tape drives – all devices supported by TSM also supported by gStore – utilized by GSI software via TSM API

slide-9
SLIDE 9

Chep12 New York gStore - GSI Data Archive System 9

gStore: software view

2. GSI Software (>100,000 lines of C-code): – interfaces to users (command, API) – interface to TSM (API) – entry servers – data mover servers – read/write cache managers:

  • meta data management
  • cache file locking
  • space management
  • data mover selection (load balancing)
slide-10
SLIDE 10

Chep12 New York gStore - GSI Data Archive System 10

gStore: Hardware Status

IBM 3584-L23 tape library (ATL)

  • 8 IBM 3592-E07 tape drives (SAN)

– 250 MB/s read/write per drive – 4 TB/medium uncompressed

  • 8.8 PB overall data capacity

– ~1 PB used

  • 40 €/TB media costs:

– add 60% for library/drives => – for reliable long term archiving: no similar inexpensive alternative for tape

  • really green IT
slide-11
SLIDE 11

Chep12 New York gStore - GSI Data Archive System 11

gStore: Hardware Status

IBM 3584-L23 tape library (ATL)

  • copies of raw experiment data
  • 4 IBM 3592-E06 tape drives (SAN)

– 160 MB/s read/write per drive – 1 TB/medium uncompressed

  • 1.2 PB overall data capacity

– 200 TB used

  • in different building: enables disaster

recovery

slide-12
SLIDE 12

Chep12 New York gStore - GSI Data Archive System 12

gStore: Hardware Status

currently 17 data movers:

  • Suse Linux
  • 3 – 20 TB disk cache
  • 4 Gb SAN connection to ATL
  • Ethernet connection to clients:

– 10 Gb (9x, limit 40 Gb switch) – 1 Gb (8x)

slide-13
SLIDE 13

Chep12 New York gStore - GSI Data Archive System 13

gStore: Hardware Status data movers overall:

  • 200 TByte disk cache (read/write/DAQ)
  • max. I/O bandwidth:

– disk cache <-> tape: 2 GByte/s – disk cache <-> clients: 5 GByte/s

slide-14
SLIDE 14

Chep12 New York gStore - GSI Data Archive System 14

gStore: how it works

. . .

S A N

. . . DM 1 DM j . . . client k client 2 client 1 cache admin query entry server data control tapes/ATLs

ATL1 tape2 ATL1 ATLn tape1

cache cache disk disk TSM Server2 Server1 TSM

tape i

data mover clients

slide-15
SLIDE 15

Chep12 New York gStore - GSI Data Archive System 15

gStore: design principles gStore:

  • reliable long-term archive storage
  • high-performance access
  • fully scalable in data capacity
  • fully scalable in I/O bandwidth
slide-16
SLIDE 16

Chep12 New York gStore - GSI Data Archive System 16

gStore: some features

  • 64 bit servers
  • 32/64 bit clients

– command clients – API clients

  • recursive file operations

– wildcards in file names – file lists

slide-17
SLIDE 17

Chep12 New York gStore - GSI Data Archive System 17

gStore: some features

  • large file transfer with single command

– performance increase by parallelization

  • staging big file sets:

– files on different tapes: copy in parallel to different data movers

  • decreases staging time
  • enables highly parallel access

– files on same tape: distribute to several data movers (sequentially)

  • enables highly parallel access
  • important as media size increases

– impossible for user (no tape info)

slide-18
SLIDE 18

Chep12 New York gStore - GSI Data Archive System 18

gStore Usage

TB transferred average MB/s

  • no. of files

transferred

Jan 1 – May 13, 2012

710 61 880,000 average day 5.3 61 6,567 top day (Aug 12,

2011)

46.7 540 21,600

slide-19
SLIDE 19

Chep12 New York gStore - GSI Data Archive System 19

gStore and lustre lustre: GSI online mass storage

  • ~ 3 PB size
  • small experiments: gStore cache, no lustre

data transfers gStore <-> lustre:

  • gStore data movers <-> lustre OSTs

– up to 500 MB/s (single file) – max bandwidth 5 GB/s

  • or tape <-> lustre

– up to 250 MB/s (single file) – max bandwidth 2 GB/s

slide-20
SLIDE 20

Chep12 New York gStore - GSI Data Archive System 20

gStore: online data storage

On-line data storage: constant, continous data streams from data acquisition over long time ranges

  • many data streams in parallel
  • e.g. Hades experiment: 16 data streams
  • distribution to DMs: load balancing
  • undisturbed by offline business
  • fast data availability in lustre
  • for online monitoring & analysis
slide-21
SLIDE 21

Chep12 New York gStore - GSI Data Archive System 21

gStore: online data storage

storage order:

1.gStore write cache 2.optionally copy to lustre: 3.finally migration to tape

  • if preset cache fill level reached
  • verall bandwidth:

– 500 MB/s: with full copy to lustre – 1 GB/s: no copy to lustre

slide-22
SLIDE 22

Chep12 New York gStore - GSI Data Archive System 22

gStore: online data storage two online copy modes to lustre:

1. parallel copy

  • data buffer level
  • problem: lustre latencies
  • > delay of DAQ read-out

2. sequential copy

  • file level
  • storage to write cache independent of

lustre

slide-23
SLIDE 23

Chep12 New York gStore - GSI Data Archive System 23

gStore: online data storage Hades experiment march/april 2012:

  • 5 weeks beam time
  • 16 data streams in parallel
  • acquisition rate ~100 MB/s

– storage to write cache – copy to lustre (all files) – migration to tape

  • verall ~200 TB of data
  • in parallel up to 3 add. experiments (~MB/s)
  • handled by gStore without problems
slide-24
SLIDE 24

lustre 3 PB gStore Data Movers 200 TB Buffer Storage

5 GB/s

gStore 2012

Tape Robot capacity 8.8 PB expandable: 50 PB

  • ffline

Clients

  • nline DAQ

Clients

0.5 GB/s 2 GB/s

slide-25
SLIDE 25

Chep12 New York gStore - GSI Data Archive System 25

gStore: Outlook current/future projects:

1.

  • ptimal utilization of available bandwidth:

automatic parallelization of large data transfers (single command) – for staging already done

  • all data transfers on server side

– senseless for some client storage, e.g.

  • desktops
  • verloaded file/group servers
  • servers with small network bandwidth
slide-26
SLIDE 26

Chep12 New York gStore - GSI Data Archive System 26

gStore: Outlook

next: parallelize transfers gStore <-> lustre

  • lustre: powerfull client system
  • no. of parallel processes limited by
  • no. of available data movers
  • no. of available tape drives

writing to lustre: effective lustre load balancing reading from lustre: file distribution on lustre OSTs depends on history

slide-27
SLIDE 27

Chep12 New York gStore - GSI Data Archive System 27

gStore: Outlook

future projects:

  • 2. HSM for lustre

– future of EOFS and lustre GPL? – not yet under investigation

slide-28
SLIDE 28

Chep12 New York gStore - GSI Data Archive System 28

gStore: Outlook

future projects:

  • 3. preparation for FAIR (Start 2018)

– storage situation 2018? – data growth: 33 PB/year (2018) – current ATL: expandable to 50 PB (E07) – with next gen. (E08): expect >=100 PB

  • E06->E07: was factor 4!

– data bandwith: need factor >10

slide-29
SLIDE 29

Chep12 New York gStore - GSI Data Archive System 29

gStore: Outlook

In the past 15 years at GSI we mastered similar increases in data capacity and bandwidth Technical progress helped in the past and will help also in the future gStore is designed for scalability

slide-30
SLIDE 30

Chep12 New York gStore - GSI Data Archive System 30

gStore: how it works

slide-31
SLIDE 31

Chep12 New York gStore - GSI Data Archive System 31

gStore: how it works

slide-32
SLIDE 32

Chep12 New York gStore - GSI Data Archive System 32

gStore: how it works

slide-33
SLIDE 33

Chep12 New York gStore - GSI Data Archive System 33

gStore: Usage Profiles mainly three use cases:

1. transfer large amounts of data, e.g. between

  • lustre
  • group/file servers
  • local disks

and gStore read/write cache, e.g.

  • stage actions to prepare a data analysis
  • archive actions after a data analysis
  • to handle lack of lustre space
slide-34
SLIDE 34

Chep12 New York gStore - GSI Data Archive System 34

gStore: Usage Profiles

  • 2. parallel transfer of many single files

between farm nodes

  • local disks
  • local memory (API)

and gStore read/write cache:

  • for data analysis
  • smaller GSI experiments not needing

lustre

slide-35
SLIDE 35

Chep12 New York gStore - GSI Data Archive System 35

gStore: Usage Profiles

  • 3. On-line data storage: constant,

continous data streams from running experiments over long time ranges

  • many data streams in parallel
  • e.g. Hades experiment: 16 data streams
  • distribution to DMs: load balancing
  • undisturbed from offline business
  • fast data availability in lustre
  • for online monitoring & analysis
slide-36
SLIDE 36

Chep12 New York gStore - GSI Data Archive System 36

gStore: online data storage

some more features of online lustre copy:

  • fraction of files: selectable by user (0 to all)
  • ptionally: new lustre subdir after n files
  • subdir naming conventions