KAGRA Data Tier Nobuyuki Kanda (Graduate School pf Science / NITEP - - PowerPoint PPT Presentation

▶
kagra data tier
SMART_READER_LITE
LIVE PREVIEW

KAGRA Data Tier Nobuyuki Kanda (Graduate School pf Science / NITEP - - PowerPoint PPT Presentation

Dec 16, 2022 261 likes •420 views

JGW-G1910278 KAGRA Data Tier Nobuyuki Kanda (Graduate School pf Science / NITEP , Osaka City U.) on behalf of the KAGRA collaboration at TAUP2019 8-14 September 2019 Toyama International Conference Cente 1 2. Distribution START mirror

slide-1
SLIDE 1

KAGRA Data Tier

Nobuyuki Kanda

(Graduate School pf Science / NITEP , Osaka City U.)

  • n behalf of the KAGRA collaboration

at TAUP2019 8-14 September 2019 Toyama International Conference Cente

1

JGW-G1910278

slide-2
SLIDE 2

KAGRA Data Tier

2

GOAL START

  • 1. Store

KAGRA data

  • 3. Mutual exchange of

data between global GW detectors, i.e. LIGO, Virgo

  • 2. Distribution

mirror or partial set of data Collaboration Users

slide-3
SLIDE 3

KAGRA Data Tier

Over View

3

Tier-0 Tier-0.5 Tier-2 Tier-1 Tier-3 Data Sharing with Other

  • bs.

faster (upstream) ↔ later (downstream) less amount ↔ larger amount

Event Alert to Counterparts/ Follow-ups in KAGRA Academia SINICA (Taiwan) KISTI (Korea)

Tier-1 archive Kashiwa Tier-0 archive Oversea GW experiments (low latency h(t)) Oversea GW experiments (bulk of data) Kamioka Osaka City U. Tier-0.5 for low latency Niigata U, RESCEU Detector (tunnel) raw data ~20MB/s raw data ~20MB/s +

  • Proc. data

~1MB/s raw + Proc. data

  • Proc. data +

partial raw data partial raw&proc. data set

  • Proc. data ~1MB/s

(option : raw data without permanent store)

end user sites

follow-ups / counterparts

…

socket (KAGRA DMG software) GRID GRID or alternative low latency (h(t)) Alart in GCN format

Nagaoka Tech.

alert data base

low latency h(t) low latency h(t)

(in preparation)

slide-4
SLIDE 4

KAGRA Data Tier

sharing

KAGRA data

raw data (file): full : full data, 32sec (will be 64sec ?) “frame format” full data is essentially ’raw’ data science : subset of full data trend : re-merged as long trend data for slow sampling datas, e.g. temperature monitors. Calibrated data (channel) : Main interferometer signal will be reconstructed as time series metric strain h(t) , that are named as “(channel definition)_C##” C0# : level 0 = primary level, generated from the interferometer control C1# : level 1 = low latency on-line generated, using calibration signal response and impulse response filters C2# : level 2 = off-line calculated with various/fine corrections as like long-time drift. It will be released later.

  • Proc. data (file) :

Lv.0 : C00 + data quality flags Lv.1 : C00 + C1# Lv.2 : C00 + C1# + C2#

4

DGS system

environmental monitors

Calibration system Transfer, Storage International GW network

full, science, trend proc

Interferometer

Kamioka KAGRA
 tunnel Kamioka surface building ICRR,
 U Tokyo (Kashiwa) LIGO Virgo

“frame” is GW experiments’ common format, that is packing time-series data.

slide-5
SLIDE 5

KAGRA Data Tier

History of Bulk Data Transfer

Main storage (2.5PB) will be able to cover O3. We are preparing to add newer system at the spring of 2021.

5

3500 3000 2500 2000 1500 1000 500 # of files / 100000 sec 2016/07/01 2017/01/01 2017/07/01 2018/01/01 2018/07/01 2019/01/01 2019/07/01 date

data cumulative (by March 2019) bKAGRA phase-I (2018/4/28-5/6) iKAGRA (2016/3/15-31, 4/11-25) raw ~450 TB ~6 TB ~3.5 TB proc. ~1.8 TB ~ 996 GB 763 GB

rate current (March 2019) bKAGRA phase-I (two servers) iKAGRA (two servers) amount ~50 GB/hour ~54 GB/hour ~20 GB/hour files (full+science) ~225 files/hour ~450 files/hour ~450 files/hour

iKAGRA ⇩ bKAGRA phase-I ⇩

slide-6
SLIDE 6

KAGRA Data Tier

Bulk Data Rate / Amount

6

25 20 15 10 5 Data Rate [MB/s] 2016/01/01 2017/01/01 2018/01/01 2019/01/01 2020/01/01 2021/01/01 Date 2.5 2.0 1.5 1.0 0.5 0.0 Cumulative Amount od Data [PB] Data Rate 'full' + 'science' + 'trend' 'full Cumulative Amount of Data 'full' + 'science' + 'trend' 'full' expected with 20MB/s

~16MB/s ~760TiB

slide-7
SLIDE 7

KAGRA Data Tier

KAGRA Data Tiers

7

LIGO-Virgo

latency : fast

KAGRA data tree (unfinished)

latency : slow

Tier-0

(primary)

Tier-0.5

(low latency analysis)

Tier-1

(full data mirror)

Analysis building (Hokubu-kaikan) RESCEU KAGRA site (tunnel) Niigata Nagaoka Tech. somewhere in Korea somewhere in Korea NCU somewhere in Taiwan

Tier-2

(analysis, development)

Tier-3

(development)

somewhere in Japan ICRR Kashiwa Osaka C.U. Academia SINICA KISTI (*) in preparation

(*) KISTI is in preparation for Tier-1 currently.

Bulk 32sec frame <1sec 3~10 sec ~day a few day / week low latency h(t)
 1sec frame ~3 sec (9 ~15 sec for LIGO/Virgo)

latency

slide-8
SLIDE 8

KAGRA Data Tier

Tier-0 : Main Data System at ICRR, U. Tokyo, Kashiwa

2.4 PiB (HHD) for

  • bservational data storage

gpfs file system 12.8 TFLOPS since March 2017

8

Storage: DDN SFA7700X + SS8460 Servers: HP ProLiant DL180 G9 HP ProLiant DL20 Gen9 HP ProLiant XL170 Gen9 Internal network: Infiniband FDR

slide-9
SLIDE 9

KAGRA Data Tier

Tier-0.5 : Osaka City Univ.

Tier-0.5 is receiving low latency data, and bulk data partially. OCU’s ‘ORION’ cluster consists of 920 cores (will be 1160 at the end of September 2019) 324 TB storage Scientific Linux 7.5 HT condor

9

slide-10
SLIDE 10

KAGRA Data Tier

Tier-1: Academia SINICA

Tier-1 is mirror site to keep all raw data (+some.) Academia SINICA, Taiwan ~20MB/s Transfer every 
 100000 sec

10

Approximately 20 MB/s. (a) recent (b) longer history Fig: Recent Tier-1 transfer rate to Academia SINICA by S.Haino

Also, KISTI (Korea) is in preparation for Tier-1 now.

slide-11
SLIDE 11

KAGRA Data Tier

Low Latency Data derived from CAL (Calibration Subsystem)

From CAL’s strain calibrated data h(t) i.e. C10 level calibrated data. 1sec frame transfer using ‘framelink’

The software on the OS is sending direct to Kashiwa system, but the physical network path is bit complicated.

The latency is ~3 sec including impulse filter calculation, format etc.

11

坑内から低遅延で観測信号を、柏キャンパスにおける サーバへ送信する。 この際、 坑内に置くキャリブレーション用の新サーバから直接に へ送信で きることが必要になる。 現状では， はプライベートアドレスを持つ サーバである。 ただし， の により，外部ネットワークから 特定のグローバルアドレスでアクセ ス可能。 神岡（北部会館）における ルーター 図中 に、内部 から への直接のルーティングを行わさせる。 利点： ルーティングを司るのは だけでよく、シンプル。 難点： セキュリティ上の注意が必要

柏キャンパス:KAGRA主データ装置 柏キャンパス:iKAGRAデータシステム KAGRA北部会館 KAGRA坑内 VPN Infiniband SINET andromeda-01 storage pegasus-01 VPN2 HUB3 perseus-01 perseus-02 HUB2 aldebaran VPN1 hyades-01 HUB1 hyades-02 new sever FW

柏キャンパスにおける ルーター 図中 に、内部 から への 直接のルーティングを行わさせる。 利点： セキュリティ上の注意は必要だが、 と の接続は の内部なので案 （１）に比較すれば安全 の外で通過する機器を少なくできる（ を出た後は、柏キャン パス内の で繋げられる） 難点： ルーティングを行うのに、 のそれぞれの設定はどうなるのか？

柏キャンパス 主データ装置 柏キャンパス データシステム 北部会館 坑内

一部のルーティングの改変の必要性

改変案（１） 改変案（２）

KAGRA tunnel Hokubu-Kaikan (surface building) ICRR Kashiwa campus iKAGRA data system KAGRA main system

CAL’s real time h(t)

: A software developed by

  • LIGO. Socket connection to

push/pull the data.

slide-12
SLIDE 12

KAGRA Data Tier

Toss the data to analysis pipelines

Low latency data have to toss the data analysis pipelines. We connect (and are preparing)

  • ‘framelink’ shared memory
  • dump to files

12

Kashiwa (ICRR)

Kashiwa dedicated server for data sharing Osaka City U. Tier-0.5 for low latency RESCEU CBC low latency h(t) by ‘framelink’ ‘framelink’

LIGO, Virgo

Calibration server (in tunnel)

CBC pipeline cluster shared memory

files (1sec buffering)

Kashiwa main system

main storage

Burst pipeline cluster

frame merge

‘framelink’

(plan)

(working for installation)

slide-13
SLIDE 13

KAGRA Data Tier Kashiwa

Data exchange between LIGO,Virgo and KAGRA

Low Latency h(t) : 1sec frame with a latency of second order Exchange (sharing) is connected and is working.

(Thanks for LIGO/Virgo great cooperation!)

with ‘framelink’

Latencies are depend on network condition.

LHO, LLO -> Kashiwa : 6~14 sec Virgo -> Kashiwa : 10~16 sec KAGRA tunnel -> Kashiwa : ~3 sec (Note : These latency include h(t) reconstruction calculation.)

Bulk data exchange is also in preparation.

13

KAGRA tunnel, Kamioka KAGRA Kashiwa main system main system (2.5 PB storage) data sharing Server Caltech server for low latency LLO LHO Virgo ….(many systems)…. shared memory shared memory framelink push/pull CBC burst Osaka C.U.

slide-14
SLIDE 14

KAGRA Data Tier

Low latency data are dumped in files.

14

KAGRA Kashiwa main system data sharing Server shared memory (1sec frame) K L,V main system (2.5 PB storage) file (4096 sec) 1sec frame 1sec frame 1sec frame 1sec frame …

2019-06-27T02:15:17 start for detector K1 2019-06-27T02:15:17 merge from 1245632809 to 1245636905 shift 0 2019-06-27T03:23:33 merge from 1245636905 to 1245641001 shift 0 2019-06-27T04:31:49 merge from 1245641001 to 1245645097 shift 0 …

  • ----------Parameters used--------------

Input Files: K-K1_llhoft-1245808937-4096.gwf First frame : 0 0 (GPS=0.0) Last frame : 2147483647 2147483647 (GPS=2147483647.0) Debug level : 1 Dump all Frame info

  • K-K1_llhoft-1245808937-4096.gwf 1245808937 4096 0 0

File(s) summary: 4096 Frames in the requested time range (0 to 10000000000 (GPS)) First frame start at:1245808937 (UTC:Sat Jun 29 02:02:00 2019) length=1.00s. Last frame end at:1245813033 (UTC:Sat Jun 29 03:10:16 2019) length=1.00s.

$ ls -lh ...

  • rw-r--r-- 1 ll dmg 1.4G 6月 27 14:40 H-H1_llhoft-1245645102-4096.gwf
  • rw-r--r-- 1 ll dmg 1.5G 6月 27 15:48 H-H1_llhoft-1245649198-4096.gwf

...

  • rw-r--r-- 1 ll dmg 66M 6月 27 12:23 K-K1_llhoft-1245636905-4096.gwf
  • rw-r--r-- 1 ll dmg 66M 6月 27 13:31 K-K1_llhoft-1245641001-4096.gwf
slide-15
SLIDE 15

KAGRA Data Tier

Summary and To do for newer future

  • 1. KAGRA Data Store

Transfer from the KAGRA site to our main server is established and well working. 2.4 PiB storage provides for these 5 years as like. ( and newer system will adjoin the current system for continuous and seamless data storage.)

  • 2. KAGRA Data Tier

Some Tier sites are already working. Some are in preparation.

  • 3. Data Sharing with the Global GW detectors

i.e. with LIGO and Virgo Low latency data sharing have been working well. We have to fix bulk data exchange.

15