KAGRA data management and analysis N.Kanda, on behalf of KAGRA - - PowerPoint PPT Presentation
1 JGW-G1706749-v2 KAGRA data management and analysis N.Kanda, on behalf of KAGRA collaboration The 3rd KAGRA International Workshop (KIW3) May 21-22, 2017 Academia Sinica (NTU campus), Taipei 2 KAGRA data related subgroups CAL
N.Kanda, on behalf of KAGRA collaboration The 3rd KAGRA International Workshop (KIW3) May 21-22, 2017 Academia Sinica (NTU campus), Taipei
1
JGW-G1706749-v2
2
Interferometer
computing environment
storage CPUs softwares Data Results
to manage and to operate ‘KAGRA data Tier’
+ Development of softwares for these operations
mirror sites and end users
leader : N.Kanda sub-leader : K.Oohara effort members : S.Haino, K.Hayama, Y.Inoue, Y.Itoh, M.Kaneyama, G. Kang, C-Y Lin, O.Miyakawa, A.Miyamoto, S.Miyoki, S.Oh, K.Sakai, Y.Sasaki, E. J. Son, H.Tagoshi, H.Takahashi, K.Tanaka, S.Ueki, T.Yamamoto, T.Yokozawa, H.Yuzurihara
3
4
Tier-0 Tier-0.5 Tier-2 Tier-1 Tier-3 Data Sharing with Other
faster (upstream) ↔ later (downstream) less amount ↔ larger amount
Overview of KAGRA data flow
Event Alert to Counterparts/ Follow-ups in KAGRA Mirrors: Academia SINICA (Taiwan) KISTI (Korea)
Tier-1 archive Kashiwa Tier-0 archive Oversea GW experiments (low latency h(t)) Oversea GW experiments (bulk of data) Kamioka Osaka City U. Tier-0.5 for low latency Niigata U, RESCEU Detector (tunnel) raw data ~20MB/s raw data ~20MB/s +
~1MB/s raw + Proc. data
partial raw data partial raw&proc. data set
~1MB/s
(option : raw data without permanent store)
end user sites
alert data base follow-ups / counterparts
… …
socket (KAGRA DMG software) GRID GRID or alternative low latency (h(t)) Alart in GCN format
Nagaoka Tech.
All data sets are in ‘frame’ format.
RAW
Data files derived from DGS’s frame writer Tier-0 : primary fully and permanently archive at Kashiwa Tier-1 : mirror of Tier-1 at Academia SINICA (Taiwan) and KISTI (Korea)
KISTI has iKAGRA data now. It will be extend to ‘full’ data in near future.
Tier-0.5 : low latency transfer to OCU (Osaka) same to Tier-0, but will not keep full sets permanently.
Proc.
Processed data set that consists of calibrated strain h(t) and some related channels for GW event analysis. at Tier-0, 1, 0.5 at Tier-2 : RESCEU (Tokyo), Niigata, +more future
——— iKAGRA data
2016/3/25-31 : lock duration 101.93 hours 2016/4/11-25 : lock duration 296.18 hours
~7.5 TB of raw data, 756 GB of proc data
5
6
HUB 10GIFO, DSG frontend Environmental Monitor (EPICS layer)
IP address low latency hostname IP address frame writer k1fw1 IP address frame writer k1fw0 IP address NDS hostname IP address DetChar hostname HUB 1G monitor hosts in control room IP address data concentrator hostname 20TB 20TB IP address login / job man. taurus-01 IP address login / job man. taurus-02 disk array Infiniband SW IP address data transfer aldebaran IP address calculation pleiades-01 IP address calculation pleiades-02 IP address calculation pleiades-03 IP address calculation pleiades-04 IP address primary data server k1dm0 / hyades-0 IP address primary data server k1dm1 / hyades-1 HUB 10G HUB 10G / 1GDedicated optical fiber 4.5 km, tunnel <-> surface build.
IP address VPN gwave_kamioka Infiniband SW IP address MDS crab-mds-01 IP address MDS crab-mds-02 IP address OSS crab-oss-01 IP address OSS crab-oss-02 disk array MDT disk array OST disk array OST IP address VPN gwave_kashiwa HUB 10G / 1G IP address login server perseus-01 IP address login server perseus-02 Infiniband SW IP address MDS/OSS algol-01 disk array MDT/OST IP address ICRR interoperable computer systemSINET
iKAGRA data system overview
Drawn by N.Kanda last update : 2014/8/19
KAGRA Tunnel site Surface building at Kamioka ICRR, U Tokyo. (Kashiwa campus)
x２sets frame writer
200 TiB luster file system mid spool of the data, HUB of data transfer Tier-0 archive iKAGRA :100TiB bKAGRA : 3PB
KAGRA’s VPN connection
We developed the transfer software
We writing the code fully. (No black-box!) Using Linux standard tools as possible as.
Speed and Stability is good for bKAGRA requirement.
bKAGRA requirement : >20MB/s Performance : >40MB/s between Kamika and Kashiwa/Osaka
7
最頻値は以下の値となった（ 幅：
転送遅延のヒストグラム
—> Performance test for 1sec data chunk from Kamioka to Osaka (by K.Sakai)
for the development of KAGRA analysis for low-latency search
8
# of cores Storage [TB] 200 400 600 800 Fiscal Year 2012 2013 2014 2015 2016
CPU [cores] (for calculation) Storage [TB] (/data, /home)
cluster system CPU/Storage
KAGRA’s VPN
9
KAGRA DMG VPN (Kamioka-Kashiwa) Tier-0 archive ICRR VPN (Kashiwa) Prototype PC GRID server (Tier-1) Checksum verification SSH access over VPN GRID transfer ASGC (Taiwan) drawing by S.Haino
10
VPN
11
Storage: DDN SFA7700X + SS8460 Servers: HP ProLiant DL180 G9 HP ProLiant DL20 Gen9 HP ProLiant XL170 Gen9 Internal network: Infiniband FDR
(working) VPN router KAGRA Private Network Internal Network (working) ICRR shared computer (working) iKAGRA data transfer and storage system
The internat (SINET)
Kamioka⇐
Kashiwa Campus
Hi-Speed Network Login Server Data Reciever Login Server (Disk Array) Mass Storage Data Sender
Compute Server Gateway Server
KAGRA Main Data Storage (System A)
Fire wall
Gateway Server Compute Server Compute Server Compute Server Compute Server Compute Server Compute Server Compute Server
2.4 PiB data storage 100 TiB for iKAGRA era
12
Extend in 2017 push==> (current) push==>
all data
spool ~100TiB
==> pull
partial data (h(t), latest raw) h(t)
13
Site Capacity Main Usage Kamioka (surface) 200 TiB spool, On-site analysis Kashiwa 100 TiB + 2.4 PiB iKAGRA data storeage Osaka City Univ. 304 TiB CBC, Burst, low latency search RESCEU 80 TiB CW Niigata Univ. 77 TiB
Academia SINICA (Taiwan Group) 220 TiB Tier-1 mirroring, etc. KISTI-GSDC (Korea Group) 150TB (800TB in 2018 and 2019) Tier-1, Detector characterization （total） ~ 3.5 PiB
（ ） Site Capacity Main Usage ICRR computer for cooperative researches proc data, event analysis KEK computer center (not yet decided)
まとめ
We also cooperate with:
14
“M31” is system-A’s login servers Osaka Kashiwa Tokyo, RESCEU “KANBAI(寒梅)”
Search for Gravitational Waves in KAGRA data Extract Science of GWs Development of Tools for them (=Software, Computers, etc.)
LIGO (including Inidia), Virgo, GEO J-GEM (Japanese collaboration for Gravitational-wave Electro-Magnetic
follow-up)
Neutrino observations etc.
We have been cooperated with these partners in these several years.
15
16
2 Chief:&H.Tagoshi SubJchiefs:&Y.Itoh,&H.Takahashi Core&members:&N.Kanda,&K.Oohara,&K.Hayama Korean&subgroup Leader:&Hyung Won&Lee
Osaka&City&Univ(13)&:&&N.Kanda,&H.&Tagoshi,&T.Narikawa,&T.&Yokozawa,& M.&Kaneyama,&N.&Uchikata,&S.&Tsuchida,& H.Yuzurihara,&T.&Yamamoto,&K.Tanaka, A.&Miyamoto,&T.&Kaji,&Y.&Kitaoka RESCEU(6)&&&&&&&&&&&&&:& Y.&Itoh,&K.&Eda,&J.&Yokoyama,&S.&Morisaki,&R.&Tsukada K.&Cannon ICRR&(1)&&&&&&&&&&&&&&&&&: K.Hayama Nagaoka&Tech&(4)&: H.Takahashi,&K.&Sakai,&Y.&Sasaki,&S.&Ueki Niigata&Univ (4)&&: K.Oohara,&Y.Hiranuma,&T.&Wakamatsu,&Y.&Watanabe Toyama&Univ (3)&&:& S.&Hirobayashi,&Dongbao Jia,&K.&Miyake NAOJ&&(3)&&&&&&&&&&&&&&:& N.&Ohishi,&A.&Shoda,&Y.&Fujii ISM&&&(1)&&&&&&&&&&&&&&&&: S.&Mano OIT&&&&(2)&&&&&&&&&&&&&&&&:&&H.&Shinkai,&S.&Yamamoto UWM&(1)&&&&&&&&&&&&&&:&&K.&Ueno CUHK(2)&&&&&&&&&&&&&&&:&&T.G.F&Li,&R.&Yeung Inje Univ.&:&Hyung Won&Lee Jeongcho Kim KASI:&&Chunglee Kim
2016/07/26
Total:&43&(Undergraduate&students&are¬&included)&
DAS viewgraphs are drawn by Tagoshi.
17
3
Projects Members
CBC&offline
Yuzurihara, Tagoshi,&(Ueno)
CBCJPE&in&KAGALI
H.W.&Lee,&Jeongcho Kim, Chunglee Kim&
Burst
Hayama
CW
Eda,&Itoh
Radiometry&for&CW&
K.Tanaka,&Kanda,&Itoh
CBCJPE&on&injected signals
Narikawa, (Tagoshi)
HHT&on&injected&signals
Ueki, Takahashi,&Oohara,&Kanda,&Yokozawa
HW&injection&signals
Yokozawa,&……
Gaussianity,& CBC&nonJGaussian&triggers,…
Kitaoka,&Sasaki,&Kanda,&Tagoshi,&…
CBC
Method: Frequency domain, matched filter - chi square analysis Mass range: 1-3Msun, no spin Template: TaylorF2 Template bank: Hexagonal template placement method
Burst
Search method : Excess power based method Event reconstruction :clustering based on mathematical morphology
Continuous
Continuous waves from 62 isolated known pulsars Analysis frequency range : 50 Hz – 1,000 Hz Search method : F-statistic
18
Horizon&distance&of&CBC&by&iKAGRA
6
10 0 10 1 10 2
Component mass [Msun]
10 0 10 1
[pc] Horizon distance of iKAGRA
ρ=8 fmin=5Hz m1=m2
(1.4,1.4)Msun =>3.2pc (10,10)Msun =>12.2pc
19
9
This&is&often&called&"Hardware&Injection&test". This&is&done&by&shaking&mirrors.& (The&signals&injected&are¬&real&gravitational&wave&signals!)&
Laser
Injected waveforms (Sine Gaussian)
○Sine Gaussian (test run) Q = 20(constant), f0 = [30, 100, 300, 1000]Hz, A = various XY differential, X/Y arm only(30,100Hz), 1count = 7.5e-13[strain] time[index] count count time[index] 100Hz SG 33Hz SG ○Sine Gaussian (injection run) Q = 20(constant), f0 = [33, 100, 333], A = 3000/f^2. 1500/f^2, XY differential
6Injected waveforms (Supernova)
○Supernova waveform Provided Y.Suwa(YTIP, Kyoto), 2D Newtonian numerical simulation (KAGRA original) t=0 : Start time of gravity collapse Progenitor mass = 11.2M , different core rotation, matter eject memory effect 0.0 π rad/s 0.2 π rad/s 0.5 π rad/s 1.0 π rad/s No core rotation strong SASI motion(from entropy motion) 50M 80M 6Models(2s) × 3 amplitude scale 18 waveforms injection
7
Injected waveforms (CBC)
○CBC waveform(prepare waveforms by H.Yuzurihara, H.Tagoshi)
mass combination : 20-20, 25-25, 30-30, 36.3-28.6 and 50-50M , no spin 3 amplitude(SNR) scales, low frequency cutoff : 10Hz To connect smoothly between 0 and waveform, we used error function
1.35-1.35 M , 2 amplitude(Scales) (KAGRA original) 20-20M “GW150914” 1.35-1.35M 36.3-28.6M⊙ Total 17 waveforms are injected
8
Hardware injection studies on
20
13
21
14
22