Grid Activities at High Energy Accelerator Research Organization - PowerPoint PPT Presentation

Grid Activities at High Energy Accelerator Research Organization (KEK) May 04. 2006 KEK Computing Research Center Setsuya Kawabata ISGC2006 at ASGC

Outline 1. LCG Testbed � Collaboration for Grid deployment with ICEPP � Testbed 2. Grid CA at KEK 3. Belle experiment � Status of Experiment � Data analysis � Activity for Grid � New B Factory Computer System 4. Deployment Plan � New Central Information System 5. Lattice QCD Grid � New Super Computer System at KEK 6. Collaboration with NAREGI 7. Summary ISGC2006 on May 04 2006 S.Kawabata: "Grid Activities at KEK" 2

1. LCG Test bed � Since Nov. 2001, KEK and ICEPP have collaborated to study and experience the function of the Regional center facility. � experienced NorduGrid before LCG available. � HPSS performance test in NorduGrid environment � . . . � ATLAS Tier-2 Center at ICEPP, U. of Tokyo � Major facility will be installed in FY.2006. ⇒ Prof. Sakamoto’s talk � KEK LCG test bed developed since Sep. 2005 � In collaboration with ICEPP � Learning and technical skill update on the LCG middleware (LCG 2.6 -) � Implementation of Atlas software for muonTrigger Simulation � Organized LCG installation training course 17-19 Nov. 2005 at KEK ISGC2006 on May 04 2006 S.Kawabata: "Grid Activities at KEK" 3

LCG-2.6 Testbed ICEPP/U. Tokyo KEK UI-VOMS / WN1 VO: Atlas_j, rcrd CE_torque / WN2 UI VO: Atlas_j, g4med SE_dpm, MON SuperSINET(1Gbps) VOMS BDII / WN3 SE_dpm, LFC_mysql PX / WN4 BDII, RB, MON, PX RB / WN5 CE_torque CE_lsf LFC_mysql / WN6 Dual cpu pc’s WN1 JST Cluster WN2 � Functional test of middleware Opteron � Dual x 20 nodes Performance measurement of data sharing � Atlas simulation software was installed to demonstrate regional resource sharing PC farm � Parallel processing of Geant4 simulator with ~ 50 CPU MPI Private network ISGC2006 on May 04 2006 S.Kawabata: "Grid Activities at KEK" 4

2. KEK Grid CA � KEK submitted a GRID CA application to APGRIDPMA in Nov. 2005. � KEK Grid CA was approved by APGRID PMA and in production since January 2006. � 3 rd production GRID CA in Japan � NAREGI CA software was modified to use at KEK. � https://gridca.kek.jp/ � KEK employees and their collaborators are eligible for this service. ISGC2006 on May 04 2006 S.Kawabata: "Grid Activities at KEK" 5

3. Belle Experiment � Belle Exp. � B meson Factory using world highest luminosity accelerator: KEKB e + e － Collider � Accelerator: � Luminosity: � Peak 1.627 × 10 34 cm -2 s -1 � e + (3.5GeV) ~ 2.0 A ; e － (8GeV) ~ 1.36A � Continuous injection from Linac � Improved banch-banch interference and electron cloud � Luminosity will be much more improved by Crab Cavity in 2006. ISGC2006 on May 04 2006 S.Kawabata: "Grid Activities at KEK" 6

Belle Detector Aerogel Cherenkov cnt. n=1.015~1.030 SC solenoid 1.5T 3.5GeV e + CsI(Tl) 16 X 0 TOF counter 8GeV e − Tracking + dE/dx small cell + He/C 2 H 5 μ / K L detection Si vtx. det. 14/15 lyr. RPC+Fe 3 lyr. DSSD ISGC2006 on May 04 2006 S.Kawabata: "Grid Activities at KEK" 7

Integrated Luminosity Trend Constraint: ∫ dt (fb -1 ) 8GeV x 3.5GeV wall plug pwr.< 100MW crossing angle< 30mrad “Super-KEKB “ 1,50 0 L = 10 36 ∫ dt = 1,000fb − 1 1,0 0 0 larger beam current Present KEKB smaller β y * L = 1.5x10 34 50 0 long bunch option crab crossing 0 2002 03 04 05 06 07 08 09 10 11 ISGC2006 on May 04 2006 S.Kawabata: "Grid Activities at KEK" 8

Experimental Data and its Sharing among Belle Collaboration by SRB � Data accumulated so far � 1.5 PB including Simulation data � Recent data acquisition rate ~ 1.0TB/day � SRB servers for real data storage system has been implemented in Aug. 2005. � Current active data sharing � among KEK, U. Melbourne (Australia), and Nagoya Univ. � Target storage space 120 TB � files registered to MCAT ~ 423 files as of Sep. 6 ISGC2006 on May 04 2006 S.Kawabata: "Grid Activities at KEK" 9

The Belle SRB system Australia KEK FW Internet KEK network SRB server at Melbourn-Univ HSM-DISC HSM Belle FW SRB Server MES server MCAT Tape server NFS Library MES server DB at ANU Router server MCAT federation Super NAGOYA Univ. SINET TOHOKU Univ. DB MCAT server HDD SRB MES SRB server server server RAID ISGC2006 on May 04 2006 S.Kawabata: "Grid Activities at KEK" 10

Pre-production LCG site for Belle (JP-KEK-CRC-01) � Pre-production site was built up with LCG2.7 March, 2006 � Certification by APROC for registration to GOCDB has been done at the end of March � New VO: Belle has been registered to the LCG/EGEE as a global VO. � Initial collaboration sites expected: � Melbourne, ASGC, Krakow, Jozef Stefan Institute (Slovenia), IHEP Vienna � Nagoya U . ISGC2006 on May 04 2006 S.Kawabata: "Grid Activities at KEK" 11

LCG2.7 Node Site: JP-KEK-CRC-01 VO: Belle, dteam Internet Registory Internet Service web site: . keklcg.jp http://hepdg.cc.kek.jp/service / . RB 192.168.1.0 DNS,NAT CE wn001~wn014 ce01.keklcg.jp(192.168.1.1) WN Farm-1 SE se01.keklcg.jp(192.168.1.2) Router/SW MON mon.keklcg.jp(192.168.1.3) VOMS UI PX/BDII WN Farm-2 LFC (Not ready) CE ce02.keklcg.jp(192.168.1.4) Not ready Disk Server 8-03-2006 1.4 TB RAID ISGC2006 on May 04 2006 S.Kawabata: "Grid Activities at KEK" 12

New B Factory Computer System � New B Factory Computer System since March 23. 2006 � History of B Factory Computer System Performance ＼ Year 1997-(4years) 2001-(5years) 2006-(6years) Computing Server ~100 ~1,250 ~42,500 （ SPECint2000 rate ） (WS) (WS+PC) (PC) Disk Capacity ~4 ~9 1,000 （ TB ） (1PB) Tape Library Capacity 160 620 3,500 （ TB ） (3.5PB) Work Group server 3+(9) 11 80+16FS （ # of hosts ） User Workstation 25WS 23WS 128PC （ # of hosts ） +68X +100PC Moore’s Law 1.5y=twice 4y=~6.3 5y=~10 ISGC2006 on May 04 2006 S.Kawabata: "Grid Activities at KEK" 13

New B computer system KEK-DMZ 130.87.104.0/22 （ *.kek.jp ） Work Group Server Net 130.87.192.0/24 （ *.kek.jp ） L ogin W eb Internet KEK Internet FireWall L3 SW WG SRB G rid KEK-FB 130.87.224.0/21 （ *.kek.jp ） 80 10 N TP ・ D NS ・ S MTP UWS W eb R ADIUS SUS 172.22.28.0/24 128 APP SQL CVS LSF ITA W indows BK LDAP WF Core SW Data Transfer Network 5 16 172.22.32 - 43.0/24 Huge Storage System SC High-Speed Data transfer system Belles Farm DAQ Server @ Exp. Hall 1140 10.34.32 - 43.0/24 Outer net SW Administration Net SC ：Computing Server WG ：Work Group Server GRID Server BK ：Backup Server Super- Super- WF ：Work File Server SINET 130.87.194.0/24 SINET UWS ：User Work Station KEK-BC AI ST SUS ：Software Update Server 172.17.X.X KEK ITA ：Administration Server Internet とう Internet FireWall Nagoya,Tohoku,Tokyo ISGC2006 on May 04 2006 S.Kawabata: "Grid Activities at KEK" 14 I nst,Tokyo Univ.

New B Factory Computer System � Computing Server (CS) � CS ＋ WG servers(80) = 1208 nodes=2416CPU =45,662 SPEC CINT 2k rate =8.7THz � DELL Power Edge 1855 Xeon3.6GHz x2, memory 1GB � Linux 1 Enclosure = 10 nodes/ 7U space (CentOS/CS,REL/WGS) 1 rack = 50 nodes 25 racks = 4 arrays ISGC2006 on May 04 2006 S.Kawabata: "Grid Activities at KEK" 15

New B Factory Computer System � Storage System (SS) –disk- � 1,000TB, 42FileServ. � Nexan + ADTeX + SystemWks � SATAII 500G dr . × ~ 2000 (~ 1.8 failures/day?) � HSM = 370TB ← Nexan SATABeast non HSM (no Bck) 42dr/4U/21TB = 630TB ← ADTeX Array Master 15dr/3U/8TB ISGC2006 on May 04 2006 S.Kawabata: "Grid Activities at KEK" 16

New B Factory Computer System � Storage System (SS) -tape- � HSM � 3.5PB + 60drv + 13srv � SAIT 500GB/volume � 30MB/s drive � Petaserv(SONY) � WFS backup � 90TB + 12drv + 3srv � LTO3 400GB/volume � NetVault ISGC2006 on May 04 2006 S.Kawabata: "Grid Activities at KEK" 17

New B Factory Computer System � System usage � User Workstations (PC) are used for the network terminal. � Users login to Work Group Server （ WGS; 4~5persons/host ） . � 1208 Computing Servers divided into 3 LSF (Batch System) clusters. � WGS(80svr) shares WFS(16srv+80TB) as NFS user home directories. � Not-so-frequently-modified applications/Libraries are hold in 50 NFS servers. 1140 CS shares the NFS server. � Exp. Data (Many and Big size) � is transferred between CS(1140) and Storage Servers （ 42) � by using a Belle self-made simple TCP/socket application. � Data is managed by cooperation with DB system. � Storage System ⇔ Computing server transfer performance spec. � CS/WGS 1/3(540) ⇔ SS = 10GB/s � CS 2/3(1080) ⇔ SS/HSM = 0.5GB/s � SS/HSM ⇔ SS/nonHSM = 0.5GB/s ISGC2006 on May 04 2006 S.Kawabata: "Grid Activities at KEK" 18