INFSO-RI-508833
Enabling Grids for E-sciencE
www.eu-egee.org
Application Assessment of Production Service Vincent Breton (CNRS) - - PowerPoint PPT Presentation
Enabling Grids for E-sciencE Application Assessment of Production Service Vincent Breton (CNRS) EGEE 1 st EU Review 9-11/02/2005 www.eu-egee.org INFSO-RI-508833 Introduction Enabling Grids for E-sciencE Talk content Objectives of
INFSO-RI-508833
www.eu-egee.org
Vincent Breton, Application Assessment of Production Service 2
Enabling Grids for E-sciencE
INFSO-RI-508833
Vincent Breton, Application Assessment of Production Service 3
Enabling Grids for E-sciencE
INFSO-RI-508833
Vincent Breton, Application Assessment of Production Service 4
Enabling Grids for E-sciencE
INFSO-RI-508833
Vincent Breton, Application Assessment of Production Service 5
Enabling Grids for E-sciencE
INFSO-RI-508833
Vincent Breton, Application Assessment of Production Service 6
Enabling Grids for E-sciencE
INFSO-RI-508833
120k Pb+Pb events produced in 56k jobs 1.3 million files (26TByte) in Castor@CERN Total CPU: 285 MSI-2k hours (2.8 GHz PC working 35 years) ~25% produced on LCG-2 Phase II 1 million jobs, 10 TB produced, 200TB transferred ,500 MSI2k hours CPU ~15% on LCG-2
Vincent Breton, Application Assessment of Production Service 7
Enabling Grids for E-sciencE
INFSO-RI-508833
ATLAS DC2 - CPU usage
41% 30% 29% LCG NorduGrid Grid3
7.7 Million events fully simulated (Geant 4) in 95.000 jobs 22 TByte Total CPU: 972 M SI-2k hours >40% produced on LCG-2 (used LCG-2, GRID2003, NorduGrid)
ATLAS DC2 - LCG - September 7
1% 2% 0% 1% 2% 14% 3% 1% 3% 9% 8% 3% 2% 5% 1% 4% 1% 1% 3% 0% 1% 1% 4% 1% 0% 12% 0% 1% 1% 2% 10% 1% 4% at.uibk ca.triumf ca.ualberta ca.umontreal ca.utoronto ch.cern cz.golias cz.skurut de.fzk es.ifae es.ific es.uam fr.in2p3 it.infn.cnaf it.infn.lnl it.infn.mi it.infn.na it.infn.na it.infn.roma it.infn.to it.infn.lnf jp.icepp nl.nikhef pl.zeus ru.msu tw.sinica uk.bham uk.ic uk.lancs uk.man
Vincent Breton, Application Assessment of Production Service 8
Enabling Grids for E-sciencE
INFSO-RI-508833
Vincent Breton, Application Assessment of Production Service 9
Enabling Grids for E-sciencE
INFSO-RI-508833
DIRAC alone Plus LCG
1.8 106/day
LCG use pause
3-5 106/day
LCG use restart
186 M events 61 TByte Total CPU: 424 CPU years (43 LCG-2 and 20 DIRAC sites) Up to 3500 concurrent running jobs in LCG-2
Vincent Breton, Application Assessment of Production Service 10
Enabling Grids for E-sciencE
INFSO-RI-508833
Vincent Breton, Application Assessment of Production Service 11
Enabling Grids for E-sciencE
INFSO-RI-508833
Vincent Breton, Application Assessment of Production Service 12
Enabling Grids for E-sciencE
INFSO-RI-508833
Cyprus
CC-IN2P3 (France):
CNAF (Italy):
IFAE (Spain):
UPV (Spain):
LAL: 1CE – 20 CPU CGG: 1CE – 8 CPU LPC: 2CE – 142 CPU UPV: 1CE – 20 CPU CNB: 1CE – 16 CPU CC-IN2P3: 1CE – 100 CPU SCAI: 1CE – 28 CPU IPP: 1CE – 2 CPU (MPICH) CNAF: 2CE (MPICH)
Vincent Breton, Application Assessment of Production Service 13
Enabling Grids for E-sciencE
INFSO-RI-508833
> 1.200
(days)
~ 29.000
(hours)
104 248 656
CNAF* IPP* SCAI* 567 701 921 CGG 1 939 18 248 246 CNB 48 1 864 UPV 1 186 2 785 370 LAL-IN2P3 2 259 750 580 CC-IN2P3 17 399 81 760 675 LPC-IN2P3
#jobs machine_time used (s) RC
* These sites have recently enabled the Biomed VO
Vincent Breton, Application Assessment of Production Service 14
Enabling Grids for E-sciencE
INFSO-RI-508833
Vincent Breton, Application Assessment of Production Service 15
Enabling Grids for E-sciencE
INFSO-RI-508833
Vincent Breton, Application Assessment of Production Service 16
Enabling Grids for E-sciencE
INFSO-RI-508833
Vincent Breton, Application Assessment of Production Service 17
Enabling Grids for E-sciencE
INFSO-RI-508833
Vincent Breton, Application Assessment of Production Service 18
Enabling Grids for E-sciencE
INFSO-RI-508833
– Encourages communities to submit a well documented proposal – Fosters discussion on the added value brought by the Grid to the applications – Points out needs and resources for migration and deployment for each application – Prioritizes the deployment of the selected applications – Monitors the progress of the selected portfolio
Vincent Breton, Application Assessment of Production Service 19
Enabling Grids for E-sciencE
INFSO-RI-508833
– 5 applications, 3 recommended for approval Computational Chemistry MAGIC, Astrophysics Earth Science
– 7 applications received, 4 recommended for approval Earth sciences (Solid Earth Physics, Hydrology) Cosmology (Planck collaboration) Drug discovery (Molecular docking) Search engines for digital knowledge (GRACE)
Vincent Breton, Application Assessment of Production Service 20
Enabling Grids for E-sciencE
INFSO-RI-508833
Vincent Breton, Application Assessment of Production Service 21
Enabling Grids for E-sciencE
INFSO-RI-508833
CENAERO British Telecom MICROSOFT CERFACS ICATIS Sun ESI EDF NOVARTIS PHARMA AG NICE SOCIETE GENERALE Sanofi-Aventis BULL LION Bioscience AG THALES DATA SYNAPSE C-S IFP TOTAL ACRI-ST France Télécom PECHINEY CRV GRIDXPERT PLATFORM COMPUTING PSA Oracle Daimler CSTB Hewlett-Packard STMicroelectronics Srl SNECMA EADS CCR DASSAULT AVIATION PALLAS-INTEL BNP CNES CREDIT LYONNAIS GRIDSYSTEMS CEA MICHELIN CSCS ORION LOGIC Ltd. DUTCH SPACE paris Office AIRBUS Telefonica Spain HLRS AGENIUM Technologies Datamat SCHLUMBERGER Compagnie Générale de Géophysique Gridwise Technologies HUTCHINSON IBM Fugitsu Arcelor GENIAS Benelux SCAI Pôle Européen Plasturgie
Vincent Breton, Application Assessment of Production Service 22
Enabling Grids for E-sciencE
INFSO-RI-508833
Vincent Breton, Application Assessment of Production Service 23
Enabling Grids for E-sciencE
INFSO-RI-508833
Vincent Breton, Application Assessment of Production Service 24
Enabling Grids for E-sciencE
INFSO-RI-508833
Vincent Breton, Application Assessment of Production Service 25
Enabling Grids for E-sciencE
INFSO-RI-508833
Vincent Breton, Application Assessment of Production Service 26
Enabling Grids for E-sciencE
INFSO-RI-508833
Vincent Breton, Application Assessment of Production Service 27
Enabling Grids for E-sciencE
INFSO-RI-508833
Vincent Breton, Application Assessment of Production Service 28
Enabling Grids for E-sciencE
INFSO-RI-508833
VO at SARA was created in July 2004 and is functional now using EGEE
countries
GEOsciences on DEmand) VO was created at IN2P3 (Lyon) in mid- October for CGG and Geocluster
migration to EGEE production service
has successfully been deployed on GILDA and demonstrated at the 2nd EGEE Conference in The Hague using the GENIUS portal
satellite experiment GOME and their validation by using LIDAR data has been run on EGEE production service
Vincent Breton, Application Assessment of Production Service 29
Enabling Grids for E-sciencE
INFSO-RI-508833
Vincent Breton, Application Assessment of Production Service 30
Enabling Grids for E-sciencE
INFSO-RI-508833
– VO server is hosted by SARA/NIKHEF – Successful first running in GILDA as well as in Crossgrid testbed using LCG-2 middleware
– CNAF will support the Magic VO with a Resource Broker – PIC will support the Magic VO with storage and the RLS – CNAF, PIC and GridKA will provide CPU – GILDA can be used for the first test as well
Vincent Breton, Application Assessment of Production Service 31
Enabling Grids for E-sciencE
INFSO-RI-508833
Vincent Breton, Application Assessment of Production Service 32
Enabling Grids for E-sciencE
INFSO-RI-508833
Vincent Breton, Application Assessment of Production Service 33
Enabling Grids for E-sciencE
INFSO-RI-508833
ORCA Aligned with APROM strategy Exploit native gLite functionality Athena DIAL High level services AliROOT PROOF ROOT Interactive analysis DaVinci GANGA GUI to Grid Middleware Experiment analysis application framework Basic prototype component Main focus LHC Experiment
Vincent Breton, Application Assessment of Production Service 34
Enabling Grids for E-sciencE
INFSO-RI-508833
INFSO-RI-508833
Interactive Session
Demo on Wednesday
Demo at Supercomputing 04
Enabling Grids for E-sciencEINFSO-RI-508833
Current Status
Demo on Wednesday
Enabling Grids for E-sciencEINFSO-RI-508833
Combined Test Beam
Example: ATLAS TRT data analysis done by PNPI St Petersburg Number of straw hits per layer
Real data processed at gLite
Standard Athena for testbeam Data from CASTOR Processed on gLite worker node
Enabling Grids for E-sciencEINFSO-RI-508833
User Job Monitoring MonAlisa
Demo at Supercomputing 04
Vincent Breton, Application Assessment of Production Service 35
Enabling Grids for E-sciencE
INFSO-RI-508833
INFSO-RI-508833
Work Load Management
ARDA has been evaluating two WMSs
– available since April – pull model – integrated with gLite shell, file catalog and package manager
– available since middle of October – currently push model (pull model not yet possible but foreseen) – not yet integrated with other gLite components (file catalogue, package manager, gLite shell)
Enabling Grids for E-sciencEINFSO-RI-508833
Performance
– Good performance due to streaming – 80 concurrent queries, 0.35 s/query, 2.6s startup time
– First attempt to use the catalog: quite high entrance fee – Good performance – Not yet stable results due to unexpected crashes
We are interacting with the developers
FC insertion Attach MD Files F ile s /s [1 /s ] 2 4 6 8 10 1000 10000 100000 Errors
Tim e to com pletion# Clients Time to Completion [s]
Selecting 2.5k files of 10k 5 10 15 20 25 30 35 40 20 40 60 80 100 Enabling Grids for E-sciencE
INFSO-RI-508833
gLiteIO tests
INFSO-RI-508833
Package management
and user private packages on the Grid – Pre-installation of the experiment software is implemented by a site manager with further publishing of the installed software. Job can run
– Installation on demand at the worker node. Installation can be removed as soon as job execution is over.
light-weight installations, close to the second approach
D M S : c a t a l
s DMS: data transfer
Vincent Breton, Application Assessment of Production Service 36
Enabling Grids for E-sciencE
INFSO-RI-508833
Vincent Breton, Application Assessment of Production Service 37
Enabling Grids for E-sciencE
INFSO-RI-508833
Vincent Breton, Application Assessment of Production Service 38
Enabling Grids for E-sciencE
INFSO-RI-508833
Vincent Breton, Application Assessment of Production Service 39
Enabling Grids for E-sciencE
INFSO-RI-508833
Vincent Breton, Application Assessment of Production Service 40
Enabling Grids for E-sciencE
INFSO-RI-508833
Final Report of Application Identification and Support Activity M24 DNA4.4 Second revision of EGEE Application Migration Progress report M21 DNA4.3.3 First revision of EGEE Application Migration Progress report M15 DNA4.3.2
with feedback M24 MNA4.3 First external review of Applications Identification and Support with feedback M12 MNA4.2
Vincent Breton, Application Assessment of Production Service 41
Enabling Grids for E-sciencE
INFSO-RI-508833