Computing Report Construction Project activities Computing - - PowerPoint PPT Presentation

▶
computing report
SMART_READER_LITE
LIVE PREVIEW

Computing Report Construction Project activities Computing - - PowerPoint PPT Presentation

Oct 20, 2023 392 likes •574 views

Computing Report Construction Project activities Computing Consortium activities Capacity provision (GridPP+IRIS) On behalf of Manchester, RAL, Edinburgh (paid partners) All other participating GridPP sites (unpaid partners) Pete Clarke /

slide-1
SLIDE 1

1

Computing Report

Pete Clarke / DUNE-UK / DL / 11th Dec 2019

Construction Project activities Computing Consortium activities Capacity provision (GridPP+IRIS)

On behalf of Manchester, RAL, Edinburgh (paid partners) All other participating GridPP sites (unpaid partners)

slide-2
SLIDE 2

2

Specific construction Project activities

slide-3
SLIDE 3

3

Data Management

  • Lead: Edinburgh
  • People: James Perry, Teng Li
  • Starts: 2019 @ 1 FTE
  • Immediate work: Mainly development of RUCIO data management for DUNE
  • Share with RUCIO work for other communities in IRIS

Offline Production Management

  • Lead: RAL
  • People: Chris Brew, Raja Nandakumar
  • Starts: 2021 @ 0.5 FTE but this is started anyway
  • Initial Work: Trialling use of DIRAC (the LHCb workload management system)

Cloud Integration

  • Lead: Manchester
  • People: Andrew McNab
  • Starts: 2020 @ 0.25FTE but this is started anyway
  • Initial Work: Augmenting DUNE capability to use diverse resources

Construction project “paid” work

slide-4
SLIDE 4

4

Rucio is the ATLAS data management system

  • Policy driven
  • It is now adopted by CMS, DUNE
  • Being trialed by SKA and possibly LSST

Work carried out (Perry)

  • Added support for S3 and Swift signed URLs to Rucio core
  • Added objectstore to Rucio development Docker image
  • Move current experiment-specific code out of Rucio core and into separate Python packages

– Each VO will maintain its own policy package

  • Basic implementation done, targeted for Rucio 1.22 release

Work carried out (Li)

  • Monitoring including data deletion/movement, RUCIO internal healthiness and accounting.

Data management @ Edi : Rucio

Perry, Li

slide-5
SLIDE 5

5

Data management: Rucio dashboard

slide-6
SLIDE 6

6

Demonstrated usability of DIRAC as a WMS for DUNE

è Run simple user jobs

l

https://indico.fnal.gov/event/21328/session/0/contribution/7/material/slides/0.pdf

è Full chain MC test “production”

l

https://indico.fnal.gov/event/21506/session/0/contribution/8/material/slides/0.pdf

è Integrated with SAM

l

http://samweb.fnal.gov:8480/station_monitor/dune/stations/dune/projects/test_nraja-5770

l

https://docs.dunescience.org/cgi- bin/sso/RetrieveFile?docid=12982&filename=DUNE_MonthlyReport_201909.pdf&version=10

This work was commissioned by DUNE computing Presented at summer workshop Gives DUNE choices

Workload Management @ RAL : DIRAC

Nandakumar, Brew

slide-7
SLIDE 7

7

Cloud resource work @ Manchester

McNab

slide-8
SLIDE 8

8

Computing Consortium

slide-9
SLIDE 9

9 3

slide-10
SLIDE 10

10 10

Andrew McNab is one of two technical coordinators

  • This work not foreseen in the construction proposal
  • …..this is the story of computing

AMc made particular effort this year for

  • Data model workshop in BNL in August
  • Organised and led computing model workshop in FNAL in Sept

PC setting up the Computing Contributions Board

  • Formal body to deliver computing capacity requirements
  • Follows LHC-like process
  • One national representative from each of the larger partner countries + FNAL + CERN

Representation of DUNE in various relevant places important to DUNE.

  • WLCG Management board (PC, AM)
  • LHCOPN Network meetings (PC)
  • WLCG GDB steering group (AM)
  • GridPP PMB (PC, AM)
  • IRIS PMB (PC-Director, AM)
slide-11
SLIDE 11

11

Computing capacity contributions

slide-12
SLIDE 12

12

Computing capacity via GridPP&IRIS

Capacity provided on Grid via GridPP

  • DUNE uses Grid resources like any other large HEP experiment
  • Part of WLCG + OSG
  • Includes FNAL and CERN of course

October 2019:

  • 42 countries
  • 65 MoU’s
  • 168 sites
  • ~ 900,000 cores
  • ~ 0.5 Exabytes disk
  • ~ 0.5 Exabytes tape
slide-13
SLIDE 13

13

Computing capacity from UK

GridPP sites jumped in (resources and people to help)

  • RAL -> R.Nandakumar (PPD) attends sites meetings
  • Manchester
  • Edinburgh
  • Imperial -> D.Bauer attends sites meetings - also IRIS scrutineer work
  • Lancaster -> M.Doidge active in enabling DUNE
  • Liverpool -> S.Jones attends sites meetings
  • Sheffield
  • Glasgow
  • Oxford -> V.Davada attends sites meetings
  • Bristol
  • QMUL -> T.Froy setting up perfSONAR

IRIS & GridPP have provided the extra resources

  • GridPP5 has 10% for non-LHC
  • GridPP6 includes DUNE
  • DUNE also submits formal annual request to IRIS (www.iris.ac.uk)
  • In 2019 : 1000 cores + 2.0 PB of disk
  • In 2020 will be ~ 2000 cores and 2.5 PB - driven by DUNE model, not UK
slide-14
SLIDE 14

14

Computing Contributions so far Shown to LBNC

Country % prod. jobs USA 52 UK 18 CH 12 NL 6 ES 2 CZ 6 FR 5

Percentage of successful production jobs over last year

  • Central ring shows countries
  • Outer ring shows sites

This shows a good trend Central Ring: Resources provided by:

  • OSG sites
  • WLCG sites
  • FNAL
  • CERN (part of WLCG)
slide-15
SLIDE 15

15 15

From EGI accounting portal

slide-16
SLIDE 16

16

User analysis jobs : a problem

Percentage of successful user analysis jobs

  • Central ring shows countries
  • Outer ring shows sites

Need to encourage analysis jobs to migrate to the world

  • Education/culture change

Last 6 months

So please consider sending your jobs to non FNAL sites

slide-17
SLIDE 17

17

How we should be building our electronics