Evolution of the LHC Computing Models Ian Fisk May 22, 2014 About - - PowerPoint PPT Presentation

Evolution of the LHC Computing Models

Ian Fisk May 22, 2014

Ian Fisk CD/FNAL

About Me

I am a scientist with Fermilab I have spent the last 14 years working on LHC Computing problems I helped build the first Tier-2 prototype computing center in the US I was responsible for Integration and Commissioning of the CMS Computing system for 2006-2010 And I was computing coordinator of CMS for LHC Run1

Ian Fisk CD/FNAL

Final Steps

Software and Computing is the final in a long series to realize the physics potential of the experiment

• As the environment has become more complex and

demanding, computing and software have had to become faster and more capable

• Storage and

Serve the data Reconstruct the physics objects Analyze the events

Ian Fisk CD/FNAL

To get this You need this

Ian Fisk FNAL/CD

Distributed Computing

• Computing models are

based roughly on the MONARC model

–Developed more than a decade ago –Foresaw Tiered Computing Facilities to meet the needs of the LHC Experiments

• Assumes poor

networking

• Hierarchy of functionality

and capability

• 16 -

622 Mbits/s 622 Mbits /s CERN/CMS 350k Si95 350 Tbytes Disk; Robot Tier2 Center 20k Si95 20 Tbytes Disk, Robot FNAL/BNL 70k Si95 70 Tbytes Disk; Robot

622 Mbits /s N X 6 2 2 M b i t s / s 622Mbits/s 622 Mbits/s

Tier3 Univ WG 1 Tier3 Univ WG M

Model Circa 2005 Model Circa 2005

Tier3 Univ WG 2

• Fig. 4-1 Computing for an LHC Experiment Based on a Hierarchy of Computing Centers. Capac

for CPU and disk are representative and are provided to give an approximate scale).

Tier-

Tier- 1 Tier- 2 Tier- 2 Tier- 2 Tier- 1 Tier- 1

5

Ian Fisk

Distributing Computing at the Beginning

• Before LHC most of the Computing

Capacity was located at the experiment at the beginning

–Most experiments evolved and added distributed computing later

6

LHC began with a global distributed computing system

OSG LCG NDG

Ian Fisk CD/FNAL

Grid Services

During the evolution the low level services are largely the same

• Most of the changes

come from the actions and expectations of the experiments

CE SE

Information System

FTS BDII WMS

Lower Level Services Providing Consistent Interfaces to Facilities Higher Level Services

VOMS

Experiment Services Site

Connection to batch (Globus and CREAM based) Connection to storage (SRM or xrootd)

Ian Fisk FNAL/CD

Successes

• When the WLCG started there was a lot of

concern about the viability of the Tier-2 Program

–A university based grid of often small sites

• Total system uses close to half a million

processor cores continuously

8

44% 38% 18%

Tier-0 Tier-1 Tier-2

47% 33% 20%

2009 2013 Capacity Grows by factor 2.5

Ian Fisk FNAL/CD

Moving Forward

–Strict hierarchy of connections becomes more of a mesh –Divisions in functionality especially for chaotic activities like analysis become more blurry –More access over the wide area

Tier- Tier- 1 Tier- 1 Tier- 1 Tier- 2 Tier- 2 Tier- 2 Tier- 2 Tier- 2 CAF

Prompt Reconstruction Storage Commissioning Re-Reconstruction/ Simulation Archiving Data Serving Simulation and User Analysis

• Model changes have been an evolution
• Not all experiments have emphasized the same things
• Each pushing farther in particular directions

9

T i e r

• 2

Ian Fisk CD/FNAL

Evolution

We have had evolution all through the history or the project Slow changes and improvements Some examples Use of Tier-2s for analysis in LHCb Full mesh transfers in ATLAS and CMS Data federation in ALICE Better use of the network by all the experiments But many things are surprisingly stable Architectures of hardware (x86 with ever increasing cores) Services both in terms of architectures and interfaces

Ian Fisk CD/FNAL

Looking back

In June of 2010 we had a workshop on Data Access and Manager in Amsterdam Areas we worried about at the time were making a less deterministic and flexible system providing better access to the data for analysis being more efficient Some things were were not worrying about New architectures for hardware Clouds Opportunistic Computing

Ian Fisk FNAL/CD

Progress Networking

• One of the areas of progress has been better use
• f wide area networking to move data and to

make efficient use of the distributed computing

–Limited dedicated network –Much shared use R&E networking

12

LHCOPN

• Dedicated resource T0->T1

and T1 to T1 LHCOne to Tier-2s LHCOne

• New initiative for Tier-2 network

Ian Fisk FNAL/CD

Mesh Transfers

–Change from

• –To

13

Transfers West Transfers East

Tier- 1 Tier- 1 Tier- 1 Tier- 1 Tier- 2 Tier- 2 Tier- 2 Tier- 2 150MB/s 150MB/s 150MB/s 300MB/s

Ian Fisk FNAL/CD

Completing the Mesh

• Tier-2 to Tier-2 transfers are now similar to

Tier-1 to Tier-2 in CMS

14

Tier- 2

Tier- 2 Tier- 2 Tier- 2

Ian Fisk

Overlay Batch

One of the challenges of the grid is despite having a consistent set of protocols actually getting access to resources takes a lot of workflow development Pilots jobs are centrally submitted and start on worker nodes, reporting back that they are available Building up an enormous batch queue Batch

Ian Fisk CD/FNAL

So what changes next?

The LHC is currently in a 2 year shutdown to improve the machine Energy will increase to ~13TeV and the luminosity will grow by a factor a ~2 Both CMS and ATLAS aim to collect about 1kHz of data Events are more complex and take longer to reconstruct All experiments need to continue to improve efficiency

Ian Fisk CD/FNAL

Resource Provisioning

The switch to pilot submissions opens other improvements in resource provisioning Instead of submitting pilots through CEs We can submit pilots through local batch systems We can submit requests to Cloud provisioning systems that start VMs with pilots Currently both ATLAS and CMS provision the use of their online trigger farms through an OpenStack cloud The CERN Tier-0 will also be provisioned this way Before the start of Run2 ~20% of the resources could be allocated with cloud interfaces

Ian Fisk FNAL/CD

Evolving the Infrastructure

• In the new resource provisioning model the pilot

infrastructure communicates with the resource provisioning tools directly

– Requesting groups of machines for periods of time

18

Resource Provisioning Resource Provisioning Pilots Resource Requests Cloud Interface CE VM with Pilots VM with Pilots VM with Pilots VM with Pilots VM with Pilots VM with Pilots VM with Pilots Batch Queue WN with Pilots WN with Pilots WN with Pilots WN with Pilots WN with Pilots WN with Pilots WN with Pilots

Ian Fisk

Local Environment

Once you arrive on a worker node, you need something to run Environment distribution has come a long way LHC experiments use the same read-only environment centrally distributed to nearly half a million processor cores CVMFS SQUID SQUID WN SQUID

Local Client FUSE Client

Ian Fisk

High Performance Computing

As modern worker nodes get more and more cores per box, these systems look like HPC All LHC Experiments are working on multi-processing and/or multi-threaded versions

• f their code

We are transitioning how we scheduled pilots. A single pilot comes in an takes over an entire box or group of cores The overlay batch then schedules the appropriate mix

• f work to use all the cores. And tightly coupled

applications can run too

Ian Fisk FNAL/CD

Wide Area Access

• All experiments are looking at sending data

directly to the worker node even from long distance

–Sending data directly to applications over the WAN

• Not immediately obvious that this increases the

wide area network transfers

21

11

m

Ian Fisk

Data Moved

Small hit in efficiency A lot of work goes into predictive read ahead and caching Currently we see about 400MB/s read

• ver the wide area

Thousands of active transfers

Ian Fisk

Network Improvements

While CPU (25%/year) and disk (20%/year) have both slowed in the performance improvements at a fixed cost, network is still above 30% improvement per year Cost of 100Gb/s optics are falling For CMS we expect 30% of our Tier-2 resources will be connected at 100Gb/s at Universities within a year

Ian Fisk

Changes at CERN

CERN recently deployed half their computing in Budapest with 2 x 100Gb/s connecting the two facilities Geneva is expensive for people, power, and space All the disks are at CERN and half the worker nodes are in Hungary We see a 5% drop in analysis efficiency

Evolution of Tier 0

2013%10& 2013%11& 2013%12& 2014%01& 2014%02& 2014%03& 0.000& 0.100& 0.200& 0.300& 0.400& 0.500& 0.600& 0.700& 0.800& 0.900& 1.000&

Analysis(job(CPU(efficiency(

Meyrin&SLC6& virtual& Wigner&SLC6& virtual&

Ian Fisk CD/FNAL

A more flexible system

Slowly you are seeing a break down of the boundaries of the sites and the hierarchy of responsibilities and functionality. Sites are not restricted to a specific set of functions Even the concept of boundaries between sites

Legacy of MONARC

• A lot of the structure and hierarchy of the MONARC computing

models remains for several of the LHC experiments

• Transparency of Data Placement and Access has been replaced with a

reasonably structured environment

• The MONARC Report just turned 10. There have been

improvements in Computing Services and it may be time to revisit

CMS

Prompt Reconstruction Storage Commissioning Re-Reconstruction/ Simulation Archiving Data Serving Simulation and User Analysis

≠

Ian Fisk FNAL/CD

Reducing Boundaries

• One of the first changes we are seeing is a flattening of the

tiered structure of LHC computing –the functional differences of what each layer can do are being reduced and we have a desire to use the system as a distributed system, and not a collection of sites

• One concrete action of this is to separate the archival

functionality from the other site functions –This can be done with separate instances like CERN has done with Castor and EOS –Or can be done with Service Classes like ATLAS has done

• 26

Ian Fisk FNAL/CD

Changes how we think of tiers

• Once you introduce the concept of an archival

services that is decoupled from the Tier-1

– The functional difference between Tier-1 and TIer-2 is based more on availability and support than size of services

• Difference between Tier-1 and Tier-2 from a functional perspective is

small

– Model begins to look less Monarc-like

27

T1 T1 T1 T1 T2 T2 T2 T2 T2

Site View

• Current site view for several of the LHC

experiments begins to look like a walled city with a couple of gates

• CE accepts jobs. We run thousands of

similar requests and we authorize all of them

• SE transfers data in and out
• Data is preloaded into the sites and

jobs come and finds it

DISK CE SE

Site TAPE

Ian Fisk FNAL/CD

Stretches into Other elements

• After Long Shutdown 1, CMS and ATLAS will likely

reconstruct a portion of the data the first time at Tier-1s in close to real time

– Very little unique about the functionality of the Tier-0

• Some prompt calibration work that uses Express data, but even that

could probably be exported

– Calibration in ATLAS was exported to a dedicated Tier-2

• Also looking at other opportunistic computing

resources

– Nature of PromptReco becomes more flexible

28

T0 T1 T1 T1 HLT Opportunistic

Ian Fisk FNAL/CD

Evolution

• After many years of operations and work our

system continues to evolve

–We are reducing how strictly we define the functionality in each tier

• Lines and capabilities are blurring together

–We have much better access to data

• With it we begin to eliminate even the boundaries between

sites

–We will have a much more diverse set of resources

• Cloud and opportunistic access will be big areas of growth

29