Regional Climate Modelling using Grid Infrastructures S. K. Dash - - PowerPoint PPT Presentation

Centre for Atmospheric Sciences Indian Institute of Technology Delhi Hauz Khas, New Delhi-110016

Regional Climate Modelling using Grid Infrastructures

• S. K. Dash and Deepika Vaddi

Acknowledgements: Stefano Cozzini, Filippo Giorgi, Abdus Salam ICTP

Points to Discuss

• Importance of Indian summer

monsoon & need of regional models

• RegCM3 simulations at IIT Delhi
• RegCM implemented on Garuda GRID
• RegCM4 simulations
• Efforts for integrating RegCM4 on

ICTP computer using EU GRID

Importance of Monsoon in India

• Indian summer monsoon during June

September is very important for Indian agriculture and water sources

• Monsoon is scientifically challenging
• Monsoon features are difficult to be

simulated by GCMs primarily because of large temporal and spatial variations

• There is a need for high resolution

Regional Models

Regional aspects of Indian surface features

Large spatial variations in monsoon rainfall in five homogeneous zones of India. The numbers inside the zones indicate mean monsoon rainfall (mm), standard deviation (mm) and coefficient of variation (%) from top to bottom respectively

Dash et al., 2002, Mausam, 53(2), 133-144

1) ICTP RegCM3 simulations at IIT Delhi showing the impact of variations in Tibetan snow on summer monsoon rainfall 2) Two experiments (i) with 10cm of snow And (ii) no snow in the extreme case of global warming

Model domain used in RegCM3 and the five homogeneous zones of India such as North West India (NWI), West Central India (WCI), Central Northeast India (CNI), North East India (NEI) and South Peninsular India (SPI) (Parthasarathy et al., 1995)

Dash et al., 2006, Theor. Appl. Climatol, special issue, 1-12.

Central Lat and Lon- 20

• N, 80
• E

101 X 115 Points along XY direction Domain covers 55

• E to 105
• E and 5
• S to 45
• N

with Grid distance- 55 Km

Region over which 10cm of snow has been introduced uniformly in the snow experiment

20 40 60 80 100 120 1993 1994 1995 1996 average no-snow snow AI 10 20 30 40 50 60 1993 1994 1995 1996 average no-snow snow NWI 10 20 30 40 50 60 1993 1994 1995 1996 average no-snow snow NEI 10 20 30 40 50 60 1993 1994 1995 1996 average no-snow snow WCI 10 20 30 40 50 60 1993 1994 1995 1996 average no-snow snow SPI 10 20 30 40 50 60 1993 1994 1995 1996 average no-snow snow CNI

Comparison of JJAS mean rainfall (cm) over All India and its five homogeneous zones as simulated by RegCM3 in no-snow and snow experiments

a b c d e f Decrease of rainfall 30% for AI 20% for WCI 25% for CNI

23% for NWI 15% for SPI

Initial Conditions: 25th April to 3rd May up to 30th September, 9-member Horizontal grid distance: 55 Km Domain chosen: 51OE to 109OE and 3OS to 43ON

Simulation of Monsoons 1982-2009

Using ICTP RegCM3

Data used:

USGS Global 30 Arc-Sec. elevation datasets at 30’ resolution to create terrain USGS Global GLCC dataset at 30’ resolution to create vegetation or landuse file Weekly analysis OISST available from NOAA for integration NCEP Reanalysis (NNRP1) are used for setting the initial and boundary conditions

• Variations in rainfall, temperature and winds

at time scales of intra-seasonal, seasonal and inter-annual examined in detail and compared with observations.

• Special emphasis given on contrasting

monsoon years such as 1982-1983, 1987- 1988, 2002-2003 etc to validate RegCM.

Clusters on Garuda (being used)

Begabung Cluster Head Node (IITD)

Garuda Head Node (gridfs)

GG-BLR (Linux Xeon machine Bangalore) GG-HYD (Linux Xeon at Hyderabad)

Resources used

Cluster Name OS ARCH Memory Job Manag er Conf Procs #procs Ajaymeru (CAS, IITD) Linux2. 6.18- 128. x86_6 4 32 GB

• 128

128 gg-blr.tfg (Garuda) Linux2. 6.18- 53. x86_6 4 16 GB PBS 240 208 begabung (Garuda

cluster, IITD)

Linux2. 6.9- 42.0 x86_6 4 4 GB PBS 36 17

Time Evaluation

Seasonal Runs

#processors Execution Time(gg-blr) Execution Time(Ajaymeru) Gain on the grid 12 processors 1:35:40 2:35:42 1:00:02 24 processors 1: 10:49 2:02:54 0:52:05 30 processors 0:54:12 1:49:23 0:55:11 40 processors 0:46:36 1:20:26 0:33:50 60 processors 0:41:02 1:05:47 0:24:45

Global Data Pre-Processing Model Execution Post Processing & Visualisation RegCM

GLCC dataset GTOPO (landuse) Dataset GISST, OISST, OIWK Global re- analysis data EIN15, ERA40, NNRP1, NNRP2

Terrain

SST ICBC RegcmMPI ATM RAD SRF CDO NCO Grads

Represents the different components of RegCM modelling system.

5.Download input data

• 7. Upload
• utput data
• 4. JDL Job

submission

• 8. Download

Final output

• 3. Upload Input data

Data for Pre- processing

HPC local resources resources User Interface

• 2. Move Input

data to UI

Storage Grid GRID Computing Element

• 6. RegCM execution
• 1. Pre-

Processing

Schematic diagram of interaction of model simulation with data management within the Grid Infrastructure

Clusters on Garuda used for Demo

Cluster Head Node(CHN) IIT Delhi Begabung connecting Garuda(via NKN)

Garuda Head Node(GHN) Gridfs

Cluster Head Node(CHN) Bangalore (CDAC) gg-blr Compute nodes Cluster Head Node(CHN) Hyderabad gg-hyd Compute nodes

Login to the Cluster Head Node Begabung Login to the Garuda Head Node Gridfs Login to the Cluster Head Node to be used for execution gg-blr/gg-hyd Transfer model and data to the clusters Compile the model -> create binaries Go back to Gridfs Submit the job using Job template Completion of the job (DONE status) Output on CHN (gg-blr/gg-hyd) Transfer output data back to Begabung

Flow chart for integrating the model

• Global data(CHN begabung to gg-blr/gg-hyd)

EIN15, 1 year data, 15GB begabung to gg-blr/gg-hyd- 120 min@625kbps begabung to gridfs -120 min @625kbps gridfs to gg-blr/gg-hyd- 20 min @11.2mbps SURFACE data, 2GB – 10 min SST data, 368MB – 2-3 min

• Output data(gg-blr/gg-hyd to begabung) @ 450kbps

1 month, SRF, 967MB - 20 min @ 450kbps SAV, 219MB- 5 min @ 380kbps ATM, 1.9GB- 45 min @ 780 kbps RAD, 1.4GB- 30 min @780 kbps

Data transfer specifications

Grid Resources Used

Name CPU type # Cores Network

Argo Intel Nehalem 2.4 GHz 8 Infiniband DDR Garuda Grid India gg-blr Intel Harpertown 3.16 GHz 8 Infiniband @20gbps full duplex EU: Briareo/Ce- 01 AMD 2.4GHz 4 Infiniband @10gbps EU:Legnaro X5650 @ 2.67GHz 12

• EU:Legnaro

AMD Opteron(tm) Processor 6174 24

APHRODITE CRU RegCM3 CMAP GPCP

Climate of JJAS precipitation (cm) in RegCM3 and observations

RegCM3 IMD

RegCM3-IMD RegCM3-CRU RegCM3-APHRODITE RegCM3-GPCP RegCM3-CMAP

Percentage differences in JJAS precipitation in RegCM3 and Observations

June July August September JJAS Correlation Coefficients between RegCM3 and IMD observed ensemble mean monsoon rainfall for the period 1982-2009 spanning 28 years. The contours are obtained with 9 point smoothing to the gridded result.

Model: RegCM4.1.1 Grid points: Y direction-160 Grid Points: X direction-224 Horizontal Resolution: 50Km Simulation Period: 6 Years 01 Jan 1998 to 31 Dec 2003

South Asia CORDEX Domain

Domain details

Central Longitude= 70oE Central Latitude= 16oN

• CORDEX domain experiments have been conducted using ARGO,

HPC cluster of ICTP for Indian summer monsoon configuration

• One year climate run on 32 processors on ICTP cluster takes about

7 hrs CPU time

Exp-0 default settings (Table 1) + with irrigated crop Exp-1 modified Grell over land and ocean + with irrigated crop Exp-2 modified Grell over land and ocean + removed irrigated crop Exp-3 modified Grell over land and ocean + removed irrigated crop + dtauc15 Exp-4 modified Grell over land and ocean + removed Irrigated crop + dtauc25 Exp-5 modified Grell over land and ocean + removed irrigated crop + rsmincrop Exp-6 modified Grell over land and ocean + removed irrigated crop + rsminforest_fcmax Exp-7 modified Grell over land and 2nd time over ocean + removed irrigated crop Exp-8 modified Grell 2nd time

• ver ocean + removed irrigated

crop Exp-9 modified Grell over land and ocean + Emanuel over land + removed irrigated crop Exp-10 modified Grell 2nd time

• ver ocean + Emanuel over land

+ removed irrigated crop Exp-11 modified Grell over

• cean + Emanuel over land +

with irrigated crop Exp-12 modified Grell over

• cean + Emanuel over land +

with irrigated crop + modified Zeng Exp-13 modified Grell 2nd time

• ver ocean + Emanuel over land

+ removed irrigated crop + modified Zeng Exp-14 modified Grell 2nd time

• ver ocean + Emanuel over

land + removed irrigated crop + modified Zeng + rsmincrop Exp-15 modified Grell over

• cean + Emanuel over land +

removed irrigated crop + modified Zeng + rsmincrop

Experiments conducted

Bias between RegCM4.1.1 and IMD observed JJAS accumulated rainfall (cm) climatology Bias between RegCM4.1.1 and CMAP observed JJAS accumulated rainfall (cm) climatology

Bias between RegCM4.1.1 simulated and CRU observed climatology of JJAS mean surface temperature (oC) from 1998-2003

The 1990–2008 mean annual cycle of precipitation(mm/day) in Indian land and in its five homogeneous regions. RegCM4.1.1 simulated precipitation climate (1990-2008) is compared with IMD (1990-2008), APHRO (1990-2007) GPCP (1997-2008) and TRMM (1998-2008)

• JJAS temperature has a slight cold bias over the

mountain and coastline compared with CRU

• dataset. Indian land area temperature is well

represented.

• The monsoon precipitation over the Indian

continent is reasonably represented by the use

• f double convection scheme.
• Comparison of RegCM4.2 precipitation with that
• f IMD dataset shows good inter-annual

variations.

Results from RegCM4 simulations

Data Management

• There are 2 approaches
• data transfer from Storage Element(SE) available on the Grid

to the Worker Node(WN).

• This approach fetches the already pre-processed data stored in the

storage elements to the worker node.

• Using OpenDAP approach
• This approach accesses the global data sets made available on the

thredds server and carries out all the pre-processing steps creating domain, sst and icbc files.

lcg recursive tools

• 5. Output

files

GridFTP

• 1. Job Submission

JDL with modified regcm.in

User Interface OpenDAP Repository Grid Computing Element

3.Pre-processing

• 4. RegCM

Storage Grid

• 2. OpenDAP

Protocol 6.Output data

Schematic diagram of data management using OpenDAP

5000 10000 15000 20000 25000 30000 35000 Argo-HPC ce-01.grid.sissa.it briareo.grid.elettra.trieste .it Garuda Grid India(gg-blr)

time (in sec) CE

Pre-installed version 4 8 16 32

• The graphs imply that the

model performance is consistent and is scalable with Pre-installed version on controlled Computing Elements.

• The performance difference

in controlled CE’s is due to the software stack used to compile the RegCM package

• n the two platforms.
• The machines showing good

performance has intel compiler whereas gnu fortran compiler has been used

• n

machine like briareo.

Number of cores Argo CE-01 gg-blr Briareo 4 11687 23129 11927.73 32369 8 6157 12025 8413.59 18397 16 3153 8170 4559.42

• 32

1726

• 2569.57
• Performance of the Model with Pre-installed version

CPU Cores Terrain SST ICBC RegcmMPI Total Time(in sec) 4 265 2 843 19077.06 20187.06 8 204 2 627 10888.94 11721.94 16 192 2 580 6014.20 6788.20 32 157 1 557 6505.97 7220.97

• data directly from THREDDS servers( graph in the next slide represents this data

This table represents the performance with OpenDAP Data management on t2-ce-05.lnl.infn.it(Legnaro)

CPU Cores Terrain SST ICBC RegcmMPI Total Time(in sec) 4 240 1 455 28826.71 29522.71 8 258 1 476 34637.60 35372.6 16 215 1 479 33229.45 33924.45

This table represents the performance with OpenDAP Data Management on Ce-01

5000 10000 15000 20000 25000 30000 35000 40000 ce-01.grid.sissa.it t2-ce- 05.lnl.infn.it

time (in sec)

CE

OpenDAP -data from thredds server 4 8 16 32

• The figure shows the overall time taken for both pre-processing and execution
• n 2 different computing elements.
• Using the OpenDAP approach pre-processing and execution is done in a single

go.

• The model fetches data from the Thredds server at each step and completes the

execution.

• The difference in performance is solely due to the difference in time taken in

fetching the data from the server, execution time is almost similar to that taken by the relocatable package. Graph representing model performance with OpenDAP

Conclusions

• Climate simulations can be performed on any

machine available on the Grid infrastructure using the relocatable package.

• Modern resources multicore architecture on grid

infrastructure can deliver the same kind of performance as the local HPC resources within the same computational nodes.

• OpenDAP

approach is implemented

• n

grid infrastructure which enables more efficient data management.

• Time taken using OpenDAP protocol is comparable

and usage of this protocol is much easier and more efficient on any computational platform.