Parallel computations for weather research and environment - - PowerPoint PPT Presentation

Parallel computations for weather research and environment protection

Alexander V. Starchenko

Tomsk State University, Institute of Atmospheric Optics

Introduction

One of the methods for research and forecast developing above a bounded territory local atmospheric processes and pollution transport is a mathematical modelling, based on application of mesoscale meteorological models and regional environmental models.

Historical facts...

• British mathematician L.F.Richardson first proposed numerical

weather prediction in 1922. Richardson attempted to perform a numerical forecast but it was not successful. He and Russian scientist A.A.Freedman were first in 20th year of the last century, who understand a necessity of application of high performance resources for meteorological forecasts.

• The first successful numerical prediction was performed in 1950 by a

team composed of the American meteorologist J. Charney, Norwegian meteorologist Ragnar Fjörtoft and applied mathematician J. Neumann, using the ENIAC digital computer. They used a simplified form of atmospheric dynamics based on the barotropic vorticity equation. This simplification greatly reduced demands on computer time and memory, so that the computations could be performed on the relatively primitive computers available at the time. Later models used more complete equations for atmospheric dynamics and thermodynamics.

• Operational numerical weather prediction (i.e., routine predictions for

practical use) began in 1955 under a joint project by the U.S. Air Force, Navy, and Weather Bureau

• A mesoscale model is a

numerical weather prediction model with sufficiently high horizontal and vertical resolution to forecast mesoscale weather phenomena. These phenomena are often forced by topography or coastlines, or are related to convection.

• Most severe weather occurs at

the mesoscale, including tornadoes and mesoscale convective systems. Visibility, turbulence, sensible weather can vary enormously over just a few kilometers and have a tremendous impact on

• perations.

Mesoscale models and processes

Mesoscale models

include, as a rule, unsteady three-dimensional equations of hydro- thermodynamics and differ by various approaches of parameterisation of atmospheric processes.

= ∂ ρ ∂ + ∂ ρ ∂ + ∂ ρ ∂ + ∂ ρ ∂ z ) w ( y ) v ( x ) u ( t

Continuity equation Equations of hydrodynamics

      ∂ ∂ Γ ∂ ∂ +         ∂ ∂ + ∂ ∂ Γ + + ∂ ∂ − =         ∂ ∂ + ∂ ∂ + ∂ ∂ + ∂ ∂ z u z y u x u fv x p z u w y u v x u u t u

m Z H 2 2 2 2

ρ ρ       ∂ ∂ Γ ∂ ∂ +         ∂ ∂ + ∂ ∂ Γ + − ∂ ∂ − =         ∂ ∂ + ∂ ∂ + ∂ ∂ + ∂ ∂ z v z y v x v fu y p z v w y v v x v u t v

m Z H 2 2 2 2

ρ ρ       ∂ ∂ Γ ∂ ∂ +         ∂ ∂ + ∂ ∂ Γ + − ∂ ∂ − =         ∂ ∂ + ∂ ∂ + ∂ ∂ + ∂ ∂ z w z y w x w g z p z w w y w v x w u t w

m Z H 2 2 2 2

ρ ρ

Basic governing equations of MM

Basic governing equations of MM

Energy equation Humidity equation Equation of state

        + − = ρ =

O H air

M q M q R R , RT p

2

1

p

c / R

) p / p ( T = θ

θ

θ θ θ θ θ θ θ ρ Φ + +       ∂ ∂ Γ ∂ ∂ +         ∂ ∂ + ∂ ∂ Γ =         ∂ ∂ + ∂ ∂ + ∂ ∂ + ∂ ∂ R z z y x z w y v x u t

h Z H 2 2 2 2 q q Z q H q H

z q z y q y x q x z q w y q v x q u t q Φ +       ∂ ∂ Γ ∂ ∂ +         ∂ ∂ Γ ∂ ∂ +       ∂ ∂ Γ ∂ ∂ =         ∂ ∂ + ∂ ∂ + ∂ ∂ + ∂ ∂ ρ

c c Z c H c H

z c z y c y x c x z c w y c v x c u t c Φ +       ∂ ∂ Γ ∂ ∂ +         ∂ ∂ Γ ∂ ∂ +       ∂ ∂ Γ ∂ ∂ =         ∂ ∂ + ∂ ∂ + ∂ ∂ + ∂ ∂ ρ

Pollutant continuity equations

Regional models of air pollution transport

Modern photochemical dispersion models are based on Eulerian approach that allows for an integrated “one-atmosphere” assessm

• f gaseous and particulate air pollution over many scales rangin

from sub-urban to continental. These models simulate the emission, dispersion, chemical reaction and removal of pollutants in the troposphere by solving the pollutant continuity equation for each chemical species on a system of nested three-dimensional grids.

Here С is a generalized variable: С = 1, u, v, w, θ, q, c. Κ, Δ are advection and diffusion terms of equation. The equations with initial and boundary conditions for dependent variables and information on continua properties comes to mathamatical formulation of the problem. Computer realization of the models is based on application of non-trivial numerical algorithms and requires high performance computational resources. f t C

z y x z y x

+ ∆ + ∆ + ∆ = Κ + Κ + Κ + ∂ ∂ In general, the considered conservation laws for mass, impulse and energy may be written as follows:

Numerical methods for solution

( )

1 , , 1 , , 1 , , , , 1 , ,

) (

+ + + +

+ Λ + Λ + Λ = Κ + Κ + Κ + −

n k j i n k j i z y x n k j i z y x n k j i n k j i

f C C τ Implicit finite-differencing scheme

n k j i n k j i z n k j i y x n k j i z y x n k j i n k j i

f C C

, , 1 , , , , , , , , , 1 , ,

) ( ) ( + Λ + Λ + Λ = Κ + Κ + Κ + −

+ +

τ Explicit-implicit differencing scheme Grids: equidistant grid in longitude and latitude, non-equidistant in vertical direction ( σ-co-ordinate). High order approximation of derivatives in space Conditionally stable: τ(sec)<3dx(km), sweeps in vertical co-ordinate Absolutely stable, non-linearity of differencing equations

Tomsk Region Tomsk City

Tomsk Ob river Tom river Kemerovo

Technology of computations in nested domains

Examples of ММ: Mesoscale Model 5

• The MM5 mesoscale model has been developed at Penn

State and NCAR as a community mesoscale model.

• The MM5 includes a multiple-nest capability,

nonhydrostatic dynamics, which allows the model to be used at a few-kilometer scale, multitasking capability on shared- and distributed-memory machines, more physics

• ptions.
• The MM5 is applied to various supercomputers such as

Compaq Cluster, CrayT3E, IBM SP, SGI Origin, SUN, PC Linux Cluster. DM parallelization is based on 2D domain decomposition and MPI standard.

MM5 is installed on Linux clusters at TSU and IAO

Financial support of RFBR, grant N 04-07-90219

Examples of ММ: WRF

• The WRF is being developed as a collaborative effort

among the NCAR, NCEP and others.

• The WRF model is designed to be a flexible, state-of-

the-art, portable code that is efficient in a massively parallel computing environment, which includes support for moving nested grids.

• It offers numerous physics options and is suitable for

use in a broad spectrum of applications across scales ranging from meters to thousands of kilometers.

• It is efficient execution on a range of computing

platforms (distributed and shared memory, vector and scalar types).

Application of the MM5 & WRF

D1 D2 D3

Novosibirsk Tomsk Kemerovo

D1 D2 D3

Novosibirsk Kemerovo

Three nested domains D1 (450х450km), D2 (150х150km), D3 (50х50km) and distribution of land use categories: Water, few vegatation, mixed forest, evergreen forest, urban area

Application of the MM5 & WRF

• Grids 52х52х31 for

domains D1, D2, D3

• Horizontal resolution:

9; 3; 1 km for D1, D2, D3

• Temporal step: 27; 9;

3 sec for D1, D2, D3

• Vertical size of

domains: 17km

• Cluster IAO & TSU
• Grids 52х52х31 for

domains D1, D2, D3

• Horizontal resolution:

9; 3; 1 km for D1, D2, D3

• Temporal step: 60; 30;

10 sec for D1, D2, D3

• Vertical size of

domains: 17 km

• Cluster IAO SB RAS

MM5 MM5 WRF WRF

MM5 domain decomposition for DM parallel

Number of processors is equal 10 = 5 x 2

• Number of nodes = 20
• Node: 2*Pentium III

1GHz, RAM 1Gb

• Network: Gigabit

Ethernet

• Rpeak=20Gflops
• RLinpack=10,5Gflops
• 2001

Distributed Memory parallel system at IAO

Distributed Memory parallel system at TSU

• Number of nodes=283
• Node: 2*Intel Xeon

Woodcrest 2,66GHz, RAM 4Gb, HDD80Gb

• Network: Infinipath
• Rpeak=12000Gflops
• RLinpack=9013Gflops
• 2007

Results of MM5 and WRF application to weather research at Tomsk

• 18-16-14-12-10 -8 -6 -4 -2 0

2 4 6 8 10 12 14 16 18 20 22 24 26 28 30

время, час

2 4 6 8 10

Скорость ветра, м/с

• 18-16-14-12-10 -8 -6 -4 -2 0

2 4 6 8 10 12 14 16 18 20 22 24 26 28 30

время, час

100 200 300 400

Направление ветра, град

• 18-16-14-12-10 -8 -6 -4 -2 0

2 4 6 8 10 12 14 16 18 20 22 24 26 28 30

время, час

• 10
• 8
• 6
• 4
• 2

Температура, С

Гидрометцентр РФ WRF MM5

20.10.2004 21.10.2004 22.10.2004

• 18-16-14-12-10 -8 -6 -4 -2

2 4 6 8 10 12 14 16 18 20 22 24 26 28 30

время, час

2 4 6 8 10

Скорость ветра, м/с

• 18-16-14-12-10 -8 -6 -4 -2

2 4 6 8 10 12 14 16 18 20 22 24 26 28 30

время, час

100 200 300 400

Направление ветра, град

• 18-16-14-12-10 -8 -6 -4 -2

2 4 6 8 10 12 14 16 18 20 22 24 26 28 30

время, час

10 20 30 40

Температура, С

Гидрометцентр РФ WRF MM5 ТОР станция ИОА

16.05.2004 17.04.2004 18.05.2004

a б

Prediction of near surface wind velocity above Tomsk region. 20:00 LST 17.05.2004.

• 25000
• 20000
• 15000
• 10000
• 5000

5000 10000 15000 20000 25000

• 25000
• 20000
• 15000
• 10000
• 5000

5000 10000 15000 20000 25000

• 25000
• 20000
• 15000
• 10000
• 5000

5000 10000 15000 20000 25000

• 25000
• 20000
• 15000
• 10000
• 5000

5000 10000 15000 20000 25000

17 May 2004, 14:00, Domain D3

MM5 MM5 WRF WRF

Speedup and time costs

2 4 6 8 10 12 14 16 18 20

Number of processors

1 2 3 4 5 6 7

Speed up

Models

Mesoscale Model of the Fifth Generation Weather Research and Forecast

2 4 6 8 10 12 14 16 18 20

Number of processors

10 20 30 40 50 60

Time of calculation, min

Temporal period of simulation 1hour

2 4 6 8 10 12 14 16 18 20

Number of processors

1 2 3 4 5 6 7

Speed up

MM5

IAO

Cyberia 2 4 6 8 10 12 14 16 18 20

Number of processors

1 10 100

Time of calculation, min

Temporal period of simulation 1hour

Single process: 6 vs 59 min: 10 processes: 1,4 vs 10 min

Example of RMAQ: TSU-IAO MS

Computer modelling system (MS) for research of weather and air quality in atmospheric boundary layer is being development at TSU and IAO. The MS is applied for numerical prediction of important with point

• f air quality meteorologic parameters (wind velocity and direction,

turbulent diffusivity) above urban territory with rough relief. To increase quality of numerical prediction technology of nested domains was used. Pollution transport model has the following properties:

• Eulerian 3D equations for basic anthropogenic pollutants of near

surface layer;

• Dry deposition (resistance model);
• Photochemical reactions of Hurley’s GRS-mechanism of

tropospheric ozone and PM10 generation (CSIRO);

• Data base of distributed point, area, mobile (linear) sources.

Example of RMAQ: TSU-IAO MS

Governing equations of pollution transport model:

; f t C

z y x z y x

+ ∆ + ∆ + ∆ = Κ + Κ + Κ + ∂ ∂

Equations for such pollutants as

• zone, NO2, NO, CO, SO2, PM2.5, PM10, H2O2, CH, Rsmog

Numerical explicit-implicit finite difference scheme:

n k j i n k j i z n k j i y x n k j i z y x n k j i n k j i

f C C

, , 1 , , , , , , , , , 1 , ,

) ( ) ( + Λ + Λ + Λ = Κ + Κ + Κ + −

+ +

τ Second order approximation in space and the first in time, τ=10sec, dx=500m

Parallel realisation of TSU-IAO MS

x y 2-th processor 3-th processor 4-th processor 1-th processor

Grids: 100x100x30=300000 Equations: 10 Period of simulation: 2days 1D domain decomposition, MPI standard for DM systems

Results of MS application

• 20
• 15
• 10
• 5

5 10 15 20 25 time, hrs 10 20 30 40 50 O3, ppb 26-27 May 2004

Akademgorodok Tomsk TOR-site IAO

• 20
• 15
• 10
• 5

5 10 15 20 25 time, hrs 10 20 30 40 50 NO, ppb

• 20
• 15
• 10
• 5

5 10 15 20 25 time, hrs 40 80 120 160 NO2, ppb 26-27 May 2004

Akademgorodok Tomsk TOR-site IAO

• 20
• 15
• 10
• 5

5 10 15 20 25 time, hrs 200 400 600 800 CO, ppb

Results of MS application

Speedup and time costs

2 4 6 8 10 12 14 16 18 20

Number of processors

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Speed up

MS TSU-IAO

IAO

CYBERIA Sp = p (ideal case) 2 4 6 8 10 12 14 16 18 20

Number of processors

1 10 100 1000

Time of calculation, min

Temporal period of simulation 2hours

Single process: 51 vs 278 min: 10 processes: 5 vs 35 min

Inverse problem of pollution transport

• 5 points of air quality
• bservation in Tomsk
• measurements of near

surface CO are made 3 times per day (7:00, 13:00, 19:00 LST)

• Research domain is

covered by grid. Each cell pollution intensity is unknown and should be found

# S # S # S # S # S # S

• r. T
• m

To define distribution of surface pollution sources and their intensities for real weather conditions and on the basis of measured pollutant concentrations (CO)

Inverse problem of pollution transport

Mathematical formulation of the problem is based on converged equations and dual representation of functional:

) ( * ) ( * ) ( * ) ( *

* * * * * * m m m m z y x z y x

z z y y x x t t t C − δ − δ − δ − δ + + ∆ + ∆ + ∆ = Κ − Κ − Κ − ∂ ∂ − ) , , , ( * = z y x C , * * , * * : = ∂ ∂ Γ + = = ∂ ∂ Γ + = x C uC Lx: x x C uC x , * * v y , * * : = ∂ ∂ Γ + = = ∂ ∂ Γ + = y C C Ly: y C vC y * z , * : = ∂ ∂ Γ = = ∂ ∂ = z C H: z C z

m=1,…Nobs ( )

∑ ∫

=

= =        

N j i m m m m ij ij T j i m

y x t c c S Q dt y x t C

1 , *

, , ) 2 , , , (

Inverse problem of pollution transport

• It is required to solve Nobs pollution

transport problems: Nobs*300000*10*2days

• But because the problems for one point of
• bservation are not dependent each from
• ther, they can be effectively solved on

supercomputer

• Each problem can be solved in parallel with

1D or 2D domain decomposition approach

Results of solution of inverse problem

• 20000 -15000 -10000
• 5000

5000 10000 15000 20000

• 20000
• 15000
• 10000
• 5000

5000 10000 15000 20000

• 20000 -15000 -10000
• 5000

5000 10000 15000 20000

• 20000
• 15000
• 10000
• 5000

5000 10000 15000 20000

• 20000 -15000 -10000
• 5000

5000 10000 15000 20000

• 20000
• 15000
• 10000
• 5000

5000 10000 15000 20000

• 20000 -15000 -10000
• 5000

5000 10000 15000 20000

• 20000
• 15000
• 10000
• 5000

5000 10000 15000 20000

• 20000 -15000 -10000
• 5000

5000 10000 15000 20000

• 20000
• 15000
• 10000
• 5000

5000 10000 15000 20000

30 problems 100x100x30 grid 600 processors SKIF Cyberia speedup 570 Test case 7-8.09.2000 Subdomains with max CO source intensity are coloured

Conclusion

Application of supercomputers makes weather research and forecast and solution of environmental problems faster, more exact and more detailed. To reach high performance of supercomputer in solution of weather research or environmental problems, one need to use in parallelization 2D domain decomposition. To reach high performance in computations of mesoscale meteorological or pollution transport processes with resolution less then 1km, implicit finite-differencing schemes are more preferable in spite of their complexity. The modern supercomputers in Russia allows to use in parallel computations of weather and environmental research hundreds of high performance processors and validated modern mesoscale models and regional pollution transport models.