Many Task Computing for Modeling the Fate of Oil Discharged from the - - PowerPoint PPT Presentation

▶
many task computing for modeling the fate of oil
SMART_READER_LITE
LIVE PREVIEW

Many Task Computing for Modeling the Fate of Oil Discharged from the - - PowerPoint PPT Presentation

Aug 27, 2023 268 likes •547 views

Many Task Computing for Modeling the Fate of Oil Discharged from the Deep Water Horizon Well Blowout Ashwanth Srinivasan, Judith Helgers, Matthieu LeHenaff, Claire Paris, HeeSook Kang, Villy Kourafalou, Carlisle Thacker, Mohamed Iskandarani, Joel

slide-1
SLIDE 1

Many Task Computing for Modeling the Fate of Oil Discharged from the Deep Water Horizon Well Blowout

Ashwanth Srinivasan, Judith Helgers, Matthieu LeHenaff, Claire Paris, HeeSook Kang, Villy Kourafalou, Carlisle Thacker, Mohamed Iskandarani, Joel Zysman, Nick Tsinoremas

University of Miami, Miami, FL, USA

and Omar Knio

Johns Hopkins University, Baltimore, MD, USA MTAGS-2010 - Nov 15th, 2010 – p. 1/27

slide-2
SLIDE 2

Marine Rapid Environmental Assessment

  • Support for marine environmental disasters, search&rescue, hurricane intensity prediction,

Naval operations and other applications.

  • Under time constraints and quasi-operational in nature
  • Typically need to combine multiple diverse models to construct the application
  • Component models are from different specialty areas –> not necessarily interoperable
  • Parametric uncertainty and model error leads to uncertainty in model predictions
  • Uncertainty quantification through Monte Carlo analysis, parameter sweeps,stochastic

spectral projections

  • Several hundreds to thousands of model runs might be needed for robust uncertainty

quantification

MTAGS-2010 - Nov 15th, 2010 – p. 2/27

slide-3
SLIDE 3

Deep Water Horizon Well Blowout

  • DWH well blowout discharged an unknown quantity of crude into the Gulf of Mexico
  • Very little is known about the oil in the deep - difficult to observe or estimate
  • Numerical models are the best repositories of our knowledge about the oceans
  • Simulations allow us to evaluate various scenarios and provide guidance for field

measurements

  • Multiple simulations will allow us to form a statistical picture while accounting for

uncertainties

MTAGS-2010 - Nov 15th, 2010 – p. 3/27

slide-4
SLIDE 4

Questions:

  • What is the fate of discharged oil?
  • Are there plumes of oil trapped below the surface?
  • Where might the subsurface oil end up?
  • What impact, if any, will this spill have on the Florida Keys?

MTAGS-2010 - Nov 15th, 2010 – p. 4/27

slide-5
SLIDE 5

Modeling the fate of the discharged oil

  • Use state-of-the art 3D ocean circulation models to obtain information on currents, density

etc.

  • Combine information from ocean circulation models with a 3D oil model which simulates

time history of the oil

  • Current operational ocean prediction systems provide information at 3-4 km / 50 m

resolution in the horizontal/vertical

  • 1-2 km / 10 m is desirable especially in the Florida Keys

MTAGS-2010 - Nov 15th, 2010 – p. 5/27

slide-6
SLIDE 6

The nested model domains

  • We adopt a nested approach - embed a higher resolution Florida Keys (900 m) within a 2

km Gulf of Mexico model.

  • Each model simulates resolves specific time and space scales – the combination is

sufficient for our purposes

  • Increases complexity - have to deal with outputs from two coupled simulations

MTAGS-2010 - Nov 15th, 2010 – p. 6/27

slide-7
SLIDE 7

The Oil Model

  • Oil is modeled as aggregation of particles - up to 10 million
  • Particles represent three hydrocarbon fractions - light, medium, heavy
  • 3D spreading due to bouyancy, currents and wind
  • Weathering, biodegradation, evaporation oil filters to remove oil

MTAGS-2010 - Nov 15th, 2010 – p. 7/27

slide-8
SLIDE 8

Uncertainty Propagation and Quantification

  • We need error bars on our calculations

⊲ All three models used here are dependent on many input parameters and empirical

constants

⊲ Almost all these parameters are very poorly known ⊲ Additional uncertainty due to conflicting reports on the leak rate, dispersant use etc. ⊲ Given all this uncertainty, we would like to propagate and quantify it

  • How to assign error bars?

⊲ Traditionally Monte Carlo type of analysis is used ⊲ We use a stochastic spectral expansion based on Wieners Polynomial Chaos

expansions

⊲ Model outputs quantified as a function of input uncertainties ⊲ Allows us to refine results based on new measurements and transfer uncertainties

through the application

⊲ Requires hundreds or thousands of model runs to explore uncertain dimensions of the

problem

MTAGS-2010 - Nov 15th, 2010 – p. 8/27

slide-9
SLIDE 9

How many runs to Quantify Uncertainty?

  • The model solutions are expanded as a polynomial series in the uncertain parameters
  • Problem is to determine the expansion coefficients
  • Number of coefficients, P

, is given by:

P = (N + K)! (N!K!) − 1

  • N is the number of uncertain parameters; K is the order of polynomial used

⊲ Each uncertain parameter adds a dimension to the probability space that must be

explored

⊲ For a 5th order polynomial: ⋆ 2 uncertain parameters - 36 runs ⋆ 3 uncertain parameters - 216 runs ⋆ 4 uncertain parameters -1024 runs ⊲ Leads to a practical limitation that Bellman termed "the curse of dimensionality" ⊲ Also leads to a big data processing problem

MTAGS-2010 - Nov 15th, 2010 – p. 9/27

slide-10
SLIDE 10

The Application

  • Two 3D ocean circulation model + 3D oil model
  • 1 realization of the ocean models; many realization of the oil model
  • Data management via a central THREDDS/OPeNDAP server
  • Runs once daily in Real-Time mode - produces output for 10 days centered on the current

day

  • In hindcast mode the application runs continuously for a time period of 85 days

MTAGS-2010 - Nov 15th, 2010 – p. 10/27

slide-11
SLIDE 11

Characteristics of the Application

  • The application consists of three loosely coupled components
  • The computational requirements and execution characteristics differ between the

components

  • Uncertainty quantification requires 12000-36000 oil model runs for periods of 30 min to 96

hours

  • The number of tasks required depends on the convergence of the results, and is therefore

dynamic

  • They models execute on a 640 core IBM P575 and a 5040 core Linux cluster operated by

the University of Miami

  • They components communicate via files using a OPeNDAP server
  • The oil application is a good example of Many Task Computing in Marine Rapid

Environment Assessment

MTAGS-2010 - Nov 15th, 2010 – p. 11/27

slide-12
SLIDE 12

Details of the application

  • Details and execution characteristics of the ocean model component
  • Details and execution characteristics of the oil model component
  • Data flow using the THREDDS/OPeNDAP Server
  • MTC Workflow

MTAGS-2010 - Nov 15th, 2010 – p. 12/27

slide-13
SLIDE 13

The Ocean Model Component

  • We use the HYbrid Coordinate Ocean Model (HYCOM) as the ocean circulation model
  • Solves the Navier-Stokes equations as applied to a thin fluid layer on a rotating Earth
  • Well tested community code; large number of users
  • The model is written in a mix of Fortran-77/Fortran-90
  • Parallelized using a SPMD approach - 2D domain decomposition
  • Implemented using MPI and OpenMP - can be run in a hybrid OpenMP+MPI mode
  • Message Passing code requires a low latency interconnect for communication
  • Model solutions bit for bit reproducible on any number of cores - no global sums/reduction
  • perations

MTAGS-2010 - Nov 15th, 2010 – p. 13/27

slide-14
SLIDE 14

The GOM-HYCOM Component

  • Model State vector - 1073 x 769 x 26
  • 2 min time step
  • Configuration requires 28 GB memory and 2 GB of data input
  • Runs daily for 10 model days (T-5 through T+4 days)
  • 40 GB output per run
  • Requires O(100) cores for efficient daily use
  • We run this model on 64 cores

MTAGS-2010 - Nov 15th, 2010 – p. 14/27

slide-15
SLIDE 15

The SFFS-HYCOM Component

  • Model State vector - 437 x 361 x 26
  • 20 sec time step - makes it compute intensive
  • Configuration requires 4 GB memory and 0.5 GB of data input
  • Runs daily for 10 model days (T-5 through T+4 days)
  • 12 GB output per run
  • Requires few 10 of cores for efficient daily use
  • We run this model on 8 cores

MTAGS-2010 - Nov 15th, 2010 – p. 15/27

slide-16
SLIDE 16

Sample Computation/IO Stat for the SFFS-HYCOM

xctilr calls = 128968 time = 200.05331 zaio** calls = 21 time = 1.38842 zaiord calls = 53 time = 0.18225 zaiowr calls = 2002 time = 3.82392 zaioIO calls = 2055 time = 0.00000 xc**** calls = 1 time = 223.18875 cnuity calls = 720 time = 180.00045 tsadvc calls = 720 time = 248.62906 momtum calls = 720 time = 614.14717 barotp calls = 720 time = 85.60713 thermf calls = 720 time = 44.78050 ic**** calls = 720 time = 0.49241 mx**** calls = 720 time = 258.04346 conv** calls = 720 time = 0.00021 diapf* calls = 720 time = 0.00016 hybgen calls = 720 time = 206.89063 restrt calls = 1 time = 0.75360 archiv calls = 8 time = 4.81984 total calls = 1 time = 1645.05471

MTAGS-2010 - Nov 15th, 2010 – p. 16/27

slide-17
SLIDE 17

The Oil Model Component

  • Is data intensive - requires full outputs (2GB) from two ocean models every two compute

steps

  • oil computations are cheap - 40 % of the time spent in IO operations
  • Parallelized using a task pool approach
  • Domain decomposition not used for this configuration
  • The tasks can also be run as non-parallelized batch of single jobs
  • In near real-time mode the model requires 60 GB input and produces 15 GB output
  • In hindcast mode the model requires 0.5 TB input and produces 1.2 TB output

MTAGS-2010 - Nov 15th, 2010 – p. 17/27

slide-18
SLIDE 18

DataFlow using OPeNDAP

  • OPeNDAP: Open-source Project for a Network Data Access Protocol
  • We use the THREDDS/OPeNDAP server for data management and data flow
  • Why OPeNDAP?

⊲ Enables remote data access ⊲ Hundreds of files are used in a single near real-time run - difficult to handle ⊲ Ocean model outputs are 3D daily files with no time axis between successive files ⊲ Oil model needs a 4D space-time sequence of files ⊲ OPeNDAP aggregation provides a 4D dataset for the oil model and simplifies

complexity

MTAGS-2010 - Nov 15th, 2010 – p. 18/27

slide-19
SLIDE 19

OPeNDAP Load Test

  • Client processes read data without writing

Concurrent Requests Time for 2 GB Time for 10 GB 1 45 secs 3 min 28 s 5 52 secs 3 min 48 secs 10 1 min 48 secs 9 min 20 secs 15 3 min 48 secs 22 min 14 secs

MTAGS-2010 - Nov 15th, 2010 – p. 19/27

slide-20
SLIDE 20

OPeNDAP Load Test

  • Client processes read and write regridded data

Concurrent Requests Time for 2 GB Time for 10 GB 1 1 min 48 secs 10 min 28 s 5 1 min 52 secs 10 min 48 secs 10 2 min 20 secs 11 min 40 secs 15 2 min 58 secs 13 min 14 secs

  • Might not be suitable to support hundreds of requests - but is good enough for 10 requests
  • Data is downloaded from OPeNDAP by the master task and staged locally before model

runs

MTAGS-2010 - Nov 15th, 2010 – p. 20/27

slide-21
SLIDE 21

Workflow

  • Many sophisticated tools to manage workflows - Kepler, Taverna and others
  • Need to deploy rapidly - so did not explore these tools yet
  • For now we use a scripting approach to implement the workflow
  • Multiple Perl scripts are used for end-to-end automation
  • A master script is used to manage the overall workflow
  • Component scripts to launch and manage the three components

MTAGS-2010 - Nov 15th, 2010 – p. 21/27

slide-22
SLIDE 22

Workflow

MTAGS-2010 - Nov 15th, 2010 – p. 22/27

slide-23
SLIDE 23

Near Real-Time MTC Application - Run for 10 days

Component No of Tasks

  • Avg. Time

Data I/O per Run GOM-HYCOM 64 200 (min) 20 (GB) / 40 (GB) SFFS-HYCOM 8 300 (min) 5 (GB) / 8 (GB) Oil-Model 12000 30 (min) 50 (GB) / 14 (GB) Entire Application 12100 600 (min) 75 (GB) / 60 (GB)

  • Real-time runs need to finish on time
  • Advance reservations were needed to guarantee timely completion
  • Oil-Model tasks clustered into 25 separate 480 task MPI jobs
  • Clustering takes advantage of local job scheduling policies

MTAGS-2010 - Nov 15th, 2010 – p. 23/27

slide-24
SLIDE 24

Hindcast MTC Application - Run for 85 days

Component No of Tasks Total Time Data I/O per Run GOM-HYCOM 64 2 days 100 (GB) / 320 (GB) SFFS-HYCOM 16 1.5 days 50 (GB) / 100 (GB) Oil-Model 37000 4 days 420 (GB) / 1.2(TB) Entire Application 37000 6 days 0.5 (TB) / 1.5 (TB)

  • Not as time critical as real-time runs
  • Resource constraints, delays, wait time and scheduling policies increase the time to

completion by 3 days

  • Better workflow with sophisticated tools can be used to schedule concurrent computations

and reduce overall time

  • Oil Model tasks clustered into 36 X 1024 task MPI jobs
  • Oil model could also be launched as 37000 single jobs if resources are available

MTAGS-2010 - Nov 15th, 2010 – p. 24/27

slide-25
SLIDE 25

Oil model and system performance on 1024 cores

  • We did not notice any significant system related bottleneck during these runs
  • The time/model increases with core count but not by much
  • Might be a significant issue more tasks are used

MTAGS-2010 - Nov 15th, 2010 – p. 25/27

slide-26
SLIDE 26

Sample Results

MTAGS-2010 - Nov 15th, 2010 – p. 26/27

slide-27
SLIDE 27

Summary

  • A multi-component application was sucessfully deployed for modeling the fate of oil

discharged from DWH. The application is a typical example of Many Task Computing for Marine Rapid Environmental Assessment

  • We are exploring more sophisticated workflow engines to utilize larger set of resources

such as the TeraGrid. The infrastructure setup here will be very useful for rapid deployment of such applications

MTAGS-2010 - Nov 15th, 2010 – p. 27/27