Many Task Computing for Modeling the Fate of Oil Discharged from the - PowerPoint PPT Presentation

Many Task Computing for Modeling the Fate of Oil Discharged from the Deep Water Horizon Well Blowout Ashwanth Srinivasan, Judith Helgers, Matthieu LeHenaff, Claire Paris, HeeSook Kang, Villy Kourafalou, Carlisle Thacker, Mohamed Iskandarani, Joel Zysman, Nick Tsinoremas University of Miami, Miami, FL, USA and Omar Knio Johns Hopkins University, Baltimore, MD, USA MTAGS-2010 - Nov 15 th , 2010 – p. 1/27

Marine Rapid Environmental Assessment • Support for marine environmental disasters, search&rescue, hurricane intensity prediction, Naval operations and other applications. • Under time constraints and quasi-operational in nature • Typically need to combine multiple diverse models to construct the application • Component models are from different specialty areas –> not necessarily interoperable • Parametric uncertainty and model error leads to uncertainty in model predictions • Uncertainty quantification through Monte Carlo analysis, parameter sweeps,stochastic spectral projections • Several hundreds to thousands of model runs might be needed for robust uncertainty quantification MTAGS-2010 - Nov 15 th , 2010 – p. 2/27

Deep Water Horizon Well Blowout • DWH well blowout discharged an unknown quantity of crude into the Gulf of Mexico • Very little is known about the oil in the deep - difficult to observe or estimate • Numerical models are the best repositories of our knowledge about the oceans • Simulations allow us to evaluate various scenarios and provide guidance for field measurements • Multiple simulations will allow us to form a statistical picture while accounting for uncertainties MTAGS-2010 - Nov 15 th , 2010 – p. 3/27

Questions: • What is the fate of discharged oil? • Are there plumes of oil trapped below the surface? • Where might the subsurface oil end up? • What impact, if any, will this spill have on the Florida Keys? MTAGS-2010 - Nov 15 th , 2010 – p. 4/27

Modeling the fate of the discharged oil • Use state-of-the art 3D ocean circulation models to obtain information on currents, density etc. • Combine information from ocean circulation models with a 3D oil model which simulates time history of the oil • Current operational ocean prediction systems provide information at 3-4 km / 50 m resolution in the horizontal/vertical • 1-2 km / 10 m is desirable especially in the Florida Keys MTAGS-2010 - Nov 15 th , 2010 – p. 5/27

The nested model domains • We adopt a nested approach - embed a higher resolution Florida Keys (900 m) within a 2 km Gulf of Mexico model. • Each model simulates resolves specific time and space scales – the combination is sufficient for our purposes • Increases complexity - have to deal with outputs from two coupled simulations MTAGS-2010 - Nov 15 th , 2010 – p. 6/27

The Oil Model • Oil is modeled as aggregation of particles - up to 10 million • Particles represent three hydrocarbon fractions - light, medium, heavy • 3D spreading due to bouyancy, currents and wind • Weathering, biodegradation, evaporation oil filters to remove oil MTAGS-2010 - Nov 15 th , 2010 – p. 7/27

Uncertainty Propagation and Quantification • We need error bars on our calculations ⊲ All three models used here are dependent on many input parameters and empirical constants ⊲ Almost all these parameters are very poorly known ⊲ Additional uncertainty due to conflicting reports on the leak rate, dispersant use etc. ⊲ Given all this uncertainty, we would like to propagate and quantify it • How to assign error bars? ⊲ Traditionally Monte Carlo type of analysis is used ⊲ We use a stochastic spectral expansion based on Wieners Polynomial Chaos expansions ⊲ Model outputs quantified as a function of input uncertainties ⊲ Allows us to refine results based on new measurements and transfer uncertainties through the application ⊲ Requires hundreds or thousands of model runs to explore uncertain dimensions of the problem MTAGS-2010 - Nov 15 th , 2010 – p. 8/27

How many runs to Quantify Uncertainty? • The model solutions are expanded as a polynomial series in the uncertain parameters • Problem is to determine the expansion coefficients • Number of coefficients, P , is given by: P = ( N + K )! − 1 ( N ! K !) • N is the number of uncertain parameters; K is the order of polynomial used ⊲ Each uncertain parameter adds a dimension to the probability space that must be explored ⊲ For a 5th order polynomial: ⋆ 2 uncertain parameters - 36 runs ⋆ 3 uncertain parameters - 216 runs ⋆ 4 uncertain parameters -1024 runs ⊲ Leads to a practical limitation that Bellman termed "the curse of dimensionality" ⊲ Also leads to a big data processing problem MTAGS-2010 - Nov 15 th , 2010 – p. 9/27

The Application • Two 3D ocean circulation model + 3D oil model • 1 realization of the ocean models; many realization of the oil model • Data management via a central THREDDS/OPeNDAP server • Runs once daily in Real-Time mode - produces output for 10 days centered on the current day • In hindcast mode the application runs continuously for a time period of 85 days MTAGS-2010 - Nov 15 th , 2010 – p. 10/27

Characteristics of the Application • The application consists of three loosely coupled components • The computational requirements and execution characteristics differ between the components • Uncertainty quantification requires 12000-36000 oil model runs for periods of 30 min to 96 hours • The number of tasks required depends on the convergence of the results, and is therefore dynamic • They models execute on a 640 core IBM P575 and a 5040 core Linux cluster operated by the University of Miami • They components communicate via files using a OPeNDAP server • The oil application is a good example of Many Task Computing in Marine Rapid Environment Assessment MTAGS-2010 - Nov 15 th , 2010 – p. 11/27

Details of the application • Details and execution characteristics of the ocean model component • Details and execution characteristics of the oil model component • Data flow using the THREDDS/OPeNDAP Server • MTC Workflow MTAGS-2010 - Nov 15 th , 2010 – p. 12/27

The Ocean Model Component • We use the HYbrid Coordinate Ocean Model (HYCOM) as the ocean circulation model • Solves the Navier-Stokes equations as applied to a thin fluid layer on a rotating Earth • Well tested community code; large number of users • The model is written in a mix of Fortran-77/Fortran-90 • Parallelized using a SPMD approach - 2D domain decomposition • Implemented using MPI and OpenMP - can be run in a hybrid OpenMP+MPI mode • Message Passing code requires a low latency interconnect for communication • Model solutions bit for bit reproducible on any number of cores - no global sums/reduction operations MTAGS-2010 - Nov 15 th , 2010 – p. 13/27

The GOM-HYCOM Component • Model State vector - 1073 x 769 x 26 • 2 min time step • Configuration requires 28 GB memory and 2 GB of data input • Runs daily for 10 model days (T-5 through T+4 days) • 40 GB output per run • Requires O(100) cores for efficient daily use • We run this model on 64 cores MTAGS-2010 - Nov 15 th , 2010 – p. 14/27

The SFFS-HYCOM Component • Model State vector - 437 x 361 x 26 • 20 sec time step - makes it compute intensive • Configuration requires 4 GB memory and 0.5 GB of data input • Runs daily for 10 model days (T-5 through T+4 days) • 12 GB output per run • Requires few 10 of cores for efficient daily use • We run this model on 8 cores MTAGS-2010 - Nov 15 th , 2010 – p. 15/27

Sample Computation/IO Stat for the SFFS-HYCOM xctilr calls = 128968 time = 200.05331 zaio** calls = 21 time = 1.38842 zaiord calls = 53 time = 0.18225 zaiowr calls = 2002 time = 3.82392 zaioIO calls = 2055 time = 0.00000 xc**** calls = 1 time = 223.18875 cnuity calls = 720 time = 180.00045 tsadvc calls = 720 time = 248.62906 momtum calls = 720 time = 614.14717 barotp calls = 720 time = 85.60713 thermf calls = 720 time = 44.78050 ic**** calls = 720 time = 0.49241 mx**** calls = 720 time = 258.04346 conv** calls = 720 time = 0.00021 diapf* calls = 720 time = 0.00016 hybgen calls = 720 time = 206.89063 restrt calls = 1 time = 0.75360 archiv calls = 8 time = 4.81984 total calls = 1 time = 1645.05471 MTAGS-2010 - Nov 15 th , 2010 – p. 16/27

The Oil Model Component • Is data intensive - requires full outputs (2GB) from two ocean models every two compute steps • oil computations are cheap - 40 % of the time spent in IO operations • Parallelized using a task pool approach • Domain decomposition not used for this configuration • The tasks can also be run as non-parallelized batch of single jobs • In near real-time mode the model requires 60 GB input and produces 15 GB output • In hindcast mode the model requires 0.5 TB input and produces 1.2 TB output MTAGS-2010 - Nov 15 th , 2010 – p. 17/27

DataFlow using OPeNDAP • OPeNDAP: Open-source Project for a Network Data Access Protocol • We use the THREDDS/OPeNDAP server for data management and data flow • Why OPeNDAP? ⊲ Enables remote data access ⊲ Hundreds of files are used in a single near real-time run - difficult to handle ⊲ Ocean model outputs are 3D daily files with no time axis between successive files ⊲ Oil model needs a 4D space-time sequence of files ⊲ OPeNDAP aggregation provides a 4D dataset for the oil model and simplifies complexity MTAGS-2010 - Nov 15 th , 2010 – p. 18/27