Some Organizing Principles for Coupling in Multiphysics and - - PowerPoint PPT Presentation

Some Organizing Principles for Coupling in Multiphysics and Multiscale Models

• J. Walter Larson

ANU Supercomputer Facility The Australian National University Presented at the Biennial Computational Techniques and Applications Conference (CTAC ’06) Townsville QLD Australia 3 July 2006

CTAC ’06, Townsville QLD Australia

Overview

• Multiphysics and Multiscale

Models

• The Coupling and Parallel

Coupling Problems

• Some Organizing Principles
• Conclusions + Future Work

CTAC ’06, Townsville QLD Australia

Multiphysics Systems

• A bold effort to move the

practice of modeling from single systems in isolation to simulation of entire systems comprising multiple mutually interacting models

• Examples abound:
• Climate and Weather
• Fusion device simulation
• Space weather
• Reactive flow (e.g.,

combustion)

• Fluid structure interactions
• Groundwater hydrology

CTAC ’06, Townsville QLD Australia

Multiscale Systems

Tropical Cyclone Larry 20 March 2006 03:55UT

• Physical systems that exhibit

novel interacting phenomena that is basically the same type

• f physics on multiple time

scales

• Fluids exhibit this property
• Weather (pictured) is a classic

example of such multiscale modeling in operational use today

CTAC ’06, Townsville QLD Australia

The Ties that Bind

• Multiphysics and Multiscale systems face a common

challenge--coupling

• The Coupling Problem--Data sharing and transformation

in models that execute von Neumann style (i.e., single process)

• The Parallel Coupling Problem--Distributed data

description, parallel data transfer and redistribution, and parallel data transformation in models that employ distributed-memory (a.k.a. message-passing parallelism)

CTAC ’06, Townsville QLD Australia

Organizing Principle #0: The Parallel Coupling Problem can be split into two distinct parts: coupled model architecture and parallel data processing

CTAC ’06, Townsville QLD Australia

Parallel Coupling Problem

Coupled Model Architecture

• Science of the system under study:
• Connectivity--who talks to

whom?

• Coupling event scheduling

(e.g., periodic?)

• Domain overlap--lower-

dimensional vs. colocation

• Timescale separation/

interaction & domain overlap

• Tightness
• Implementation choices:
• Resource allocation
• Scheduling of model execution
• Number of executables
• Mechanism

Parallel Data Processing

• Description of data to be exchanged

during coupling

• Physical fields/variables
• Mesh or representation associated

with the data

• Domain decomposition
• Transfer of data--a.k.a. the MxN

problem

• Transformation of data
• Intermesh interpolation/

transformation between representations (and associated conservation issues)

• Time transformation
• Diagnostic/variable

transformations

• Merging of data from multiple

sources

CTAC ’06, Townsville QLD Australia

Organizing Principle #1: The Parallel Data Processing part of the parallel coupling problem is largely solved--concentrate instead on parallel coupled model architecture

CTAC ’06, Townsville QLD Australia

Organizing Principle #2: Graph Theory is relevant

CTAC ’06, Townsville QLD Australia

Notions from Graph Theory

• A coupled model has a directed graph representation
• Nodes represent individual models/components
• A directed arc from model A to model B represents a data

dependency of input to model B on output from model A

• In-valency of a node corresponds to number of distinct inputs from
• ther models
• Out-valency of a node corresponds to the number of distinct outputs

to other models

• The representation can not be factored into two or more disjoint

graphs; otherwise one is representing multiple isolated coupled models

• Typically, the directed graph representation is cyclic

CTAC ’06, Townsville QLD Australia

C3 C1 C2 C4 A Famous Example

Atmosphere Sea-Ice Ocean Land

Coupled Climate Model (CCSM)

CTAC ’06, Townsville QLD Australia

Organizing Principle #3: The matrix representation of Directed Graphs is relevant

CTAC ’06, Townsville QLD Australia

Connectivity

• Connectivity of a coupled model can be viewed as a

directed graph, where nodes represent subsystem models and directed edges point from a source of coupling data to a target

• Can be represented in terms of a connectivity matrix K

Kij =

• 1,

if Ci receives data from Cj, 0, if i = j, or if Ci does not receive data from Cj

• Model C is directly dependent on C if K = 1
• Model C is first-order indirectly dependent on C if K = 0 but

C is directly dependent on another model C that is directly dependent on C

• Possible to define higher-order indirect dependencies as well

i j k ij i j ij i j

CTAC ’06, Townsville QLD Australia

Organizing Principle #4: Coupling tightness can be quantified

CTAC ’06, Townsville QLD Australia

The Load Matrix

A logical starting point is the cost break-down structure for the coupled model. Define the load matrix L as Lij =

• cost of running Ci in isolation,

if i = j, cost of getting output from Cj to Ci if i = j where costs are defined in a suitable unit such as cpu node- seconds, and can be decomposed as

Lij = Rij + Sji

Cost to model i to receive from model j Cost to model j to send to model i

CTAC ’06, Townsville QLD Australia

Tightness vs. Looseness

The tightness of an individual model’s coupling to the rest of the system is given by its coupling overhead The tightness of the coupling between models i and j

τij = τji = Lij + Lji Lii + Ljj Oi = N

j,j=i(Sij + Rij)

Lii

• Tight and loose coupling are often mentioned, but, there is no clear

definition

• Can be defined following communications/compute ratio from HPC

CTAC ’06, Townsville QLD Australia

Organizing Principle #5: Definition of terms for Resource Allocation and Component Scheduling

CTAC ’06, Townsville QLD Australia

Resource Allocation and Component Scheduling

• Single component multiple data (SCMD) approach, a.k.a. Serial Composition
• Single global resource (e.g. MPI processes) pool
• Sequential scheduling of components as an event loop
• Example: NCAR/DOE Parallel Climate Model (PCM)
• Advantage: Load balancing more straightforward
• Disadvantage: “Weakest link” Amdahl’s law scaling limitations and wasted processor time
• Multiple component multiple data (MCMD) approach, a.k.a. Parallel Composition
• Partition global resource pool into disjoint sets
• Components scheduled concurrently, each on its respective processor pool
• Example: NCAR Community Climate System Model (CCSM)
• Advantage: Mitigation of disparity in component scaling problem
• Disadvantage: Intercomponent load balance challenge--the stalling problem
• Hybrid SCMD/MCMD

CTAC ’06, Townsville QLD Australia

Organizing Principle #6: Definition of terms for Dynamic Load Balance in Parallel Coupled Models

CTAC ’06, Townsville QLD Australia

Levels of Dynamic Load

Strategy Intra-component Inter-component Global Level 0 (Static) Static Static Static Level 1 Dynamic Static Static Level 2 Dynamic Dynamic Static Level 3 Dynamic Dynamic Dynamic

CTAC ’06, Townsville QLD Australia

Requirements Stemming from Dynamic Load Balance

Cumulative growth in requirements as constraints on scheduling and resource allocation are loosened: Level 0: No additional requirements Level 1: Fast handshaking between components to cope with changing decompositions between coupling events Level 2: Ability of framework or coupling mechanism to checkpoint models and re-instantiate/restart them on their new processor pools Level 3: Ability of underlying communications mechanism to cope with dynamically varying global resource pool (e.g., dynamic MPI_COMM_WORLD)

CTAC ’06, Townsville QLD Australia

Conclusions + Future Work

• Multiscale and Multiphysics models almost always confront the parallel

coupling problem

• The data processing part of the parallel coupling problem is largely

amenable to automation

• The Model Architecture part of the problem has some organizing

principles

• Graph theory offers a good jumping-off point to define a number of

quantities for parallel coupled models

• Future work is to cement these notions and exploit them in larger

systems comprising greater numbers of components