Applying AD to the Community Land Model Laboratory for Advanced - PowerPoint PPT Presentation

Applying AD to the Community Land Model Laboratory for Advanced Numerical Simulations Mathematics and Computer Science Argonne National Laboratory Azamat Mametjanov , Boyana Norris , Xiaoyan Zeng, Beth Drewniak, Jean Utke , Mihai Anitescu, and Paul Hovland Supported by DOE’s BER under Climate Science for Sustainable Energy Future (CSSEF) 1 / 24

Outline CESM and CLM AD and OpenAD AD Process Results 2 / 24

CESM and CLM 3 / 24

CESM: Community Earth System Model ◮ Is a global model for simulations of the climate system. ◮ Is composed of five fully-coupled components of 1. atmosphere 2. land 3. ocean 4. land-ice 5. sea-ice ◮ Each component is configurable for one the modes ◮ active (fully prognostic) ◮ data (intercomponent data cycling) ◮ inactive (interface) ◮ Provides state-of-the-art simulations of past, present and future climate states (1850–2100) 4 / 24

CLM: Community Land Model Is a CESM component for simulations of 1. energy fluxes in land bio-geophysics 2. chemical compound fluxes in land bio-geochemistry 3. water fluxes in land hydrology Divides the modeled surface grid into grid-cells ◮ land units ◮ columns ◮ plant functional types (PFT) 5 / 24

CLM: Structure Land unit models ◮ glacier ◮ lake ◮ wetland ◮ urban area ◮ vegetated area Column models vertical layers of ◮ soil – up to 15 layers ◮ snow – up to 5 layers Plant functional type models ◮ trees and shrubs – 11 types ◮ grasses – 3 types ◮ crops – 1 type 6 / 24

CLM extension: CLM-Crop Extends CLM with irrigated/managed crop species ◮ corn ◮ wheat ◮ soybean Models pools and fluxes of Carbon and Nitrogen (CN) in ◮ leaves ◮ stems ◮ roots ◮ harvested organs 7 / 24

CLM-Crop 8 / 24

CLM-Crop Parameter Calibration ◮ Crop growth is substantially affected by CN fluxes ◮ CN fluxes are determined by CN ratios of a given crop species ◮ There is a large uncertainty in regards to CN ratios ◮ We study sensitivity of CN ratios to calibrate the parameters for further improvement of model accuracy 9 / 24

AD and OpenAD 10 / 24

Algorithmic Differentiation ◮ Computation encoded by a program can be broken into a sequence of elemental operations ◮ Independent (and varied ) variable values are transformed into ( useful and) dependent variable values ◮ AD computes derivatives of elemental operations acting on active variables ◮ Based on the chain rule, derivative values are accumulated to obtain a derivative of the entire program 11 / 24

OpenAD: modular Fortran/C/C++ AD tool Open64 EDG/ROSE front-ends whirl Sage III Open whirlToXAIF Open64 SageToXAIF Analysis XAIF OpenAD Angel xaifBooster boost (AD source transformation) xerces 12 / 24

AD Process 13 / 24

CLM source code Modules Files Lines Biogeochemistry 39 18568 Biogeophysics 24 13931 Riverroute 3 1435 Couplers/drivers 53 34416 Total 119 68350 14 / 24

CLM module dependencies m_sparsematrixtomaps.mod CNDVEcosystemDynIniMod.F90 m_globaltolocal.mod CNVegStructUpdateMod.o lnd_comp_mct.o CNDVEcosystemDynIniMod.o BalanceCheckMod.F90 clm_driver.F90 m_globalsegmap.mod UrbanMod.o seq_flds_indices.mod spmdMod.o DryDepVelocity.F90 clm_comp.o CNNStateUpdate1Mod.F90 FrictionVelocityMod.o initGridCellsMod.o RtmMod.o CNCStateUpdate2Mod.F90 decompInitMod.o surfrdMod.F90 SoilT CNiniSpecial.F90 emperatureMod.F90 histFileMod.o Biogeophysics2Mod.F90 CNSetValueMod.o scam_setlatlonidx.o SoilHydrologyMod.o shr_strdata_mod.mod m_string.mod fileutils.F90 m_generalgrid.mod clmtype.F90 CNFireMod.o subgridRestMod.o abortutils.o C13SummaryMod.o pftdynMod.F90 STATICEcosysDynMod.o getdatetime.o spmdGathScatMod.o SnowHydrologyMod.F90 CNC13StateUpdate2Mod.o shr_infnan_mod.mod BiogeophysicsLakeMod.o SurfaceRadiationMod.F90 CNPhenologyMod.o CNFireMod.F90 STATICEcosysDynMod.F90 CNC13StateUpdate3Mod.o CNBalanceCheckMod.F90 Biogeophysics1Mod.o filterMod.F90 m_attrvectcomms.mod DUSTMod.o decompMod.F90 m_matattrvectmul.mod SoilHydrologyMod.F90 accumulMod.F90 CNGapMortalityMod.o UrbanInitMod.o CNCStateUpdate1Mod.o clm_time_manager.o shr_kind_mod.mod pftdynMod.o histFldsMod.F90 clm_atmlnd.F90 CNGRespMod.F90 pftvarcon.F90 SNICARMod.o clm_initializeMod.F90 accFldsMod.F90 pftvarcon.o CNWoodProductsMod.o subgridAveMod.F90 clmtypeInitMod.F90 CNAnnualUpdateMod.F90 inicFileMod.F90 CanopyFluxesMod.F90 fileutils.o BiogeophysicsLakeMod.F90 ndepStreamMod.F90 CNGRespMod.o m_mctworld.mod controlMod.F90 CNWoodProductsMod.F90 initSurfAlbMod.o nanMod.o CNDVEstablishmentMod.F90 CNDVLightMod.o CNMRespMod.F90 m_sparsematrixplus.mod C13SummaryMod.F90 Biogeophysics1Mod.F90 CNC13FluxMod.o iniTimeConst.o aerdepMod.F90 BiogeophysRestMod.F90 HydrologyLakeMod.F90 CNNDynamicsMod.F90 CNCStateUpdate2Mod.o system_messages.F90 m_list.mod shr_orb_mod.mod clm_atmlnd.o seq_cdata_mod.mod clm_mct_mod.o filterMod.o lnd_comp_mct.F90 snowdp2lev.o shr_string_mod.mod CNMRespMod.o ndepFileMod.o CNNStateUpdate2Mod.o clm_initializeMod.o restFileMod.F90 m_sparsematrixcomms.mod controlMod.o BiogeophysRestMod.o do_close_dispose.F90 Hydrology1Mod.o CNSetValueMod.F90 CNGapMortalityMod.F90 inicFileMod.o histFileMod.F90 DryDepVelocity.o CNPrecisionControlMod.o ncdio.o ndepFileMod.F90 BareGroundFluxesMod.F90 CNDecompMod.o CNC13StateUpdate2Mod.F90 clm_varorb.o esmf_mod.mod accFldsMod.o CASAMod.o CNAllocationMod.F90 CNDVEstablishmentMod.o surfrdMod.o VOCEmissionMod.F90 scam_setlatlonidx.F90 Hydrology1Mod.F90 SurfaceAlbedoMod.o ncdio.F90 iniTimeConst.F90 shr_mct_mod.mod CNEcosystemDynMod.o clm_varsur.o CNSummaryMod.o SNICARMod.F90 VOCEmissionMod.o UrbanInitMod.F90 dynlandMod.o restFileMod.o clm_mct_mod.F90 shr_ncread_mod.mod SoilT emperatureMod.o SurfaceRadiationMod.o subgridRestMod.F90 CNrestMod.o CNSummaryMod.F90 BalanceCheckMod.o CNiniTimeVar.o initGridCellsMod.F90 m_die.mod areaMod.o pft2colMod.F90 clm_driverInitMod.o CanopyFluxesMod.o CASAPhenologyMod.o CNNDynamicsMod.o spmdMod.F90 CNAllocationMod.o areaMod.F90 CNC13StateUpdate3Mod.F90 clm_time_manager.F90 CNBalanceCheckMod.o SurfaceAlbedoMod.F90 CNC13StateUpdate1Mod.F90 clm_comp.F90 CNNStateUpdate2Mod.F90 CNDVMod.o aerdepMod.o clm_varcon.o accumulMod.o preproc.h Hydrology2Mod.o HydrologyLakeMod.o FracWetMod.o CASAPhenologyMod.F90 CNPhenologyMod.F90 clm_varctl.F90 system_messages.o subgridMod.o subgridAveMod.o RtmMod.F90 RunoffMod.o FrictionVelocityMod.F90 seq_infodata_mod.mod pft2colMod.o CNVegStructUpdateMod.F90 clm_varsur.F90 snowdp2lev.F90 Biogeophysics2Mod.o seq_drydep_mod.mod domainMod.F90 CASAMod.F90 CNCStateUpdate1Mod.F90 abortutils.F90 pio_kinds.mod SnowHydrologyMod.o mct_mod.mod CNiniSpecial.o decompMod.o clm_glclnd.F90 DUSTMod.F90 ndepStreamMod.o organicFileMod.o QSatMod.F90 CNrestMod.F90 seq_timemgr_mod.mod CASAiniTimeVarMod.F90 CNC13FluxMod.F90 m_sparsematrix.mod misc.h clm_varpar.o clmtype.o CNDVMod.F90 shr_const_mod.mod clmtypeInitMod.o iobinary.o UrbanInputMod.F90 m_stdio.mod clm_driverInitMod.F90 shr_stream_mod.mod mkarbinitMod.o TridiagonalMod.o TridiagonalMod.F90 clm_glclnd.o CNiniTimeVar.F90 pio_types.mod shr_sys_mod.mod domainMod.o clm_varorb.F90 Hydrology2Mod.F90 iobinary.F90 CNNStateUpdate3Mod.F90 RunoffMod.F90 m_rearranger.mod clm_varpar.F90 organicFileMod.F90 getdatetime.F90 CNC13StateUpdate1Mod.o seq_flds_mod.mod CNCStateUpdate3Mod.F90 CNAnnualUpdateMod.o do_close_dispose.o clm_varcon.F90 FracWetMod.F90 m_generalgridcomms.mod CASAiniTimeVarMod.o perf_mod.mod spmdGathScatMod.F90 shr_cal_mod.mod shr_mpi_mod.mod initSurfAlbMod.F90 CNNStateUpdate1Mod.o CNCStateUpdate3Mod.o dynlandMod.F90 CNNStateUpdate3Mod.o decompInitMod.F90 clm_varctl.o subgridMod.F90 UrbanInputMod.o clm_driver.o CNDecompMod.F90 piolib_mod.mod m_mergesorts.mod shr_file_mod.mod m_attrvect.mod CNDVLightMod.F90 histFldsMod.o nanMod.F90 BareGroundFluxesMod.o mkarbinitMod.F90 CNEcosystemDynMod.F90 CNPrecisionControlMod.F90 QSatMod.o UrbanMod.F90 15 / 24

Call graphs ◮ Help to understand control flow dependencies ◮ CESM is supported by PGI complier suite ◮ Ported to GNU suite to profile with gprof 16 / 24

AD pattern: start small, iterative increments ◮ AD is excellent when applied to numeric programming of smooth functions ◮ Real codes represent complex compositions of (non-smooth) computations ◮ Pattern ◮ start with a small numerical core C ◮ differentiate: C ′ ◮ validate C ′ ◮ add more functions/subroutines ◮ repeat until the desired scope is achieved 17 / 24

Local AD: CNAllocation module 18 / 24

OpenAD work-flow ◮ Identify module dependencies ◮ Collect source files ◮ Preprocess with proper build sequence parameters ◮ Concatenate the sources into a single file ◮ Insert annotations marking independent and dependent variables ◮ Invoke transformations: 1. preprocess 2. fortran → whirl 3. whirl → xaif 4. xaif → xaif’ 5. xaif’ → whirl’ 6. whirl’ → fortran’ 7. postprocess 19 / 24

Results 20 / 24

AD of CNAllocation calculation 12 inputs ◮ final leaf/stem/root CN ratio ◮ leaf/organ/live-wood/dead-wood CN ratio ◮ new fine-root/stem C per new leaf C ◮ new coarse-root C per new stem C ◮ fraction of new wood that is live ◮ fraction of allocation that goes to current growth 6 outputs ◮ leaf/stem/organ carbon/nitrogen 21 / 24