Increasing the Interoperability of an Earth System Model:
Atmospheric-Ocean Dynamics and Tracer Transports
NCC4-624

Milestone J: Use ESMF services to couple the UCLA AGCM and POP, and integrate them into the ESMF framework for performance evaluation relative to milestones E, F and G. Update Test Plan/Procedures for integrating the AGCM/POP with the ESMF. Prepare reports on interoperability tests and an initial User’s Guide

1. Introduction

Our current Earth System Model (ESM) comprises four models representing the coupled dynamics, physics and chemistry of the global atmosphere and world oceans: 1) atmospheric general circulation model (UCLA AGCM), 2) oceanic general circulation model (LANL Parallel Ocean Model: POP), 3) atmospheric chemistry and transport model (UCLA ACTM), and 4) Ocean Chemical Transport Model (JPL OCTM). A novel Distributed Data Broker (DDB) has been designed for data exchanges. We also use the MIT ocean GCM as an option. The ESM is described in Milestone A.

In the present round of the CAN project we focus on the coupled atmosphere-ocean system. Therefore, our work as early adaptors of the ESMF consists of integrating into this framework the ESM gridded components that represent the coupled system: 1) UCLA AGCM, and 2) LANL/POP and MIT OGCM. The ESMF functionality will be demonstrated in Milestone K by analyzing the El Niño prediction capability of the AGCM coupled to POP and MIT OGCM, in combination with NASA/JPL ocean data and optimization products.

2. ESM Driver Program (EDP)

ESM Driver Program (EDP) was designed to control and define the ESMF environment and execution sequence of gridded and coupling components. The EDP consists of five routines, all of which are ESM compliant having initialize, run and finalize methods:

ESM driver
AGCM driver
OGCM driver
Atmosphere to Ocean coupling driver
Ocean to Atmosphere coupling driver

The AGCM and OGCM drivers include the calls to initialize, run, and terminate (finalize) the selected gridded components (UCLA AGCM, POP, MIT OGCM). The Atmosphere to Ocean and Ocean to Atmosphere coupling drivers control the data transfers between gridded components. The transfer of execution between EDP routines is done with an ESMF entry point registration and an ESMF call to branch to the appropriate routine. Entry points are registered for each of the drivers and their initialize, run and finalize methods.

The passing of variables from gridded components into the EDP is component-dependent. The UCLA AGCM uses a set of subroutines that are called by the model itself. This simplifies the argument lists and allows the model to transfer the data at the point where it is generated or wanted, without the needs of calling ESMF routines. This also allows for a very clean interface between the model and EDP. The LANL POP uses pointers to insert and extract data directly to/from the corresponding export and import states data structure. The MIT OGCM passes variables through its argument list.

Variables are communicated between gridded components as fields (data with grid specification) in an import/export state. Fields are processed in the coupling routines by reference to their symbolic name, thus greatly simplifying and generalizing the coupling drivers. The name registration is done by each of the gridded components in order to specify to the EDP the variables that the component will produce and consume.

We next describe the EDP subroutines in more detail. The ESM driver subroutine initializes the ESMF and specifies the coupled system, i.e. the gridded components to be used and the execution sequence. The initialize method creates and registers each of the gridded and coupling components, and the import/export states. The run method has the time integration loop for the application and transfers control, using ESMF calls, to each of the gridded and coupling components in the appropriate sequence. The finalize method terminates the execution of components.

The ESM gridded components drivers are component-dependent. For the UCLA AGCM and MIT OGCM the drivers are very similar. Changes in the components themselves are minimized by passing information from the model into these routines so the corresponding ESMF routines can be called to create the grid(s), and register the variables. The data organization is via symbolic field name recognition. The paradigm used for POP is more intrusive into the code, with the variables being accessed via pointer lists and ESMF grid specifications defined internally.

Specifically for the UCLA AGCM, the initialize method consists of two stages. In the first stage the model initializes itself (i.e. defines constants, decomposition/layout, geometry, communications). This information is used to create the ESMF grids, fields and layout. In the second stage, the model obtains the initial and boundary conditions and executes the first physics time step. The initialize method for the OGCMs consists of one stage only, in which the model selected obtains the initial and boundary conditions.

The run method extracts the variables and missing data mask from the import state using a data mask to flag missing/invalid data. After that, the appropriate gridded component is called to advance the simulation time until the next instance of coupling. At the completion of the coupling interval, missing data is replaced as a function of the existing information, and a missing data mask is created. The variables are now inserted into the export state so they are ready to be transferred when coupling driver is called.

The finalize method uses termination routines that close files, write restarts and print out statistics.
The coupling drivers routines use a very simple and general-purpose algorithm. The transfer of variables between the import to export state within the coupling routine is done by utilizing the list of symbolic variable names that was generated during the initialization. The routine extracts the data pointer, data shape, and field pointer, from both the import and export state. Regridding is currently accomplished by using ESMF rectilinear interpolation from the import state to the export state.

3. The UCLA AGCM

Extensive work was performed with the UCLA AGCM to make it ESMF compliant. The model’s own control code and coupling routines were removed, since these functions are now performed by the EDP routines. To conform to ESMF methodology, entry points were defined in the model to allow for discrete calls to the initialize, run, and finalize methods. The coupling interface, which was based on the methodology of concurrent execution, was rewritten due to limitations of the ESMF version available at the time when the work was started (2.1.0) and that only allowed for sequential execution. (More recent ESMF versions allow for concurrent execution.) In addition, routines were created so the EDP can extract grid, geometry and variable information from the model.

The UCLA AGCM was tested using the EDP. No functional or performance degradation was found. The code can be still be run stand alone (without ESMF) by setting a flag.

4. The LANL POP

The LANL POP model was modified by Phil Jones to be ESMF compliant. A number of modifications needed to be done to make it run with ESMF version 2.0.1 and to remove a number of bugs in the code and it's coupling interface. The model was linked and run using ESMF version 2.01, EDP and UCLA AGCM. The coupled system was run for 30 days with a coupling interval of 6 hours. The ESMF adds a negligible overhead to the coupled system.

5. MIT OGCM

The MIT OGCM was modified by Chris Hill to be ESMF compliant. The model was linked and run using ESMF version 2.0.1, EDP and UCLA AGCM. As with the LANL POP model the coupled system was successfully integrated out to 30 days with a 6 hour coupling interval with negligible overhead.

6. Conclusions

Milestone J required the use of ESMF services to couple the UCLA AGCM with the MIT OGCM and LANL POP ocean modes. We successfully accomplished this task without any significant loss of code performance. For the most part the ESMF meets our needs admirably. The validity of using the ESMF framework for coupling of inter-disciplinary models is readily apparent to us. The following list enumerates our findings.

Suggestions:

- A data flow diagram would be nice to facilitate understanding of the relationship between functions.
- Function and variable dependency list.
- Better documentation with dependency listed as notes.
- Notes on unusual behavior of ESMF and issues that are potentially misleading to users.
- Standardization of model interfaces (too much to ask?).

7. Deliverables and documentation

Simplified ESM Driver Program (EDP) flow chart.

OGCM_init
|
call ESMF_GridCompGet ! ESMF initialization
|
call get_domain_size ! set domain size
|
ocnGrid = ESMF_GridCreateHorzLatLon ! create grid
|
ocnLayout = ESMF_DELayoutCreate ! create logical
| ! layout
call ESMF_GridDistribute
|
call ESMF_GridCompSet
|
|
call ESMF_ArraySpecSet
|
fieldRef = ESMF_FieldCreate ! create fields
call ESMF_StateAddField ! add fields to export state
. . .
|
fieldRef = ESMF_FieldCreate ! create fields
call ESMF_StateAddField ! add fields to import state
. . .
|
|
call esmf_ogcm_coupler ! initialize internal coupler
|
call driver_init ! initialize ocean model
|
call ins_var_o2a ! collect SST
. . .
|
call var_store ! send variables to export state


OGCM_run
|
call var_retrieve ! retrieve vars from import state
|
call DO_OCN_RUN -----
| |
call var_store | !store vars to export state
|
|----> call ext_var_a2o ! ext vars
. . .
|
call driver_run ! run OGCM
|
call ins_var_o2a ! ins vars
. . .


OGCM_final




CPLa2o_init
|
call ESMF_CplCompGet
|
cplLayout = ESMF_DELayoutCreate
cplLayout = ESMF_DELayoutCreate


CPLa2o_run
|
do
call STATE_GET_FADP ! import State
call STATE_GET_FADP ! export State
|
call ESMF_FieldRegrid ! regrid field
enddo


CPLa2o_final




CPLo2a_init
|
call ESMF_CplCompGet
|
cplLayout = ESMF_DELayoutCreate


CPLo2a_run
|
do
call STATE_GET_FADP ! import state
call STATE_GET_FADP ! export state
|
call ESMF_FieldRegrid ! regrid
enddo


CPLo2a_final

Appendix B

Table: List of risks and risk mitigation strategies at the completion of Milestone J

Name
Description of Risk
Level of Impact (H, M, L)
Prob. Of Impact (H, M, L) Type of Mitigation (Avoid, Mitigate, Accept)
Mitigation Actions PBL parameterization for AGCM
Completed El Niño simulation Poor performance of AGCM coupled to MIT OGCM M L Accept The simulated climate by the UCLA AGCM coupled to the MITogcm is realistic. HYPOP Timely availability M H Avoid Use POP version 2.0 in a quasi-global domain El Niño Prediction Poor model simulation of El Niño.
H
L
Mitigate Model parameterizations will be further revised/tuned to improve performance. Availability of appropriate ESMF interfaces Available Integration of ESM into ESMF Completed

Appendix C
UCLA ESM/ESMF Software Test PlanScope
The scope of this project is the coupling of existing gridded and coupler components that form the UCLA Earth System Model (ESM) using the Earth System Modeling Framework (ESMF).
System Overview
The software system to be tested includes three gridded components: 1) UCLA AGCM 7.1/7.2 and later, 2) MITogcm, and 3) POP 2.0 OGCM. It also includes the ESMF libraries.
2. Referenced Documents
UCLA AGCM Users Guide
http://www.atmos.ucla.edu/~mechoso/esm
POP 2.0 Users Guide Version 2.0, Los Alamos National Laboratory, LACC 98-18, 23 March 2003.
MITogcm Users Guide. Available from http://mitgcm.org/pelican/online_documents/manual.html
3. Software Test Environment
Our test hardware environment will be the SGI Origin 3000 system at NASA/Ames, and DEC/COMPAC/HP ALPHA systems at NASA/GSFC and at UCLA. We will utilize the current compilers and system libraries provided in these environments. Personnel conducting the tests will be Joseph A. Spahr from UCLA. We intend to focus on a single grid resolution for each component. The atmosphere model will have a 5 degree longitude by 4 degree latitude resolution and 15 levels. The ocean models will have a rectilinear grid with approximately 1 degree longitude resolution and between 1/3 to 1 degree latitude resolution and 46 levels. We will write drivers and diagnostic routines primarily to test the correctness of the integration of the component models (verification).
4. Test Identification
This section describes the tests to be performed under this plan.
4.1 General Information
Tests to be performed to assess the reliability and performance of the UCLA ESM integrated with the ESMF.
4.1.1 Test Levels
Tests 1, 2, and 3 are at the component/integration level, Test 4 at the integration level, and Test 5 at the system level.
4.1.2 Test Classes
The test classes to be performed are 1) integration of the models into the ESMF, 2) operational in climate simulations, and 3) code performance, The table in Section 5 summarizes the tests classes.
4.2 Planned Tests
This section describes the specific tests to be performed under this plan
4.2.1 Stand Alone AGCM
Test purpose: Verify the implementation of ESMF-compliant control structures for the AGCM. These structures control initialization, sequencing of tasks during the time integration part of the execution, and the finalize.
Test type: Functional
Test inputs: Boundary conditions, restart history, run namelist.
Verification method: Compare output with a control AGCM integration without ESMF.
Special Requirements: Availability of a working, stable version of the ESMF.
Assumptions/Constraints: Codes available in versions appropriate for the available test-hardware environment.
Expected results: AGCM will start, run, and terminate successfully producing correct results.
Actual results: Seasonal simulation of the atmospheric circulation.
4.2.2 Stand-Alone OGCMs (MITogcm or POP)
Test purpose: Verify the implementation of ESMF-compliant control structures for the OGCMs. These structures control initialization, sequencing of tasks during the time integration part of the execution, and the finalize.
Test type: Functional
Test inputs: Boundary conditions, restart history, run namelist.
Verification method: Compare output with a control OGCM integration without ESMF.
Special Requirements: Availability of a working, stable version of the ESMF.
Assumptions/Constraints: Codes available in versions appropriate for our test-hardware environment.
Expected results: OGCM will start, run, and terminate successfully.
Actual results: Seasonal simulation of the oceanic circulation.
4.2.3 Side-by-side AGCM/OGCMs
Test: Boundary conditions, restart history, run namelist. Test purpose: Verify the implementation of the ESMF-compliant control structures when the application is configured with two components, but without data exchanges (coupling) between components.
Test type: Functional
Test inputs: Boundary conditions, restart history, run namelist.Test purpose: Verify the implementation of the ESMF-compliant control structures when the application is configured with two components, but without data exchanges (coupling) between components.
Verification method: Compare output with a control AGCM/OGCM integration without ESMF.
Special Requirements: Availability of a working, stable version of the ESMF.
Assumptions/Constraints: Codes available in versions appropriate for the available test-hardware environment.
Expected results: AGCM/OGCM will start, run, and terminate successfully.
Actual results: Seasonal climate simulation
4.2.4 Coupled AGCM/OGCM (Functional)
Test purpose: Verify the implementation of the ESM-compliant structures for data exchanges (coupling).
Test type: Functional
Test inputs: Boundary conditions, restart history, run namelist.
Verification method: Compare output with a control AGCM/OGCM integration without ESMF.
Special Requirements: Availability of a working, stable version of the ESMF.
Assumptions/Constraints: Codes available in versions appropriate for the available test-hardware environment.
Expected results: AGCM/OGCM will start, run, and terminate successfully producing correct results.
Actual results: Seasonal climate simulation
4.2.5 Coupled AGCM/OGCM (Performance)
Test purpose: Verify that integration to the ESMF does not impose more than 10% overhead to the AGCM/OGCM.
Test type: Performance
Test inputs: Boundary conditions, restart history, run namelist.
Verification method: Comparison of timings
Special Requirements: Several sessions of dedicated machine time, one-hour long each. Availability of a working, stable version of the ESMF.
Assumptions/Constraints: Codes available in versions appropriate for the available test-hardware environment.
Expected results: Overhead to the AGCM/OGCM by ESMF will be lower than 10%.
Actual results: Seasonal climate simulation.
4.3 Test Schedules
The tests will be performed sequentially. Tests 4.2.1 through 4.2.3 have been completed successfully for the UCLA AGCM and the MIT and LANL OGCM. Test 4.2.4 was completed for small decompositions and is currently being test on Halem with a larger number of nodes. Test 4.2.5 will be performed after the completion of test 4.2.4.
5.0 Requirements Traceability

Requirement

TEST

Stand-alone AGCM
(functional)
Stand-alone OGCM
(functional) Side-by-side AGCM/
OGCM
(functional) Coupled AGCM/
OGCM
(functional/ operational) Coupled
AGCM/
OGCM
(performance) Organization using the ESMF component design model 1.1-1.5
1.7-1.11 1.1-1.5
1.7-1.11
1.6


Inclusion of an ESMF compliant application component


2.1-2.4.1

Inclusion of an ESMF compliant coupler

3.1-3.1.1

System contains atmosphere and ocean components
4.1-4.2.4

Verification of ESMF timing overhead
5.1

PAGE 13