4.12. Adjoint Sensitivity Analysis for Tracer Injection

(in directory: verification/tutorial_tracer_adjsens/)

MITgcm has been adapted to enable AD using TAMC or TAF. The present description is specific to the use of TAMC or TAF as AD tool. The following sections describe the steps which are necessary to generate a tangent linear or adjoint model of MITgcm. We take as an example the sensitivity of carbon sequestration in the ocean. The AD-relevant hooks in the code are outlined in Section 7.2 and Section 7.2.4.4.

4.12.1. Overview of the experiment

We describe an adjoint sensitivity analysis of out-gassing from the ocean into the atmosphere of a carbon-like tracer injected into the ocean interior (see Hill et al. 2004 [HBFM04]).

4.12.1.1. Passive tracer equation

For this work, MITgcm was augmented with a thermodynamically inactive tracer, \(C\). Tracer residing in the ocean model surface layer is out-gassed according to a relaxation time scale, \(\mu\). Within the ocean interior, the tracer is passively advected by the ocean model currents. The full equation for the time evolution

(4.59)\[\frac{\partial C}{\partial t} \, = \, -\mathbf{U} \cdot \nabla C \, - \, \mu C \, + \, \Gamma(C) \,+ \, S\]

also includes a source term \(S\). This term represents interior sources of \(C\) such as would arise due to direct injection. The velocity term, \(\mathbf{U}\), is the sum of the model Eulerian circulation and an eddy-induced velocity, the latter parameterized according to Gent/McWilliams (Gent and McWilliams 1990 [GM90]; Gent et al. (1995) [GWMM95]). The convection function, \(\Gamma\), mixes \(C\) vertically wherever the fluid is locally statically unstable.

The out-gassing time scale, \(\mu\), in (4.59) is set so that \(1/\mu \sim 1\) year for the surface ocean and \(\mu=0\) elsewhere. With this value, (4.59) is valid as a prognostic equation for small perturbations in oceanic carbon concentrations. This configuration provides a powerful tool for examining the impact of large-scale ocean circulation on CO2 out-gassing due to interior injections. As source we choose a constant in time injection of \(S = 1\) mol s-1.

4.12.1.2. Model configuration

The model configuration employed has a constant \(4^\circ \times 4^\circ\) resolution horizontal grid and realistic geography and bathymetry. Twenty vertical layers are used with vertical spacing ranging from 50 m near the surface to 815 m at depth. Driven to steady-state by climatological wind-stress, heat and fresh-water forcing, the model reproduces well known large-scale features of the ocean general circulation.

4.12.1.3. Out-gassing cost function

To quantify and understand out-gassing due to injections of \(C\) in (4.59), we define a cost function \({\cal J}\) that measures the total amount of tracer out-gassed at each timestep:

(4.60)\[{\cal J}(t=T)=\int_{t=0}^{t=T}\int_{A} \mu C \, dA \, dt\]

(4.60) integrates the out-gassing term, \(\mu C\), from (4.59) over the entire ocean surface area, \(A\), and accumulates it up to time \(T\). Physically, \({\cal J}\) can be thought of as representing the amount of CO2 that our model predicts would be out-gassed following an injection at rate \(S\). The sensitivity of \({\cal J}\) to the spatial location of \(S\), \(\frac{\partial {\cal J}}{\partial S}\), can be used to identify regions from which circulation would cause CO2 to rapidly out-gas following injection and regions in which CO2 injections would remain effectively sequestered within the ocean.

4.12.2. Code configuration

The code customization routines are in verification/tutorial_tracer_adjsens/code_ad:

The runtime flag and parameters settings are contained in verification/tutorial_tracer_adjsens/input/ and verification/tutorial_tracer_adjsens/input_ad/, together with the forcing fields and and restart files:

Below we describe the customizations of this files which are specific to this experiment.

4.12.2.1. File code_ad/COST_OPTIONS.h /

This file contains package-specific CPP-options (see Section 7.2.4).

4.12.2.2. File code_ad/CTRL_OPTIONS.h /

This file contains package-specific CPP-options (see Section 10.3).

4.12.2.3. File code_ad/CPP_OPTIONS.h /

This file contains model-specific CPP options (see Section 3.7). Most options are related to the forward model setup. They are identical to the global steady circulation setup of verification/global_ocean.90x40x15/. The three options specific to this experiment are as follows. #define ALLOW_PASSIVE_TRACER enables the code to carry through the advection/diffusion of a passive tracer along the model integration. #define ALLOW_MIT_ADJOINT_RUN enables the inclusion of some AD-related fields concerning initialization, link between control variables and forward model variables, and the call to the top-level forward/adjoint subroutine adthe_main_loop.F instead of model/src/the_main_loop.F. #define ALLOW_GRADIENT_CHECK enables the gradient check package. After computing the unperturbed cost function and its gradient, a series of computations are performed for which:

  • an element of the control vector is perturbed

  • the cost function w.r.t. the perturbed element is computed

  • the difference between the perturbed and unperturbed cost function is computed to compute the finite difference gradient

  • the finite difference gradient is compared with the adjoint-generated gradient.

The gradient check package is further described in Section 7.3.

4.12.2.4. File ECCO_OPTIONS.h

The CPP options of several AD-related packages are grouped in this file:

4.12.2.5. File SIZE.h

Listing 4.34 verification/tutorial_global_oce_latlon/code/SIZE.h
 1CBOP
 2C    !ROUTINE: SIZE.h
 3C    !INTERFACE:
 4C    include SIZE.h
 5C    !DESCRIPTION: \bv
 6C     *==========================================================*
 7C     | SIZE.h Declare size of underlying computational grid.
 8C     *==========================================================*
 9C     | The design here supports a three-dimensional model grid
10C     | with indices I,J and K. The three-dimensional domain
11C     | is comprised of nPx*nSx blocks (or tiles) of size sNx
12C     | along the first (left-most index) axis, nPy*nSy blocks
13C     | of size sNy along the second axis and one block of size
14C     | Nr along the vertical (third) axis.
15C     | Blocks/tiles have overlap regions of size OLx and OLy
16C     | along the dimensions that are subdivided.
17C     *==========================================================*
18C     \ev
19C
20C     Voodoo numbers controlling data layout:
21C     sNx :: Number of X points in tile.
22C     sNy :: Number of Y points in tile.
23C     OLx :: Tile overlap extent in X.
24C     OLy :: Tile overlap extent in Y.
25C     nSx :: Number of tiles per process in X.
26C     nSy :: Number of tiles per process in Y.
27C     nPx :: Number of processes to use in X.
28C     nPy :: Number of processes to use in Y.
29C     Nx  :: Number of points in X for the full domain.
30C     Ny  :: Number of points in Y for the full domain.
31C     Nr  :: Number of points in vertical direction.
32CEOP
33      INTEGER sNx
34      INTEGER sNy
35      INTEGER OLx
36      INTEGER OLy
37      INTEGER nSx
38      INTEGER nSy
39      INTEGER nPx
40      INTEGER nPy
41      INTEGER Nx
42      INTEGER Ny
43      INTEGER Nr
44      PARAMETER (
45     &           sNx =  45,
46     &           sNy =  20,
47     &           OLx =   3,
48     &           OLy =   3,
49     &           nSx =   2,
50     &           nSy =   2,
51     &           nPx =   1,
52     &           nPy =   1,
53     &           Nx  = sNx*nSx*nPx,
54     &           Ny  = sNy*nSy*nPy,
55     &           Nr  =  20)
56
57C     MAX_OLX :: Set to the maximum overlap region size of any array
58C     MAX_OLY    that will be exchanged. Controls the sizing of exch
59C                routine buffers.
60      INTEGER MAX_OLX
61      INTEGER MAX_OLY
62      PARAMETER ( MAX_OLX = OLx,
63     &            MAX_OLY = OLy )
64

The file contains the grid point dimensions of the forward model. It is identical to the verification/exp2/.

4.12.2.6. File /pkg/autodiff/adcommon.h

This file contains common blocks of some adjoint variables that are generated by TAMC. The common blocks are used by the adjoint support routine /pkg/autodiff/addummy_in_stepping.F which needs to access those variables:

common /addynvars_r/

is related to model/inc/DYNVARS.h

common /addynvars_cd/

is related to model/inc/DYNVARS.h

common /addynvars_diffkr/

is related to model/inc/DYNVARS.h

common /addynvars_kapgm/

is related to model/inc/DYNVARS.h

common /adtr1_r/

is related to TR1.h

common /adffields/

is related to model/inc/FFIELDS.h

Note that if the structure of the common block changes in the above header files of the forward code, the structure of the adjoint common blocks will change accordingly. Thus, one must make sure that the structure of the adjoint common block in the hand-written file /pkg/autodiff/adcommon.h complies with the automatically generated adjoint common blocks in adjoint_model.F. The header file is enabled via the CPP-option ALLOW_AUTODIFF_MONITOR.

4.12.2.7. File code_ad/tamc.h

This routine contains the dimensions for TAMC checkpointing and some indices relevant for storing ky computations.

The following parameters may be worth describing: isbyte, maxpass.

4.12.2.8. File makefile

This file contains all relevant parameter flags and lists to run TAMC or TAF. It is assumed that TAMC is available to you, either locally, being installed on your network, or remotely through the ’TAMC Utility’. TAMC is called with the command tamc followed by a number of options. They are described in detail in the TAMC manual (Giering 1999 [Gie99]). Here we briefly discuss the main flags used in the makefile. The standard output for TAF is written to file taf.log.

TAMC:

-input «variable names» -output «variable name» -i4 -r4 ...
-toplevel «S/R name» -reverse «file names»

TAF:

-input «variable names» -output «variable name» -i4 -r4 ...
-toplevel «S/R name» -reverse «file names»
-flow taf_flow.log -nonew_arg

  • -toplevel «S/R name»

    Name of the toplevel routine, with respect to which the control flow analysis is performed.

  • input «variable names»

    List of independent variables \(u\) with respect to which the dependent variable \(J\) is differentiated.

  • -output «variable name»

    Dependent variable \(J\) which is to be differentiated.

  • -reverse «file names»

    Adjoint code is generated to compute the sensitivity of an independent variable w.r.t. many dependent variables. In the discussion of Section 7 the generated adjoint top-level routine computes the product of the transposed Jacobian matrix \(M^T\) times the gradient vector \(\nabla_v J\). «file names» refers to the list of files .f which are to be analyzed by TAMC. This list is generally smaller than the full list of code to be compiled. The files not contained are either above the top-level routine (some initializations), or are deliberately hidden from TAMC, either because hand-written adjoint routines exist, or the routines must not (or don’t have to) be differentiated. For each routine which is part of the flow tree of the top-level routine, but deliberately hidden from TAMC (or for each package which contains such routines), a corresponding file .flow exists containing flow directives for TAMC.

  • -i4 -r4

  • -flow taf_flow.log

    Will cause TAF to produce a flow listing file named taf_flow.log in which the set of active and passive variables are identified for each subroutine.

  • -nonew_arg

    The default in the order of the parameter list of adjoint routines has changed. Before TAF 1.3 the default was compatible with the TAMC-generated list. As of TAF 1.3 the order of adjoint routine parameter lists is no longer compatible with TAMC. To restore compatibility when using TAF 1.3 and higher, this argument is needed. It is currently crucial to use since all hand-written adjoint routines refer to the TAMC default.

4.12.2.8.1. File input/topog.bin

Contains 2-D bathymetry information.

4.12.2.8.2. Files input/windx.bin, input/windy.bin, input/salt.bin, input/theta.bin, input/SSS.bin, input/SST.bin

These contain the initial values of salnity and potential temperature (salt.bin, theta.bin), surface boundary values (surface wind stresses windx.bin, windy.bin), and surface restoring fields (SSS.bin, SST.bin).

4.12.3. Compiling the model and its adjoint

The build process of the adjoint model is slightly more complex than that of compiling the forward code. The main reason is that the adjoint code generation requires a specific list of routines that are to be differentiated (as opposed to the automatic generation of a list of files to be compiled by genmake2). This list excludes routines that don’t have to be or must not be differentiated. For some of the latter routines flow directives may be necessary, a list of which has to be given as well. For this reason, a separate makefile is currently maintained in the directory adjoint/. This makefile is responsible for the adjoint code generation.

In the following we describe the build process step by step, assuming you are in the directory bin/. A summary of steps to follow is given at the end.

4.12.3.1. Adjoint code generation and compilation – step by step

  1. ln -s ../verification/???/code/.genmakerc .
    ln -s ../verification/???/code/*.[Fh] .
    Link your customized genmake options, header files, and modified code to the compile directory.
  2. ../tools/genmake -makefile
    Generate your Makefile (see Section 3.5.2).
  3. make depend
    Dependency analysis for the CPP pre-compiler (see Section 3.5.1).
  4. cd ../adjoint
    make adtaf or make adtamc
    Depending on whether you have TAF or TAMC at your disposal, you’ll choose adtaf or adtamc as your make target for the makefile in the directory adjoint/. Several things happen at this stage.
  • make adrestore
    make ftlrestore
    The initial template files adjoint_model.F and tangentlinear_model.F in pkg/autodiff which are part of the compiling list created by genmake2 are restored.
  • make depend, make small_f
    The bin/ directory is brought up to date, i.e., for recent changes in header or source code .[Fh], corresponding .f routines are generated or re-generated. Note that here, only CPP pre-compiling is performed; no object code .o is generated as yet. Pre-compiling is necessary for TAMC to see the full code.
  • make allcode
    All Fortran routines .f in bin/ are concatenated into a single file called tamc_code.f.
  • make admodeltaf/admodeltamc
    Adjoint code is generated by TAMC or TAF. The adjoint code is written to the file tamc_code_ad.f. It contains all adjoint routines of the forward routines concatenated in tamc_code.f. For a given forward routine subroutine routinename the adjoint routine is named adsubroutine routinename by default (that default can be changed via the flag -admark «markname»). Furthermore, it may contain modified code which incorporates the translation of adjoint store directives into specific Fortran code. For a given forward routines subroutine routinename the modified routine is named mdsubroutine routinename. TAMC or TAF info is written to file tamc_code.prot or taf.log, respectively.
  • make adchange
    The multi-threading capability of MITgcm requires a slight change in the parameter list of some routines that are related to to active file handling. This post-processing invokes the sed script tools/adjoint_sed to insert the threading counter myThId into the parameter list of those subroutines. The resulting code is written to file tamc_code_sed_ad.f and appended to the file adjoint_model.F. This concludes the adjoint code generation.
  1. cd ../bin
    make
    The file adjoint_model.F cnow contains the full adjoint code. All routines are now compiled.

N.B.: The targets make adtaf/adtamc now comprise a series of targets that in previous versions had to be invoked separately. This was probably preferable at a more experimental stage, but has now been dropped in favor of a more straightforward build process.

4.12.3.1.1. Adjoint code generation and compilation – summary

cd bin
ln -s ../verification/my_experiment/code/.genmakerc .
ln -s ../verification/my_experiment/code/*.[Fh] .
../tools/genmake -makefile
make depend
cd ../adjoint
make adtaf <OR: make adtamc>
     contains the targets:
     adrestore small_f allcode admodeltaf/admodeltamc adchange
cd ../bin
make