4.13. Offline Experiments

This document describes two experiments using the offline form of the MITgcm.

4.13.1. Overview

The first experiment demonstrates use of the offline form of the MITgcm to study advection of a passive tracer. Time-averaged flow-fields and mixing coefficients, deriving from a prior online run, are re-used leaving only the tracer equation to be integrated. This first experiment’s run configuration is specified in directory verification/tutorial_cfc_offline/input_tutorial.

Figure — missing figure — shows a movie of tracer being advected using the offline package of the MITgcm. In the top panel the frames of the movie show the monthly surface evolution of an initially local source of passive tracer. In the lower panel, the frames of the movie show the changing monthly surface evolution where the initial tracer field had a global distribution.

The second experiment, a more complicated example exploring contamination of the global ocean through surface exposure to CFCs during the last century, is described after this more simple first example. The run configuration for this second experiment is specified in directory verification/tutorial_cfc_offline/input.

4.13.2. Time-stepping of tracers

See Section 2.16 and Section 2.17 for details of available tracer time-stepping schemes and their characteristics.

4.13.3. Code Configuration

The experiment files

contain the code customizations and parameter settings required to run the example. In addition the following binary data files are required: File input_tutorial/data

Listing 4.35 verification/tutorial_cfc_offline/input_tutorial/data
 1# ====================
 2# | Model parameters |
 3# ====================
 5# Continuous equation parameters
 6 &PARM01
 7 implicitDiffusion=.TRUE.,
 8 &
10# Elliptic solver parameters
11 &PARM02
12 cg2dMaxIters=1000,
13 cg2dTargetResidual=1.E-13,
14 &
16# Time stepping parameters
17 &PARM03
18 nIter0  = 4269600,
19 nTimeSteps = 4,
20 deltaTtracer= 43200.0,
21 deltaTClock = 43200.0,
22 pChkptFreq=3110400000.,
23 chkptFreq= 3110400000.,
24 dumpFreq=  31104000.,
25 taveFreq=  31104000.,
26 monitorFreq= 1.,
27 periodicExternalForcing=.TRUE.,
28 externForcingPeriod=2592000.,
29 externForcingCycle=31104000.,
30 &
32# Gridding parameters
33 &PARM04
34 usingSphericalPolarGrid=.TRUE.,
35 delR=  50.,  70., 100., 140., 190., 
36       240., 290., 340., 390., 440., 
37       490., 540., 590., 640., 690.,
38 ygOrigin=-90.,
39 dxSpacing=2.8125,
40 dySpacing=2.8125,
41 &
43# Input datasets
44 &PARM05
45 bathyFile=      'depth_g77.bin',
46 &

This file specifies the main parameters for the experiment.

  • Lines 18-19,

    nIter0  = 4269600,
    nTimeSteps = 4,

nIter0 and nTimesteps control the start time and the length of the run (in timesteps). Given at nIter0 is non-zero, the model requires appropriate pickup files to be present in the run directory. For testing purposes, the model has been prescribed to run for 4 timesteps; for a longer run, increase nTimesteps.

  • Line 20,

    deltaTtracer= 43200.0,

deltaTtracer is the tracer timestep in seconds, in this case, 12 hours (43200 seconds = 12 hours). Note that deltatTracer must be specified in input_tutorial/data as well as specified in deltaToffline in input_tutorial/data.off.

  • Line 21,

    deltaTClock= 43200.0,

When using the MITgcm in offline mode, deltaTClock (an internal model counter) should be made equal to the value assigned to deltatTtracer.

  • Line 27,


periodicExternalForcing is a flag telling the model whether to cyclically re-use forcing data where there is external forcing (see Section 4.13.5 below). Where there is no external forcing, as here, but where there is to be cyclic re-use of the offline flow and mixing fields, periodicExternalForcing must be assigned the value .TRUE..

  • Line 28,


externForcingPeriod specifies the period of the external forcing data in seconds. In the absence of external forcing, as in this example, it must be made equal to the value of externForcingPeriod in input_tutorial/data.off, in this case, monthly (2592000 seconds = 1 month).

  • Line 29,


externForcingCycle specifies the duration of the external forcing data cycle in seconds. In the absence of external forcing, as in this example, it must be made equal to the value of externForcingCycle in input_tutorial/data.off, in this case, the cycle is one year (31104000 seconds = 1 year).

  • Line34,


This line requests that the simulation be performed in a spherical polar coordinate system. It affects the interpretation of grid input parameters and causes the grid generation routines to initialize an internal grid based on spherical polar geometry.

  • Lines 35-37,

    delR=  50.,  70., 100., 140., 190.,
          240., 290., 340., 390., 440.,
          490., 540., 590., 640., 690.,

This line sets the vertical grid spacing between each \(z\)-coordinate line in the discrete grid. Here the total model depth is 5200 m.

  • Line 38,


This line sets the southern boundary of the modeled domain to -90o latitude N (= 90o S). This value affects both the generation of the locally orthogonal grid that the model uses internally and affects the initialization of the Coriolis force. Note: it is not required to set a longitude boundary, since the absolute longitude does not alter the kernel equation discretization.

  • Line 39,


This line sets the horizontal grid spacing between each \(y\)-coordinate line in the discrete grid to 2.8125o in longitude.

  • Line 40,


This line sets the vertical grid spacing between each \(x\)-coordinate line in the discrete grid to 2.8125o in latitude.

  • Line 45,


This line specifies the name of the file from which the domain bathymetry is read. This file contains a 2-D (\(x,y\)) map of (assumed 64-bit) binary numbers giving the depth of the model at each grid cell, ordered with the \(x\) coordinate varying fastest. The points are ordered from low coordinate to high coordinate for both axes. The units and orientation of the depths in this file are the same as used in the MITgcm code. In this experiment, a depth of 0 m indicates land. File input_tutorial/data.off

Listing 4.36 verification/tutorial_cfc_offline/input_tutorial/data.off
 2  UvelFile= 'input_off/uVeltave',
 3  VvelFile= 'input_off/vVeltave',
 4  WvelFile= 'input_off/wVeltave',
 5  GMwxFile= 'input_off/GM_Kwx-T',
 6  GMwyFile= 'input_off/GM_Kwy-T',
 7  GMwzFile= 'input_off/GM_Kwz-T',
 8  ConvFile= 'input_off/Convtave',
 9 &
12  offlineIter0=4248000,
13  deltaToffline=43200.,
14  offlineForcingPeriod=2592000.,
15  offlineForcingCycle=31104000.,
16 &

input_tutorial/data.off provides the MITgcm offline package with package specific parameters. Specifically, it contains the location (relative to the run directory) and prefix of files describing the flow field (UvelFile, VvelFile, WvelFile) and the corresponding convective mixing coefficients (ConvFile) which together prescribe the 3-D, time varying dynamic system within which the offline model will advect the tracer.

  • Lines 2-4,8

    UvelFile= '../input/input_off/uVeltave',
    VvelFile= '../input/input_off/vVeltave',
    WvelFile= '../input/input_off/wVeltave',
    ConvFile= '../input/input_off/Convtave',

In the example the offline data is located in the sub-directory verification/tutorial_cfc_offline/input/input_off. In this directory are fields describing the velocity and convective mixing histories of a prior forward integration of the MITgcm, required for the offline package. Based on the values of deltaToffline, offlineForcingPeriod and offlineForcingCycle specified in verification/tutorial_cfc_offline/input/input_off, since offlineForcingCycle corresponds to twelve forcing periods offlineForcingPeriod and since offlineIter0 is zero, there needs to be twelve uVeltave, twelve vVeltave, twelve wVeltave and twelve Convtave files each having a 10 digit sequence identifier between 0000000001 to 0000000012, that is, a total of 48 files.

  • Line 12,


offlineIter0, here specified to be 4248000 timesteps, corresponds to the timestep at which the tracer model is initialized. Note that offlineIter0 and nIter0 (set in input_tutorial/data) need not be the same.

  • Line 13,


deltatToffline sets the timestep associated with the offline model data in seconds, here 12 hours (43200 seconds = 12 hours).

  • Line 14,


offlineForcingPeriod sets the forcing period associated with the offline model data in seconds.

  • Line 15,


offlineForcingCycle sets the forcing cycle length associated with the offline model data in seconds. In this example the offline forcing cycle is 6 days, or twelve offline forcing periods. Together deltatToffline, offlineForcingPeriod and offlineForcingCycle determine the value of the ten digit sequencing tag the model expects files in input_tutorial/data.off to have. File input_tutorial/data.pkg

Listing 4.37 verification/tutorial_cfc_offline/input_tutorial/data.pkg
1# Packages
3 useGMRedi=.TRUE.,
5 useGCHEM=.TRUE.,
6 &

This file specifies which MITgcm packages are to be used.

  • Line 4,


usePTRACERS is a flag invoking pkg/ptracers which is responsible for the advection of the tracer within the model. File input_tutorial/data.ptracers

Listing 4.38 verification/tutorial_cfc_offline/input_tutorial/data.ptracers
 2 PTRACERS_numInUse=2,
 3 PTRACERS_Iter0= 4248000,
 5# tracer 1 - CFC11
 6 PTRACERS_advScheme(1)=77,
 7 PTRACERS_diffKh(1)=0.E3,
 8 PTRACERS_diffKr(1)=5.E-5,
 9 PTRACERS_useGMRedi(1)=.TRUE. ,
11 PTRACERS_initialFile(1)=' ',
12# tracer 2 - CFC12
13 PTRACERS_advScheme(2)=77,
14 PTRACERS_diffKh(2)=0.E3,
15 PTRACERS_diffKr(2)=5.E-5,
16 PTRACERS_useGMRedi(2)=.TRUE. ,
18 PTRACERS_initialFile(2)=' ',
19 &

This file provides the MITgcm ptracers package with package specific parameters, prescribing the nature of the the tracer/tracers as well as the variables associated with their advection.

  • Line 2,


PTRACERS_numInUse tells the model how many separate tracers are to be advected, in this case 2. Note: The value of PTRACERS_numInUse must agree with the value specified in code/PTRACERS_SIZE.h (see below).

  • Line 3,

    PTRACERS_Iter0= 4248000,

PTRACERS_Iter0 specifies the iteration at which the tracer is to be introduced.

  • Lines 6 and 13,


PTRACERS_advScheme(n) identifies which advection scheme will be used for tracer n, where n is the number of the tracer up to PTRACERS_numInUse. See Section 2.17 to identify the numerical codes used to specify different advection schemes (e.g. centered 2nd order, 3rd order upwind) as well as details of each.

  • Lines 7 and 14,


PTRACERS_diffKh(n) is the horizontal diffusion coefficient for tracer n, where n is the number of the tracer up to PTRACERS_numInUse.

  • Lines 8 and 15,


PTRACERS_diffKr(n) is the vertical diffusion coefficient for tracer n, where n is the number of the tracer up to PTRACERS_numInUse.

  • Lines 11 and 18,

    PTRACERS_initialFile(1)=' ',

PTRACERS_initialFile(n) identifies the initial tracer field to be associated with tracer n, where n is the number of the tracer up to PTRACERS_numInUse. Note that no initial file is specified here.

Note input_tutorial/data.ptracers requires a set of entries for each tracer. File input_tutorial/eedata

This file uses standard default values and does not contain customizations for this experiment. File code/packages.conf

Listing 4.39 verification/tutorial_cfc_offline/code/packages.conf
 1#-- list of packages (or group of packages) to compile for this experiment:

This file is used to invoke the model components required for a particular implementation of the MITgcm. File code/PTRACERS_SIZE.h

Listing 4.40 verification/tutorial_cfc_offline/code/PTRACERS_SIZE.h
 6C #include PTRACERS_SIZE.h
 9C Contains passive tracer array size (number of tracers).
11C PTRACERS_num defines how many passive tracers are allocated/exist.
12C  and is set here (default 1)
14C     Number of tracers
16      PARAMETER(PTRACERS_num = 2 )
19#endif /* ALLOW_PTRACERS */
  • Line 16,


This line sets the parameters PTRACERS_num (the number of tracers to be integrated) to 2 (in agreement with input_tutorial/data.ptracers). File code/SIZE.h

Listing 4.41 verification/tutorial_cfc_offline/code/SIZE.h
 4C    include SIZE.h
 5C    !DESCRIPTION: \bv
 6C     *==========================================================*
 7C     | SIZE.h Declare size of underlying computational grid.
 8C     *==========================================================*
 9C     | The design here supports a three-dimensional model grid
10C     | with indices I,J and K. The three-dimensional domain
11C     | is comprised of nPx*nSx blocks (or tiles) of size sNx
12C     | along the first (left-most index) axis, nPy*nSy blocks
13C     | of size sNy along the second axis and one block of size
14C     | Nr along the vertical (third) axis.
15C     | Blocks/tiles have overlap regions of size OLx and OLy
16C     | along the dimensions that are subdivided.
17C     *==========================================================*
18C     \ev
20C     Voodoo numbers controlling data layout:
21C     sNx :: Number of X points in tile.
22C     sNy :: Number of Y points in tile.
23C     OLx :: Tile overlap extent in X.
24C     OLy :: Tile overlap extent in Y.
25C     nSx :: Number of tiles per process in X.
26C     nSy :: Number of tiles per process in Y.
27C     nPx :: Number of processes to use in X.
28C     nPy :: Number of processes to use in Y.
29C     Nx  :: Number of points in X for the full domain.
30C     Ny  :: Number of points in Y for the full domain.
31C     Nr  :: Number of points in vertical direction.
33      INTEGER sNx
34      INTEGER sNy
35      INTEGER OLx
36      INTEGER OLy
37      INTEGER nSx
38      INTEGER nSy
39      INTEGER nPx
40      INTEGER nPy
41      INTEGER Nx
42      INTEGER Ny
43      INTEGER Nr
44      PARAMETER (
45     &           sNx =  64,
46     &           sNy =  32,
47     &           OLx =   4,
48     &           OLy =   4,
49     &           nSx =   2,
50     &           nSy =   2,
51     &           nPx =   1,
52     &           nPy =   1,
53     &           Nx  = sNx*nSx*nPx,
54     &           Ny  = sNy*nSy*nPy,
55     &           Nr  =  15)
57C     MAX_OLX :: Set to the maximum overlap region size of any array
58C     MAX_OLY    that will be exchanged. Controls the sizing of exch
59C                routine buffers.
62      PARAMETER ( MAX_OLX = OLx,
63     &            MAX_OLY = OLy )

Several lines are customized in this file for the current experiment:

  • Line 45,


    this line sets the lateral domain extent in grid points for the axis aligned with the \(x\)-coordinate.

  • Line 46,


    this line sets the lateral domain extent in grid points for the axis aligned with the \(y\)-coordinate.

  • Line 55,


    this line sets the vertical domain extent in grid points.

4.13.4. Running the Experiment

In your run directory, as per usual, a copy of all files from the input directory (here, input_tutorial/) are required. In addition, you will also need to copy .data and .meta files from directory input/input_off.

4.13.5. A more complicated example

The previous example demonstrated simple advection of a passive tracer using the offline form of the MITgcm. Now we present a more complicated example in which the model is used to explore contamination of the global ocean through surface exposure to CFCs during the last century. In invoking packages pkg/gchem, pkg/gmredi and pkg/cfc it provides a starting point and template for more complicated offline modeling, involving as it does surface forcing through wind and ice fields, more sophisticated mixing, and a time-varying forcing function.

The run configuration for this experiment resides under the directory verification/tutorial_cfc_offline/input/ (the code configuration is the same as in the first example, so the same model executable can be used, i.e., no need to re-compile). The files

contain all the parameter settings required. File input/data

Listing 4.42 verification/tutorial_cfc_offline/input/data
 1# ====================
 2# | Model parameters |
 3# ====================
 5# Continuous equation parameters
 6 &PARM01
 7 tRef=15*20.,
 8 sRef=15*35.,
 9 viscA4=0.,
10 viscAh=5.E5,
11 diffKhT=0.E3,
12 diffKhS=0.E3,
13 viscAr=1.E-3,
14 diffKrT=5.E-5,
15 diffKrS=5.E-5,
16 gravity=9.81,
17 rhoConst=1035.,
18 rigidLid=.FALSE.,
19 implicitFreeSurface=.TRUE.,
20 eosType='POLY3',
21 implicitDiffusion=.TRUE.,
22 implicitViscosity=.TRUE.,
23 ivdc_kappa=100.,
24 multiDimAdvection   = .FALSE.
26 useCDscheme=.FALSE.,
27 &
29# Elliptic solver parameters
30 &PARM02
31 cg2dMaxIters=1000,
32 cg2dTargetResidual=1.E-13,
33 &
35# Time stepping parameters
36 &PARM03
37 nIter0  = 4269600,
38 nTimeSteps = 4,
39# 100 years starting from a spinup of 5900 years:
40#startTime = 1.835136E+11,
41#endTime   = 1.866240E+11,
42 deltaTmom = 900.0,
43#tauCD =     321428.,
44 deltaTtracer= 43200.0,
45 deltaTClock = 43200.0,
46 abEps = 0.1,
47#cAdjFreq = -1,
48 pChkptFreq=3110400000.,
49#chkptFreq= 3110400000.,
50 dumpFreq=  31104000.,
51 taveFreq=  31104000.,
52#monitorFreq= 4853865600.,
53 monitorFreq= 2592000.,
54#tauThetaClimRelax = 5184000.0,
55#tauSaltClimRelax =  7776000.0,
56 periodicExternalForcing=.TRUE.,
57 externForcingPeriod=2592000.,
58 externForcingCycle=31104000.,
59 &
61# Gridding parameters
62 &PARM04
63 usingSphericalPolarGrid=.TRUE.,
64 delR=  50.,  70., 100., 140., 190.,
65       240., 290., 340., 390., 440.,
66       490., 540., 590., 640., 690.,
67 ygOrigin=-90.,
68 dxSpacing=2.8125,
69 dySpacing=2.8125,
70 &
72# Input datasets
73 &PARM05
74 bathyFile=      'depth_g77.bin',
76#hydrogSaltFile= 'lev_clim_salt.bin',
77#zonalWindFile=  'tren_taux.bin',
78#meridWindFile=  'tren_tauy.bin',
79#thetaClimFile=  'lev_monthly_temp.bin',
80#saltClimFile=   'lev_monthly_salt.bin',
81#surfQnetFile=   'shi_qnet.bin',
82#EmPmRFile=      'shi_empmr_year.bin',
83 the_run_name=   'Testing CFC and OFFLINE code',
84 &

A single line must be added (under PARM01, line 21) from the previous example


When pkg/gmredi is used, the flag implicitDiffusion must be assigned the value .TRUE.

In this example the starting timestep nIter0 is set to 4269600 requiring model access to pickup files with the suffix 0004269600. The model will run for 4 timesteps (nTimeSteps = 4). In this case the frequencies with which permanent and rolling checkpoints (pChkptFreq and chkptFreq) have been set is sufficiently long to ensure that only one from the last timestep will be written. This is also true of the values that have been assigned to the frequency with which dumps are written (dumpFreq) and time averaging (taveFreq) is performed. However, since the model always dumps the state of the model when it stops without error, a dump will be written with suffix 0004269604 upon completion. File input/data.off

Listing 4.43 verification/tutorial_cfc_offline/input/data.off
 2  UvelFile= '../input/input_off/uVeltave',
 3  VvelFile= '../input/input_off/vVeltave',
 4  WvelFile= '../input/input_off/wVeltave',
 5  GMwxFile= '../input/input_off/GM_Kwx-T',
 6  GMwyFile= '../input/input_off/GM_Kwy-T',
 7  GMwzFile= '../input/input_off/GM_Kwz-T',
 8  ConvFile= '../input/input_off/Convtave',
 9  SaltFile= '../input/input_off/Stave',
10  ThetFile= '../input/input_off/Ttave',
11# SFluxFile='../input/input_off/sFluxtave',
12# HFluxFile=' ',
13 &
16  offlineIter0=4248000,
17  deltaToffline=43200.,
18  offlineForcingPeriod=2592000.,
19  offlineForcingCycle=31104000.,
20 &

This file specifies the prefixes and locations of additional input files required to run the offline model. Note that directory input/input_off contains only as many offline files as are required to successfully run for 4 timesteps. Where the GMREDI scheme was used in the forward run, as here, package GMREDI must again be invoked when running offline. In this example, tracer is specified as having been introduced with a non-zero starttime, at timestep 4248000. File input/data.pkg

Listing 4.44 verification/tutorial_cfc_offline/input/data.pkg
1# Packages
3 useGMRedi=.TRUE.,
5 useGCHEM=.TRUE.,
6 useOffLine=.TRUE.,
8 &

This file specifies which MITgcm packages are to be used. It now invokes additional packages pkg/gmredi and pkg/gchem. File input/data.ptracers

Listing 4.45 verification/tutorial_cfc_offline/input/data.ptracers
 2 PTRACERS_numInUse=2,
 3 PTRACERS_Iter0= 4248000,
 4# for verification:
 5 PTRACERS_monitorFreq=43200.,
 6#- for each tracers:
 7# tracer 1 - dic
 8 PTRACERS_names(1)='cfc11',
 9 PTRACERS_long_names(1)='CFC11',
10 PTRACERS_units(1)='mol/m^3',
11 PTRACERS_advScheme(1)=77,
12 PTRACERS_diffKh(1)=0.E3,
13 PTRACERS_diffKr(1)=5.E-5,
14 PTRACERS_useGMRedi(1)=.TRUE. ,
16 PTRACERS_initialFile(1)=' ',
17# tracer 2 - alk
18 PTRACERS_names(2)='cfc12',
19 PTRACERS_units(2)='mol/m^3',
20 PTRACERS_advScheme(2)=77,
21 PTRACERS_diffKh(2)=0.E3,
22 PTRACERS_diffKr(2)=5.E-5,
23 PTRACERS_useGMRedi(2)=.TRUE. ,
25 PTRACERS_initialFile(2)=' ',
26 &

This file specifies parameters associated with the CFC11 and CFC12 tracer fields advected in this example.

  • Line 3,

    PTRACERS_Iter0= 4248000,

In this example the tracers were introduced at iteration 4248000.

  • Lines 12 and 21,


Since package GMREDI is being used, regular horizontal diffusion is set to zero.

  • Lines 14-15 and 23-24,

    PTRACERS_useGMRedi(n)=.TRUE. ,
    PTRACERS_useKPP(n)=.FALSE. ,

Setting flag PTRACERS_useGMRedi(n) to .TRUE. identifies that /pkg/gmredi is to be used. Setting flag PTRACERS_useKPP(n) to .FALSE. explicitly turns off KPP mixing.

  • Lines 16 and 25,

    PTRACERS_initialFile(n)=' ',

Since this is a ‘pickup’ run the initial tracer files PTRACERS_initialFile are not needed. The model will obtain the tracer state from pickup_ptracers.0004269600.data File input/data.gchem

Listing 4.46 verification/tutorial_cfc_offline/input/data.gchem
2  useCFC=.TRUE.,
3  nsubtime=1,
4 &

This file specifies the parameters used in /pkg/gchem. File input/data.gmredi

Listing 4.47 verification/tutorial_cfc_offline/input/data.gmredi
 1# from MOM
 2# GM_background_K: 	isopycnal diffusion coefficien
 3# GM_maxSlope:		max slope of isopycnals
 4# GM_Scrit:		transition for scaling diffusion coefficient
 5# GM_Sd:		half width scaling for diffusion coefficient
 6# real background diff:	horizontal diffusion
 8# ifdef GM_VISBECK_VARIABLE_K, include following in GM_PARM01
 9# GM_Visbeck_alpha   = 0.,
10# GM_Visbeck_length  = 2.e+5,
11# GM_Visbeck_depth   = 1.e+3,
12# GM_Visbeck_maxval_K= 2.5e+3,
14 &GM_PARM01 
15  GM_background_K    = 1.e+3,
16  GM_taper_scheme    = 'gkw91',
17  GM_maxSlope        = 1.e-2,
18  GM_Kmin_horiz      = 100.,
19  GM_Scrit           = 4.e-3,
20  GM_Sd              = 1.e-3,
21 &

This file specifies parameters required for /pkg/gmredi. File input/cfc1112.atm

This is a ASCII data file containing the CFC source functions over the northern and southern hemispheres annually from 1931 through 1998. Running the Experiment

The model is run as before.