# 3. Getting Started with MITgcm¶

This chapter is divided into two main parts. The first part, which is covered in sections Section 3.1 through Section 3.6, contains information about how to download, build and run MITgcm. We believe the best way to familiarize yourself with the model is to run one of the tutorial examples provided in the MITgcm repository (see Section 4), so would suggest newer MITgcm users jump there following a read-through of the first part of this chapter. Information is also provided in this chapter on how to customize the code when you are ready to try implementing the configuration you have in mind, in the second part (Section 3.8). The code and algorithm are described more fully in Section 2 and Section 6 and chapters thereafter.

In this chapter and others (e.g., chapter Contributing to the MITgcm), for arguments where the user is expected to replace the text with a user-chosen name, userid, etc., our convention is to show these as upper-case text surrounded by « », such as «USER_MUST_REPLACE_TEXT_HERE». The « and » characters are NOT typed when the text is replaced.

## 3.1. Where to find information¶

There is a web-archived support mailing list for the model that you can email at MITgcm-support@mitgcm.org once you have subscribed.

## 3.2. Obtaining the code¶

The MITgcm code and documentation are under continuous development and we generally recommend that one downloads the latest version of the code. You will need to decide if you want to work in a “git-aware” environment (Method 1) or with a one-time “stagnant” download (Method 2). We generally recommend method 1, as it is more flexible and allows your version of the code to be regularly updated as MITgcm developers check in bug fixes and new features. However, this typically requires at minimum a rudimentary understanding of git in order to make it worth one’s while.

Periodically we release an official checkpoint (or “tag”). We recommend one download the latest code, unless there are reasons for obtaining a specific checkpoint (e.g. duplicating older results, collaborating with someone using an older release, etc.)

### 3.2.1. Method 1¶

This section describes how to download git-aware copies of the repository. In a terminal window, cd to the directory where you want your code to reside. Type:

% git clone https://github.com/MITgcm/MITgcm.git


This will download the latest available code. If you now want to revert this code to a specific checkpoint release, first cd into the MITgcm directory you just downloaded, then type git checkout checkpoint«XXX» where «XXX» is the checkpoint version.

Alternatively, if you prefer to use ssh keys (say for example, you have a firewall which won’t allow a https download), type:

% git clone git@github.com:MITgcm/MITgcm.git


You will need a GitHub account for this, and will have to generate a ssh key though your GitHub account user settings.

The fully git-aware download is over several hundred MB, which is considerable if one has limited internet download speed. In comparison, the one-time download zip file (Method 2, below) is order 100 MB. However, one can obtain a truncated, yet still git-aware copy of the current code by adding the option --depth=1 to the git clone command above; all files will be present, but it will not include the full git history. However, the repository can be updated going forward.

### 3.2.2. Method 2¶

This section describes how to do a one-time download of MITgcm, NOT git-aware. In a terminal window, cd to the directory where you want your code to reside. To obtain the current code, type:

% wget https://github.com/MITgcm/MITgcm/archive/master.zip


For specific checkpoint release XXX, instead type:

% wget https://github.com/MITgcm/MITgcm/archive/checkpoint«XXX».zip


## 3.3. Updating the code¶

There are several different approaches one can use to obtain updates to MITgcm; which is best for you depends a bit on how you intend to use MITgcm and your knowledge of git (and/or willingness to learn). Below we outline three suggested update pathways:

This approach is the most simple, and virtually foolproof. Whether you downloaded the code from a static zip file (Method 2) or used the git clone command (Method 1), create a new directory and repeat this procedure to download a current copy of MITgcm. Say for example you are starting a new research project, this would be a great time to grab the most recent code repository and keep this new work entirely separate from any past simulations. This approach requires no understanding of git, and you are free to make changes to any files in the MIT repo tree (although we generally recommend that you avoid doing so, instead working in new subdirectories or on separate scratch disks as described here, for example).

1. Using git pull to update the (unmodified) MITgcm repo tree

If you have downloaded the code through a git clone command (Method 1 above), you can incorporate any changes to the source code (including any changes to any files in the MITgcm repository, new packages or analysis routines, etc.) that may have occurred since your original download. There is a simple command to bring all code in the repository to a ‘current release’ state. From the MITgcm top directory or any of its subdirectories, type:

% git pull


and all files will be updated to match the current state of the code repository, as it exists at GitHub. (Note: if you plan to contribute to MITgcm and followed the steps to download the code as described in Section 5, you will need to type git pull upstream instead.)

This update pathway is ideal if you are in the midst of a project and you want to incorporate new MITgcm features into your executable(s), or take advantage of recently added analysis utilties, etc. After the git pull, any changes in model source code and include files will be updated, so you can repeat the build procedure (Section 3.5) and you will include all these new features in your new executable.

Be forewarned, this will only work if you have not modified ANY of the files in the MITgcm repository (adding new files is ok; also, all verification run subdirectories build and run are also ignored by git). If you have modified files and the git pull fails with errors, there is no easy fix other than to learn something about git (continue reading…)

1. Fully embracing the power of git!

Git offers many tools to help organize and track changes in your work. For example, one might keep separate projects on different branches, and update the code separately (using git pull) on these separate branches. You can even make changes to code in the MIT repo tree; when git then tries to update code from upstream (see Figure 5.1), it will notify you about possible conflicts and even merge the code changes together if it can. You can also use git commit to help you track what you are modifying in your simulations over time. If you’re planning to submit a pull request to include your changes, you should read the contributing guide in Section 5, and we suggest you do this model development in a separate, fresh copy of the code. See Section 5.2 for more information and how to use git effectively to manage your workflow.

## 3.4. Model and directory structure¶

The “numerical” model is contained within a execution environment support wrapper. This wrapper is designed to provide a general framework for grid-point models; MITgcm is a specific numerical model that makes use of this framework (see Section 6.2 for additional detail). Under this structure, the model is split into execution environment support code and conventional numerical model code. The execution environment support code is held under the eesupp directory. The grid point model code is held under the model directory. Code execution actually starts in the eesupp routines and not in the model routines. For this reason the top-level main.F is in the eesupp/src directory. In general, end-users should not need to worry about the wrapper support code. The top-level routine for the numerical part of the code is in model/src/the_model_main.F. Here is a brief description of the directory structure of the model under the root tree.

• model: this directory contains the main source code. Also subdivided into two subdirectories: model/inc (includes files) and model/src (source code).

• eesupp: contains the execution environment source code. Also subdivided into two subdirectories: eesupp/inc and eesupp/src.

• pkg: contains the source code for the packages. Each package corresponds to a subdirectory. For example, pkg/gmredi contains the code related to the Gent-McWilliams/Redi scheme, pkg/seaice the code for a dynamic seaice model which can be coupled to the ocean model. The packages are described in detail in Section 8 and Section 9].

• doc: contains MITgcm documentation in reStructured Text (rst) format.

• tools: this directory contains various useful tools. For example, genmake2 is a script written in bash that should be used to generate your makefile. The subdirectory tools/build_options contains ‘optfiles’ with the compiler options for many different compilers and machines that can run MITgcm (see Section 3.5.2.2). This directory also contains subdirectories tools/adjoint_options and tools/OAD_support that are used to generate the tangent linear and adjoint model (see details in Section 7).

• utils: this directory contains various utilities. The utils/matlab subdirectory contains matlab scripts for reading model output directly into matlab. The subdirectory utils/python contains similar routines for python. utils/scripts contains C-shell post-processing scripts for joining processor-based and tiled-based model output.

• verification: this directory contains the model examples. See Section 4.

• jobs: contains sample job scripts for running MITgcm.

• lsopt: Line search code used for optimization.

• optim: Interface between MITgcm and line search code.

## 3.5. Building the model¶

### 3.5.1. Quickstart Guide¶

To compile the code, we use the make program. This uses a file (Makefile) that allows us to pre-process source files, specify compiler and optimization options and also figures out any file dependencies. We supply a script (genmake2), described in section Section 3.5.2, that automatically generates the Makefile for you. You then need to build the dependencies and compile the code (Section 3.5.3).

As an example, assume that you want to build and run experiment verification/exp2. Let’s build the code in verification/exp2/build:

% cd verification/exp2/build


First, generate the Makefile:

% ../../../tools/genmake2 -mods ../code -optfile «/PATH/TO/OPTFILE»


The -mods command line option tells genmake2 to override model source code with any files in the subdirectory ../code (here, you need to configure the size of the model domain by overriding MITgcm’s default SIZE.h with an edited copy ../code/SIZE.h containing the specific domain size for exp2).

The -optfile command line option tells genmake2 to run the specified optfile, a bash shell script, during genmake2’s execution. An optfile typically contains definitions of environment variables, paths, compiler options, and anything else that needs to be set in order to compile on your local computer system or cluster with your specific Fortan compiler. As an example, we might replace «/PATH/TO/OPTFILE» with ../../../tools/build_options/linux_amd64_ifort11 for use with the Intel Fortran compiler (version 11 and above) on a linux x86_64 platform. This and many other optfiles for common systems and Fortran compilers are located in tools/build_options.

-mods, -optfile, and many additional genmake2 command line options are described more fully in Section 3.5.2.1. Detailed instructions on building with MPI are given in Section 3.5.4.

Once a Makefile has been generated, we create the dependencies with the command:

% make depend


It is important to note that the make depend stage will occasionally produce warnings or errors if the dependency parsing tool is unable to find all of the necessary header files (e.g., netcdf.inc, or worse, say it cannot find a Fortran compiler in your path). In some cases you may need to obtain help from your system administrator to locate these files.

Next, one can compile the code using:

% make


Assuming no errors occurred, the make command creates an executable called mitgcmuv.

Now you are ready to run the model. General instructions for doing so are given in section Section 3.6.

### 3.5.2. Generating a Makefile using genmake2¶

A shell script called genmake2 for generating a Makefile is included as part of MITgcm. Typically genmake2 is used in a sequence of steps as shown below:

% ../../../tools/genmake2 -mods ../code -optfile «/PATH/TO/OPTFILE»
% make depend
% make


The first step above creates a unix-style Makefile. The Makefile is used by make to specify how to compile the MITgcm source files (for more detailed descriptions of what the make tools are, and how they are used, see here).

This section describes details and capabilities of genmake2, located in the tools directory. genmake2 is a shell script written to work in bash (and with all “sh”–compatible shells including Bourne shells). Like many unix tools, there is a help option that is invoked thru genmake2 -h. genmake2 parses information from the following sources, in this order:

1. Command-line options (see Section 3.5.2.1)

2. A genmake_local file if one is found in the current directory. This is a bash shell script that is executed prior to the optfile (see step #3), used in some special model configurations and/or to set some options that can affect which lines of the optfile are executed. For example, this genmake_local file is required for a special setup, building a ‘MITgcm coupler’ executable; in a more typical setup, one will not require a genmake_local file.

3. An “options file” a.k.a. optfile (a bash shell script) specified by the command-line option –optfile «/PATH/TO/OPTFILE», as mentioned briefly in Section 3.5.1 and described in detail in Section 3.5.2.2.

4. A packages.conf file (if one is found) with the specific list of packages to compile (see Section 8.1.1). The search path for file packages.conf is first the current directory, and then each of the -mods directories in the given order (as described here).

When you run the genmake2 script, typical output might be as follows:

% ../../../tools/genmake2 -mods ../code -optfile ../../../tools/build_options/linux_amd64_gfortran

GENMAKE :

A program for GENerating MAKEfiles for the MITgcm project.
For a quick list of options, use "genmake2 -h"
or for more detail see the documentation, section "Building the model"

===  Processing options files and arguments  ===
getting local config information:  none found
Warning: ROOTDIR was not specified ; try using a local copy of MITgcm found at "../../.."
getting OPTFILE information:
using OPTFILE="../../../tools/build_options/linux_amd64_gfortran"
check Fortran Compiler...  pass  (set FC_CHECK=5/5)
check makedepend (local: 0, system: 1, 1)

===  Checking system libraries  ===
Do we have the system() command using gfortran...  yes
Do we have the fdate() command using gfortran...  yes
Do we have the etime() command using gfortran... c,r: yes (SbR)
Can we call simple C routines (here, "cloc()") using gfortran...  yes
Can we unlimit the stack size using gfortran...  yes
Can we register a signal handler using gfortran...  yes
Can we use stat() through C calls...  yes
Can we create NetCDF-enabled binaries...  yes
skip check for LAPACK Libs
Can we call FLUSH intrinsic subroutine...  yes

===  Setting defaults  ===
Making source files in eesupp from templates
Making source files in pkg/exch2 from templates
Making source files in pkg/regrid from templates

===  Determining package settings  ===
getting package dependency info from  ../../../pkg/pkg_depend
getting package groups info from      ../../../pkg/pkg_groups
checking list of packages to compile:
using PKG_LIST="../code/packages.conf"
before group expansion packages are: oceanic -kpp -gmredi cd_code
replacing "oceanic" with:  gfd gmredi kpp
replacing "gfd" with:  mom_common mom_fluxform mom_vecinv generic_advdiff debug mdsio rw monitor
after group expansion packages are:  mom_common mom_fluxform mom_vecinv generic_advdiff debug mdsio rw monitor gmredi kpp -kpp -gmredi cd_code
applying DISABLE settings
applying ENABLE settings
packages are:  cd_code debug generic_advdiff mdsio mom_common mom_fluxform mom_vecinv monitor rw
applying package dependency rules
packages are:  cd_code debug generic_advdiff mdsio mom_common mom_fluxform mom_vecinv monitor rw
Searching for *OPTIONS.h files in order to warn about the presence
of "#define "-type statements that are no longer allowed:
found CPP_EEOPTIONS="../../../eesupp/inc/CPP_EEOPTIONS.h"
found CPP_OPTIONS="../../../model/inc/CPP_OPTIONS.h"
Creating the list of files for the adjoint compiler.

===  Creating the Makefile  ===
setting INCLUDES
Determining the list of source and include files
Writing makefile: Makefile
Making list of "exceptions" that need ".p" files
Making list of NOOPTFILES

===  Done  ===
original 'Makefile' generated successfully
=> next steps:
> make depend
> make       (<-- to generate executable)


In the above, notice:

• we did not specify ROOTDIR, i.e., a path to your MITgcm repository, but here we are building code from within the repository (specifically, in one of the verification subdirectory experiments). As such, genmake2 was smart enough to locate all necessary files on its own. To specify a remote ROOTDIR, see here.

• we specified the optfile linux_amd64_gfortran based on the computer system and Fortran compiler we used (here, a linux 64-bit machine with gfortran installed).

• genmake2 did some simple checking on availability of certain system libraries; all were found (except LAPACK, which was not checked since it is not needed here). NetCDF only requires a ‘yes’ if you want to write netCDF output; more specifically, a ‘no’ response to “Can we create NetCDF-enabled binaries” will disable including pkg/mnc and switch to output plain binary files. While the makefile can still be built with other ‘no’ responses, sometimes this will foretell errors during the make depend or make commands.

• any .F or .h files in the -mods directory ../code will also be compiled, overriding any MITgcm repository versions of files, if they exist.

• a handful of packages are being used in this build; see Section 8.1.1 for more detail about how to enable and disable packages.

• genmake2 terminated without error (note output at end after ===  Done  ===), generating Makefile and a log file genmake.log. As mentioned, this does not guarantee that your setup will compile properly, but if there are errors during make depend or make, these error messages and/or the standard output from genmake2 or genmake.log may provide clues as to the problem. If instead genmake2 finishes with a warning message Warning: FORTRAN compiler test failed , this means that genmake2 is unable to locate the Fortran compiler or pass a trivial “hello world” Fortran compilation test. In this case, you should see genmake.log for errors and/or seek assistance from your system administrator; these tests need to pass in order to proceed to the make steps.

#### 3.5.2.1. Command-line options:¶

genmake2 supports a number of helpful command-line options. A complete list of these options can be obtained by:

% genmake2 -h


The most important command-line options are:

–optfile «/PATH/TO/OPTFILE»

(or shorter: -of ) specifies the optfile that should be used for a particular build.

If no optfile is specified through the command line, genmake2 will try to make a reasonable guess from the list provided in tools/build_options. The method used for making this guess is to first determine the combination of operating system and hardware and then find a working Fortran compiler within the user’s path. When these three items have been identified, genmake2 will try to find an optfile that has a matching name. See Section 3.5.2.2.

–mods '«DIR1 DIR2 DIR3 ...»'

specifies a list of directories containing “modifications”. These directories contain files with names that may (or may not) exist in the main MITgcm source tree but will be overridden by any identically-named sources within the -mods directories. Note the quotes around the list of directories, necessary given multiple arguments.

The order of precedence for versions of files with identical names:

• “mods” directories in the order given (e.g., will use copy of file located in DIR1 instead of DIR2)

• Packages either explicitly specified or included by default

• Packages included due to package dependencies

• The “standard dirs” (which may have been specified by the -standarddirs option below)

-rootdir «/PATH/TO/MITGCMDIR»

specify the location of the MITgcm repository top directory (ROOTDIR). By default, genmake2 will try to find this location by looking in parent directories from where genmake2 is executed (up to 5 directory levels above the current directory).

In the quickstart example above (Section 3.5.1) we built the executable in the build directory of the experiment. Below, we show how to configure and compile the code on a scratch disk, without having to copy the entire source tree. The only requirement is that you have genmake2 in your $PATH, or you know the absolute path to genmake2. In general, one can compile the code in any given directory by following this procedure. Assuming the model source is in ~/MITgcm, then the following commands will build the model in /scratch/exp2-run1: % cd /scratch/exp2-run1 % ~/MITgcm/tools/genmake2 -rootdir ~/MITgcm -mods ~/MITgcm/verification/exp2/code % make depend % make  As an alternative to specifying the MITgcm repository location through the -rootdir command-line option, genmake2 recognizes the environment variable $MITGCM_ROOTDIR.

-standarddirs «/PATH/TO/STANDARDDIR»

specify a path to the standard MITgcm directories for source and includes files. By default, model and eesupp directories (src and inc) are the “standard dirs”. This command can be used to reset these default standard directories, or instead NOT include either model or eesupp as done in some specialized configurations.

-oad

generates a makefile for an OpenAD build (see Section 7.5)

–adoptfile «/PATH/TO/FILE»

(or shorter: -adof ) specifies the “adjoint” or automatic differentiation options file to be used. The file is analogous to the optfile defined above but it specifies information for the AD build process. See Section 7.2.3.4.

The default file is located in tools/adjoint_options/adjoint_default and it defines the “TAF” and “TAMC” compiler options.

–mpi

enables certain MPI features (using CPP #define) within the code and is necessary for MPI builds (see Section 3.5.4).

–omp

enables OpenMP code and compiler flag OMPFLAG (see Section 3.5.5).

–ieee

use IEEE numerics (requires support in optfile). This option is typically a good choice if one wants to compare output from different machines running the same code. Note using IEEE disables all compiler optimizations.

-devel

use IEEE numerics (requires support in optfile) and add additional compiler options to check array bounds and add other additional warning and debugging flags.

–make «/PATH/TO/GMAKE»

due to the poor handling of soft-links and other bugs common with the make versions provided by commercial unix vendors, GNU make (sometimes called gmake) may be preferred. This option provides a means for specifying the make executable to be used.

While it is possible to use genmake2 command-line options to set the Fortran or C compiler name (-fc and -cc respectively), we generally recommend setting these through an optfile, as discussed in Section 3.5.2.2. Other genmake2 options are available to enable performance/timing analyses, etc.; see genmake2 -h for more info.

#### 3.5.2.2. Optfiles in tools/build_options directory:¶

The purpose of the optfiles is to provide all the compilation options for particular “platforms” (where “platform” roughly means the combination of the hardware and the compiler) and code configurations. Given the combinations of possible compilers and library dependencies (e.g., MPI or netCDF) there may be numerous optfiles available for a single machine. The naming scheme for the majority of the optfiles shipped with the code is OS_HARDWARE_COMPILER where

OS

is the name of the operating system (generally the lower-case output of a linux terminal uname command)

HARDWARE

is a string that describes the CPU type and corresponds to output from a uname -m command. Some common CPU types:

amd64

use this code for x86_64 systems (most common, including AMD and Intel 64-bit CPUs)

ia64

is for Intel IA64 systems (eg. Itanium, Itanium2)

ppc

is for (old) Mac PowerPC systems

COMPILER

is the compiler name (generally, the name of the Fortran compiler executable). MITgcm is primarily written in FORTRAN 77. Compiling the code requires a FORTRAN 77 compiler. Any more recent compiler which is backwards compatible with FORTRAN 77 can also be used; for example, the model will build successfully with a Fortran 90 or Fortran 95 compiler. A C99 compatible compiler is also need, together with a C preprocessor . Some optional packages make use of Fortran 90 constructs (either free-form formatting, or dynamic memory allocation); as such, setups which use these packages require a Fortran 90 or later compiler build.

There are existing optfiles that work with many common hardware/compiler configurations; we first suggest you peruse the list in tools/build_options and try to find your platform/compiler configuration. These are the most common:

The above optfiles are all for linux x86_64 (64-bit) systems, utilized in many large high-performance computing centers. All of the above will work with single-threaded, MPI, or shared memory (OpenMP) code configurations. gfortran is GNU Fortran, ifort is Intel Fortran, pgf77 is PGI Fortran (formerly known as “The Portland Group”). Note in the above list there are two ifort optfiles: linux_amd64_ifort+impi is for a specific case of using ifort with the Intel MPI library (a.k.a. impi), which requires special define statements in the optfile (in contrast with Open MPI or MVAPICH2 libraries; see Section 3.5.4). Note that both ifort optfiles require ifort version 11 or higher. Many clusters nowadays use environment modules, which allows one to easily choose which compiler to use through module load «MODULENAME», automatically configuring your environment for a specific compiler choice (type echo $PATH to see where genmake2 will look for compilers and system software). In most cases, your platform configuration will be included in the available optfiles list and will result in a usable Makefile being generated. If you are unsure which optfile is correct for your configuration, you can try not specifying an optfile; on some systems the genmake2 program will be able to automatically recognize the hardware, find a compiler and other tools within the user’s path, and then make a best guess as to an appropriate optfile from the list in the tools/build_options directory. However, for some platforms and code configurations, new optfiles must be written. To create a new optfile, it is generally best to start with one of the defaults and modify it to suit your needs. Like genmake2, the optfiles are all written in bash (or using a simple sh–compatible syntax). While nearly all environment variables used within genmake2 may be specified in the optfiles, the critical ones that should be defined are: FC the Fortran compiler (executable) to use on .F files, e.g., ifort or gfortran, or if using MPI, the mpi-wrapper equivalent, e.g., mpif77 F90C the Fortran compiler to use on .F90 files (only necessary if your setup includes a package which contains .F90 source code) CC similarly, the C compiler to use, e.g., icc or gcc, or if using MPI, the mpi-wrapper equivalent, e.g., mpicc DEFINES command-line options passed to the compiler CPP the C preprocessor to use, and any necessary command-line options, e.g. cpp -traditional -P CFLAGS, FFLAGS command-line compiler flags required for your C and Fortran compilers, respectively, to compile and execute properly. See your C and Fortran compiler documentation for specific options and syntax. FOPTIM command-line optimization Fortran compiler settings. See your Fortran compiler documentation for specific options and syntax. NOOPTFLAGS command-line settings for special files that should not be optimized using the FOPTIM flags NOOPTFILES list of source code files that should be compiled using NOOPTFLAGS settings INCLUDES path for additional files (e.g., netcdf.inc, mpif.h) to include in the compilation using the command-line -I option INCLUDEDIRS path for additional files to be included in the compilation LIBS path for additional library files that need to be linked to generate the final executable, e.g., libnetcdf.a For example, an excerpt from an optfile which specifies several of these variables (here, for the linux-amd64 architecture using the PGI Fortran compiler) is as follows: if test "x$MPI" = xtrue ; then
CC=mpicc
FC=mpif77
F90C=mpif90
else
CC=pgcc
FC=pgf77
F90C=pgf90
fi

DEFINES="-DWORDLENGTH=4"
if test "x$ALWAYS_USE_F90" = x1 ; then FC=$F90C
else
DEFINES="$DEFINES -DNML_EXTENDED_F77 -DEXCLUDE_OPEN_ACTION" fi CPP='cpp -traditional -P' F90FIXEDFORMAT='-Mfixed' EXTENDED_SRC_FLAG='-Mextend' GET_FC_VERSION="-V" OMPFLAG='-mp' NOOPTFLAGS='-O0' NOOPTFILES='' FFLAGS="$FFLAGS -byteswapio -Ktrap=fp"
#- might want to use '-r8' for fizhi pkg:
#FFLAGS="$FFLAGS -r8" if test "x$IEEE" = x ; then	#- with optimisation:
FOPTIM='-tp k8-64 -pc=64 -O2 -Mvect=sse'
#FOPTIM="$FOPTIM -fastsse -O3 -Msmart -Mvect=cachesize:1048576,transform" else #- no optimisation + IEEE : #FFLAGS="$FFLAGS -Mdclchk"  #- pkg/zonal_filt does not pass with declaration-check
FOPTIM='-pc=64 -O0 -Kieee'
fi

F90FLAGS=$FFLAGS F90OPTIM=$FOPTIM


The above list of environment variables typically specified in an optfile is by no means complete; additional variables may be required for your specific setup and/or your specific Fortran (or C) compiler.

If you write an optfile for an unrepresented machine or compiler, you are strongly encouraged to submit the optfile to the MITgcm project for inclusion. MITgcm developers are willing to provide help writing or modifing optfiles. Please submit the file through the GitHub issue tracker or email the MITgcm-support@mitgcm.org list.

Instructions on how to use optfiles to build MPI-enabled executables is presented in Section 3.5.4.

### 3.5.3. make commands¶

Following a successful build of Makefile, type make depend. This command modifies the Makefile by attaching a (usually, long) list of files upon which other files depend. The purpose of this is to reduce re-compilation if and when you start to modify the code. The make depend command also creates local links for all source files from the source directories (see “-mods” description in Section 3.5.2.1), so that all source files to be used are visible from the local build directory, either as hardcopy or as symbolic link.

IMPORTANT NOTE: Editing the source code files in the build directory will not edit a local copy (since these are just links) but will edit the original files in model/src (or model/inc) or in the specified -mods directory. While the latter might be what you intend, editing the master copy in model/src is usually NOT what is intended and may cause grief somewhere down the road. Rather, if you need to add to the list of modified source code files, place a copy of the file(s) to edit in the -mods directory, make the edits to these -mods directory files, go back to the build directory and type make Clean, and then re-build the makefile (these latter steps critical or the makefile will not link to this newly edited file).

The final make invokes the C preprocessor to produce the “little f” files (*.f and *.f90) and then compiles them to object code using the specified Fortran compiler and options. The C preprocessor step converts a number of CPP macros and #ifdef statements to actual Fortran and expands C-style #include statements to incorporate header files into the “little f” files. CPP style macros and #ifdef statements are used to support generating different compile code for different model configurations. The result of the build process is an executable with the name mitgcmuv.

Additional make “targets” are defined within the makefile to aid in the production of adjoint (Section 7.2.2) and other versions of MITgcm.

On computers with multiple processor cores, the build process can often be sped up appreciably using the command:

% make -j 2


where the “2” can be replaced with a number that corresponds to the number of cores (or discrete CPUs) available.

In addition, there are several housekeeping make clean options that might be useful:

• make clean removes files that make generates (e.g., *.o and *.f files)

• make Clean removes files and links generated by make and make depend; strongly recommended for “un-clean” directories which may contain the (perhaps partial) results of previous builds

• make CLEAN removes pretty much everything, including any executables and output from genmake2

### 3.5.4. Building with MPI¶

Building MITgcm to use MPI libraries can be complicated due to the variety of different MPI implementations available, their dependencies or interactions with different compilers, and their often ad-hoc locations within file systems. For these reasons, its generally a good idea to start by finding and reading the documentation for your machine(s) and, if necessary, seeking help from your local systems administrator.

The steps for building MITgcm with MPI support are:

1. Make sure you have MPI libraries installed on your computer system or cluster. Different Fortran compilers (and different versions of a specific compiler) will generally require a custom version (of a MPI library) built specifically for it. On environment module-enabled clusters, one typically must first load a Fortran compiler, then specific MPI libraries for that compiler will become available to load. If libraries are not installed, MPI implementations and related tools are available including:

2. Determine the location of your MPI library “wrapper” Fortran compiler, e.g., mpif77 or mpifort etc. which will be used instead of the name of the fortran compiler (gfortran, ifort, pgi77 etc.) to compile your code. Often the directory in which these wrappers are located will be automatically added to your $PATH environment variable when you perform a module load «SOME_MPI_MODULE»; thus, you will not need to do anything beyond the module load itself. If you are on a cluster that does not support environment modules, you may have to manually add this directory to your path, e.g., type PATH=$PATH:«ADD_ADDITIONAL_PATH_TO_MPI_WRAPPER_HERE» in a bash shell.

3. Determine the location of the includes file mpif.h and any other MPI-related includes files. Often these files will be located in a subdirectory off the main MPI library include/. In all optfiles in tools/build_options, it is assumed environment variable $MPI_INC_DIR specifies this location; $MPI_INC_DIR should be set in your terminal session prior to generating a Makefile.

4. Determine how many processors (i.e., CPU cores) you will be using in your run, and modify your configuration’s SIZE.h (located in a “modified code” directory, as specified in your genmake2 command-line). In SIZE.h, you will need to set variables nPx*nPy to match the number of processors you will specify in your run script’s MITgcm execution statement (i.e., typically mpirun or some similar command, see Section 3.6.1). Note that MITgcm does not use dynamic memory allocation (a feature of Fortran 90, not FORTRAN 77), so all array sizes, and hence the number of processors to be used in your MPI run, must be specified at compile-time in addition to run-time. More information about the MITgcm WRAPPER, domain decomposition, and how to configure SIZE.h can be found in Section 6.3.

5. Build the code with the genmake2 -mpi option using commands such as:

%  ../../../tools/genmake2 -mods=../code -mpi -of=«/PATH/TO/OPTFILE»
%  make depend
%  make


### 3.5.5. Building with OpenMP¶

Unlike MPI, which requires installation of additional software support libraries, using shared memory (OpenMP) for multi-threaded executable builds can be accomplished simply through the genmake2 command-line option -omp:

%  ../../../tools/genmake2 -mods=../code -omp -of=«/PATH/TO/OPTFILE»
%  make depend
%  make


While the most common optfiles specified in Section 3.5.2.2 include support for the -omp option, some optfiles in tools/build_options do not include support for multi-threaded executable builds. Before using one of the less common optfiles, check whether OMPFLAG is defined.

Note that one does not need to specify the number of threads until runtime (see Section 3.6.2). However, the default maximum number of threads in MITgcm is set to a (low) value of 4, so if you plan on more you will need to change this value in eesupp/inc/EEPARAMS.h in your modified code directory.

## 3.6. Running the model¶

If compilation finished successfully (Section 3.5) then an executable called mitgcmuv will now exist in the local (build) directory.

To run the model as a single process (i.e., not in parallel) simply type (assuming you are still in the build directory):

% cd ../run
% ln -s ../input/* .
% cp ../build/mitgcmuv .
% ./mitgcmuv


Here, we are making a link to all the support data files (in ../input/) needed by the MITgcm for this experiment, and then copying the executable from the the build directory. The ./ in the last step is a safe-guard to make sure you use the local executable in case you have others that might exist in your $PATH. The above command will spew out many lines of text output to your screen. This output contains details such as parameter values as well as diagnostics such as mean kinetic energy, largest CFL number, etc. It is worth keeping this text output with the binary output so we normally re-direct the stdout stream as follows: % ./mitgcmuv > output.txt  In the event that the model encounters an error and stops, it is very helpful to include the last few line of this output.txt file along with the (stderr) error message within any bug reports. For the example experiment in verification/exp2, an example of the output is kept in verification/exp2/results/output.txt for comparison. You can compare your output.txt with the corresponding one for that experiment to check that your set-up indeed works. Congratulations! ### 3.6.1. Running with MPI¶ Run the code with the appropriate MPI “run” or “exec” program provided with your particular implementation of MPI. Typical MPI packages such as Open MPI will use something like: % mpirun -np 4 ./mitgcmuv  Sightly more complicated scripts may be needed for many machines since execution of the code may be controlled by both the MPI library and a job scheduling and queueing system such as Slurm, PBS/TORQUE, LoadLeveler, or any of a number of similar tools. See your local cluster documentation or system administrator for the specific syntax required to run on your computing facility. ### 3.6.2. Running with OpenMP¶ Assuming the executable mitgcmuv was built with OpenMP (see Section 3.5.5), the syntax to run a multi-threaded simulation is the same as running single-threaded (see Section 3.6), except that the following additional steps are required beforehand: 1. Environment variables for the number of threads and the stacksize need to be set prior to executing the model. The exact names of these environment variables differ by Fortran compiler, but are typically some variant of OMP_NUM_THREADS and OMP_STACKSIZE, respectively. For the latter, in your run script we recommend adding the line export OMP_STACKSIZE=400M (or for a C shell-variant, setenv OMP_STACKSIZE 400M). If this stacksize setting is insufficient, MITgcm will crash, in which case a larger number can be used. Similarly, OMP_NUM_THREADS should be set to the exact number of threads you require. 2. In file eedata you will need to change namelist parameters nTx and nTy to reflect the number of threads in x and y, respectively (for a single-threaded run, nTx =nTy=1). The value of nTx *nTy must equal the value of environment variable OMP_NUM_THREADS (or its name-equivalent for your Fortan compiler) or MITgcm will terminate during its initialization with an error message. MITgcm will take the number of tiles used in the model (as specified in SIZE.h) and the number of threads (nTx and nTy from file eedata), and in running will spread the tiles out evenly across the threads. This is done independently for x and y. As such, the number of tiles in x (variable nSx as defined in SIZE.h) must divide evenly by the number of threads in x (namelist parameter nTx), and similarly for nSy and nTy, else MITgcm will terminate on initialization. More information about the MITgcm WRAPPER, domain decomposition, and how to configure SIZE.h can be found in Section 6.3. ### 3.6.3. Output files¶ The model produces various output files and, when using pkg/mnc (i.e., netCDF), sometimes even directories. Depending upon the I/O package(s) selected at compile time (either pkg/mdsio, pkg/mnc, or both as determined by packages.conf) and the run-time flags set (in data.pkg), the following output may appear. More complete information describing output files and model diagnostics is described in Section 9. #### 3.6.3.1. Raw binary output files¶ The “traditional” output files are generated by the pkg/mdsio (see Section 9.2).The pkg/mdsio model data are written according to a “meta/data” file format. Each variable is associated with two files with suffix names .data and .meta. The .data file contains the data written in binary form (big endian by default). The .meta file is a “header” file that contains information about the size and the structure of the .data file. This way of organizing the output is particularly useful when running multi-processors calculations. At a minimum, the instantaneous “state” of the model is written out, which is made of the following files: • U.00000nIter - zonal component of velocity field (m/s and positive eastward). • V.00000nIter - meridional component of velocity field (m/s and positive northward). • W.00000nIter - vertical component of velocity field (ocean: m/s and positive upward, atmosphere: Pa/s and positive towards increasing pressure i.e., downward). • T.00000nIter - potential temperature (ocean: $$^{\circ}\mathrm{C}$$, atmosphere: $$\mathrm{K}$$). • S.00000nIter - ocean: salinity (g/kg), atmosphere: water vapor (g/kg). • Eta.00000nIter - ocean: surface elevation (m), atmosphere: surface pressure anomaly (Pa). The chain 00000nIter consists of ten figures that specify the iteration number at which the output is written out. For example, U.0000000300 is the zonal velocity at iteration 300. In addition, a “pickup” or “checkpoint” file called: • pickup.00000nIter is written out. This file represents the state of the model in a condensed form and is used for restarting the integration (at the specific iteration number). Some additional parameterizations and packages also produce separate pickup files, e.g., • pickup_cd.00000nIter if the C-D scheme is used (see C_D Scheme) • pickup_seaice.00000nIter if the seaice package is turned on (see SEAICE Package) • pickup_ptracers.00000nIter if passive tracers are included in the simulation (see PTRACERS Package) Rolling checkpoint files are the same as the pickup files but are named differently. Their name contain the chain ckptA or ckptB instead of 00000nIter. They can be used to restart the model but are overwritten every other time they are output to save disk space during long integrations. #### 3.6.3.2. NetCDF output files¶ pkg/mnc is a set of routines written to read, write, and append netCDF files. Unlike the pkg/mdsio output, the pkg/mnc–generated output is usually placed within a subdirectory with a name such as mnc_output_ (by default, netCDF tries to append, rather than overwrite, existing files, so a unique output directory is helpful for each separate run). The pkg/mnc output files are all in the “self-describing” netCDF format and can thus be browsed and/or plotted using tools such as: • ncdump is a utility which is typically included with every netCDF install, and converts the netCDF binaries into formatted ASCII text files. • ncview is a very convenient and quick way to plot netCDF data and it runs on most platforms. Panoply is a similar alternative. • MATLAB, GrADS, IDL and other common post-processing environments provide built-in netCDF interfaces. ### 3.6.4. Looking at the output¶ #### 3.6.4.1. MATLAB¶ ##### 3.6.4.1.1. Raw binary output¶ The repository includes a few MATLAB utilities to read binary output files written in the /pkg/mdsio format. The MATLAB scripts are located in the directory utils/matlab under the root tree. The script utils/matlab/rdmds.m reads the data. Look at the comments inside the script to see how to use it. Some examples of reading and visualizing some output in MATLAB: % matlab >> H=rdmds('Depth'); >> contourf(H');colorbar; >> title('Depth of fluid as used by model'); >> eta=rdmds('Eta',10); >> imagesc(eta');axis ij;colorbar; >> title('Surface height at iter=10'); >> [eta,iters,M]=rdmds('Eta',NaN); % this will read all dumped iterations >> % iter numbers put in variable 'iters'; 'M' is a character string w/metadata >> for n=1:length(iters); imagesc(eta(:,:,n)');axis ij;colorbar;pause(.5);end  Typing help rdmds in MATLAB will pull up further information on how to use the rdmds utility. ##### 3.6.4.1.2. NetCDF output¶ Similar scripts for netCDF output (e.g., utils/matlab/rdmnc.m) are available and they are described in Section 9.3. #### 3.6.4.2. Python¶ Install the MITgcmutils python package following the instructions in Section 11.1. ##### 3.6.4.2.1. Raw binary output¶ The following example shows how to load in some data: # python from MITgcmutils import mds Eta = mds.rdmds('Eta', itrs=10)  For more information about this function and its options, see the API docs, MITgcmutils.mds.rdmds(). ##### 3.6.4.2.2. NetCDF output¶ The netCDF output is currently produced with one file per processor. This means the individual tiles need to be stitched together to create a single netCDF file that spans the model domain. The script utils/python/MITgcmutils/scripts/gluemncbig can do this efficiently from the command line. If you have installed the MITgcmutils package, a copy of gluemncbig should be on your path. For usage information, see Section 11.1.9. The following example shows how to use the xarray python package to read the resulting netCDF file into Python: # python import xarray as xr Eta = xr.open_dataset('Eta.nc')  ## 3.7. Customizing the Model Configuration - Code Parameters and Compilation Options¶ ### 3.7.1. Model Array Dimensions¶ MITgcm’s array dimensions need to be configured for each unique model domain. The size of each tile (in dimensions $$x$$, $$y$$, and vertical coordinate $$r$$) the “overlap” region of each tile (in $$x$$ and $$y$$), the number of tiles in the $$x$$ and $$y$$ dimensions, and the number of processes (using MPI) in the $$x$$ and $$y$$ dimensions all need to be specified in SIZE.h. From these parameters, global domain-size variables Nx, Ny are computed by the model. See a more technical discussion of SIZE.h parameters in in Section 6.3.1, and a detailed explanation of an example SIZE.h setup in tutorial Baroclinic Ocean Gyre. Parameter Default SIZE.h Description sNx 30 number of points in $$x$$ dimension in a single tile sNy 15 number of points in $$y$$ dimension in a single tile Nr 4 number of points in $$r$$ dimension OLx 2 number of “overlap” points in $$x$$ dimension for a tile OLy 2 number of “overlap” points in $$y$$ dimension for a tile nSx 2 number of tile per process in $$x$$ dimension nSy 4 number of tile per process in $$y$$ dimension nPx 1 number of processes in $$x$$ dimension nPy 1 number of processes in $$y$$ dimension Note the repository version of SIZE.h includes several lines of text at the top that will halt compilation with errors. Thus, to use MITgcm you will need to copy SIZE.h to a code modification directory and make edits, including deleting or commenting out the offending lines of text. ### 3.7.2. C Preprocessor Options¶ The CPP flags relative to the “numerical model” part of the code are defined and set in the file CPP_OPTIONS.h in the directory model/inc/. In the parameter tables in Section 3.8 we have noted CPP options that need to be changed from the default to enable specific runtime parameter to be used properly. Also note many of the options below are for less-common situations or are somewhat obscure, so newer users of the MITgcm are encouraged to jump to Section 3.8 where more basic runtime parameters are discussed. CPP Flag Name Default Description SHORTWAVE_HEATING #undef provide separate shortwave heating file, allowing shortwave to penetrate below surface layer ALLOW_GEOTHERMAL_FLUX #undef include code for applying geothermal heat flux at the bottom of the ocean ALLOW_FRICTION_HEATING #undef include code to allow heating due to friction (and momentum dissipation) ALLOW_ADDFLUID #undef allow mass source or sink of fluid in the interior (3D generalization of oceanic real-fresh water flux) ATMOSPHERIC_LOADING #define include code for atmospheric pressure-loading (and seaice-loading) on ocean surface ALLOW_BALANCE_FLUXES #undef include balancing surface forcing fluxes code ALLOW_BALANCE_RELAX #undef include balancing surface forcing relaxation code CHECK_SALINITY_FOR_NEGATIVE_VALUES #undef include code checking for negative salinity EXCLUDE_FFIELDS_LOAD #undef exclude external forcing-fields load; code allows reading and simple linear time interpolation of oceanic forcing fields, if no specific pkg (e.g., pkg/exf) is used to compute them INCLUDE_PHIHYD_CALCULATION_CODE #define include code to calculate $$\phi_{\rm hyd}$$ INCLUDE_CONVECT_CALL #define include code for convective adjustment mixing algorithm INCLUDE_CALC_DIFFUSIVITY_CALL #define include codes that calculates (tracer) diffusivities and viscosities ALLOW_3D_DIFFKR #undef allow full 3D specification of vertical diffusivity ALLOW_BL79_LAT_VARY #undef allow latitudinally varying Bryan and Lewis 1979 [BL79] vertical diffusivity EXCLUDE_PCELL_MIX_CODE #undef exclude code for partial-cell effect (physical or enhanced) in vertical mixing; this allows accounting for partial-cell in vertical viscosity and diffusion, either from grid-spacing reduction effect or as artificially enhanced mixing near surface & bottom for too thin grid-cell ALLOW_SMAG_3D_DIFFUSIVITY #undef include code for isotropic 3-D Smagorinsky diffusivity for tracers (viscosity scaled by constant Prandtl number) ALLOW_SOLVE4_PS_AND_DRAG #undef include code for combined surface pressure and drag implicit solver INCLUDE_IMPLVERTADV_CODE #define include code for implicit vertical advection ALLOW_ADAMSBASHFORTH_3 #undef include code for Adams-Bashforth 3rd-order ALLOW_QHYD_STAGGER_TS #undef include code for quasi-hydrostatic stagger time-step Adams-Bashforth code EXACT_CONSERV #define include code for “exact conservation” of fluid in free-surface formulation (recompute divergence after pressure solver) NONLIN_FRSURF #undef allow the use of non-linear free-surface formulation; implies that grid-cell thickness (hFactors) varies with time ALLOW_NONHYDROSTATIC #undef include non-hydrostatic and 3D pressure solver codes ALLOW_EDDYPSI #undef include GM-like eddy stress in momentum code (untested, not recommended) ALLOW_CG2D_NSA #undef use non-self-adjoint (NSA) conjugate-gradient solver ALLOW_SRCG #define include code for single reduction conjugate gradient solver SOLVE_DIAGONAL_LOWMEMORY #undef low memory footprint (not suitable for AD) choice for implicit solver routines solve_*diagonal.F SOLVE_DIAGONAL_KINNER #undef choice for implicit solver routines solve_*diagonal.F suitable for AD COSINEMETH_III #define selects implementation form of $$\cos{\varphi}$$ scaling of bi-harmonic term for viscosity (note, CPP option for tracer diffusivity set independently in GAD_OPTIONS.h) ISOTROPIC_COS_SCALING #undef selects isotropic scaling of harmonic and bi-harmonic viscous terms when using the $$\cos{\varphi}$$ scaling (note, CPP option for tracer diffusivity set independently in GAD_OPTIONS.h) By default, MITgcm includes several core packages, i.e., these packages are enabled during genmake2 execution if a file packages.conf is not found. See Section 8.1.1 for more information about packages.conf, and see pkg/pkg_groups for more information about default packages and package groups. These default packages are as follows: Additional CPP options that affect the model core code are set in files ${PKG}_OPTIONS.h located in these packages’ directories. Similarly, optional (non-default) packages also include package-specific CPP options that must be set in files ${PKG}_OPTIONS.h. ### 3.7.3. Preprocessor Execution Environment Options¶ Most MITgcm users can skip this section; many of these flags were intended for very specific platform environments, and not meant to be changed for more general environments (an exception being if you are using a coupled setup, see below). The file CPP_EEOPTIONS.h in the directory eesupp/inc/ contains a number of CPP flags related to the execution environment where the model will run. Below we describe the subset of user-editable CPP flags: CPP Flag Name Default Description GLOBAL_SUM_ORDER_TILES #define always cumulate tile local-sum in the same order by applying MPI allreduce to array of tiles CG2D_SINGLECPU_SUM #undef alternative way of doing global sum on a single CPU to eliminate tiling-dependent roundoff errors SINGLE_DISK_IO #undef to write STDOUT, STDERR and scratch files from process 0 only USE_FORTRAN_SCRATCH_FILES #undef flag to turn on old default of opening scratch files with the STATUS=’SCRATCH’ option COMPONENT_MODULE #undef control use of communication with other components, i.e., sets component to work with a coupler interface DISCONNECTED_TILES #undef use disconnected tiles (no exchange between tiles, just fill-in edges assuming locally periodic subdomain) REAL4_IS_SLOW #define if undefined, force _RS variables to be declared as real*4 The default setting of #define GLOBAL_SUM_ORDER_TILES in CPP_EEOPTIONS.h provides a way to achieve numerically reproducible global sums for a given tile domain decomposition. As implemented however, this approach will increase the volume of network traffic in a way that scales with the total number of tiles. Profiling has shown that letting the code fall through to a baseline approach that simply uses MPI_Allreduce() can provide significantly improved performance for certain simulations 1. The fall-though approach is activated by #undef GLOBAL_SUM_ORDER_TILES. In order to get bit-wise reproducible results between different tile domain decompositions (e.g., single tile on single processor versus multiple tiles either on single or multiple processors), one can choose to #define option CG2D_SINGLECPU_SUM to use the MUCH slower global_sum_singlecpu.F for the key part of MITgcm algorithm CG2D that relies on global sum. This option is not only much slower but also requires a large volume of communications so it is practically unusable for a large set-up; furthermore, it does not address reproducibility when global sum is used outside CG2D, e.g., in non-hydrostatic simulations. In a default multi-processor configuration, each process opens and reads its own set of namelist files and open and writes its own standard output. This can be slow or even problematic for very large processor counts. Defining the CPP-flag SINGLE_DISK_IO suppresses this behavior and lets only the master process (process 0) read namelist files and write a standard output stream. This may seem advantageous, because it reduces the amount of seemingly redundant output, but use this option with caution and only when absolutely confident that the setup is working since any message (error/warning/print) from any processor $$\ne 0$$ will be lost. The way the namelist files are read requires temporary (scratch) files in the initialization phase. By default, the MITgcm does not use intrinsic Fortran scratch files (STATUS='scratch') because they can lead to conflicts in multi-processor simulations on some HPC-platforms, when the processors do not open scratch files with reserved names. However, the implemented default scheme for the scratch files can be slow for large processor counts. If this is a problem in a given configuration, defining the CPP-flag USE_FORTRAN_SCRATCH_FILES may help by making the code use the intrinsic Fortran scratch files. The CPP-flag COMPONENT_MODULE needs to be set to #define only for builds in which the MITgcm executable (for either an oceanic or atmospheric simulation) is configured to communicate with a coupler. This coupler can be a specially configured build of MITgcm itself; see, for example, verification experiment cpl_aim+ocn. The CPP-flag DISCONNECTED_TILES should not be #define unless one wants to run simultaneously several small, single-tile ensemble members from a single process, as each tile will be disconnected from the others and considered locally as a doubly periodic patch. MITgcm _RS variables are forced to be declared as real*4 if CPP-flag REAL4_IS_SLOW to is set to #undef in CPP_EEOPTIONS.h (_RS is a macro used in declaring real variables that, in principle, do not require double precision). However, this option is not recommended except for computational benchmarking or for testing the trade-off between memory footprint and model precision. And even for these specialized tests, there is no need to edit CPP_EEOPTIONS.h since this feature can be activated using the genmake2 command line option -use_r4, as done in some regression tests (see testing results page tests with optfile suffix .use_r4). 1 One example is the llc_540 case located at https://github.com/MITgcm-contrib/llc_hires/tree/master/llc_540. This case was run on the Pleiades computer for 20 simulated days using 767 and 2919 MPI ranks. At 767 ranks, the fall-through approach provided a throughput of to 799.0 simulated days per calendar day (dd/d) while the default approach gave 781.0. The profiler showed the speedup was directly attributable to spending less time in MPI_Allreduce. The volume of memory traffic associated with MPI_Allreduce dropped by 3 orders (22.456T -> 32.596G). At 2819 MPI ranks the fall-through approach gave a throughput of 1300 dd/d while the default approach gave 800.0 dd/d. Put another way, this case did not scale at all from 767p to 2819p unless the fall-though approach was utilized. The profiler showed the speedup was directly attributable to spending less time in MPI_Allreduce. The volume of memory traffic associated with MPI_Allreduce dropped by 3 orders (303.70T ->121.08G ). ## 3.8. Customizing the Model Configuration - Runtime Parameters¶ When you are ready to run the model in the configuration you want, the most straightforward approach is to use and adapt the setup of a tutorial or verification experiment (described in Section 4) that is the closest to your configuration. Then, the amount of setup will be minimized. In this section, we document the complete list of MITgcm model namelist runtime parameters set in file data, which needs to be located in the directory where you will run the model. Model parameters are defined and declared in the file PARAMS.h and their default values are generally set in the routine set_defaults.F, otherwise when initialized in the routine ini_parms.F. Section 3.8.9 documents the “execution environment” namelist parameters in file eedata, which must also reside in the current run directory. Note that runtime parameters used by (non-default) MITgcm packages are not documented here but rather in Section 8 and Section 9, and prescribed in package-specific data.${pkg} namelist files which are read in via package-specific ${pkg}_readparms.F where ${pkg} is the package name (see Section 8.1.1).

In what follows, model parameters are grouped into categories related to configuration/computational domain, algorithmic parameters, equations solved in the model, parameters related to model forcing, and simulation controls. The tables below specify the namelist parameter name, the namelist parameter group in data (and eedata in Section 3.8.9), the default value, and a short description of its function. Runtime parameters that require non-default CPP options to be set prior to compilation (see Section 3.7) for proper use are noted.

### 3.8.1. Parameters: Configuration, Computational Domain, Geometry, and Time-Discretization¶

#### 3.8.1.1. Model Configuration¶

buoyancyRelation is set to OCEANIC by default, which employes a $$z$$-coordinate vertical axis. To simulate an ocean using pressure coordinates in the vertical, set it to OCEANICP. For atmospheric simulations, buoyancyRelation needs to be set to ATMOSPHERIC, which also uses pressure as the vertical coordinate. The default model configuration is hydrostatic; to run a non-hydrostatic simulation, set the logical variable nonHydrostatic to .TRUE..

Parameter

Group

Default

Description

buoyancyRelation

PARM01

OCEANIC

buoyancy relation (OCEANIC, OCEANICP, or ATMOSPHERIC)

quasiHydrostatic

PARM01

FALSE

quasi-hydrostatic formulation on/off flag

rhoRefFile

PARM01

' '

filename for reference density profile (kg/m3); activates anelastic form of model

nonHydrostatic

PARM01

FALSE

non-hydrostatic formulation on/off flag; requires #define ALLOW_NONHYDROSTATIC

#### 3.8.1.2. Grid¶

Four different grids are available: Cartesian, spherical polar, cylindrical, and curvilinear (which includes the cubed sphere). The grid is set through the logical variables usingCartesianGrid, usingSphericalPolarGrid, usingCylindricalGrid, and usingCurvilinearGrid. Note that the cylindrical grid is designed for modeling a rotating tank, so that $$x$$ is the azimuthual direction, $$y$$ is the radial direction, and $$r$$ is vertical coordinate (see tutorial rotating tank).

The variable xgOrigin sets the position of the western most gridcell face in the $$x$$ dimension (Cartesian, meters; spherical and cyclindrical, degrees). For a Cartesian or spherical grid, the southern boundary is defined through the variable ygOrigin which corresponds to the latitude of the southern most gridcell face (Cartesian, meters; spherical, degrees). For a cyclindrical grid, a positive ygOrigin (m) adds an inner cylindrical boundary at the center of the tank. The resolution along the $$x$$ and $$y$$ directions is controlled by the 1-D arrays delX (meters for a Cartesian grid, degrees otherwise) and delY (meters for Cartesian and cyclindrical grids, degrees spherical). On a spherical polar grid, you might decide to set the variable cosPower which is set to 0 by default and which represents $$n$$ in $$(\cos\varphi)^n$$, the power of cosine of latitude to multiply horizontal viscosity and tracer diffusivity. The vertical grid spacing is set through the 1-D array delR ($$z$$-coordinates: in meters; $$p$$-coordinates, in Pa). Using a curvilinear grid requires complete specification of all horizontal MITgcm grid variables, either through a default filename (link to new doc section) or as specified by horizGridFile.

The variable seaLev_Z represents the standard position of sea level, in meters. This is typically set to 0 m for the ocean (default value). If instead pressure is used as the vertical coordinate, the pressure at the top (of the atmosphere or ocean) is set through top_Pres, typically 0 Pa. As such, these variables are analogous to xgOrigin and ygOrigin to define the vertical grid axis. But they also are used for a second purpose: in a $$z$$-coordinate setup, top_Pres sets a reference top pressure (required in a non-linear equation of state computation, for example); note that 1 bar (i.e., typical Earth atmospheric sea-level pressure) is added already, so the default is 0 Pa. Similarly, for a $$p$$-coordinate setup, seaLev_Z is used to set a reference geopotential (after gravity scaling) at the top of the ocean or bottom of the atmosphere.

Parameter

Group

Default

Description

usingCartesianGrid

PARM04

TRUE

use Cartesian grid/coordinates on/off flag

usingSphericalPolarGrid

PARM04

FALSE

use spherical grid/coordinates on/off flag

usingCylindricalGrid

PARM04

FALSE

use cylindrical grid/coordinates on/off flag

usingCurvilinearGrid

PARM04

FALSE

use curvilinear grid/coordinates on/off flag

xgOrigin

PARM04

0.0

west edge $$x$$-axis origin (Cartesian: m; spherical and cyclindrical: degrees longitude)

ygOrigin

PARM04

0.0

South edge $$y$$-axis origin (Cartesian and cyclindrical: m; spherical: degrees latitude)

dxSpacing

PARM04

unset

$$x$$-axis uniform grid spacing, separation between cell faces (Cartesian: m; spherical and cyclindrical: degrees)

delX

PARM04

dxSpacing

1D array of $$x$$-axis grid spacing, separation between cell faces (Cartesian: m; spherical and cyclindrical: degrees)

delXFile

PARM04

' '

filename containing 1D array of $$x$$-axis grid spacing

dySpacing

PARM04

unset

$$y$$-axis uniform grid spacing, separation between cell faces (Cartesian and cyclindrical: m; spherical: degrees)

delY

PARM04

dySpacing

1D array of $$x$$-axis grid spacing, separation between cell faces (Cartesian and cyclindrical: m; spherical: degrees)

delYFile

PARM04

' '

filename containing 1D array of $$y$$-axis grid spacing

cosPower

PARM01

0.0

power law $$n$$ in $$(\cos\varphi)^n$$ factor for horizontal (harmonic or biharmonic) viscosity and tracer diffusivity (spherical polar)

delR

PARM04

computed using delRc

vertical grid spacing 1D array ([$$r$$] unit)

delRc

PARM04

computed using delR

vertical cell center spacing 1D array ([$$r$$] unit)

delRFile

PARM04

' '

filename for vertical grid spacing 1D array ([$$r$$] unit)

delRcFile

PARM04

' '

filename for vertical cell center spacing 1D array ([$$r$$] unit)

rSphere

PARM04

6.37E+06

radius of sphere for spherical polar or curvilinear grid (m)

seaLev_Z

PARM04

0.0

reference height of sea level (m)

top_Pres

PARM04

0.0

top pressure ($$p$$-coordinates) or top reference pressure ($$z$$-coordinates) (Pa)

selectFindRoSurf

PARM01

0

select method to determine surface reference pressure from orography (atmos.-only)

horizGridFile

PARM04

' '

filename containing full set of horizontal grid variables (curvilinear)

PARM04

rSphere

radius of sphere used in input curvilinear horizontal grid file (m)

phiEuler

PARM04

0.0

Euler angle, rotation about original $$z$$-axis (spherical polar) (degrees)

thetaEuler

PARM04

0.0

Euler angle, rotation about new $$x$$-axis (spherical polar) (degrees)

psiEuler

PARM04

0.0

Euler angle, rotation about new $$z$$-axis (spherical polar) (degrees)

#### 3.8.1.3. Topography - Full and Partial Cells¶

For the ocean, the topography is read from a file that contains a 2-D($$x,y$$) map of bathymetry, in meters for $$z$$-coordinates, in pascals for $$p$$-coordinates. The bathymetry is specified by entering the vertical position of the ocean floor relative to the surface, so by convention in $$z$$-coordinates bathymetry is specified as negative numbers (“depth” is defined as positive-definite) whereas in $$p$$-coordinates bathymetry data is positive. The file name is represented by the variable bathyFile. See our introductory tutorial setup Section 4.1 for additional details on the file format. Note no changes are required in the model source code to represent enclosed, periodic, or double periodic domains: periodicity is assumed by default and is suppressed by setting the depths to 0 m for the cells at the limits of the computational domain.

To use the partial cell capability, the variable hFacMin needs to be set to a value between 0.0 and 1.0 (it is set to 1.0 by default) corresponding to the minimum fractional size of a gridcell. For example, if a gridcell is 500 m thick and hFacMin is set to 0.1, the minimum thickness for a “thin-cell” for this specific gridcell is 50 m. Thus, if the specified bathymetry depth were to fall exactly in the middle of this 500m thick gridcell, the initial model variable hFacC($$x,y,r$$) would be set to 0.5. If the specified bathymetry depth fell within the top 50m of this gridcell (i.e., less than hFacMin), the model bathymetry would snap to the nearest legal value (i.e., initial hFacC($$x,y,r$$) would be equal to 0.0 or 0.1 depending if the depth was within 0-25 m or 25-50 m, respectively). Also note while specified bathymetry bottom depths (or pressures) need not coincide with the model’s levels as deduced from delR, any depth falling below the model’s defined vertical axis is truncated.

Parameter

Group

Default

Description

bathyFile

PARM05

' '

filename for 2D bathymetry (ocean) ($$z$$-coor.: m, negative; $$p$$-coor.: Pa, positive)

topoFile

PARM05

' '

filename for 2D surface topography (atmosphere) (m)

PARM05

' '

filename for 2D western cell-edge “thin-wall”

PARM05

' '

filename for 2D southern cell-edge “thin-wall”

hFacMin

PARM01

1.0E+00

minimum fraction size of a cell

hFacMinDr

PARM01

1.0E+00

minimum dimensional size of a cell ([$$r$$] unit)

hFacInf

PARM01

2.0E-01

lower threshold fraction for surface cell; for non-linear free surface only, see parameter nonlinFreeSurf

hFacSup

PARM01

2.0E+00

upper threshold fraction for surface cell; for non-linear free surface, only see parameter nonlinFreeSurf

useMin4hFacEdges

PARM04

FALSE

set hFacW, hFacS as minimum of adjacent hFacC on/off flag

pCellMix_select

PARM04

0

option/factor to enhance mixing at the surface or bottom (0- 99)

pCellMix_maxFac

PARM04

1.0E+04

maximum enhanced mixing factor for too thin partial-cell (non-dim.)

pCellMix_delR

PARM04

0.0

thickness criteria for too thin partial-cell ([$$r$$] unit)

#### 3.8.1.4. Physical Constants¶

Parameter

Group

Default

Description

rhoConst

PARM01

rhoNil

vertically constant reference density (Boussinesq) (kg/m3)

gravity

PARM01

9.81E+00

gravitational acceleration (m/s2)

gravityFile

PARM01

' '

filename for 1D gravity vertical profile (m/s2)

gBaro

PARM01

gravity

gravity constant in barotropic equation (m/s2)

#### 3.8.1.5. Rotation¶

For a Cartesian or cylindical grid, the Coriolis parameter $$f$$ is set through the variables f0 (in s–1) and beta ($$\frac{\partial f}{ \partial y}$$; in m–1s–1), which corresponds to a Coriolis parameter $$f = f_o + \beta y$$ (the so-called $$\beta$$-plane).

Parameter

Group

Default

Description

rotationPeriod

PARM01

8.6164E+04

rotation period (s)

omega

PARM01

$$2\pi/$$rotationPeriod

selectCoriMap

PARM01

depends on grid (Cartesian and cylindrical=1, spherical and curvilinear=2)

Coriolis map options

• 0: f-plane

• 1: beta-plane

• 2: spherical Coriolis ($$=2\Omega\sin{\varphi}$$)

• 3: read 2D field from file

f0

PARM01

1.0E-04

reference Coriolis parameter (Cartesian or cylindrical grid) (1/s)

beta

PARM01

1.0E-11

$$\beta$$ (Cartesian or cylindrical grid) (m–1s–1)

fPrime

PARM01

0.0

$$2 \Omega \cos{\phi}$$ parameter (Cartesian or cylindical grid) (1/s); i.e., for $$\cos{\varphi}$$ Coriolis terms from horizontal component of rotation vector (also sometimes referred to as reciprocal Coriolis parm.)

#### 3.8.1.6. Free Surface¶

The logical variables rigidLid and implicitFreeSurface specify your choice for ocean upper boundary (or lower boundary if using $$p$$-coordinates); set one to .TRUE. and the other to .FALSE.. These settings affect the calculations of surface pressure (for the ocean) or surface geopotential (for the atmosphere); see Section 3.8.2.

Parameter

Group

Default

Description

implicitFreeSurface

PARM01

TRUE

implicit free surface on/off flag

rigidLid

PARM01

FALSE

rigid lid on/off flag

useRealFreshWaterFlux

PARM01

FALSE

use true E-P-R freshwater flux (changes free surface/sea level) on/off flag

implicSurfPress

PARM01

1.0E+00

implicit fraction of the surface pressure gradient (0-1)

implicDiv2Dflow

PARM01

1.0E+00

implicit fraction of the barotropic flow divergence (0-1)

implicitNHPress

PARM01

implicSurfPress

implicit fraction of the non-hydrostatic pressure gradient (0-1); for non-hydrostatic only, see parameter nonHydrostatic

nonlinFreeSurf

PARM01

0

non-linear free surface options (-1,0,1,2,3; see Table 2.1); requires #define NONLIN_FRSURF

select_rStar

PARM01

0

vertical coordinate option

• 0: use r

• >0: use $$r^*$$

see Table 2.1; requires #define NONLIN_FRSURF

selectNHfreeSurf

PARM01

0

non-hydrostatic free surface formulation option

• 0: don’t use

• >0: use

requires non-hydrostatic formulation, see parameter nonHydrostatic

exactConserv

PARM01

FALSE

exact total volume conservation (recompute divergence after pressure solver) on/off flag

#### 3.8.1.7. Time-Discretization¶

The time steps are set through the real variables deltaTMom and deltaTtracer (in seconds) which represent the time step for the momentum and tracer equations, respectively (or you can prescribe a single time step value for all parameters using deltaT). The model “clock” is defined by the variable deltaTClock (in seconds) which determines the I/O frequencies and is used in tagging output. Time in the model is thus computed as:

model time = baseTime + iteration number * deltaTClock

Parameter

Group

Default

Description

deltaT

PARM03

0.0

default value used for model time step parameters (s)

deltaTClock

PARM03

deltaT

timestep used for model clock (s): used for I/O frequency and tagging output and checkpoints

deltaTmom

PARM03

deltaT

momentum equation timestep (s)

deltaTtracer

PARM03

deltaT

tracer equation timestep (s)

dTtracerLev

PARM03

deltaTtracer

tracer equation timestep specified at each vertical level (s)

deltaTfreesurf

PARM03

deltaTmom

free-surface equation timestep (s)

baseTime

PARM03

0.0

model base time corresponding to iteration 0 (s)

### 3.8.2. Parameters: Main Algorithmic Parameters¶

#### 3.8.2.1. Pressure Solver¶

By default, a hydrostatic simulation is assumed and a 2-D elliptic equation is used to invert the pressure field. If using a non-hydrostatic configuration, the pressure field is inverted through a 3-D elliptic equation (note this capability is not yet available for the atmosphere). The parameters controlling the behavior of the elliptic solvers are the variables cg2dMaxIters and cg2dTargetResidual for the 2-D case and cg3dMaxIters and cg3dTargetResidual for the 3-D case.

Parameter

Group

Default

Description

cg2dMaxIters

PARM02

150

upper limit on 2D conjugate gradient solver iterations

cg2dTargetResidual

PARM02

1.0E-07

2D conjugate gradient target residual (non-dim. due to RHS normalization )

cg2dTargetResWunit

PARM02

-1.0E+00

2D conjugate gradient target residual ($$\dot{r}$$ units); <0: use RHS normalization, i.e., cg2dTargetResidual instead

cg2dPreCondFreq

PARM02

1

frequency (in number of iterations) for updating cg2d pre-conditioner; for non-linear free surface only, see parameter nonlinFreeSurf

cg2dUseMinResSol

PARM02

0 unless flat-bottom, Cartesian

• 0: use last-iteration/converged cg2d solution

• 1: use solver minimum-residual solution

cg3dMaxIters

PARM02

150

upper limit on 3D conjugate gradient solver iterations; requires #define ALLOW_NONHYDROSTATIC

cg3dTargetResidual

PARM02

1.0E-07

3D conjugate gradient target residual (non-dim. due to RHS normalization ); requires #define ALLOW_NONHYDROSTATIC

useSRCGSolver

PARM02

FALSE

use conjugate gradient solver with single reduction (single call of mpi_allreduce)

printResidualFreq

PARM02

1 unless debugLevel >4

frequency (in number of iterations) of printing conjugate gradient residual

integr_GeoPot

PARM01

2

select method to integrate geopotential

• 1: finite volume

• $$\neq$$1: finite difference

uniformLin_PhiSurf

PARM01

TRUE

use uniform $$b_s$$ relation for $$\phi_s$$ on/off flag

deepAtmosphere

PARM04

FALSE

don’t make the thin shell/shallow water approximation

nh_Am2

PARM01

1.0E+00

non-hydrostatic terms scaling factor; requires #define ALLOW_NONHYDROSTATIC

#### 3.8.2.2. Time-Stepping Algorithm¶

The Adams-Bashforth stabilizing parameter is set through the variable abEps (dimensionless). The stagger baroclinic time stepping algorithm can be activated by setting the logical variable staggerTimeStep to .TRUE..

Parameter

Group

Default

Description

abEps

PARM03

1.0E-02

alph_AB

PARM03

0.5E+00

beta_AB

PARM03

5/12

staggerTimeStep

PARM01

FALSE

use staggered time stepping (thermodynamic vs. flow variables) on/off flag

PARM01

TRUE

use multi-dim. advection algorithm in schemes where non multi-dim. is possible on/off flag

implicitIntGravWave

PARM01

FALSE

treat internal gravity waves implicitly on/off flag; requires #define ALLOW_NONHYDROSTATIC

### 3.8.3. Parameters: Equation of State¶

The form of the equation of state is controlled by the model configuration and eosType.

For the atmosphere, eosType must be set to IDEALGAS.

For the ocean, several forms of the equation of state are available:

• For a linear approximation, set eosType to LINEAR), and you will need to specify the thermal and haline expansion coefficients, represented by the variables tAlpha (in K–1) and sBeta (in (g/kg)–1). Because the model equations are written in terms of perturbations, a reference thermodynamic state needs to be specified. This is done through the 1-D arrays tRef and sRef. tRef specifies the reference potential temperature profile (in oC for the ocean and K for the atmosphere) starting from the level k=1. Similarly, sRef specifies the reference salinity profile (in g/kg) for the ocean or the reference specific humidity profile (in g/kg) for the atmosphere.

• MITgcm offers several approximations to the full (oceanic) non-linear equation of state that can be selected as eosType:

'POLYNOMIAL':

This approximation is based on the Knudsen formula (see Bryan and Cox 1972 [BC72]). For this option you need to generate a file of polynomial coefficients called POLY3.COEFFS. To do this, use the program utils/knudsen2/knudsen2.f under the model tree (a Makefile is available in the same directory; you will need to edit the number and the values of the vertical levels in knudsen2.f so that they match those of your configuration).

’UNESCO’:

The UNESCO equation of state formula (IES80) of Fofonoff and Millard (1983) [FRM83]. This equation of state assumes in-situ temperature, which is not a model variable; its use is therefore discouraged.

’JMD95Z’:

A modified UNESCO formula by Jackett and McDougall (1995) [JM95], which uses the model variable potential temperature as input. The ’Z’ indicates that this equation of state uses a horizontally and temporally constant pressure $$p_{0}=-g\rho_{0}z$$.

’JMD95P’:

A modified UNESCO formula by Jackett and McDougall (1995) [JM95], which uses the model variable potential temperature as input. The ’P’ indicates that this equation of state uses the actual hydrostatic pressure of the last time step. Lagging the pressure in this way requires an additional pickup file for restarts.

’MDJWF’:

A more accurate and less expensive equation of state than UNESCO by McDougall et al. (2003) [MJWF03], also using the model variable potential temperature as input. It also requires lagging the pressure and therefore an additional pickup file for restarts.

’TEOS10’:

TEOS-10 is based on a Gibbs function formulation from which all thermodynamic properties of seawater (density, enthalpy, entropy sound speed, etc.) can be derived in a thermodynamically consistent manner; see http://www.teos-10.org. See IOC et al. (2010) [ISI10], McDougall and Parker (2011) [MB11], and Roquet et al. (2015) [RMMB15] for implementation details. It also requires lagging the pressure and therefore an additional pickup file for restarts. Note at this time a full implementation of TEOS10 (i.e., ocean variables of conservative temperature and practical salinity, including consideration of surface forcings) has not been implemented; also note the original 48-term polynomial term is used, not the newer, preferred 75-term polynomial.

For these non-linear approximations, neither a reference profile of temperature or salinity is required, except for a setup where implicitIntGravWave is set to .TRUE. or selectP_inEOS_Zc=1.

Note that for simplicity, salinity is expressed as a ratio in g/kg (thus effectively unitless) regardless of the choice of equation of state, despite “Practical Salinity” not precisely equal to salinity expressed as a dissolved mass fraction. If TEOS-10 is selected, the model variable salt can be interpreted as “Absolute Salinity”. See Millero (2010) [Mil10] and Pawlowicz (2013) [Paw13] for detailed discussion of salinity measurements, and why being expressed as g/kg is preferred, in the context of the ocean equation of state.

Parameter

Group

Default

Description

eosType

PARM01

LINEAR

equation of state form

tRef

PARM01

20.0 oC (ocn) or 300.0 K (atm)

1D vertical reference temperature profile (oC or K)

tRefFile

PARM01

' '

filename for reference temperature profile (oC or K)

thetaConst

PARM01

tRef(k=1)

vertically constant reference temp. for atmosphere $$p^*$$ coordinates (oK); for ocean, specify instead of tRef or tRefFile for vertically constant reference temp. (oC )

sRef

PARM01

30.0 (g/kg) (ocn) or 0.0 (atm)

1D vertical reference salinity profile (g/kg)

sRefFile

PARM01

' '

filename for reference salinity profile (g/kg)

selectP_inEOS_Zc

PARM01

depends on eosType

select which pressure to use in EOS for $$z$$-coor.

• 0: use $$-g \rho_c z$$

• 1: use $$p_{ref} = -\int{-g\rho(T_{ref},S_{ref},p_{ref})dz}$$

• 2: hydrostatic dynamical pressure

• 3: use full hyd.+non-hyd. pressure

for JMD95P, UNESCO, MDJWF, TEOS10 default=2, otherwise default =0

rhonil

PARM01

9.998E+02

reference density for linear EOS (kg/m3)

tAlpha

PARM01

2.0E-04

linear EOS thermal expansion coefficient (1/oC)

sBeta

PARM01

7.4E-04

linear EOS haline contraction coefficient ((g/kg)-1)

#### 3.8.3.1. Thermodynamic Constants¶

Parameter

Group

Default

Description

HeatCapacity_Cp

PARM01

3.994E+03

specific heat capacity Cp (ocean) (J/kg/K)

celsius2K

PARM01

2.7315E+02

conversion constant oC to Kelvin

atm_Cp

PARM01

1.004E+03

specific heat capacity Cp dry air at const. press. (J/kg/K)

atm_Rd

PARM01

atm_Cp*(2/7)

gas constant for dry air (J/kg/K)

atm_Rq

PARM01

0.0

water vapor specific volume anomaly relative to dry air (g/kg)

atm_Po

PARM01

1.0E+05

atmosphere standard reference pressure (for potential temp. defn.) (Pa)

### 3.8.4. Parameters: Momentum Equations¶

#### 3.8.4.1. Configuration¶

There are a few logical variables that allow you to turn on/off various terms in the momentum equation. These variables are called momViscosity, momAdvection, useCoriolis, momStepping, metricTerms, and momPressureForcing and by default are set to .TRUE.. Vertical diffusive fluxes of momentum can be computed implicitly by setting the logical variable implicitViscosity to .TRUE.. The details relevant to both the momentum flux-form and the vector-invariant form of the equations and the various (momentum) advection schemes are covered in Section 2.

Parameter

Group

Default

Description

momStepping

PARM01

TRUE

momentum equation time-stepping on/off flag

momViscosity

PARM01

TRUE

momentum friction terms on/off flag

PARM01

TRUE

momPressureForcing

PARM01

TRUE

pressure term in momentum equation on/off flag

metricTerms

PARM01

TRUE

include metric terms (spherical polar, momentum flux-form) on/off flag

useNHMTerms

PARM01

FALSE

use “non-hydrostatic form” of metric terms on/off flag; (see Section 2.14.4; note these terms are non-zero in many model configurations beside non-hydrostatic)

PARM01

FALSE

implicitViscosity

PARM01

FALSE

implicit vertical viscosity on/off flag

interViscAr_pCell

PARM04

FALSE

account for partial-cell in interior vertical viscosity on/off flag

momDissip_In_AB

PARM03

TRUE

use Adams-Bashforth time stepping for dissipation tendency

useCoriolis

PARM01

TRUE

include Coriolis terms on/off flag

use3dCoriolis

PARM01

TRUE

include $$\cos{\varphi}$$ Coriolis terms on/off flag

selectCoriScheme

PARM01

0

Coriolis scheme selector

• 0: original scheme

• 1: wet-point averaging method

• 2: Flux-Form: energy conserving; Vector-Inv: hFac weighted average

• 3: Flux-Form: energy conserving using wet-point method; Vector-Inv: energy conserving with hFac weight

vectorInvariantMomentum

PARM01

FALSE

use vector-invariant form of momentum equations flag

PARM01

FALSE

use Jamart wetpoints method for relative vorticity advection (vector invariant form) on/off flag

selectVortScheme

PARM01

1

vorticity scheme (vector invariant form) options

• 0,1: enstrophy conserving forms

• 2: energy conserving form

• 3: energy and enstrophy conserving form

upwindVorticity

PARM01

FALSE

bias interpolation of vorticity in the Coriolis term (vector invariant form) on/off flag

useAbsVorticity

PARM01

FALSE

use $$f + \zeta$$ in Coriolis terms (vector invariant form) on/off flag

highOrderVorticity

PARM01

FALSE

use 3rd/4th order interpolation of vorticity (vector invariant form) on/off flag

upwindShear

PARM01

FALSE

use 1st order upwind for vertical advection (vector invariant form) on/off flag

selectKEscheme

PARM01

0

kinetic energy computation in Bernoulli function (vector invariant form) options

• 0: standard form

• 1: area-weighted standard form

• 2: as 0 but account for partial cells

• 3: as 1 w/partial cells

#### 3.8.4.2. Initialization¶

The initial horizontal velocity components can be specified from binary files uVelInitFile and vVelInitFile. These files should contain 3-D data ordered in an ($$x,y,r$$) fashion with k=1 as the first vertical level (surface level). If no file names are provided, the velocity is initialized to zero. The initial vertical velocity is always derived from the horizontal velocity using the continuity equation. In the case of a restart (from the end of a previous simulation), the velocity field is read from a pickup file (see Section 3.8.7) and the initial velocity files are ignored.

Parameter

Group

Default

Description

uVelInitFile

PARM05

' '

filename for 3D specification of initial zonal velocity field (m/s)

vVelInitFile

PARM05

' '

filename for 3D specification of initial meridional velocity field (m/s)

pSurfInitFile

PARM05

' '

filename for 2D specification of initial free surface position ([$$r$$] unit)

#### 3.8.4.3. General Dissipation Scheme¶

The lateral eddy viscosity coefficient is specified through the variable viscAh (in m2s–1). The vertical eddy viscosity coefficient is specified through the variable viscAr (in [$$r$$]2s–1, where [$$r$$] is the dimension of the vertical coordinate). In addition, biharmonic mixing can be added as well through the variable viscA4 (in m4s–1).

Parameter

Group

Default

Description

viscAh

PARM01

0.0

lateral eddy viscosity (m2/s)

viscAhD

PARM01

viscAh

lateral eddy viscosity acts on divergence part (m2/s)

viscAhZ

PARM01

viscAh

lateral eddy viscosity acts on vorticity part ($$\zeta$$ points) (m2/s)

viscAhW

PARM01

viscAhD

lateral eddy viscosity for mixing vertical momentum (non-hydrostatic form) (m2/s); for non-hydrostatic only, see parameter nonHydrostatic

viscAhDfile

PARM05

' '

filename for 3D specification of lateral eddy viscosity (divergence part) (m2/s); requires #define ALLOW_3D_VISCAH in pkg/mom_common/MOM_COMMON_OPTIONS.h

viscAhZfile

PARM05

' '

filename for 3D specification of lateral eddy viscosity (vorticity part, $$\zeta$$ points); requires #define ALLOW_3D_VISCAH in pkg/mom_common/MOM_COMMON_OPTIONS.h

viscAhGrid

PARM01

0.0

grid-dependent lateral eddy viscosity (non-dim.)

viscAhMax

PARM01

1.0E+21

maximum lateral eddy viscosity (m2/s)

viscAhGridMax

PARM01

1.0E+21

maximum lateral eddy (grid-dependent) viscosity (non-dim.)

viscAhGridMin

PARM01

0.0

minimum lateral eddy (grid-dependent) viscosity (non-dim.)

viscAhReMax

PARM01

0.0

minimum lateral eddy viscosity based on Reynolds number (non-dim.)

viscC2leith

PARM01

0.0

Leith harmonic viscosity factor (vorticity part, $$\zeta$$ points) (non-dim.)

viscC2leithD

PARM01

0.0

Leith harmonic viscosity factor (divergence part) (non-dim.)

viscC2LeithQG

PARM01

0.0

Quasi-geostrophic Leith viscosity factor (non-dim.)

viscC2smag

PARM01

0.0

Smagorinsky harmonic viscosity factor (non-dim.)

viscA4

PARM01

0.0

lateral biharmonic viscosity (m4/s)

viscA4D

PARM01

viscA4

lateral biharmonic viscosity (divergence part) (m4/s)

viscA4Z

PARM01

viscA4

lateral biharmonic viscosity (vorticity part, $$\zeta$$ points) (m4/s)

viscA4W

PARM01

viscA4D

lateral biharmonic viscosity for mixing vertical momentum (non-hydrostatic form) (m4/s); for non-hydrostatic only, see parameter nonHydrostatic

viscA4Dfile

PARM05

' '

filename for 3D specification of lateral biharmonic viscosity (divergence part) (m4/s); requires #define ALLOW_3D_VISCA4 in pkg/mom_common/MOM_COMMON_OPTIONS.h

viscA4Zfile

PARM05

' '

filename for 3D specification of lateral biharmonic viscosity (vorticity part, $$\zeta$$ points); requires #define ALLOW_3D_VISCA4 in pkg/mom_common/MOM_COMMON_OPTIONS.h

viscA4Grid

PARM01

0.0

grid dependent biharmonic viscosity (non-dim.)

viscA4Max

PARM01

1.0E+21

maximum biharmonic viscosity (m4/s)

viscA4GridMax

PARM01

1.0E+21

maximum biharmonic (grid-dependent) viscosity (non-dim.)

viscA4GridMin

PARM01

0.0

minimum biharmonic (grid-dependent) viscosity (mon-dim.)

viscA4ReMax

PARM01

0.0

minimum biharmonic viscosity based on Reynolds number (non-dim.)

viscC4leith

PARM01

0.0

Leith biharmonic viscosity factor (vorticity part, $$\zeta$$ points) (non-dim.)

viscC4leithD

PARM01

0.0

Leith biharmonic viscosity factor (divergence part) (non-dim.)

viscC4smag

PARM01

0.0

Smagorinsky biharmonic viscosity factor (non-dim.)

useFullLeith

PARM01

FALSE

use full form of Leith viscosities on/off flag

useSmag3D

PARM01

FALSE

use isotropic 3D Smagorinsky harmonic viscosities flag; requires #define ALLOW_SMAG_3D in pkg/mom_common/MOM_COMMON_OPTIONS.h

smag3D_coeff

PARM01

1.0E-02

isotropic 3D Smagorinsky coefficient (non-dim.); requires #define ALLOW_SMAG_3D in pkg/mom_common/MOM_COMMON_OPTIONS.h

useStrainTensionVisc

PARM01

FALSE

flag to use strain-tension form of viscous operator

useAreaViscLength

PARM01

FALSE

flag to use area for viscous $$L^2$$ instead of harmonic mean of $${L_x}^2, {L_y}^2$$

viscAr

PARM01

0.0

vertical eddy viscosity ([$$r$$]2/s)

viscArNr

PARM01

0.0

vertical profile of vertical eddy viscosity ([$$r$$]2/s)

pCellMix_viscAr

PARM04

viscArNr

vertical viscosity for too thin partial-cell ([$$r$$]2/s)

#### 3.8.4.4. Sidewall/Bottom Dissipation¶

Slip or no-slip conditions at lateral and bottom boundaries are specified through the logical variables no_slip_sides and no_slip_bottom. If set to .FALSE., free-slip boundary conditions are applied. If no-slip boundary conditions are applied at the bottom, a bottom drag can be applied as well. Two forms are available: linear (set the variable bottomDragLinear in [$$r$$]/s, ) and quadratic (set the variable bottomDragQuadratic, [$$r$$]/m).

Parameter

Group

Default

Description

no_slip_sides

PARM01

TRUE

viscous BCs: no-slip sides on/off flag

sideDragFactor

PARM01

2.0E+00

side-drag scaling factor (2.0: full drag) (non-dim.)

no_slip_bottom

PARM01

TRUE

viscous BCs: no-slip bottom on/off flag

bottomDragLinear

PARM01

0.0

linear bottom-drag coefficient ([$$r$$]/s)

PARM01

0.0

quadratic bottom-drag coefficient ([$$r$$]/m)

PARM01

-1

select quadratic bottom drag discretization option

• -1: not used

• 0: average KE from grid center to $$u,v$$ location

• 1: use local velocity norm @ $$u,v$$ location

• 2: as 1 with wet-point averaging of other velocity component

if bottomDragQuadratic $$\neq 0.$$ then default is 0

selectImplicitDrag

PARM01

0

top/bottom drag implicit treatment options

• 0: fully explicit

• 1: implicit on provisional velocity, i.e., before $$\nabla \eta$$ increment

• 2: fully implicit

if =2, requires #define ALLOW_SOLVE4_PS_AND_DRAG

bottomVisc_pCell

PARM01

FALSE

account for partial-cell in bottom viscosity (using no_slip_bottom = .TRUE.) on/off flag

### 3.8.5. Parameters: Tracer Equations¶

This section covers the tracer equations, i.e., the potential temperature equation and the salinity (for the ocean) or specific humidity (for the atmosphere) equation.

#### 3.8.5.1. Configuration¶

The logical variables tempAdvection, and tempStepping allow you to turn on/off terms in the temperature equation (similarly for salinity or specific humidity with variables saltAdvection etc.). These variables all default to a value of .TRUE.. The vertical diffusive fluxes can be computed implicitly by setting the logical variable implicitDiffusion to .TRUE..

Parameter

Group

Default

Description

tempStepping

PARM01

TRUE

temperature equation time-stepping on/off flag

PARM01

TRUE

PARM01

2

temperature horizontal advection scheme selector (see Table 2.2)

PARM01

temperature vertical advection scheme selector (see Table 2.2)

PARM01

FALSE

temperature implicit vertical advection on/off flag

PARM01

FALSE

include frictional heating in temperature equation on/off flag; requires #define ALLOW_FRICTION_HEATING

temp_stayPositive

PARM01

FALSE

saltStepping

PARM01

TRUE

salinity equation time-stepping on/off flag

PARM01

TRUE

PARM01

2

salinity horizontal advection scheme selector (see Table 2.2)

PARM01

salinity vertical advection scheme selector (see Table 2.2)

PARM01

FALSE

salinity implicit vertical advection on/off flag

salt_stayPositive

PARM01

FALSE

implicitDiffusion

PARM01

FALSE

implicit vertical diffusion on/off flag

interDiffKr_pCell

PARM04

FALSE

account for partial-cell in interior vertical diffusion on/off flag

linFSConserveTr

PARM01

FALSE

correct source/sink of tracer due to use of linear free surface on/off flag

doAB_onGtGs

PARM03

TRUE

apply Adams-Bashforth on tendencies (rather than on T,S) on/off flag

#### 3.8.5.2. Initialization¶

The initial tracer data can be contained in the binary files hydrogThetaFile and hydrogSaltFile. These files should contain 3-D data ordered in an ($$x,y,r$$) fashion with k=1 as the first vertical level. If no file names are provided, the tracers are then initialized with the values of tRef and sRef discussed in Section 3.8.3. In this case, the initial tracer data are uniform in $$x$$ and $$y$$ for each depth level.

Parameter

Group

Default

Description

hydrogThetaFile

PARM05

' '

filename for 3D specification of initial potential temperature (oC)

hydrogSaltFile

PARM05

' '

filename for 3D specification of initial salinity (g/kg)

PARM05

TRUE

apply (center-point) mask to initial hydrographic theta data on/off flag

PARM05

TRUE

apply (center-point) mask to initial hydrographic salinity on/off flag

checkIniTemp

PARM05

TRUE

check if initial theta (at wet-point) identically zero on/off flag

checkIniSalt

PARM05

TRUE

check if initial salinity (at wet-point) identically zero on/off flag

#### 3.8.5.3. Tracer Diffusivities¶

Lateral eddy diffusivities for temperature and salinity/specific humidity are specified through the variables diffKhT and diffKhS (in m2/s). Vertical eddy diffusivities are specified through the variables diffKrT and diffKrS. In addition, biharmonic diffusivities can be specified as well through the coefficients diffK4T and diffK4S (in m4/s). The Gent and McWilliams parameterization for advection and mixing of oceanic tracers is described in Section 8.4.1.

Parameter

Group

Default

Description

diffKhT

PARM01

0.0

Laplacian diffusivity of heat laterally (m2/s)

diffK4T

PARM01

0.0

biharmonic diffusivity of heat laterally (m4/s)

diffKrT

PARM01

0.0

Laplacian diffusivity of heat vertically (m2/s)

diffKr4T

PARM01

0.0

biharmonic diffusivity of heat vertically (m2/s)

diffKrNrT

PARM01

0.0 at k=top

vertical profile of vertical diffusivity of temperature (m2/s)

pCellMix_diffKr

PARM04

diffKrNr

vertical diffusivity for too thin partial-cell ([r]2/s)

diffKhS

PARM01

0.0

Laplacian diffusivity of salt laterally (m2/s)

diffK4S

PARM01

0.0

biharmonic diffusivity of salt laterally (m4/s)

diffKrS

PARM01

0.0

Laplacian diffusivity of salt vertically (m2/s)

diffKr4S

PARM01

0.0

biharmonic diffusivity of salt vertically (m2/s)

diffKrNrS

PARM01

0.0 at k=top

vertical profile of vertical diffusivity of salt (m2/s)

diffKrFile

PARM05

' '

filename for 3D specification of vertical diffusivity (m2/s); requires #define ALLOW_3D_DIFFKR

diffKrBL79surf

PARM01

0.0

surface diffusivity for Bryan & Lewis 1979 [BL79] (m2/s)

diffKrBL79deep

PARM01

0.0

deep diffusivity for Bryan & Lewis 1979 [BL79] (m2/s)

diffKrBL79scl

PARM01

2.0E+02

depth scale for Bryan & Lewis 1979 [BL79] (m)

diffKrBL79Ho

PARM01

-2.0E+03

turning depth for Bryan & Lewis 1979 [BL79] (m)

diffKrBLEQsurf

PARM01

0.0

same as diffKrBL79surf but at equator; requires #define ALLOW_BL79_LAT_VARY

diffKrBLEQdeep

PARM01

0.0

same as diffKrBL79deep but at equator; requires #define ALLOW_BL79_LAT_VARY

diffKrBLEQscl

PARM01

2.0E+02

same as diffKrBL79scl but at equator; requires #define ALLOW_BL79_LAT_VARY

diffKrBLEQHo

PARM01

-2.0E+03

same as diffKrBL79Ho but at equator; requires #define ALLOW_BL79_LAT_VARY

BL79LatVary

PARM01

3.0E+01

transition from diffKrBLEQ to diffKrBL79 parms at this latitude; requires #define ALLOW_BL79_LAT_VARY

#### 3.8.5.4. Ocean Convection¶

In addition to specific packages that parameterize ocean convection, two main model options are available. To use the first option, a convective adjustment scheme, you need to set the variable cadjFreq, the frequency (in seconds) with which the adjustment algorithm is called, to a non-zero value (note, if cadjFreq set to a negative value by the user, the model will set it to the model clock time step). The second option is to parameterize convection with implicit vertical diffusion. To do this, set the logical variable implicitDiffusion to .TRUE. and the real variable ivdc_kappa (in m2/s) to an appropriate tracer vertical diffusivity value for mixing due to static instabilities (typically, several orders of magnitude above the background vertical diffusivity). Note that cadjFreq and ivdc_kappa cannot both have non-zero value.

Parameter

Group

Default

Description

ivdc_kappa

PARM01

0.0

implicit vertical diffusivity for convection (m2/s)

PARM03

0

frequency of convective adj. scheme; <0: sets value to deltaTclock (s)

hMixCriteria

PARM01

-0.8E+00

• <0: specifies $$\Delta T$$ (oC) to define ML depth where $$\Delta\rho = \Delta T*d\rho/dT$$ occurs

• >1: define ML depth where local strat. exceeds mean strat. by this factor (non-dim.)

hMixSmooth

PARM01

0.0

use this fraction of neighboring points (for smoothing) in ML calculation (0-1; 0: no smoothing)

### 3.8.6. Parameters: Model Forcing¶

The forcing options that can be prescribed through runtime parameters in data are easy to use but somewhat limited in scope. More complex forcing setups are possible with optional packages such as pkg/exf or pkg/rbcs, in which case most or all of the parameters in this section can simply be left at their default value.

#### 3.8.6.1. Momentum Forcing¶

This section only applies to the ocean. You need to generate wind-stress data into two files zonalWindFile and meridWindFile corresponding to the zonal and meridional components of the wind stress, respectively (if you want the stress to be along the direction of only one of the model horizontal axes, you only need to generate one file). The format of the files is similar to the bathymetry file. The zonal (meridional) stress data are assumed to be in pascals and located at U-points (V-points). See the MATLAB program gendata.m in the input directories of verification for several tutorial example (e.g. gendata.m in the barotropic gyre tutorial) to see how simple analytical wind forcing data are generated for the case study experiments.

Parameter

Group

Default

Description

momForcing

PARM01

TRUE

included external forcing of momentum on/off flag

zonalWindFile

PARM05

' '

filename for 2D specification of zonal component of wind forcing (N/m2)

meridWindFile

PARM05

' '

filename for 2D specification of meridional component of wind forcing (N/m2)

momForcingOutAB

PARM03

0

1: take momentum forcing out of Adams-Bashforth time stepping

momTidalForcing

PARM01

TRUE

tidal forcing of momentum equation on/off flag (requires tidal forcing files)

PARM05

' '

filename for 2D specification of atmospheric pressure loading (ocean $$z$$-coor. only) (Pa)

#### 3.8.6.2. Tracer Forcing¶

A combination of flux data and relaxation terms can be used for driving the tracer equations. For potential temperature, heat flux data (in W/m2) can be stored in the 2-D binary file surfQnetfile. Alternatively or in addition, the forcing can be specified through a relaxation term. The SST data to which the model surface temperatures are restored are stored in the 2-D binary file thetaClimFile. The corresponding relaxation time scale coefficient is set through the variable tauThetaClimRelax (in seconds). The same procedure applies for salinity with the variable names EmPmRfile, saltClimFile, and tauSaltClimRelax for freshwater flux (in m/s) and surface salinity (in g/kg) data files and relaxation timescale coefficient (in seconds), respectively.

Parameter

Group

Default

Description

tempForcing

PARM01

TRUE

external forcing of temperature forcing on/off flag

surfQnetFile

PARM05

' '

filename for 2D specification of net total heat flux (W/m2)

surfQswFile

PARM05

' '

filename for 2D specification of net shortwave flux (W/m2); requires #define SHORTWAVE_HEATING

tauThetaClimRelax

PARM03

0.0

temperature (surface) relaxation time scale (s)

lambdaThetaFile

PARM05

' '

filename for 2D specification of inverse temperature (surface) relaxation time scale (1/s)

ThetaClimFile

PARM05

' '

filename for specification of (surface) temperature relaxation values (oC)

balanceThetaClimRelax

PARM01

FALSE

subtract global mean heat flux due to temp. relaxation flux every time step on/off flag; requires #define ALLOW_BALANCE_RELAX

balanceQnet

PARM01

FALSE

subtract global mean Qnet every time step on/off flag; requires #define ALLOW_BALANCE_FLUXES

geothermalFile

PARM05

' '

filename for 2D specification of geothermal heating flux through bottom (W/m2); requires #define ALLOW_GEOTHERMAL_FLUX

temp_EvPrRn

PARM01

UNSET

temperature of rain and evaporated water (unset, use local temp.) (oC)

allowFreezing

PARM01

FALSE

limit (ocean) temperature at surface to >= -1.9oC

saltForcing

PARM01

TRUE

external forcing of salinity forcing on/off flag

convertFW2Salt

PARM01

3.5E+01

salinity used to convert freshwater flux to salt flux (-1: use local S) (g/kg) (note default is -1 if useRealFreshWaterFlux= .TRUE.)

rhoConstFresh

PARM01

rhoConst

constant reference density for fresh water (rain) (kg/m3)

EmPmRFile

PARM05

' '

filename for 2D specification of net freshwater flux (m/s)

saltFluxFile

PARM05

' '

filename for 2D specification of salt flux (from seaice) ((g/kg).kg/m2/s)

tauSaltClimRelax

PARM03

0.0

salinity (surface) relaxation time scale (s)

lambdaSaltFile

PARM05

' '

filename for 2D specification of inverse salinity (surface) relaxation time scale (1/s)

saltClimFile

PARM05

' '

filename for specification of (surface) salinity relaxation values (g/kg)

balanceSaltClimRelax

PARM01

FALSE

subtract global mean flux due to salt relaxation every time step on/off flag

selectBalanceEmPmR

PARM01

0

option to balance net surface freshwater flux every time step

• 0: off

• 1: uniform surface correction

• 2: non-uniform surface correction, scaled using wghtBalancedFile for local weighting

if =1 or 2, requires #define ALLOW_BALANCE_FLUXES

wghtBalanceFile

PARM05

' '

filename for 2D specification of weights used in selectBalanceEmPmR =2 correction

salt_EvPrRn

PARM01

0.0

salinity of rain and evaporated water (g/kg)

PARM01

0

add fluid to ocean interior options (-1, 0: off, or 1); requires #define ALLOW_ADDFLUID

PARM01

temp_EvPrRn

PARM01

salt_EvPrRn

PARM05

' '

filename for 3D specification of mass source/sink (+=source, kg/s); requires #define ALLOW_ADDFLUID

balancePrintMean

PARM01

FALSE

print subtracted balancing means to STDOUT on/off flag; requires #define ALLOW_BALANCE_FLUXES and/or #define ALLOW_BALANCE_RELAX

latBandClimRelax

PARM03

whole domain

relaxation to (T,S) climatology equatorward of this latitude band is applied

tracForcingOutAB

PARM03

0

1: take T, S, and pTracer forcing out of Adams-Bashforth time stepping

#### 3.8.6.3. Periodic Forcing¶

To prescribe time-dependent periodic forcing, concatenate successive time records into a single file ordered in a ($$x,y$$,time) fashion and set the following variables: periodicExternalForcing to .TRUE., externForcingPeriod to the period (in seconds between two records in input files) with which the forcing varies (e.g., 1 month), and externForcingCycle to the repeat time (in seconds) of the forcing (e.g., 1 year; note externForcingCycle must be a multiple of externForcingPeriod). With these variables specified, the model will interpolate the forcing linearly at each iteration.

Parameter

Group

Default

Description

periodicExternalForcing

PARM03

FALSE

allow time-dependent periodic forcing on/off flag

externForcingPeriod

PARM03

0.0

period over which forcing varies (e.g. monthly) (s)

externForcingCycle

PARM03

0.0

period over which the forcing cycle repeats (e.g. one year) (s)

### 3.8.7. Parameters: Simulation Controls¶

#### 3.8.7.1. Run Start and Duration¶

The beginning of a simulation is set by specifying a start time (in seconds) through the real variable startTime or by specifying an initial iteration number through the integer variable nIter0. If these variables are set to non-zero values, the model will look for a ”pickup” file (by default, pickup.0000nIter0) to restart the integration. The end of a simulation is set through the real variable endTime (in seconds). Alternatively, one can instead specify the number of time steps to execute through the integer variable nTimeSteps. Iterations are referenced to deltaTClock, i.e., each iteration is deltaTClock seconds of model time.

Parameter

Group

Default

Description

nIter0

PARM03

0

starting timestep iteration number for this integration

nTimeSteps

PARM03

0

number of (model clock) timesteps to execute

nEndIter

PARM03

0

run ending timestep iteration number (alternate way to prescribe nTimeSteps)

startTime

PARM03

baseTime

run start time for this integration (s) (alternate way to prescribe nIter0)

endTime

PARM03

0.0

run ending time (s) (with startTime, alternate way to prescribe nTimeSteps)

#### 3.8.7.2. Input/Output Files¶

The precision with which to read binary data is controlled by the integer variable readBinaryPrec, which can take the value 32 (single precision) or 64 (double precision). Similarly, the precision with which to write binary data is controlled by the integer variable writeBinaryPrec. By default, MITgcm writes output (snapshots, diagnostics, and pickups) separately for individual tiles, leaving it to the user to reassemble these into global files, if needed (scripts are available in utils/). There are two options however to have the model do this for you. Setting globalFiles to .TRUE. should always work in a single process setup (including multi-threaded processes), but for MPI runs this will depend on the platform – it requires simultaneous write access to a common file (permissible in typical Lustre setups, but not on all file systems). Alternatively, one can set useSingleCpuIO to .TRUE. to generate global files, which should always work, but requires additional mpi-passing of data and may result in slower execution.

Parameter

Group

Default

Description

globalFiles

PARM01

FALSE

write output “global” (i.e. not per tile) files on/off flag

useSingleCpuIO

PARM01

FALSE

only master MPI process does I/O (producing global output files)

the_run_name

PARM05

' '

string identifying the name of the model “run” for meta files

PARM01

32

precision used for reading binary files (32 or 64)

writeBinaryPrec

PARM01

32

precision used for writing binary files (32 or 64)

outputTypesInclusive

PARM03

FALSE

allows writing of output files in multiple formats (i.e. pkg/mdsio and pkg/mnc)

rwSuffixType

PARM03

0

controls the format of the pkg/mdsio binary file “suffix”

• 0: use iteration number (myIter, I10.10)

• 1: 100*myTime

• 2: myTime

• 3: myTime/360

• 4: myTime/3600

where myTime is model time in seconds

mdsioLocalDir

PARM05

' '

if not blank, read-write output tiled files from/to this directory name (+four-digit processor-rank code)

#### 3.8.7.3. Frequency/Amount of Output¶

The frequency (in seconds) with which output is written to disk needs to be specified. dumpFreq controls the frequency with which the instantaneous state of the model is written. monitorFreq controls the frequency with which monitor output is dumped to the standard output file(s). The frequency of output is referenced to deltaTClock.

Parameter

Group

Default

Description

dumpFreq

PARM03

0.0

interval to write model state/snapshot data (s)

dumpInitAndLast

PARM03

TRUE

write out initial and last iteration model state on/off flag

diagFreq

PARM03

0.0

interval to write additional intermediate (debugging cg2d/3d) output (s)

monitorFreq

PARM03

lowest of other output *Freq parms

interval to write monitor output (s)

monitorSelect

PARM03

2 (3 if fluid is water)

select group of monitor variables to output

• 1: dynamic variables only

debugLevel

PARM01

depends on debugMode

level of printing of MITgcm activity messages/statistics (1-5, higher -> more activity messages)

plotLevel

PARM01

debugLevel

controls printing of field maps (1-5, higher -> more fields)

#### 3.8.7.4. Restart/Pickup Files¶

chkPtFreq and pchkPtFreq control the output frequency of rolling and permanent pickup (a.k.a. checkpoint) files, respectively. These frequencies are referenced to deltaTClock.

Parameter

Group

Default

Description

pChkPtFreq

PARM03

0.0

permanent restart/pickup checkpoint file write interval ( s )

chkPtFreq

PARM03

0.0

rolling restart/pickup checkpoint file write interval ( s )

pickupSuff

PARM03

' '

force run to use pickups (even if nIter0 =0) and read files with this suffix (10 char. max)

pickupStrictlyMatch

PARM03

TRUE

force pickup (meta) file formats to exactly match (or terminate with error) on/off flag

writePickupAtEnd

PARM03

FALSE

write a (rolling) pickup file at run completion on/off flag

usePickupBeforeC54

PARM01

FALSE

initialize run using old pickup format from code prior to checkpoint54a

startFromPickupAB2

PARM03

FALSE

### 3.8.8. Parameters Used In Optional Packages¶

Some optional packages were not written with package-specific namelist parameters in a data.\${pkg} file; or for historical and/or other reasons, several package-specific namelist parameters remain in data.

#### 3.8.8.1. C-D Scheme¶

(package pkg/cd_code)

If you run at a sufficiently coarse resolution, you might choose to enable the C-D scheme for the computation of the Coriolis terms. The variable tauCD, which represents the C-D scheme coupling timescale (in seconds) needs to be set.

Parameter

Group

Default

Description

useCDscheme

PARM01

FALSE

use C-D scheme for Coriolis terms on/off flag

tauCD

PARM03

deltaTMom

C-D scheme coupling timescale (s)

rCD

PARM03

1 - deltaTMom/tauCD

C-D scheme normalized coupling parameter (non-dim.)

epsAB_CD

PARM03

abEps

Adams-Bashforth-2 stabilizing weight used in C-D scheme

#### 3.8.8.2. Automatic Differentiation¶

(package pkg/autodiff; see Section 7)

Parameter

Group

Default

Description

nTimeSteps_l2

PARM03

4

number of inner timesteps to execute per timestep

PARM03

0.0

interval to write model state/snapshot data adjoint run (s)

PARM03

0.0

interval to write monitor output adjoint run (s)

PARM05

' '

if not blank, read-write checkpointing files from/to this directory name

### 3.8.9. Execution Environment Parameters¶

If running multi-threaded (i.e., using shared memory/OpenMP), you will need to set nTx and/or nTy so that nTx*nTy is the total number of threads (per process).

The parameter useCubedSphereExchange needs to be changed to .TRUE. if you are using any type of grid composed of interconnected individual faces, including the cubed sphere topology or a lat-lon cap grid. See (needs section to be written).

Note that setting flag debugMode to .TRUE. activates a separate set of debugging print statements than parameter debugLevel (see Section 3.8.7.3). The latter controls print statements that monitor model activity (such as opening files, etc.), whereas the former produces a more coding-oriented set of print statements (e.g., entering and exiting subroutines, etc.)

Parameter

Group

Default

Description

useCubedSphereExchange

EEPARMS

FALSE

use cubed-sphere topology domain on/off flag

nTx

EEPARMS

1

number of threads in the $$x$$ direction

nTy

EEPARMS

1

number of threads in the $$y$$ direction

useCoupler

EEPARMS

FALSE

communicate with other model components through a coupler on/off flag

useSETRLSTK

EEPARMS

FALSE

call C routine to set environment stacksize to ‘unlimited’

useSIGREG

EEPARMS

FALSE

enable signal handler to receive signal to terminate run cleanly on/off flag

debugMode

EEPARMS

FALSE

print additional debugging messages; also “flush” STDOUT file unit after each print

printMapIncludesZeros

EEPARMS

FALSE

text map plots of fields should ignore exact zero values on/off flag

maxLengthPrt1D

EEPARMS

65

maximum number of 1D array elements to print to standard output

## 3.9. MITgcm Input Data File Format¶

MITgcm input files for grid-related data (e.g., delXFile), forcing fields (e.g., tauThetaClimRelax), parameter fields (e.g., viscAhZfile), etc. are assumed to be in “flat” or “unblocked” binary format. For historical reasons, MITgcm files use big-endian byte ordering, NOT little-endian which is the more common default for today’s computers. Thus, some care is required to create MITgcm-readable input files.

• Using MATLAB: When writing binary files, MATLAB’s fopen command includes a MACHINEFORMAT option 'b' which instructs MATLAB to read or write using big-endian byte ordering. 2-D arrays should be index-ordered in MATLAB as ($$x$$, $$y$$) and 3-D arrays as ($$x$$, $$y$$, $$z$$); data is ordered from low to high in each index, with $$x$$ varying most rapidly.

An example to create a bathymetry file of single-precision, floating point values (from tutorial Barotropic Ocean Gyre, a simple enclosed, flat-bottom domain) is as follows:

ieee = 'b';           % big-endian format
accuracy = 'float32'; % this is single-precision (='real*4')

Ho=5000;  % ocean depth in meters
nx=62;    % number of gridpoints in x-direction
ny=62;    % number of gridpoints in y-direction

% Flat bottom at z = -Ho
h = -Ho * ones(nx, ny);

% Walls (surrounding domain)
h([1 end], :) = 0;   % set ocean depth to zero at east and west walls
h(:, [1 end]) = 0;   % set ocean depth to zero at south and north walls

% save as single-precision (float32) with big-endian byte ordering
fid = fopen('bathy.bin', 'w', ieee);
fwrite(fid, h, accuracy);
fclose(fid);


To read this bathymetry file back into MATLAB, reshaped back to (nx, ny):

fid = fopen('bathy.bin', 'r', ieee);
h = reshape(fread(fid, Inf, accuracy), nx, ny);
fclose(fid);

• Using Python:

A python version of the above script to create a bathymetry file is as follows:

import numpy as np

Ho = 5000  # ocean depth in meters
nx = 62    # number of gridpoints in x-direction
ny = 62    # number of gridpoints in y-direction

# Flat bottom at z = -Ho
h = -Ho * np.ones((ny, nx))

# Walls (surrounding domain)
h[:, [0,-1]] = 0   # set ocean depth to zero at east and west walls
h[[0,-1], :] = 0   # set ocean depth to zero at south and north walls

# save as single-precision (NumPy type float32) with big-endian byte ordering
h.astype('>f4').tofile('bathy.bin')


The dtype specification '>f4' above instructs Python to write the file with big-endian byte ordering (specifically, due to the ‘>’) as single-precision real numbers (due to the ‘f4’ which is NumPy float32 or equivalently, Fortran real*4 format).

To read this bathymetry file back into Python, reshaped back to (ny, nx):

h = np.fromfile('bathy.bin', '>f4').reshape(ny, nx)


where again the dtype spec instructs Python to read a big-endian file of single-precision, floating point values.

Note that 2-D and 3-D arrays should be index-ordered as ($$y$$, $$x$$) and ($$z$$, $$y$$, $$x$$), respectively, to be written in proper ordering for MITgcm.

A more complicated example of using Python to generate input date is provided in verification/tutorial_baroclinic_gyre/input/gendata.py.

• Using Fortran: To create flat binary files in Fortran, open with syntax OPEN(..., ACCESS='DIRECT', ...) (i.e., NOT ACCESS='SEQUENTIAL' which includes additional metadata). By default Fortran will use the local computer system’s native byte ordering for reading and writing binary files, which for most systems will be little-endian. One therefore has two options: after creating a binary file in Fortran, use MATLAB or Python (or some other utility) to read in and swap the bytes in the process of writing a new file; or, determine if your local Fortran has a compiler flag to control byte-ordering of binary files. Similar to MATLAB, 2-D and 3-D arrays in Fortran should be index-ordered as ($$x$$, $$y$$) and ($$x$$, $$y$$, $$z$$), respectively.

Using NetCDF format for input files is only partially implemented at present in MITgcm, and use is thus discouraged.

Input files are by default single-precision real numbers (32-bit, real*4), but can be switched to double precision by setting namelist parameter readBinaryPrec (PARM01 in file data) to a value of 64.