5. Contributing to the MITgcm

The MITgcm is an open source project that relies on the participation of its users, and we welcome contributions. This chapter sets out how you can contribute to the MITgcm.

5.1. Bugs and feature requests

If you think you’ve found a bug, the first thing to check that you’re using the latest version of the model. If the bug is still in the latest version, then think about how you might fix it and file a ticket in the GitHub issue tracker. Please include as much detail as possible. At a minimum your ticket should include:

  • what the bug does;

  • the location of the bug: file name and line number(s); and

  • any suggestions you have for how it might be fixed.

To request a new feature, or guidance on how to implement it yourself, please open a ticket with the following details:

  • a clear explanation of what the feature will do; and

  • a summary of the equations to be solved.

5.2. Using Git and Github

To contribute to the source code of the model you will need to fork the repository and place a pull request on GitHub. The two following sections describe this process in different levels of detail. If you are unfamiliar with git, you may wish to skip the quickstart guide and use the detailed instructions. All contributions to the source code are expected to conform with the Coding style guide. Contributions to the manual should follow the same procedure and conform with Section 5.6.

5.2.1. Quickstart Guide

0. As a precursor, if you have not done so already, set up ssh keys for GitHub command line authentication (or alteratively, authenticate using a personal access token).

1. Fork the project on GitHub (using the fork button).

2. Create a local clone (we strongly suggest keeping a separate repository for development work). If you are using ssh keys for command line authentication:

% git clone git@github.com:«GITHUB_USERNAME»/MITgcm.git

Alternatively, if you are using a personal access token for authentication:

% git clone https://github.com/«GITHUB_USERNAME»/MITgcm.git

3. Move into your local clone directory (cd MITgcm) and and set up a remote that points to the original:

% git remote add upstream https://github.com/MITgcm/MITgcm.git

4. Make a new branch from upstream/master (name it something appropriate, such as ‘bugfix’ or ‘newfeature’ etc.) and make edits on this branch:

% git fetch upstream
% git checkout -b «YOUR_NEWBRANCH_NAME» upstream/master

5. When edits are done, do all git add’s and git commit’s. In the commit message, make a succinct (<70 char) summary of your changes. If you need more space to describe your changes, you can leave a blank line and type a longer description, or break your commit into multiple smaller commits. Reference any outstanding issues addressed using the syntax #«ISSUE_NUMBER».

6. Push the edited branch to the origin remote (i.e. your fork) on GitHub:

% git push -u origin «YOUR_NEWBRANCH_NAME»

7. On GitHub, go to your fork and hit the compare and pull request (PR) button, provide the requested information about your PR (in particular, a non-trivial change to the model requires a suggested addition to doc/tag-index) and wait for the MITgcm head developers with merge privileges to review your proposed changes. In addition to the MITgcm head developers, a broader group of reviewers and developers will peruse and try to respond to a new PR within a week or two. The reviewers may accept the PR as is, or more typically, may request edits and changes. Occasionally the review team will reject changes that are not sufficiently aligned with and do not fit with the code structure. The review team is always happy to discuss their decision, but want to avoid people investing extensive effort in code that has a fundamental design flaw. As such, we strongly suggest opening an issue on GitHub to discuss any proposed contributions beforehand.

The current pull request discussion and review team is Jean-Michel Campin, Ed Doddridge, Chris Hill, Oliver Jahn, Jon Lauderdale, Martin Losch, Jeff Scott, Timothy Smith, and Ou Wang. Please contact anyone on this team with questions about a proposed pull request.

If you want to update your code branch before submitting a PR (or any point in development), follow the recipe below. It will ensure that your GitHub repo stays up to date with the main repository. Note again that your edits should always be to your development branch, not the master branch.

% git checkout master
% git pull upstream master
% git push origin master
% git checkout «YOUR_NEWBRANCH_NAME»
% git merge master

If you prefer, you can rebase rather than merge in the final step above; just be careful regarding your rebase syntax!

5.2.2. Detailed guide for those less familiar with Git and GitHub

What is Git? Git is a version control software tool used to help coordinate work among the many MITgcm model contributors. Version control is a management system to track changes in code over time, not only facilitating ongoing changes to code, but also as a means to check differences and/or obtain code from any past time in the project history. Without such a tool, keeping track of bug fixes and new features submitted by the global network of MITgcm contributors would be virtually impossible. If you are familiar with the older form of version control used by the MITgcm (CVS), there are many similarities, but we now take advantage of the modern capabilities offered by Git.

Git itself is open source linux software (typically included with any new linux installation, check with your sys-admin if it seems to be missing) that is necessary for tracking changes in files, etc. through your local computer’s terminal session. All Git-related terminal commands are of the form git «arguments». Important functions include syncing or updating your code library, adding files to a collection of files with edits, and commands to “finalize” these changes for sending back to the MITgcm head developers. There are numerous other Git command-line tools to help along the way (see man pages via man git).

The most common git commands are:

  • git clone download (clone) a repository to your local machine

  • git status obtain information about the local git repository

  • git diff highlight differences between the current version of a file and the version from the most recent commit

  • git add stage a file, or changes to a file, so that they are ready for git commit

  • git commit create a commit. A commit is a snapshot of the repository with an associated message that describes the changes.

What is GitHub then? GitHub is a website that has three major purposes: 1) Code Viewer: through your browser, you can view all source code and all changes to such over time; 2) “Pull Requests”: facilitates the process whereby code developers submit changes to the primary MITgcm head developers; 3) the “Cloud”: GitHub functions as a cloud server to store different copies of the code. The utility of #1 is fairly obvious. For #2 and #3, without GitHub, one might envision making a big tarball of edited files and emailing the head developers for inclusion in the main repository. Instead, GitHub effectively does something like this for you in a much more elegant way. Note unlike using (linux terminal command) git, GitHub commands are NOT typed in a terminal, but are typically invoked by hitting a button on the web interface, or clicking on a webpage link etc.

To contribute edits to MITgcm, the first step is to obtain a GitHub account, if you have not done so already; it’s free. Second, as a ‘developer’ you will need to authenticate your terminal command line sessions in GitHub. There are two ways this can be done, either using ssh keys or via a personal access token. A personal access token functions similar to a password; ssh keys require some upfront configuration (generating the key and then adding to your Github account), but most developers prefer the ease of this approach once it is set up.

Before you start working with git, make sure you identify yourself. From your terminal, type:

% git config --global user.email «your_email@example.edu»
% git config --global user.name «‘John Doe’»

(note the required quotes around your name). You should also personalize your profile associated with your GitHub account.

There are many online tutorials to using Git and GitHub (see for example https://akrabat.com/the-beginners-guide-to-contributing-to-a-github-project ); here, we are just communicating the basics necessary to submit code changes to the MITgcm. Spending some time learning the more advanced features of Git will likely pay off in the long run, and not just for MITgcm contributions, as you are likely to encounter it in all sorts of different projects.

To better understand this process, Figure 5.1 shows a conceptual map of the Git setup. Note three copies of the code: the main MITgcm repository sourcecode “upstream” (i.e., owned by the MITgcm head developers) in the GitHub cloud, a copy of the repository “origin” owned by you, also residing in the GitHub cloud, and a local copy on your personal computer or compute cluster (where you intend to compile and run). The Git and GitHub commands to create this setup are explained more fully below.

Conceptual model of GitHub

Figure 5.1 A conceptual map of the GitHub setup. Git terminal commands are shown in red, GitHub commands are shown in green.

One other aspect of Git that requires some explanation to the uninitiated: your local linux copy of the code repository can contain different “branches”, each branch being a different copy of the code repository (this can occur in all git-aware directories). When you switch branches, basic unix commands such as ls or cat will show a different set of files specific to current branch. In other words, Git interacts with your local file system so that edits or newly created files only appear in the current branch, i.e., such changes do not appear in any other branches. So if you swore you made some changes to a particular file, and now it appears those changes have vanished, first check which branch you are on (git status is a useful command here), all is probably not lost. NOTE: for a file to be “assigned” to a specific Git branch, Git must first be “made aware” of the file, which occurs after a git add and git commit (see below). Prior to this, the file will appear in the current folder independently, i.e., regardless of which git branch you are on.

A detailed explanation of steps for contributing MITgcm repository edits:

1. On GitHub, create a local copy of the repository in your GitHub cloud user space: from the main repository (https://github.com/MITgcm/MITgcm) hit the Fork button. As mentioned, your GitHub copy “origin” is necessary to streamline the collaborative development process – you need to create a place for your edits in the GitHub cloud, for developers to peruse. (Note: this step is only necessary the first time you contribute a pull request, as this forked copy will remain permanently in your Github space.)

2. Download the code onto your local computer using the git clone command. Even if you previously downloaded the code through a “git-aware” method (i.e., a git clone command, see Section 3.2.1), we STRONGLY SUGGEST you download a fresh repository, to a separate disk location, for your development work (keeping your research work separate). If you are using ssh keys for command line authentication (see above), in your terminal window type:

% git clone git@github.com:«GITHUB_USERNAME»/MITgcm.git

Alternatively, if you are using a personal access token for authentication:

% git clone https://github.com/«GITHUB_USERNAME»/MITgcm.git

(technically, here you are copying the forked “origin” version from the cloud, not the “upstream” version, but these will be identical at this point).

3. Move into the local clone directory on your computer:

% cd MITgcm

We need to set up a remote that points to the main repository:

% git remote add upstream https://github.com/MITgcm/MITgcm.git

This means that we now have two “remotes” of the project. A remote is just a pointer to a repository not on your computer, i.e., in the GitHub cloud, one pointing to your GitHub user space (“origin”), and this new remote pointing to the original (“upstream”). You can read and write into your “origin” version (since it belongs to you, in the cloud), but not into the “upstream” version. This command just sets up this remote, which is needed in step #4 – no actual file manipulation is done at this point. If in doubt, the command git remote -v will list what remotes have been set up.

4. Next make a new branch.

% git fetch upstream
% git checkout -b «YOUR_NEWBRANCH_NAME» upstream/master

You will make edits on this new branch, to keep these new edits completely separate from all files on the master branch. The first command git fetch upstream makes sure your new branch is the latest code from the main repository; as such, you can redo step 4 at any time to start additional, separate development projects (on a separate, new branch). Note that this second command above not only creates this new branch, from the upstream/master branch, it also switches you onto this newly created branch. Naming the branch something descriptive like ‘newfeature’ or ‘bugfix’ (preferably, be even more descriptive) is helpful.

5. Doing stuff! This usually comes in one of three flavors:

i) cosmetic changes, formatting, documentation, etc.;
ii) fixing bug(s), or any change to the code which results in different numerical output; or
iii) adding a feature or new package.

To do this you should:
  • edit the relevant file(s) and/or create new files. Refer to Coding style guide for details on expected documentation standards and code style requirements. Of course, changes should be thoroughly tested to ensure they compile and run successfully!

  • type git add «FILENAME1» «FILENAME2» ... to stage the file(s) ready for a commit command (note both existing and brand new files need to be added). “Stage” effectively means to notify Git of the the list of files you plan to “commit” for changes into the version tracking system. Note you can change other files and NOT have them sent to model developers; only staged files will be sent. You can repeat this git add command as many times as you like and it will continue to augment the list of files. git diff and git status are useful commands to see what you have done so far.

  • use git commit to commit the files. This is the first step in bundling a collection of files together to be sent off to the MITgcm head developers. When you enter this command, an editor window will pop up. On the top line, type a succinct (<70 character) summary of what these changes accomplished. If your commit is non-trivial and additional explanation is required, leave a blank line and then type a longer description of why the action in this commit was appropriate etc. It is good practice to link with known issues using the syntax #ISSUE_NUMBER in either the summary line or detailed comment. Note that all the changes do not have to be handled in a single commit (i.e. you can git add some files, do a commit, than continue anew by adding different files, do another commit etc.); the git commit command itself does not (yet) submit anything to head developers.

  • if you are fixing a more involved bug or adding a new feature, such that many changes are required, it is preferable to break your contribution into multiple commits (each documented separately) rather than submitting one massive commit; each commit should encompass a single conceptual change to the code base, regardless of how many files it touches. This will allow the MITgcm head developers to more easily understand your proposed changes and will expedite the review process.

When your changes are tested and documented, continue on to step #6, but read all of step #6 and #7 before proceeding; you might want to do an optional “bring my development branch up to date” sequence of steps before step #6.

6. Now we “push” our modified branch with committed changes onto the origin remote in the GitHub cloud. This effectively updates your GitHub cloud copy of the MITgcm repo to reflect the wonderful changes you are contributing.

% git push -u origin «YOUR_NEWBRANCH_NAME»

Some time might elapse during step #5, as you make and test your edits, during which continuing development occurs in the main MITgcm repository. In contrast with some models that opt for static, major releases, the MITgcm is in a constant state of improvement and development. It is very possible that some of your edits occur to files that have also been modified by others. Your local clone however will not know anything about any changes that may have occurred to the MITgcm repo in the cloud, which may cause an issue in step #7 below, when one of three things will occur:

  • the files you have modified in your development have NOT been modified in the main repo during this elapsed time, thus git will have no conflicts in trying to update (i.e. merge) your changes into the main repo.

  • during the elapsed time, the files you have modified have also been edited/updated in the main repo, but you edited different places in these files than those edits to the main repo, such that git is smart enough to be able to merge these edits without conflict.

  • during the elapsed time, the files you have modified have also been edited/updated in the main repo, but git is not smart enough to know how to deal with this conflict (it will notify you of this problem during step #7).

One option is to NOT attempt to bring your development code branch up to date, instead simply proceed with steps #6 and #7 and let the maintainers (i.e., the MITgcm head developers with merge priviledges) assess and resolve any conflict(s), should such occur (there is a checkbox ‘Allow edits by maintainers’ that is checked by default when you do step #7). If very little time elapsed during step #5, such conflict is less likely. However, if step #5 takes on the order of months, we do suggest you follow this recipe below to update the code and merge yourself. And/or during the development process, you might have reasons to bring the latest changes in the main repo into your development branch, and thus might opt to follow these same steps.

Development branch code update recipe:

% git checkout master
% git pull upstream master
% git push origin master
% git checkout «YOUR_NEWBRANCH_NAME»
% git merge master

This first command switches you from your development branch to the master branch. The second command above will synchronize your local master branch with the main MITgcm repository master branch (i.e. “pull” any new changes that might have occurred in the upstream repository into your local clone). Note you should not have made any changes to your clone’s master branch; in other words, prior to the pull, master should be a stagnant copy of the code from the day you performed step #1 above. The git push command does the opposite of pull, so in the third step you are synchronizing your GitHub cloud copy (“origin”) master branch to your local clone’s master branch (which you just updated). Then, switch back to your development branch via the second git checkout command. Finally, the last command will merge any changes into your development branch. If conflicts occur that git cannot resolve, git will provide you a list of the problematic file names, and in these files, areas of conflict will be demarcated. You will need to edit these files at these problem spots (while removing git’s demarcation text), then do a git add «FILENAME» for each of these files, followed by a final git commit to finish off the merger.

Some additional git diff commands to help sort out file changes, in case you want to assess the scope of development changes, are as follows. git diff master upstream/master will show you all differences between your local master branch and the main MITgcm repo, i.e., so you can peruse what parallel MITgcm changes have occurred while you were doing your development (this assumes you have not yet updated your clone’s master branch). You can check for differences on individual files via git diff master upstream/master  «FILENAME». If you want to see all differences in files you have modified during your development, the command is git diff master. Similarly, to see a combined list of both your changes and those occurring to the main repo, git diff upstream/master.

Aside comment: if you are familiar with git, you might realize there is an alternate way to merge, using the “rebase” syntax. If you know what you are doing, feel free to use this command instead of our suggested merge command above.

7. Finally create a “pull request” (a.k.a. “PR”; in other words, you are requesting that the MITgcm head developers with merge privileges pull your changes into the main code repository). In GitHub, go to the fork of the project that you made (https://github.com/«GITHUB_USERNAME»/MITgcm.git). There is a button for “Compare and Pull” in your newly created branch. Click the button! Now you can add a final succinct summary description of what you’ve done in your commit(s), flag up any issues, and respond to the remaining questions on the PR template form. If you have made non-trivial changes to the code or documentation, we will note this in the MITgcm change log, doc/tag-index. Please suggest how to note your changes in doc/tag-index; we will not accept the PR if this field is left blank. The MITgcm PR discussion team (in addition to the MITgcm head developers, a broader group of reviewers) will now be notified and be able to peruse your changes! In general, the PR review team will try to respond to a new PR within a week or two. While the PR remains open, you can go back to step #5 and make additional edits, git adds, git commits, and then redo step #6; such changes will be added to the PR (and head developers re-notified), no need to redo step #7.

Your pull request remains open until either the MITgcm head developers with merge privileges fully accept and merge your code changes into the main repository, or decide to reject your changes. Occasionally, the review team will reject changes that are not sufficiently aligned with and do not fit with the code structure; the review team is always happy to discuss their decision, but want to avoid people investing extensive additional effort in code that has a fundamental design flaw. But much more likely than outright rejection, you will instead be asked to respond to feedback, modify your code changes in some way, and/or clean up your code to better satisfy our style requirements, etc., and the pull request will remain open. In some cases, the head developers might take initiative to make some changes to your pull request (such changes can then be incorporated back into your local branch simply by typing git pull from your branch), but more typically you will be asked to undertake the majority of the necessary changes. Note we strongly suggest opening an issue on GitHub to discuss any proposed contributions beforehand.

It is possible for other users (besides the PR review team) to examine or even download your pull request; see Reviewing pull requests.

The current pull request discussion and review team is Jean-Michel Campin, Ed Doddridge, Chris Hill, Oliver Jahn, Jon Lauderdale, Martin Losch, Jeff Scott, Timothy Smith, and Ou Wang. Please feel free to contact anyone on this team with questions about a proposed pull request.

5.3. Coding style guide

Detailed instructions or link to be added.

5.4. Creating MITgcm packages

Optional parts of code are separated from the MITgcm core driver code and organized into packages. The packaging structure provides a mechanism for maintaining suites of code, specific to particular classes of problem, in a way that is cleanly separated from the generic fluid dynamical engine. An overview of available MITgcm packages is presented in Section 8, as illustrated in Figure 8.1. An overview of how to include and use MITgcm packages in your setup is presented in Section 8.1.1, with specific details on using existing packages spread throughout Section 8, Section 9, and Section 10. This sub-section includes information necessary to create your own package for use with MITgcm.

The MITgcm packaging structure is described below using generic package names ${pkg}. A concrete examples of a package is the code for implementing GM/Redi mixing: this code uses the package names ${PKG} = GMREDI, ${pkg} = gmredi, and ${Pkg} = gmRedi.

5.4.1. Package structure

  • Compile-time state: Given that each package is allowed to be compiled or not (e.g., all ${pkg} listed in packages.conf are compiled, see Section 8.1.1.1), genmake2 keeps track of each package’s compile-state in PACKAGES_CONFIG.h with CPP option ALLOW_${PKG} being defined (#define) or not (#undef). Therefore, in the MITgcm core code (or code from other included packages), calls to package-specific subroutines and package-specific header file #include statements must be protected within #ifdef ALLOW_${PKG} … … #endif /* ALLOW_${PKG} */ (see below) to ensure that the model compiles when this ${pkg} is not compiled.

  • Run-time state: The core driver part of the model can check for a run-time on/off switch of individual package(s) through the Fortran logical flag use${Pkg}. The information is loaded from a global package setup file called data.pkg. Note a use${Pkg} flag is NOT used within the package-local subroutine code (i.e., ${pkg}_«DO_SOMETHING».F package source code).

  • Each package gets its runtime configuration parameters from a file named data.${pkg}. Package runtime configuration options are imported into a common block held in a header file called ${PKG}.h. Note in some packages, the header file ${PKG}.h is split into ${PKG}_PARAMS.h, which contains the package parameters, and ${PKG}_VARS.h for the field arrays. The ${PKG}.h header file(s) can be imported by other packages to check dependencies and requirements from other packages (see Section 5.4.2).

In order for a package’s run-time state use${Pkg} to be set to true (i.e., “on”), the code build must have its compile-time state ALLOW_${PKG} defined (i.e., “included”), else mitgcmuv will terminate (cleanly) during initialization. A package’s run-time state is not permitted to change during a model run.

Every call to a package routine from outside the package requires a check on BOTH compile-time and run-time states:

#include "PACKAGES_CONFIG.h"
#include "CPP_OPTIONS.h"
      .
      .
#ifdef ALLOW_${PKG}
#  include "${PKG}_PARAMS.h"
#endif
      .
      .
      .

#ifdef ALLOW_${PKG}
      IF ( use${Pkg} ) THEN
         .
         .
         CALL ${PKG}_DO_SOMETHING(...)
         .
      ENDIF
#endif

Within an individual package, the header file ${PKG}_OPTIONS.h is used to set CPP flags specific to that package. This header file should include PACKAGES_CONFIG.h and CPP_OPTIONS.h, as shown in this example:

#ifndef ${PKG}_OPTIONS_H
#define ${PKG}_OPTIONS_H
#include "PACKAGES_CONFIG.h"
#include "CPP_OPTIONS.h"

#ifdef ALLOW_${PKG}
      .
      .
      .
#define ${PKG}_SOME_PKG_SPECIFIC_CPP_OPTION
      .
      .
      .
#endif /* ALLOW_${PKG} */
#endif /* ${PKG}_OPTIONS_H */

See for example GMREDI_OPTIONS.h.

5.4.2. Package boot sequence

All packages follow a required “boot” sequence outlined here:

S/R PACKAGES_BOOT()

S/R PACKAGES_READPARMS()
   #ifdef ALLOW_${PKG}
      IF ( use${Pkg} ) CALL ${PKG}_READPARMS( retCode )
   #endif

S/R PACKAGES_INIT_FIXED()
   #ifdef ALLOW_${PKG}
      IF ( use${Pkg} ) CALL ${PKG}_INIT_FIXED( retCode )
   #endif

S/R PACKAGES_CHECK()
   #ifdef ALLOW_${PKG}
      IF ( use${Pkg} ) CALL ${PKG}_CHECK( retCode )
   #else
      IF ( use${Pkg} ) CALL PACKAGES_CHECK_ERROR('${PKG}')
   #endif

S/R PACKAGES_INIT_VARIABLES()
   #ifdef ALLOW_${PKG}
      IF ( use${Pkg} ) CALL ${PKG}_INIT_VARIA( )
   #endif
  • PACKAGES_BOOT()

    determines the logical state of all use${Pkg} variables, as defined in the file data.pkg.

  • ${PKG}_READPARMS()

    is responsible for reading in the package parameters file data.${pkg} and storing the package parameters in ${PKG}.h (or in ${PKG}_PARAMS.h). ${PKG}_READPARMS is called in S/R packages_readparms.F, which in turn is called from S/R initialise_fixed.F.

  • ${PKG}_INIT_FIXED()

    is responsible for completing the internal setup of a package, including adding any package-specific variables available for output in pkg/diagnostics (done in S/R ${PKG}_DIAGNOSTICS_INIT). ${PKG}_INIT_FIXED is called in S/R packages_init_fixed.F, which in turn is called from S/R initialise_fixed.F. Note: some packages instead use CALL ${PKG}_INITIALISE (or the old form CALL ${PKG}_INIT).

  • ${PKG}_CHECK()

    is responsible for validating basic package setup and inter-package dependencies. ${PKG}_CHECK can also import parameters from other packages that it may need to check; this is accomplished through header files ${PKG}.h. (It is assumed that parameters owned by other packages will not be reset during ${PKG}_CHECK !!!) ${PKG}_CHECK is called in S/R packages_check.F, which in turn is called from S/R initialise_fixed.F.

  • ${PKG}_INIT_VARIA()

    is responsible for initialization of all package variables, called after the core model state has been completely initialized but before the core model timestepping starts. This routine calls ${PKG}_READ_PICKUP, where any package variables required to restart the model will be read from a pickup file. ${PKG}_INIT_VARIA is called in packages_init_variables.F, which in turn is called from S/R initialise_varia.F. Note: the name ${PKG}_INIT_VARIA is not yet standardized across all packages; one can find other S/R names such as ${PKG}_INI_VARS or ${PKG}_INIT_VARIABLES or ${PKG}_INIT.

5.4.3. Package S/R calls

Calls to package subroutines within the core code timestepping loop can vary. Below we show an example of calls to do calculations, generate output and dump the package state (for pickup):

S/R DO_OCEANIC_PHYS()
   #ifdef ALLOW_${PKG}
      IF ( use${Pkg} ) CALL ${PKG}_DO_SOMETHING( )
   #endif

S/R DO_THE_MODEL_IO()
   #ifdef ALLOW_${PKG}
      IF ( use${Pkg} ) CALL ${PKG}_OUTPUT( )
   #endif

S/R PACKAGES_WRITE_PICKUP()
   #ifdef ALLOW_${PKG}
      IF ( use${Pkg} ) CALL ${PKG}_WRITE_PICKUP( )
   #endif
  • ${PKG}_DO_SOMETHING()

    refers to any local package source code file, which may be called from any model/src routine (or, from any subroutine in another package). An specific example would be the S/R call gmredi_calc_tensor.F from within the core S/R model/src/do_oceanic_phys.F.

  • ${PKG}_OUTPUT()

    is responsible for writing time-average fields to output files (although the cumulating step is done within other package subroutines). May also call other output routines (e.g., CALL ${PKG}_MONITOR) and write snapshot fields that are held in common blocks. Other temporary fields are directly dumped to file where they are available. Note that pkg/diagnostics output of ${PKG} variables is generated in pkg/diagnostics subroutines. ${PKG}_OUTPUT() is called in S/R do_the_model_io.F NOTE: 1) the S/R ${PKG}_DIAGS is used in some packages but is being replaced by ${PKG}_OUTPUT to avoid confusion with pkg/diagnostics functionality. 2) the output part is not yet in a standard form.

  • ${PKG}_WRITE_PICKUP()

    is responsible for writing a package pickup file, used in packages where such is necessary for a restart. ${PKG}_WRITE_PICKUP is called in packages_write_pickup.F which in turn is called from the_model_main.F.

Note: In general, subroutines in one package (pkgA) that only contains code which is connected to a 2nd package (pkgB) will be named pkgA_pkgB_something.F (e.g., gmredi_diagnostics_init.F).

5.4.4. Package “mypackage”

In order to simply creating the infrastructure required for a new package, we have created pkg/mypackage as essentially an existing package (i.e., all package variables defined, proper boot sequence, output generated) that does not do anything. Thus, we suggest you start with this “blank” package’s code infrastructure and add your new package functionality to it, perusing the existing mypackage routines and editing as necessary, rather than creating a new package from scratch.

5.5. MITgcm code testing protocols

verification directory includes many examples intended for regression testing (some of which are tutorial experiments presented in detail in Section 4). Each one of these test-experiment directories contains “known-good” standard output files (see Section 5.5.2.1) along with all the input (including both code and data files) required for their re-calculation. Also included in verification is the shell script testreport to perform regression tests.

5.5.1. Test-experiment directory content

Each test-experiment directory («TESTDIR», see verification for the full list of choices) contains several standard subdirectories and files which testreport recognizes and uses when running a regression test. The directories and files that testreport uses are different for a forward test and an adjoint test (testreport -adm, see Section 5.5.2) and some test-experiments are set-up for only one type of regression test whereas others allow both types of tests (forward and adjoint). Also some test-experiments allow, using the same MITgcm executable, multiple tests using different parameters and input files, with a primary input set-up (e.g., input/ or input_ad/) and corresponding results (e.g., results/output.txt or results/output_adm.txt) and with one or several secondary inputs (e.g., input.«OTHER»/ or input_ad.«OTHER»/) and corresponding results (e.g., results/output.«OTHER».txt or results/output_adm.«OTHER».txt).

directory «TESTDIR»/code/

Contains the test-experiment specific source code (i.e., files that have been modified from the standard MITgcm repository version) used to build the MITgcm executable (mitgcmuv) for forward-test (using genmake2 -mods=../code).

It can also contain specific source files with the suffix _mpi to be used in place of the corresponding file (without suffix) for an MPI test (see Section 5.5.2). The presence or absence of SIZE.h_mpi determines whether or not an MPI test on this test-experiment is performed or skipped. Note that the original code/SIZE.h_mpi is not directly used as SIZE.h to build an MPI-executable; instead, a local copy build/SIZE.h.mpi is derived from code/SIZE.h_mpi by adjusting the number of processors (nPx, nPy) according to «NUMBER_OF_PROCS» (see Section 5.5.2, testreport -MPI); then it is linked to SIZE.h (ln -s SIZE.h.mpi SIZE.h) before building the MPI-executable.

directory «TESTDIR»/code_ad/

Contains the test-experiment specific source code used to build the MITgcm executable (mitgcmuv_ad) for adjoint-test (using genmake2 -mods=../code_ad). It can also contain specific source files with the suffix _mpi (see above).

directory «TESTDIR»/build/

Directory where testreport will build the MITgcm executable for forward and adjoint tests. It is initially empty except in some cases will contain an experiment specific genmake_local file (see Section 3.5.2).

directory TESTDIR/input/

Contains the input and parameter files used to run the primary forward test of this test-experiment.

It can also contain specific parameter files with the suffix .mpi to be used in place of the corresponding file (without suffix) for MPI tests, or with suffix .mth to be used for multi-threaded tests (see Section 5.5.2). The presence or absence of eedata.mth determines whether or not a multi-threaded test on this test-experiment is performed or skipped, respectively.

To save disk space and reduce downloading time, multiple copies of the same input file are avoided by using a shell script prepare_run. When such a script is found in TESTDIR/input/, testreport runs this script in directory TESTDIR/run/ after linking all the input files from TESTDIR/input/.

directory «TESTDIR»/input_ad/

Contains the input and parameter files used to run the primary adjoint test of this test-experiment. It can also contain specific parameter files with the suffix .mpi and shell script prepare_run as described above.

directory «TESTDIR»/input.«OTHER»/

Contains the input and parameter files used to run the secondary OTHER forward test of this test-experiment. It can also contain specific parameter files with suffix .mpi or .mth and shell script prepare_run (see above).

The presence or absence the file eedata.mth determines whether or not a secondary multi-threaded test on this test-experiment is performed or skipped.

directory «TESTDIR»/input_ad.«OTHER»/

Contains the input and parameter files used to run the secondary OTHER adjoint test of this test-experiment. It can also contain specific parameter files with the suffix .mpi and shell script prepare_run (see above).

directory «TESTDIR»/results/

Contains reference standard output used for test comparison. results/output.txt and results/output_adm.txt, respectively, correspond to primary forward and adjoint test run on the reference platform (currently villon.mit.edu) on one processor (no MPI, single thread) using the reference compiler (currently the GNU Fortran compiler gfortran). The presence of these output files determines whether or not testreport is testing or skipping this test-experiment. Reference standard output for secondary tests (results/output.«OTHER».txt or results/output_adm.«OTHER».txt) are also expected here.

directory «TESTDIR»/run/

Initially empty directory where testreport will run the MITgcm executable for primary forward and adjoint tests.

Symbolic links (using command ln -s) are made for input and parameter files (from ../input/ or from ../input_ad/) and for MITgcm executable (from ../build/) before the run proceeds. The sequence of links (function linkdata within shell script testreport) for a forward test is:

  • link and rename or remove links to special files with suffix .mpi or .mth from ../input/

  • link files from ../input/

  • execute ../input/prepare_run (if it exists)

The sequence for an adjoint test is similar, with ../input_ad/ replacing ../input/.

directory «TESTDIR»/tr_run.«OTHER»/

Directory created by testreport to run the MITgcm executable for secondary “OTHER” forward or adjoint tests.

The sequence of links for a forward secondary test is:

  • link and rename or remove links to special files with suffix .mpi or .mth from ../input.OTHER/

  • link files from ../input.OTHER/

  • execute ../input.OTHER/prepare_run (if it exists)

  • link files from ../input/

  • execute ../input/prepare_run (if it exists)

The sequence for an adjoint test is similar, with ../input_ad.OTHER/ and ../input_ad/ replacing ../input.OTHER/ and ../input/.

5.5.2. The testreport utility

The shell script testreport, which was written to work with genmake2, can be used to build different versions of MITgcm code, run the various examples, and compare the output. On some systems, the testreport script can be run with a command line as simple as:

% cd verification
% ./testreport -optfile ../tools/build_options/linux_amd64_gfortran

The testreport script accepts a number of command-line options which can be listed using the -help option. The most important ones are:

-ieee (default) / -fast

If allowed by the compiler (as defined in the specified optfile), use IEEE arithmetic (genmake2 -ieee). In contrast, -fast uses the optfile default for compiler flags.

-devel

Use optfile development flags (assumes specified in optfile).

-optfile «/PATH/FILENAME» (or -optfile ’«/PATH/F1» «/PATH/F2» ...’)

This specifies a list of “options files” that will be passed to genmake2. If multiple options files are used (for example, to test different compilers or different sets of options for the same compiler), then each options file will be used with each of the test directories.

-tdir «TESTDIR» (or -tdir ’«TDIR1» «TDIR2» ...’)

This option specifies the test directory or list of test directories that should be used. Each of these entries should exactly match (note: they are case sensitive!) the names of directories in verification. If this option is omitted, then all directories that are properly formatted (that is, containing an input subdirectory and a results/output.txt file) will be used.

-skipdir «TESTDIR» (or -skipdir ’«TDIR1» «TDIR2» ...’)

This option specifies a test directory or list of test directories to skip. The default is to test ALL directories in verification.

-MPI «NUMBER_OF_PROCS» (or -mpi)

If the necessary file «TESTDIR»/code/SIZE.h_mpi exists, then use it (and all TESTDIR/code/*_mpi files) for an MPI-enabled run. The option -MPI followed by the maximum number of processors enables to build and run each test-experiment using different numbers of MPI processors (specific number chosen by: multiple of nPx*nPy from «TESTDIR»/code/SIZE.h_mpi and not larger than «NUMBER_OF_PROCS»). The short option (-mpi) can only be used to build and run on 2 MPI processors (equivalent to -MPI 2).

Note that the use of MPI typically requires a special command option (see “-command” below) to invoke the MPI executable.

-command=’«SOME COMMANDS TO RUN»’

For some tests, particularly MPI runs, a specific command might be needed to run the executable. This option allows a more general command (or shell script) to be invoked.

The default here is for «SOME COMMANDS TO RUN» to be replaced by mpirun -np TR_NPROC mitgcmuv. If on your system you require something other than mpirun, you will need to use the option and specify your computer’s syntax. Because the number of MPI processors varies according to each test-experiment, the keyword TR_NPROC will be replaced by its effective value, the actual number of MPI processors needed to run the current test-experiment.

-mth

Compile with genmake2 -omp and run with multiple threads (using eedata.mth).

-adm

Compile and test the adjoint suite of verification runs using TAF.

-clean

Clean out all files/progress from any previously executed testreport runs.

-match «NUMBER»

Set matching criteria to «NUMBER» of significant digits (default is 10 digits).

Additional testreport options are available to pass options to genmake2 (called during testreport execution) as well as additional options to skip specific steps of the testreport shell script. See testreport -help for a detailed list.

In the verification/ directory, the testreport script will create an output directory «tr_NAME_DATE_N», with your computer hostname substituted for NAME, the current date for DATE, followed by a suffix number N to distinguish from previous testreport output directories. Unless you specify otherwise using the -tdir or -skipdir options described above, all sub-directories (i.e., TESTDIR experiments) in verification will be tested. testreport writes progress to the screen (stdout) and reports into the «tr_NAME_DATE_N/TESTDIR» sub-directories as it runs. In particular, one can find, in each TESTDIR subdirectory, a summary.txt file in addition to log and/or error file(s) (depending how the run failed, if this occurred). summary.txt contains information about the run and a comparison of the current output with “reference output” (see below for information on how this reference output is generated). The test comparison involves several output model variables. By default, for a forward test, these are the 2D solver initial residual cg2d_init_res and 3D state variables (T, S, U, V) from pkg/monitor output; by default for an adjoint test, the cost-function and gradient-check. However, some test-experiments use some package-specific variables from pkg/monitor according to the file «TESTDIR»/input[_ad][.«OTHER»]/tr_checklist specification. Note that at this time, the only variables that are compared by testreport are those dumped in standard output via pkg/monitor, not output produced by pkg/diagnostics. Monitor output produced from ALL run time steps are compared to assess significant digit match; the worst match is reported. At the end of the testing process, a composite summary.txt file is generated in the top «tr_NAME_DATE_N» directory as a compact, combined version of the summary.txt files located in all TESTDIR sub-directories (a slightly more condensed version of this information is also written to file tr_out.txt in the top verification/ directory; note this file is overwritten upon subsequent testreport runs). Figure 5.2 shows an excerpt from the composite summary.txt, created by running the full testreport suite (in the example here, on a linux cluster, using gfortran):

output text from summary.txt

Figure 5.2 Example output from testreport summary.txt

The four columns on the left are build/run results (successful=Y, unsuccessful=N). Explanation of these columns is as follows:

  • Gen2: did genmake2 build the makefile for this experiment without error?

  • Dpnd: did the make depend for this experiment complete without error?

  • Make: did the make successfully generate a mitgcmuv executable for this experiment?

  • Run: did execution of this experiment startup and complete successfully?

The next sets of columns shows the number of significant digits matched from the monitor output “cg2d”, “min”, “max”, “mean”, and “s d” (standard deviation) for variables T, S, U, and V (see column headings), as compared with the reference output. NOTE: these column heading labels are for the default list of variables, even if different variables are specified in a tr_checklist file (for reference, the list of actual variables tested for a specific TESTDIR experiment is output near the end of the file summary.txt appearing in the specific TESTDIR experiment directory). For some experiments, additional variables are tested, as shown in “PTR 01”, “PTR 02” sets of columns; testreport will detect if tracers are active in a given experiment and check digit match on their concentration values. A match to near-full machine precision is 15-16 digits; this generally will occur when a similar type of computer, similar operating system, and similar version of Fortran compiler are used for the test. Otherwise, different round-off can occur, and due to the chaotic nature of ocean and climate models, fewer digits (typically, 10-13 digits) are matched. A match of 22 digits generally is due to output being exactly 0.0. In some experiments, some variables may not be used or meaningful, which causes the ‘0’ and ‘4’ match results in several of the adjustment experiments above.

While the significant digit match for many variables is tested and displayed in summary.txt, only one of these is used to assess pass/fail (output to the right of the match test results) – the number bracketed by > and <. For example, see above for experiment advect_cs the pass/fail test occurs on variable “T: s d” (i.e., standard deviation of potential temperature), the first variable in the list specified in verification/advect_cs/input/tr_checklist. By default (i.e., if no file tr_checklist is present), pass/fail is assessed on the cg2d monitor output. See the testreport script for a list of permissible variables to test and a guide to their abbreviations. See tr_checklist files in the input subdirectories of several TESTDIR experiments (e.g., verification/advect_xz/input/tr_checklist) for examples of syntax (note, a + after a variable in a tr_checklist file is shorthand to compare the mean, minimum, maximum, and standard deviation for the variable).

5.5.2.1. Reference Output

Reference output is currently generated using the linux server villon.mit.edu which employs an Intel Haswell processor running Ubuntu 18.04.3 LTS. For each verification experiment in the MITgcm repository, this reference output is stored in the file «TESTDIR»/results/output.txt, which is the standard output generated by running testreport (using a single process) on villon.mit.edu using the gfortran (GNU Fortran) compiler version 7.4.0.

Using a different gfortran version (or a different Fortran compiler entirely), and/or running with MPI, a different operating system, or a different processor (cpu) type will generally result in output that differs to machine precision. The greater the number of such differences between your platform and this reference platform, typically the fewer digits of matching output precision.

5.5.3. The do_tst_2+2 utility

The shell script tools/do_tst_2+2 can be used to check the accuracy of the restart procedure. For each experiment that has been run through testreport, do_tst_2+2 executes three additional short runs using the tools/tst2+2 script. The first run makes use of the pickup files output from the run executed by testreport to restart and run for four time steps, writing pickup files upon completion. The second run is similar except only two time steps are executed, writing pickup files. The third run restarts from the end of the second run, executing two additional time steps, writing pickup files upon completion. In order to successfully pass do_tst_2+2, not only must all three runs execute and complete successfully, but the pickups generated at the end the first run must be identical to the pickup files from the end of the third run. Note that a prerequisite to running do_tst_2+2 is running testreport, both to build the executables used by do_tst_2+2, and to generate the pickup files from which do_tst_2+2 begins execution.

The tools/do_tst_2+2 script should be called from the verification/ directory, e.g.:

% cd verification
% ../tools/do_tst_2+2

The do_tst_2+2 script accepts a number of command-line options which can be listed using the -help option. The most important ones are:

-t «TESTDIR»

Similar to testreport option -tdir, specifies the test directory or list of test directories that should be used. If omitted, the test is attempted in all sub-directories.

-skd «TESTDIR»

Similar to testreport option -skipdir, specifies a test directory or list of test directories to skip.

-mpi

Run the tests using MPI; requires the prerequisite testreport run to have been executed with the -mpi or -MPI «NUMBER_OF_PROCS» flag. No argument is necessary, as the do_tst_2+2 script will determine the correct number of processes to use for your executable.

-clean

Clean up any output generated from the do_tst_2+2. This step is necessary if one wants to do additional testreport runs from these directories.

Upon completion, do_tst_2+2 will generate a file tst_2+2_out.txt in the verification/ directory which summarizes the results. The top half of the file includes information from the composite summary.txt file from the prerequisite testreport run. In the bottom half, new results from each verification experiment are given: each line starts with four Y/N indicators indicating if pickups from the testreport run were available, and whether runs 1, 2 and 3, completely successfully, respectively, followed by a pass or fail from the output pickup file comparison test, followed by the TESTDIR experiment name. In each «TESTDIR»/run subdirectory do_tst_2+2 also creates a log file tst_2+2_out.log which contains additional information. During do_tst_2+2 execution a separate directory of summary information, including log files for all failed tests, is created in an output directory «rs_NAME_DATE_N» similar to the syntax for the testreport output directory name. Note however this directory is deleted by default upon do_tst_2+2 completion, but can be saved by adding the do_tst_2+2 command line option -a NONE.

5.5.4. Daily Testing of MITgcm

On a daily basis, MITgcm runs a full suite of testreport (i.e., forward and adjoint runs, single process, single-threaded and mpi) on an array of different clusters, running using different operating systems, testing several different Fortran compilers. The reference machine villon.mit.edu is one of such daily test machines. When changes in output occur from previous runs, even if as minor as changes in numeric output to machine precision, MITgcm head developers are automatically notified.

Links to summary results from the daily testing are posted at http://mitgcm.org/public/testing.html.

5.5.5. Required Testing for MITgcm Code Contributors

5.5.5.1. Using testreport to check your new code

Before submitting your pull request for approval, if you have made any changes to MITgcm code, however trivial, you MUST complete the following:

  • Run testreport (on all experiments) on an unmodified master branch of MITgcm. We suggest using the -devel option and gfortran (typically installed in most linux environments) although neither is strictly necessary for this test. Depending how different your platform is from our reference machine setup, typically most tests will pass but some match tests may fail; it is possible one or more experiments might not even build or run successfully. But even if there are multiple experiment fails or unsuccessful builds or runs, do not despair, the purpose at this stage is simply to generate a reference report on your local platform using the master code. It may take one or more hours for testreport to complete.

  • Save a copy of this summary output from running testreport on the mastrer branch: from the verification directory, type cp tr_out.txt tr_out_master.txt. The file tr_out.txt is simply a condensed version of the composite summary.txt file located in the «tr_NAME_DATE_N» directory. Note we are not making this file “git-aware”, as we have no desire to check this into the repo, so we are using an old-fashioned copy to save the output here for later comparison.

  • Switch to your pull request branch, and repeat the testreport sequence using the same options.

  • From the verification directory, type diff tr_out_master.txt tr_out.txt which will report any differences in testreport output from the above tests. If no differences occur (other than timestamp-related), see below if you are required to do a do_tst_2+2 test; otherwise, you are clear for submitting your pull request.

Differences might occur due to one or more of the following reasons:

  • Your modified code no longer builds properly in one or more experiments. This is likely due to a Fortran syntax error; examine output and log files in the failed experiment TESTDIR to identify and fix the problem.

  • The run in the modified code branch terminates due to a numerical exception error. This too requires further investigation into the cause of the error, and a remedy, before the pull request should be submitted.

  • You have made changes which require changes to input parameters (e.g., renaming a namelist parameter, changing the units or function of an input parameter, etc.) This by definition is a “breaking change”, which must be noted when completing the PR template – but should not deter you from submitting your PR. Ultimately, you and the MITgcm head developers will likely have to make changes to one or more verification experiments, but as a first step we will want to review your PR.

  • You have made algorithmic changes which change model output in some or all setups; this too is a “breaking change” that should be noted in the PR template. As usual recourse, if the PR is accepted, the head developers will re-generate reference output and push to the affected «TESTDIR»/results/ directories when the PR is merged.

Most typically, running testreport using a single process is a sufficient test. However, any code changes which call MITgcm routines (such as eesupp/src/global_sum.F) employing low-level MPI-directives should run testreport with the -mpi option enabled.

5.5.5.2. Using do_tst_2+2 to check your new code

If you make any kind of algorithmic change to the code, or modify anything related to generating or reading pickup files, you are also required to also complete a do_tst_2+2. Again, run the test on both the unmodified master branch and your pull request branch (after you have run testreport on both branches). Verify that the output tst_2+2_out.txt file is identical between branches, similar to the above procedure for the file tr_out.txt. If the files differ, attempt to identify and fix what is causing the problem.

5.5.5.3. Automatic testing with GitHub Actions

Once your PR is submitted onto GitHub, the continuous integration service GitHub Actions runs additional tests on your PR submission. On the ‘Pull request’ tab in GitHub (https://github.com/MITgcm/MITgcm/pulls), find your pull request; initially you will see a yellow circle to the right of your PR title, indicating testing in progress. Eventually this will change to a green checkmark (pass) or a red X (fail). If you get a red X, click the X and then click on ‘Details’ to list specifics tests that failed; these can be clicked to produce a screenshot with error messages.

Note that GitHub Actions builds documentation (both html and latex) in addition to code testing, so if you have introduced syntax errors into the documentation files, these will be flagged at this stage. Follow the same procedure as above to identify the error messages so the problem(s) can be fixed. Make any appropriate edits to your pull request, re-git add and re-git commit any newly modified files, re-git push. Anytime changes are pushed to the PR, GitHub Actions will re-run its tests.

The MITgcm head developers will not review your PR until all GitHub Actions tests pass.

5.6. Contributing to the manual

Whether you are simply correcting typos or describing undocumented packages, we welcome all contributions to the manual. The following information will help you make sure that your contribution is consistent with the style of the MITgcm documentation. (We know that not all of the current documentation follows these guidelines - we’re working on it)

The manual is written in rst format, which is short for ReStructuredText directives. rst offers many wonderful features: it automatically does much of the formatting for you, it is reasonably well documented on the web (e.g., primers available here and here), it can accept raw latex syntax and track equation labelling for you, in addition to numerous other useful features. On the down side however, it can be very fussy about formatting, requiring exact spacing and indenting, and seemingly innocuous things such as blank spaces at ends of lines can wreak havoc. We suggest looking at the existing rst files in the manual to see exactly how something is formatted, along with the syntax guidelines specified in this section, prior to writing and formatting your own manual text.

The manual can be viewed either of two ways: interactively (i.e., web-based), as hosted by read-the-docs (https://readthedocs.org/), requiring an html format build, or downloaded as a pdf file. When you have completed your documentation edits, you should double check both versions are to your satisfaction, particularly noting that figure sizing and placement may be rendered differently in the pdf build. See Building the manual for detail.

5.6.1. Section headings

  • Chapter headings - these are the main headings with integer numbers - underlined with ****

  • section headings - headings with number format X.Y - underlined with ====

  • Subsection headings - headings with number format X.Y.Z - underlined with ----

  • Subsubsection headings - headings with number format X.Y.Z.A - underlined with ~~~~

  • Paragraph headings - headings with no numbers - underlined with ^^^^

N.B. all underlinings should be the same length as the heading. If they are too short an error will be produced.

5.6.2. Internal document references

rst allows internal referencing of figures, tables, section headings, and equations, i.e. clickable links that bring the reader to the respective figure etc. in the manual. To be referenced, a unique label is required. To reference figures, tables, or section headings by number, the rst (inline) directive is :numref:`«LABELNAME»`. For example, this syntax would write out Figure XX on a line (assuming «LABELNAME» referred to a figure), and when clicked, would relocate your position in the manual to figure XX. Section headings can also be referenced so that the name is written out instead of the section number, instead using this directive :ref:`«LABELNAME»`.

Equation references have a slightly different inline syntax: :eq:`«LABELNAME»` will produce a clickable equation number reference, surrounded by parentheses.

For instructions how to assign a label to tables and figures, see below. To label a section heading, labels go above the section heading they refer to, with the format .. _«LABELNAME»:. Note the necessary leading underscore. You can also place a clickable link to any spot in the text (e.g., mid-section), using this same syntax to make the label, using the syntax :ref:`«SOME TEXT TO CLICK ON» <«LABELNAME»>` for the link.

5.6.3. Citations

In the text, references should be given using the standard “Author(s) (Year)” shorthand followed by a link to the full reference in the manual bibliography. This link is accomplished using the syntax :cite:`«BIB_REFERENCE»`; this will produce clickable text, usually some variation on the authors’ initials or names, surrounded by brackets.

Full references are specified in the file doc/manual_references.bib using standard BibTeX format. Even if unfamiliar with BibTeX, it is relatively easy to add a new reference by simply examining other entries. Furthermore, most publishers provide a means to download BibTex formatted references directly from their website. Note this file is in approximate alphabetic order by author name. For all new references added to the manual, please include a DOI or a URL in addition to journal name, volume and other standard reference infomation. An example JGR journal article reference is reproduced below; note the «BIB_REFERENCE» here is “bryan:79” so the syntax in the rst file format would be “Bryan and Lewis (1979) :cite:`bryan:79`, which will appear in the manual as Bryan and Lewis (1979) [BL79].

@Article{bryan:79,
author = {Bryan, K. and L.J. Lewis},
title = {A water mass model of the world ocean},
journal = jgr,
volume = 84,
number = {C5},
pages = {2503–2517},
doi = {10.1029/JC084iC05p02503},
year = 1979,
}

5.6.5. Symbolic Notation

Inline math is done with :math:`«LATEX_HERE»`

Separate equations, which will be typeset on their own lines, are produced with:

.. math::
   «LATEX_HERE»
   :label: «EQN_LABEL_HERE»

Labelled separate equations are assigned an equation number, which may be referenced elsewhere in the document (see Section 5.6.2). Omitting the :label: above will still produce an equation on its own line, except without an equation label. Note that using latex formatting \begin{aligned}\end{aligned} across multiple lines of equations will not work in conjunction with unique equation labels for each separate line (any embedded formatting & characters will cause errors too). Latex alignment will work however if you assign a single label for the multiple lines of equations.

There is a software tool ‘universal document converter’ named pandoc that we have found helpful in converting raw latex documents into rst format. To convert a .tex file into .rst, from a terminal window type:

% pandoc -f latex -t rst -o «OUTPUT_FILENAME».rst «INPUT_FILENAME».tex

Additional conversion options are available, for example if you have your equations or text in another format; see the pandoc documentation.

Note however we have found that a fair amount of clean-up is still required after conversion, particularly regarding latex equations/labels (pandoc has the unfortunate tendency to add extra spaces, sometimes confusing the rst :math: directive, other times creating issues with indentation).

5.6.6. Figures

The syntax to insert a figure is as follows:

.. figure:: «PATHNAME/FILENAME».*
   :width: 80%
   :align: center
   :alt: «TEXT DESCRIPTION OF FIGURE HERE»
   :name: «MY_FIGURE_NAME»

   The figure caption goes here as a single line of text.

figure::: The figure file is located in subdirectory pathname above; in practice, we have located figure files in subdirectories figs off each manual chapter subdirectory. The wild-card * is used here so that different file formats can be used in the build process. For vector graphic images, save a pdf for the pdf build plus a svg file for the html build. For bitmapped images, gif, png, or jpeg formats can be used for both builds, no wild-card necessary, just substitute the actual extension (see here for more info on compatible formats). [Note: A repository for figure source .eps needs to be created]

:width:: used to scale the size of the figure, here specified as 80% scaling factor (check sizing in both the pdf and html builds, as you may need to adjust the figure size within the pdf file independently).

:align:: can be right, center, or left.

:name: use this name when you refer to the figure in the text, i.e. :numref:`«MY_FIGURE_NAME»`.

Note the indentation and line spacing employed above.

5.6.7. Tables

There are two syntaxes for tables in reStructuredText. Grid tables are more flexible but cumbersome to create. Simple tables are easy to create but limited (no row spans, etc.). The raw rst syntax is shown first, then the output.

Grid Table Example:

+------------+------------+-----------+
| Header 1   | Header 2   | Header 3  |
+============+============+===========+
| body row 1 | column 2   | column 3  |
+------------+------------+-----------+
| body row 2 | Cells may span columns.|
+------------+------------+-----------+
| body row 3 | Cells may  | - Cells   |
+------------+ span rows. | - contain |
| body row 4 |            | - blocks. |
+------------+------------+-----------+

Header 1

Header 2

Header 3

body row 1

column 2

column 3

body row 2

Cells may span columns.

body row 3

Cells may span rows.

  • Cells

  • contain

  • blocks.

body row 4

Simple Table Example:

=====  =====  ======
   Inputs     Output
------------  ------
  A      B    A or B
=====  =====  ======
False  False  False
True   False  True
False  True   True
True   True   True
=====  =====  ======

Inputs

Output

A

B

A or B

False

False

False

True

False

True

False

True

True

True

True

True

Note that the spacing of your tables in your .rst file(s) will not match the generated output; rather, when you build the final output, the rst builder (Sphinx) will determine how wide the columns need to be and space them appropriately.

5.6.8. Other text blocks

Conventionally, we have used the rst ‘inline literal’ syntax around any literal computer text (commands, labels, literal computer syntax etc.) Surrounding text with double back-quotes `` results in output html like this.

To set several lines apart in an whitespace box, e.g. useful for showing lines in from a terminal session, rst uses :: to set off a ‘literal block’. For example:

::

    % unix_command_foo
    % unix_command_fum

(note the :: would not appear in the output html or pdf) A splashier way to outline a block, including a box label, is to employ what is termed in rst as an ‘admonition block’. In the manual these are used to show calling trees and for describing subroutine inputs and outputs. An example of a subroutine input/output block is as follows:

This is an admonition block showing subroutine in/out syntax

.. admonition:: SUBROUTINE_NAME
:class: note

| \(var1\) : VAR1 ( WHERE_VAR1_DEFINED.h)
| \(var2\) : VAR1 ( WHERE_VAR2_DEFINED.h )
| \(var3\) : VAR1 ( WHERE_VAR3_DEFINED.h )

An example of a subroutine in/out admonition box in the documentation is here.

An example of a calling tree in the documentation is here.

To show text from a separate file (e.g., to show lines of code, show comments from a Fortran file, show a parameter file etc.), use the literalinclude directive. Example usage is shown here:

.. literalinclude:: «FILE_TO_SHOW»
    :start-at: String indicating where to start grabbing text
    :end-at: String indicating where to stop grabbing text

Unlike the :filelink: and :varlink: directives, which assume a file path starting at the top of the MITgcm repository, one must specify the path relative to the current directory of the file (for example, from the doc directory, it would require ../../ at the start of the file path to specify the base directory of the MITgcm repository). Note one can instead use :start-after: and :end-before: to get text from the file between (not including) those lines. If one omits the start-at or start-after, etc. options the whole file is shown. More details for this directive can be found here. Example usage in this documentation is here, where the lines to generate this are:

.. literalinclude:: ../../model/src/the_model_main.F
    :start-at: C Invocation from WRAPPER level...
    :end-at: C    |                 :: events.

5.6.9. Other style conventions

Units should be typeset in normal text, with a space between a numeric value and the unit, and exponents added with the :sup: command.

9.8 m/s\ :sup:`2`

will produce 9.8 m/s2. If the exponent is negative use two dashes -- to make the minus sign sufficiently long. The backslash removes the space between the unit and the exponent. Similarly, for subscripts the command is :sub:.

Alternatively, latex :math: directives (see above) may also be used to display units, using the \text{} syntax to display non-italic characters.

Line length: as recommended in the sphinx style guide, lines of raw rst text should be kept to fewer than 80 characters (this restriction does not apply to tables, URLs, etc. where a line break might cause difficulty).

  • Todo: determine how to break up sections into smaller files

5.6.10. Building the manual

Once you’ve made your changes to the manual, you should build it locally to verify that it works as expected. To do this you will need a working python installation with the following packages installed:

  • sphinx

  • sphinxcontrib-bibtex

  • sphinxcontrib-programoutput

  • sphinx_rtd_theme

  • numpy

These packages can be installed from the Python Package Index using pip. If you have an existing python installation using Anaconda or one of its variants (e.g., miniconda), we recommend that you can create (and use) a clean environment with the required packages like this:

cd MITgcm
conda create --name mitgcm_build_the_docs --channel conda-forge --file doc/requirements.txt
conda activate mitgcm_build_the_docs
[...] # do the work
conda deactivate

If you don’t yet have a python installation on your computer, we recommend following the Anaconda installation procedure, then following the recipe above. You do not need to learn python to build the manual; just note you should type conda activate mitgcm_build_the_docs in a shell when starting up a manual editing session, and conda deactivate when you finish (also note you only need to perform the conda create ... step above when you initially follow the recipe). This will maintain a clean, separate python virtual environment for manual compilation and won’t interfere with your python setup should you decide to learn python in the future.

Once these modules are installed you can build the html version of the manual by running make html in the doc directory.

To build the pdf version of the manual you will also need a working version of LaTeX that includes several packages that are not always found in minimal LaTeX installations. The command to build the pdf version is make latexpdf, which should also be run in the doc directory.

5.7. Reviewing pull requests

The only people with write access to the main repository are a small number of core MITgcm developers. They are the people that will eventually merge your pull requests. However, before your PR gets merged, it will undergo the automated testing on Travis-CI, and it will be assessed by the MITgcm community.

Everyone can review and comment on pull requests. Even if you are not one of the core developers you can still comment on a pull request.

The simplest way to examine a pull request is to use GitHub. You can look at changes made to files (GitHub will show you a standard linux diff for each file changed), read though commit messages, and/or peruse any comments the MITgcm community has made regarding this pull request.

If you are reviewing changes to the documentation, most likely you will also want to review the rendered manual in html format. While this is not available at GitHub, you can view html builds based on the pull request documentation using this link at readthedocs.org. Here you will need to click on the appropriate pull request (as labeled by the pull request number), then click on “View docs” (not the green button near the top of the page, but the text in the middle of the page on the right side).

Finally, if you want to test pull requests locally (i.e., to compile or run the code), you should download the pull request branch. You can do this either by cloning the branch from the pull request. If you are using ssh keys for command line authentication:

git clone -b «THEIR_DEVELOPMENT_BRANCHNAME» git@github.com:«THEIR_GITHUB_USERNAME»/MITgcm.git

If you are using a personal access token for authentication:

git clone -b «THEIR_DEVELOPMENT_BRANCHNAME» https://github.com/«THEIR_GITHUB_USERNAME»/MITgcm.git

where «THEIR_GITHUB_USERNAME» is replaced by the username of the person proposing the pull request, and «THEIR_DEVELOPMENT_BRANCHNAME» is the branch from their pull request.

Alternatively, you can add the repository of the user proposing the pull request as a remote to your existing local repository. Navigate to your local repository and type

git remote add «THEIR_GITHUB_USERNAME» https://github.com/«THEIR_GITHUB_USERNAME»/MITgcm.git

where «THEIR_GITHUB_USERNAME» is replaced by the user name of the person who has made the pull request. Then download their pull request changes

git fetch «THEIR_GITHUB_USERNAME»

and switch to the desired branch

git checkout --track «THEIR_GITHUB_USERNAME»/«THEIR_DEVELOPMENT_BRANCHNAME»

You now have a local copy of the code from the pull request and can run tests locally. If you have write access to the main repository you can push fixes or changes directly to the pull request.

None of these steps, apart from pushing fixes back to the pull request, require write access to either the main repository or the repository of the person proposing the pull request. This means that anyone can review pull requests. However, unless you are one of the core developers you won’t be able to directly push changes. You will instead have to make a comment describing any problems you find.