Welcome to MITgcm’s user manual

Authors

Alistair Adcroft, Jean-Michel Campin, Ed Doddridge, Stephanie Dutkiewicz, Constantinos Evangelinos, David Ferreira, Mick Follows, Gael Forget, Baylor Fox-Kemper, Daniel Goldberg, Patrick Heimbach, Chris Hill, Ed Hill, Helen Hill, Oliver Jahn, Jody Klymak, Martin Losch, John Marshall, Guillaume Maze, Matt Mazloff, Dimitris Menemenlis, Andrea Molod, Jeff Scott, and Ariane Verdy

Contents:

• 1. Overview
  • 1.1. Introduction
  • 1.2. Illustrations of the model in action
    • 1.2.1. Global atmosphere: ‘Held-Suarez’ benchmark
    • 1.2.2. Ocean gyres
    • 1.2.3. Global ocean circulation
    • 1.2.4. Convection and mixing over topography
    • 1.2.5. Boundary forced internal waves
    • 1.2.6. Parameter sensitivity using the adjoint of MITgcm
    • 1.2.7. Global state estimation of the ocean
    • 1.2.8. Ocean biogeochemical cycles
    • 1.2.9. Simulations of laboratory experiments
  • 1.3. Continuous equations in ‘r’ coordinates
    • 1.3.1. Kinematic Boundary conditions
    • 1.3.2. Atmosphere
    • 1.3.3. Ocean
    • 1.3.4. Hydrostatic, Quasi-hydrostatic, Quasi-nonhydrostatic and Non-hydrostatic forms
    • 1.3.5. Solution strategy
    • 1.3.6. Finding the pressure field
    • 1.3.7. Forcing/dissipation
    • 1.3.8. Vector invariant form
    • 1.3.9. Adjoint
  • 1.4. Appendix ATMOSPHERE
    • 1.4.1. Hydrostatic Primitive Equations for the Atmosphere in Pressure Coordinates
  • 1.5. Appendix OCEAN
    • 1.5.1. Equations of Motion for the Ocean
  • 1.6. Appendix OPERATORS
    • 1.6.1. Coordinate systems
• 2. Discretization and Algorithm
  • 2.1. Notation
  • 2.2. Time-stepping
  • 2.3. Pressure method with rigid-lid
  • 2.4. Pressure method with implicit linear free-surface
  • 2.5. Explicit time-stepping: Adams-Bashforth
    • 2.5.1. Adams-Bashforth II
    • 2.5.2. Adams-Bashforth III
  • 2.6. Implicit time-stepping: backward method
  • 2.7. Synchronous time-stepping: variables co-located in time
  • 2.8. Staggered baroclinic time-stepping
  • 2.9. Non-hydrostatic formulation
  • 2.10. Variants on the Free Surface
    • 2.10.1. Crank-Nicolson barotropic time stepping
    • 2.10.2. Non-linear free-surface
  • 2.11. Spatial discretization of the dynamical equations
    • 2.11.1. The finite volume method: finite volumes versus finite difference
    • 2.11.2. C grid staggering of variables
    • 2.11.3. Grid initialization and data
    • 2.11.4. Horizontal grid
    • 2.11.5. Vertical grid
    • 2.11.6. Topography: partially filled cells
  • 2.12. Continuity and horizontal pressure gradient term
  • 2.13. Hydrostatic balance
  • 2.14. Flux-form momentum equations
    • 2.14.1. Advection of momentum
    • 2.14.2. Coriolis terms
    • 2.14.3. Curvature metric terms
    • 2.14.4. Non-hydrostatic metric terms
    • 2.14.5. Lateral dissipation
    • 2.14.6. Vertical dissipation
    • 2.14.7. Derivation of discrete energy conservation
    • 2.14.8. Mom Diagnostics
  • 2.15. Vector invariant momentum equations
    • 2.15.1. Relative vorticity
    • 2.15.2. Kinetic energy
    • 2.15.3. Coriolis terms
    • 2.15.4. Shear terms
    • 2.15.5. Gradient of Bernoulli function
    • 2.15.6. Horizontal divergence
    • 2.15.7. Horizontal dissipation
    • 2.15.8. Vertical dissipation
  • 2.16. Tracer equations
    • 2.16.1. Time-stepping of tracers: ABII
  • 2.17. Advection schemes
    • 2.17.1. Linear advection schemes
    • 2.17.2. Non-linear advection schemes
    • 2.17.3. Comparison of advection schemes
  • 2.18. Shapiro Filter
    • 2.18.1. SHAP Diagnostics
  • 2.19. Nonlinear Viscosities for Large Eddy Simulation
    • 2.19.1. Eddy Viscosity
    • 2.19.2. Mercator, Nondimensional Equations
• 3. Getting Started with MITgcm
  • 3.1. Where to find information
  • 3.2. Obtaining the code
    • 3.2.1. Method 1
    • 3.2.2. Method 2
  • 3.3. Updating the code
  • 3.4. Model and directory structure
  • 3.5. Building the model
    • 3.5.1. Quickstart Guide
    • 3.5.2. Generating a Makefile using genmake2
    • 3.5.3. make commands
    • 3.5.4. Building with MPI
    • 3.5.5. Building with OpenMP
  • 3.6. Running the model
    • 3.6.1. Running with MPI
    • 3.6.2. Running with OpenMP
    • 3.6.3. Output files
    • 3.6.4. Looking at the output
  • 3.7. Customizing the Model Configuration - Code Parameters and Compilation Options
    • 3.7.1. Model Array Dimensions
    • 3.7.2. C Preprocessor Options
    • 3.7.3. Preprocessor Execution Environment Options
  • 3.8. Customizing the Model Configuration - Runtime Parameters
    • 3.8.1. Parameters: Configuration, Computational Domain, Geometry, and Time-Discretization
    • 3.8.2. Parameters: Main Algorithmic Parameters
    • 3.8.3. Parameters: Equation of State
    • 3.8.4. Parameters: Momentum Equations
    • 3.8.5. Parameters: Tracer Equations
    • 3.8.6. Parameters: Model Forcing
    • 3.8.7. Parameters: Simulation Controls
    • 3.8.8. Parameters Used In Optional Packages
    • 3.8.9. Execution Environment Parameters
  • 3.9. MITgcm Input Data File Format
• 4. MITgcm Tutorial Example Experiments
  • 4.1. Barotropic Ocean Gyre
    • 4.1.1. Equations Solved
    • 4.1.2. Discrete Numerical Configuration
    • 4.1.3. Configuration
    • 4.1.4. Building and running the model
    • 4.1.5. Model Solution
  • 4.2. Baroclinic Ocean Gyre
    • 4.2.1. Equations solved
    • 4.2.2. Discrete Numerical Configuration
    • 4.2.3. Configuration
    • 4.2.4. Building and running the model
    • 4.2.5. Running with MPI
    • 4.2.6. Running with OpenMP
    • 4.2.7. Model solution
  • 4.3. Southern Ocean Reentrant Channel Example
    • 4.3.1. Equations Solved
    • 4.3.2. Discrete Numerical Configuration
    • 4.3.3. Configuration
    • 4.3.4. Building and running the model
    • 4.3.5. Model Solution
  • 4.4. Ocean Gyre Advection Schemes
    • 4.4.1. Advection and tracer transport
    • 4.4.2. Introducing a tracer into the flow
    • 4.4.3. Selecting an advection scheme
    • 4.4.4. Comparison of different advection schemes
  • 4.5. Global Ocean Simulation
    • 4.5.1. Overview
    • 4.5.2. Discrete Numerical Configuration
    • 4.5.3. Experiment Configuration
  • 4.6. Global Ocean Simulation in Pressure Coordinates
    • 4.6.1. Overview
    • 4.6.2. Discrete Numerical Configuration
    • 4.6.3. Experiment Configuration
  • 4.7. Held-Suarez Atmosphere
    • 4.7.1. Overview
    • 4.7.2. Forcing
    • 4.7.3. Set-up description
    • 4.7.4. Experiment Configuration
  • 4.8. Deep Convection
    • 4.8.1. Overview
    • 4.8.2. Equations solved
    • 4.8.3. Discrete numerical configuration
    • 4.8.4. Numerical stability criteria and other considerations
    • 4.8.5. Experiment configuration
  • 4.9. Gravity Plume On a Continental Slope
    • 4.9.1. Configuration
    • 4.9.2. Binary input data
    • 4.9.3. Code configuration
    • 4.9.4. Model parameters
  • 4.10. Biogeochemistry Simulation
    • 4.10.1. Overview
    • 4.10.2. Equations Solved
    • 4.10.3. Code configuration
    • 4.10.4. Running the example
  • 4.11. Global Ocean State Estimation
    • 4.11.1. Overview
    • 4.11.2. Implementation of the control variable and the cost function
    • 4.11.3. Code Configuration
    • 4.11.4. Compiling
    • 4.11.5. Running the estimation
  • 4.12. Adjoint Sensitivity Analysis for Tracer Injection
    • 4.12.1. Overview of the experiment
    • 4.12.2. Code configuration
    • 4.12.3. Compiling the model and its adjoint
  • 4.13. Offline Experiments
    • 4.13.1. Overview
    • 4.13.2. Time-stepping of tracers
    • 4.13.3. Code Configuration
    • 4.13.4. Running the Experiment
    • 4.13.5. A more complicated example
  • 4.14. Rotating Tank
    • 4.14.1. Equations Solved
    • 4.14.2. Discrete Numerical Configuration
    • 4.14.3. Code Configuration
  • 4.15. Additional Example Experiments: Forward Model Setups
  • 4.16. Additional Example Experiments: Adjoint Model Setups
• 5. Contributing to the MITgcm
  • 5.1. Bugs and feature requests
  • 5.2. Using Git and Github
    • 5.2.1. Quickstart Guide
    • 5.2.2. Detailed guide for those less familiar with Git and GitHub
  • 5.3. Coding style guide
  • 5.4. Creating MITgcm packages
    • 5.4.1. Package structure
    • 5.4.2. Package boot sequence
    • 5.4.3. Package S/R calls
    • 5.4.4. Package “mypackage”
  • 5.5. MITgcm code testing protocols
    • 5.5.1. Test-experiment directory content
    • 5.5.2. The testreport utility
    • 5.5.3. The do_tst_2+2 utility
    • 5.5.4. Daily Testing of MITgcm
    • 5.5.5. Required Testing for MITgcm Code Contributors
  • 5.6. Contributing to the manual
    • 5.6.1. Section headings
    • 5.6.2. Internal document references
    • 5.6.3. Citations
    • 5.6.4. Other embedded links
    • 5.6.5. Symbolic Notation
    • 5.6.6. Figures
    • 5.6.7. Tables
    • 5.6.8. Other text blocks
    • 5.6.9. Other style conventions
    • 5.6.10. Building the manual
  • 5.7. Reviewing pull requests
• 6. Software Architecture
  • 6.1. Overall architectural goals
  • 6.2. WRAPPER
    • 6.2.1. Target hardware
    • 6.2.2. Supporting hardware neutrality
    • 6.2.3. WRAPPER machine model
    • 6.2.4. Machine model parallelism
    • 6.2.5. Communication mechanisms
    • 6.2.6. Communication primitives
    • 6.2.7. Memory architecture
    • 6.2.8. Summary
  • 6.3. Using the WRAPPER
    • 6.3.1. Specifying a domain decomposition
    • 6.3.2. Starting the code
    • 6.3.3. Controlling communication
  • 6.4. MITgcm execution under WRAPPER
    • 6.4.1. Annotated call tree for MITgcm and WRAPPER
    • 6.4.2. Measuring and Characterizing Performance
    • 6.4.3. Estimating Resource Requirements
• 7. Automatic Differentiation
  • 7.1. Some basic algebra
    • 7.1.1. Forward or direct sensitivity
    • 7.1.2. Reverse or adjoint sensitivity
    • 7.1.3. Storing vs. recomputation in reverse mode
  • 7.2. TLM and ADM generation in general
    • 7.2.1. General setup
    • 7.2.2. Building the AD code using TAF
    • 7.2.3. The AD build process in detail
    • 7.2.4. The cost function (dependent variable)
    • 7.2.5. The control variables (independent variables)
  • 7.3. The gradient check package
    • 7.3.1. Code description
    • 7.3.2. Code configuration
  • 7.4. Adjoint dump & restart – divided adjoint (DIVA)
    • 7.4.1. Introduction
    • 7.4.2. Recipe for divided adjoint code generation
    • 7.4.3. Special considerations for multi processor (MPI) runs
  • 7.5. Adjoint code generation using OpenAD
    • 7.5.1. Introduction
    • 7.5.2. Downloading and installing OpenAD
    • 7.5.3. Building MITgcm adjoint with OpenAD
    • 7.5.4. Building the MITgcm adjoint using an OpenAD Singularity container
  • 7.6. Adjoint code generation using Tapenade
    • 7.6.1. Introduction
    • 7.6.2. Downloading and installing Tapenade
    • 7.6.3. Prerequisites for Linux or Mac OS
    • 7.6.4. Steps for Mac OS
    • 7.6.5. Steps for Linux
    • 7.6.6. Prerequisites for Windows
    • 7.6.7. Steps for Windows
    • 7.6.8. Prerequisites for Tapenade setup
    • 7.6.9. Building MITgcm TLM with Tapenade
    • 7.6.10. Building MITgcm adjoint with Tapenade
• 8. Packages I - Physical Parameterizations
  • 8.1. Overview
    • 8.1.1. Using MITgcm Packages
  • 8.2. Packages Related to Hydrodynamical Kernel
    • 8.2.1. Generic Advection/Diffusion
    • 8.2.2. Momentum Packages
    • 8.2.3. Shapiro Filter
    • 8.2.4. FFT Filtering Code
    • 8.2.5. exch2: Extended Cubed Sphere Topology
    • 8.2.6. Gridalt - Alternate Grid Package
  • 8.3. General purpose numerical infrastructure packages
    • 8.3.1. OBCS: Open boundary conditions for regional modeling
    • 8.3.2. RBCS Package
    • 8.3.3. PTRACERS Package
  • 8.4. Ocean Packages
    • 8.4.1. GMREDI: Gent-McWilliams/Redi Eddy Parameterization
    • 8.4.2. KPP: Nonlocal K-Profile Parameterization for Vertical Mixing
    • 8.4.3. GGL90: a TKE vertical mixing scheme
    • 8.4.4. OPPS: Ocean Penetrative Plume Scheme
    • 8.4.5. KL10: Vertical Mixing Due to Breaking Internal Waves
    • 8.4.6. BULK_FORCE: Bulk Formula Package
    • 8.4.7. EXF: The external forcing package
    • 8.4.8. CAL: The calendar package
  • 8.5. Atmosphere Packages
    • 8.5.1. Atmospheric Intermediate Physics: AIM
    • 8.5.2. Land package
    • 8.5.3. Fizhi: High-end Atmospheric Physics
  • 8.6. Ice and Sea Ice Packages
    • 8.6.1. THSICE: The Thermodynamic Sea Ice Package
    • 8.6.2. SEAICE Package
    • 8.6.3. SHELFICE Package
    • 8.6.4. SHELFICE Remeshing
    • 8.6.5. STREAMICE Package
  • 8.7. Biogeochemistry Packages
    • 8.7.1. GCHEM Package
    • 8.7.2. DIC Package
• 9. Packages II - Diagnostics and I/O
  • 9.1. pkg/diagnostics – A Flexible Infrastructure
    • 9.1.1. Introduction
    • 9.1.2. Equations
    • 9.1.3. Key Subroutines and Parameters
    • 9.1.4. Usage Notes
  • 9.2. Fortran Native I/O: pkg/mdsio and pkg/rw
    • 9.2.1. pkg/mdsio
    • 9.2.2. pkg/rw basic binary I/O utilities
  • 9.3. NetCDF I/O: pkg/mnc
    • 9.3.1. Using pkg/mnc
    • 9.3.2. pkg/mnc Troubleshooting
    • 9.3.3. pkg/mnc Internals
  • 9.4. Monitor: Simulation State Monitoring Toolkit
    • 9.4.1. Introduction
    • 9.4.2. Using pkg/monitor
  • 9.5. Grid Generation
    • 9.5.1. Using SPGrid
    • 9.5.2. Example Grids
  • 9.6. Pre– and Post–Processing Scripts and Utilities
    • 9.6.1. Utilities Supplied With the Model
    • 9.6.2. Pre-Processing Software
  • 9.7. Potential Vorticity Matlab Toolbox
    • 9.7.1. Introduction
    • 9.7.2. Equations
    • 9.7.3. Key routines
    • 9.7.4. Technical details
    • 9.7.5. Notes on the flux form of the PV equation and vertical PV fluxes
  • 9.8. pkg/flt – Simulation of float / parcel displacements
    • 9.8.1. Introduction
    • 9.8.2. Compile-time options in FLT_OPTIONS.h
    • 9.8.3. Compile-time parameters in FLT_SIZE.h include:
    • 9.8.4. Run-time options in data.flt include:
    • 9.8.5. Input Files
    • 9.8.6. Output Files
    • 9.8.7. Verification Experiment
    • 9.8.8. Algorithm details
• 10. Packages III - Ocean State Estimation
  • 10.1. ECCO: model-data comparisons using gridded data sets
    • 10.1.1. Generic Cost Function
    • 10.1.2. Generic Integral Function
    • 10.1.3. Custom Cost Functions
    • 10.1.4. Key Routines
    • 10.1.5. Compile Options
  • 10.2. PROFILES: model-data comparisons at observed locations
  • 10.3. OBSFIT: grid-independent model-data comparisons
    • 10.3.1. Introduction
    • 10.3.2. Description
    • 10.3.3. OBSFIT configuration and compiling
    • 10.3.4. Run-time requirements
    • 10.3.5. General flags and parameters
    • 10.3.6. Experiments and tutorials that use OBSFIT
  • 10.4. CTRL: Model Parameter Adjustment Capability
    • 10.4.1. Generic Control Parameters
    • 10.4.2. Shelfice Control Parameters
    • 10.4.3. Logarithmic Control Parameters
  • 10.5. SMOOTH: Smoothing And Covariance Model
  • 10.6. The line search optimisation algorithm
    • 10.6.1. General features
    • 10.6.2. The online vs. offline version
    • 10.6.3. Number of iterations vs. number of simulations
    • 10.6.4. Alternative code to optim and lsopt
  • 10.7. Test Cases For Estimation Package Capabilities
• 11. Utilities
  • 11.1. MITgcmutils
    • 11.1.1. mds
    • 11.1.2. mnc
    • 11.1.3. diagnostics
    • 11.1.4. ptracers
    • 11.1.5. density
    • 11.1.6. miscellaneous utilities
    • 11.1.7. conversion
    • 11.1.8. cs
    • 11.1.9. llc
    • 11.1.10. examples
    • 11.1.11. gluemncbig
  • 11.2. Bash scripts
    • 11.2.1. gluemnc
• 12. Related Projects and Highlighted Papers
  • 12.1. Projects Related to MITgcm
    • 12.1.1. Estimating the Circulation and Climate of the Ocean (ECCO)
    • 12.1.2. Gcmfaces: Gridded Earth Variables In Matlab And Octave
    • 12.1.3. MITprof: In-Situ Ocean Data In Matlab And Octave
    • 12.1.4. OceanParcels - Lagrangian Particle Tracker
    • 12.1.5. Southern Ocean State Estimation (SOSE)
    • 12.1.6. Xgcm: General Circulation Model Postprocessing with xarray
    • 12.1.7. Xmitgcm
  • 12.2. Highlighted Papers
• 13. References