NEMO-3.6 Run Description File

Warning

Versions 2.1 and later of the SalishSeaCmd package introduced changes in the run description file structure that are not backward compatible. Older run description files for NEMO-3.6 runs need to be updated before they can be successfully used with the salishsea run command. Please see SalishSeaCmd Changes That Break Backward Compatibility for details.

Example Run Description YAML File

Example (from SS-run-sets/SalishSea/nemo3.6/SalishSea.yaml):

# salishsea command processor run description example for NEMO-3.6

# Name of the NEMO-3.6 configuration to use for the run;
# i.e. your NEMO-3.6-code/NEMOGCM/CONFIG/ configuration directory
config_name: SalishSea

# How is the domain to be distributed over the processors in the
# i and j grid dimensions?
# These values are used to set the nammpp namelist jpni & jpnj values
# and to set the number of processors and nodes in the PBS script
MPI decomposition: 8x18

# For runs on systems that use qsub/PBS/TORQUE or sbatch/SLURM
#
# The run_id value appears in the list of jobs display by the showq command
run_id: example
# Maximum run time requested/allowed for job
# Too low and you job will be terminated before it finishes
# Too high and you will have to wait longer on the queue for your job to start
# You have to experiment to find the "just right" value
walltime: 10:00:00
# Email address to send job begin, end, and abort notifications to
# If missing, email address is set to ${USER}@eos.ubc.ca
email: you@example.com

# For runs on systems that use sbatch/SLURM
#
# **Optional**
# Account name to charge resources used by the job to
# If missing, account defaults to rrg-allen on cedar and def-allen on
# other systems
account: def-allen

paths:
  # Absolute path to NEMO-3.6 CONFIG/ directory.
  # You can use ~, $USER, $HOME if you wish.
  NEMO code config: $HOME/MEOPAR/NEMO-3.6-code/NEMOGCM/CONFIG
  # If relative, paths below are taken from current directory
  # You can use ~, $USER, $HOME if you wish.
  XIOS: $HOME/MEOPAR/XIOS/
  # Directory in which to create temporary run directories
  # Typically NOT inside a version control repository
  runs directory: $HOME/MEOPAR/SalishSea/

grid:
  # If relative, paths are taken from the grid/ directory in the forcing
  # path above
  coordinates: $HOME/MEOPAR/grid/coordinates_seagrid_SalishSea201702.nc
  bathymetry: $HOME/MEOPAR/grid/bathymetry_201702.nc
  # Optional path/filename for land processor elimination MPI-LPE mapping
  # file that matches bathymetry;
  # If "land processor elimination:" key is absent or has the value "False",
  # land processor elimination is disabled
  land processor elimination: $HOME/MEOPAR/grid/bathymetry_201702.csv

forcing:
  # The keys below are the names of symlinks that will be created.
  # The targets of those symlinks will be the paths named by the associated
  # "link to:" keys;
  # e.g. a symlink named NEMO-atmos will be created to
  # /results/forcing/atmospheric/GEM2.5/operational/
  #
  # The keys are directory names that you use as "cn_dir" values in your
  # namelists.
  #
  # You only need to include keys that are used in the namelist(s) for
  # your run.
  #
  # If relative, paths are taken from forcing path above
  NEMO-atmos:
    link to: /results/forcing/atmospheric/GEM2.5/operational/
    check link:
      type: atmospheric
      namelist filename: namelist_cfg
  rivers:
    link to: $HOME/MEOPAR/rivers
  tides:
    link to: $HOME/MEOPAR/tides
  tracers:
    link to: $HOME/MEOPAR/tracers

restart:
  # The keys below are the names of symlinks that will be created.
  # The targets of those symlinks will be the paths/filenames associated
  # with the keys;
  # e.g. a symlink named restart.nc will be created to
  # /results/SalishSea/nowcast/05may15/SalishSea_02056320_restart.nc
  #
  # You only need to include keys for the types of restart files
  # that are used in your run.
  #
  # If relative, paths are taken from current directory
  restart.nc: /results/SalishSea/nowcast/05may15/SalishSea_02056320_restart.nc
  restart_trc.nc: /results/SalishSea/nowcast-green/06dec15/SalishSea_00004320_restart_trc.nc

namelists:
  # The namelist section files in the lists below will be concatenated
  # to create a namelist file whose name is the key under which the files
  # are listed. The keys are the names of the namelist files that NEMO-3.6
  # expects.
  #
  # The only required key is namelist_cfg.
  #
  # If relative, paths are taken from current directory
  namelist_cfg:
    - namelist.time
    - namelist.domain
    - namelist.surface
    - namelist.lateral
    - namelist.bottom
    - namelist.tracer
    - namelist.dynamics
    - namelist.vertical
    - namelist.compute
  namelist_top_cfg:
    - namelist_top_cfg
  namelist_pisces_cfg:
    - namelist_pisces_cfg

output:
  # If relative, paths are taken from current directory
  iodefs: iodef.xml
  filedefs: file_def.xml
  domaindefs: ../domain_def.xml
  fielddefs: field_def.xml
  separate XIOS server: True
  XIOS servers: 1

vcs revisions:
  git:
    - $HOME/MEOPAR/grid
    - $HOME/MEOPAR/NEMO-Cmd
    - $HOME/MEOPAR/rivers-climatology
    - $HOME/MEOPAR/SalishSeaCmd
    - $HOME/MEOPAR/SS-run-sets
    - $HOME/MEOPAR/tides
    - $HOME/MEOPAR/tracers
    - $HOME/MEOPAR/XIOS-ARCH

Basic Run Configuration

The following key-value pairs provide the basic configuration for the run:

config_name

The name of the NEMO configuration to use for runs. It is the name of a directory in NEMOGCM/CONFIG/ in the NEMO-3.6-code code directory given by the NEMO key in the paths Section.

MPI decomposition

Specify how the domain is to be distributed over the processors in the i (longitude) and j (latitude) directions; e.g. 8x18. Those values are used to set the namelist.compute nammpp namelist jpni and jpnj values, to set the number of processors and nodes in the SalishSeaNEMO.sh script that is generated by the salishsea run command, and to tell the REBUILD_NEMO tool how many files to process.

run_id

The job identifier that appears in the qstat listing.

walltime

The wall-clock time requested for the run. It limits the time that the job will run for on all machines, and it also affects queue priority for runs on Westgrid machines. It is important to allow some buffer time when calculating your walltime limits to allow for indeterminacy of the NEMO run itself, as well as the time required to combine the per-processor results files into run results files at the end of the run.

email

The email address at which you want to receive notification of the beginning and end of execution of the run, as well as notification of abnormal abort messages. The email key is only required if the address is different than would be constructed by combining your user id on the machine that the job runs on with @eos.ubc.ca.

account

Optional. The account name to include in the #SBATCH directives section of the SalishSeaNEMO.sh job script. This key-value pair is required on systems like cedar.computecanada.ca and graham.computecanada.ca that use the Slurm Workload Manager and that you submit runs to with the sbatch command.

paths Section

The paths section of the run description file is a collection of directory paths that salishsea uses to find files in other repos that it needs.

NEMO code config

The path to the CONFIG/ directory in the NEMO-3.6-code clone where the NEMO configuration directories are to be found; e.g. $HOME/MEOPAR/NEMO-3.6-code/NEMOGCM/CONFIG/.

An absolute path is required because the path is used in both the current directory and the temporary run directory created in the runs directory. The path may contain ~ or $HOME as alternative spellings of the user’s home directory, and $USER as an alternative spelling of the user’s userid.

This key may also be spelled NEMO-code-config for backward compatibility.

XIOS

The path to the XIOS-1.0 (XML I/O Server) clone where the XIOS executable for the run is to be found.

This path may be either absolute or relative. The path may contain ~ or $HOME as alternative spellings of the user’s home directory, and $USER as an alternative spelling of the user’s userid.

runs directory

The path to the directory where run directories will be created by the salishsea run (or salishsea prepare) sub-command.

This path may be relative or absolute, and may contain ~ or $HOME as alternative spellings of the user’s home directory, and $USER as an alternative spelling of the user’s userid.

forcing

Optional path to the directory where the repository clones containing the netCDF files for the grid coordinates, bathymetry, initial conditions, open boundary conditions, etc. are to be found.

This path is used as the base for relative paths in the grid Section and forcing Section sections below.

This path may be relative or absolute, and may contain ~ or $HOME as alternative spellings of the user’s home directory, and $USER as an alternative spelling of the user’s userid.

grid Section

The grid section of the run description file contains 3 key-value pairs that provide the paths/filenames of the grid files to use for the run:

  • coordinates

  • bathymetry

  • land processor elimination configuration

An example grid section:

grid:
  coordinates: $HOME/MEOPAR/grid/coordinates_seagrid_SalishSea201702.nc
  bathymetry: $HOME/MEOPAR/grid/bathymetry_201702.nc
  land processor elimination: $HOME/MEOPAR/grid/bathymetry_201702.csv

Relative or absolute paths may contain ~ or $HOME as alternative spellings of the user’s home directory, and $USER as an alternative spelling of the user’s userid.

If simple file names are provided, those files are presumed to be in the grid/ sub-directory of the directory tree pointed to by the forcing key in the paths Section.

If relative paths are given, they are appended to the grid/ directory of the forcing path.

coordinates

The name of, or path to, the coordinates file to use for the run. It is symlinked in the run directory as coordinates.nc (the file name required by NEMO).

bathymetry

The name of, or path to, the bathymetry file to use for the run. It is symlinked in the run directory as bathy_meter.nc (the file name required by NEMO).

land processor elimination

The name of, or path to, the MPI-LPE mapping CSV file that corresponds to the bathymetry to use for the run. That file contains the information necessary to configure NEMO-3.6 to perform calculations only on the MPI subdomains that contain water. Please see Land Processor Elimination for a detailed explanation, and use the Preferred MPI LPE Decompositions table to guide your choice of value for MPI decomposition. The SalishSeaCast NEMO Command Processor takes care of setting the correct number of processors based on your chosen MPI decomposition.

A value of False disables Land Processor Elimination

If the land processor elimination key is missing, a warning message is printed to let you know that the run is proceeding on the assumption that you want to run without land processor elimination.

This key may also be spelled Land processor elimination for backward compatibility.

forcing Section

The forcing section of the run description file contains sub-sections that provide the names of directories and files that are to be symlinked in the run directory for NEMO to use to read initial conditions and forcing values from.

An example forcing section:

forcing:
  NEMO-atmos:
    link to: /results/forcing/atmospheric/GEM2.5/operational/
  rivers:
    link to: $HOME/MEOPAR/rivers
  tides:
    link to: $HOME/MEOPAR/tides
  tracers:
    link to: $HOME/MEOPAR/tracers

The sub-section keys (NEMO-atmos, rivers, tides, and tracers above) are the names of the symlinks that will be created in the run directory. Those names are expected to appear in the appropriate places in the namelists. The values associated with the link to keys are the targets of the symlinks that will be created in the run directory.

The paths may be relative or absolute, and may contain ~ or $HOME as alternative spellings of the user’s home directory, and $USER as an alternative spelling of the user’s userid.

If relative paths are given, they are appended to the forcing path given in the paths Section.

To provide links to directories containing initial fields for a run to use, include a sub-section like:

forcing:
  ...
  initial_strat:
    link to: initial_strat/
  initial_green:
    link to: initial_green/

You are free to use any keys that you wish with the understanding that the key will be the name of the symlink that will be created in the run directory, and that name will also need to appear as a directory name in the appropriate namelist.

The salishsea run and salishsea prepare commands and the salishsea_cmd.api.prepare() API function confirm that the targets of the symlinks exist, and exit with an error message if not.

Atmospheric Forcing File Checks

Additional checking can be performed on the files in the atmospheric forcing directory. That checking confirms the existence of all of the atmospheric forcing files for the date range of the run. Doing so ensures that a run won’t fail part way through due to a missing atmospheric forcing file. To enable the additional checking add a check link section at the same level as the link to key:

forcing:
  NEMO-atmos:
    link to: /results/forcing/atmospheric/GEM2.5/operational/
    check link:
      type: atmospheric
      namelist filename: namelist_cfg

The type key provides the type of checking to perform on the link. The value associated with the namelist filename key is the name of the namelist file in which the atmospheric forcing link is used.

Link checking can be disabled by excluding the check link section, or by setting the value associated with the type key to None.

restart Section

The optional restart section of the run description file contains key-value pairs that provide paths and file names of restart files to be used to initialize the run.

The paths may be relative or absolute, and may contain ~ or $HOME as alternative spellings of the user’s home directory, and $USER as an alternative spelling of the user’s userid.

If relative paths are given, they are appended to the directory containing the run description file.

Here is an example restart section:

restart:
  restart.nc: /results/SalishSea/nowcast/SalishSea_00475200_restart.nc
  restart_trc.nc: /results/SalishSea/nowcast-green/06dec15/SalishSea_00004320_restart_trc.nc

NEMO requires that the name of the model restart file be restart.nc, so that is the key that you must use. For an (optional) tracers restart file the required file name (key) is restart_trc.nc.

The restart section is optional because it is not required for runs that are initialized with fields provided in a directory linked in the forcing Section.

The salishsea run and salishsea prepare commands and the salishsea_cmd.api.prepare() API function confirm that the targets of the symlinks exist, and exit with an error message if not.

namelists Section

The namelists section of the run description file contains one or more dicts of lists of NEMO namelist section files that will be concatenated to construct namelist*_cfg files (the file names required by NEMO) file for the run.

The paths may be relative or absolute, and may contain ~ or $HOME as alternative spellings of the user’s home directory, and $USER as an alternative spelling of the user’s userid.

If relative paths are given, they are appended to the directory containing the run description file.

Here is an example namelist section:

namelists:
  namelist_cfg:
    - namelist.time
    - namelist.domain
    - namelist.surface
    - namelist.lateral
    - namelist.bottom
    - namelist.tracer
    - namelist.dynamics
    - namelist.vertical
    - namelist.compute
  namelist_top_cfg:
    - namelist_top_cfg
  namelist_pisces_cfg:
    - namelist_pisces_cfg

A namelist_cfg key must be present, other namelist*_cfg keys are optional. Each namelist*_cfg section must be a list containing at least 1 namelist section file.

Namelist sections that are specific to the run such as namelist.time where the starting and ending timesteps and the restart configuration are defined are typically stored in the same directory as the run description file. That mean that they are simply listed by name in the appropriate namelist*_cfg section:

namelists:
  namelist_cfg:
    - namelist.time

On the other hand, when you want to use a namelist section that contains the group’s current consensus best values, list it as a relative or absolute path from the location of your run description file to the “standard” nameslist section files in SS-run-sets/SalishSea/nemo3.6/:

namelists:
  namelist_cfg:
    - ../../nemo3.6/namelist.bottom

For each namelist*_cfg key a NEMOGCM/CONFIG/config_name/EXP00/namelist*_ref file is symlinked into the run directory to provide default values that will be used for any namelist variables not included in the namelist section files listed in the namelists section. config_name is the value of the config name key in the run description file.

So, NEMOGCM/CONFIG/config_name/EXP00/namelist_ref will always be symlinked and, if the namelist_top_cfg key is present, the NEMOGCM/CONFIG/config_name/EXP00/namelist_top_ref file will also be symlinked into the run directory.

You can override the use of *_ref namelists from CONFIG/config_name/EXP00/ by including a *_ref namelist key. For example:

config name: SMELT

...

namelists:
  namelist_ref:
    - $HOME/MEOPAR/test-sponge/namelist_ref

will cause the namelist_ref file in the $HOME/MEOPAR/test-sponge/namelist_ref directory to be symlinked into the temporary run directory instead of CONFIG/SMELT/EXP00/namelist_ref.

output Section

The output section of the run description file contains key-value pairs that provide the names of the files that define the output files, domains, and fields to be used by the XIOS server for the run.

The paths may be relative or absolute, and may contain ~ or $HOME as alternative spellings of the user’s home directory, and $USER as an alternative spelling of the user’s userid.

If relative paths are given, they are appended to the directory containing the run description file.

iodefs

The path and name of the iodef.xml output files definitions file to use for the run. It is copied into the run directory as iodef.xml (the file name required by XIOS). The value is typically either:

  • a relative or absolute run-specific output files definitions file

  • a relative or absolute path to SS-run-sets/SalishSea/nemo3.6/iodef.xml

This key may also be spelled files for backward compatibility.

domaindefs

The path and name of the domain_def.xml output domains definitions file to use for the run. It is copied into the run directory as domain_def.xml (the file name required by XIOS). The value is typically either:

  • a relative or absolute path to SS-run-sets/SalishSea/nemo3.6/domain_def.xml

  • a relative or absolute run-specific output domains definitions file

This key may also be spelled domain for backward compatibility.

fielddefs

The path and name of the field_def.xml output fields definitions file to use for the run. It is copied into the run directory as field_def.xml (the file name required by XIOS). The value is typically a relative or absolute path to NEMO-3.6-code/NEMOGCM/CONFIG/SHARED/field_def.xml.

This key may also be spelled fields for backward compatibility.

filedefs (optional)

The path and name of the file_def.xml output domains definitions file to use for the run. This item is optional because it is only used by XIOS-2 (but it is required by XIOS-2). It is copied into the run directory as file_def.xml (the file name required by XIOS-2). The value is typically either:

  • a relative or absolute run-specific output domains definitions file

  • a relative or absolute path to SS-run-sets/SalishSea/nemo3.6/file_def.xml

The output section also contains key-value pairs that control how the XIOS server is run and, in the case where it is run as a separate server, the number of XIOS servers to run.

separate XIOS server

Boolean flag indicating whether the XIOS server should be run on separate processors from NEMO (True), or in attached mode on every NEMO processor (False). The salishsea prepare command sets the value of the using_server variable in the xios context in the copy of the iodef.xml file in the temporary run directory to reflect the separate XIOS server value.

XIOS server

The number of XIOS servers to run when the value of separate XIOS server it True. The number of XIOS servers is added to the number of NEMO processors calculated from the MPI decomposition value to specify the total number of processors requested in the #PBS directives section of the SalishSeaNEMO.sh script generated by the salishsea run command.

vcs revisions Section

The optional vcs revisions section of the run description file contains lists of version control system repositories for which the revision and status will be recorded in the temporary run and run results directories.

Note

Revision and status record files for the NEMO-3.6-code, and XIOS Mercurial repositories listed in the paths Section section are always generated, so those repository paths should not be included in the vcs revisions section.

Here is a (somewhat contrived) example vcs revisions section:

vcs revisions:
  git:
    - $HOME/MEOPAR/SS-run-sets
    - $HOME/MEOPAR/grid
    - $HOME/MEOPAR/rivers
    - $HOME/MEOPAR/tides
    - $HOME/MEOPAR/tracers
    - $HOME/MEOPAR/NEMO-Cmd
    - $HOME/MEOPAR/SalishSeaCmd
    - $HOME/MEOPAR/XIOS-ARCH
  hg:
    - $HOME/MEOPAR/FVCOM-VHFR-config

The sub-section keys (hg and git above) are the names of the version control tools to use for the repositories listed below them. At present only Mercurial (hg) and Git (git) are supported.

The paths listed under the version control tool keys are the repositories of which the revision and status will be recorded.

The repository paths may be relative or absolute, and may contain ~ or $HOME as alternative spellings of the user’s home directory, and $USER as an alternative spelling of the user’s userid.

Absolute paths with environment variables are strongly recommended for portability and re-usability.

For each repository, a file will be created in the temporary run directory. The file names are the repository directory names with _rev.txt appended. So, from the example above, the files created will be:

SS-run-sets_rev.txt
grid_rev.txt
rivers_rev.txt
tides_rev.txt
tracers_rev.txt
NEMO-Cmd_rev.txt
SalishSeaCmd_rev.txt
XIOS-ARCH_rev.txt
FVCOM-VHFR-config_rev.txt

For Git repositories, each _rev.txt file will contain the output of the commands:

git branch --show-current
git log -1
git show --pretty="" --name-only

for the repository. That is a record of the last committed revision of the repository that will be in effect for the run. For example, SS-run-sets_rev.txt might contain:

branch: main
commit: b43b962a441971def65490620f59a168c2a44a3a
author: Doug Latornell
date:   Tue Jan 31 11:14:50 2017 -0800
files:  SalishSea/SalishSea.yaml SalishSea/nemo3.6/SalishSea.yaml SalishSea/nemo3.6/SalishSea_orcinus.yaml SalishSea/nemo3.6/nowcast-like/SalishSea_nowcast_green.yaml SalishSea/nemo3.6/nowcast-like/SalishSea_nowcast_green_orcinus.yaml
message:
Add 'NEMO code config' key to selected run description YAML files.

If any of the listed repositories contain uncommitted changes, the nemo prepare command that salishsea run uses will generate a warning message like:

nemo_cmd.prepare WARNING: There are uncommitted changes in $HOME/MEOPAR/SS-run-sets/

and the list of uncommitted changes and their status codes, the output of the git diff --name-status command, will be appended to the _rev.txt file.

For Mercurial repositories, each _rev.txt file will contain the output of the hg parents -v command for the repository. That is a record of the last committed revision of the repository that will be in effect for the run. For example, FVCOM-VHFR-config_rev.txt might contain:

changset:   950:35fc362f3d77866df8c0a8b743aca81359295d59
tag:        tip
user:       Doug Latornell <djl@douglatornell.ca>
date:       Fri Mar 08 15:08:15 2019 -08:00
files:      namelists/namelist.numerics.template
description:
Change namelist.numerics to a template file.

If any of the listed repositories contain uncommitted changes, the nemo prepare command that salishsea run uses will generate a warning message like:

nemo_cmd.prepare WARNING: There are uncommitted changes in $HOME/MEOPAR/FVCOM-VHFR-config/

and the list of uncommitted changes and their status codes, the output of the hg status -mardC command, will be appended to the _rev.txt file.