NEMO-3.6 Run Description File
Warning
Versions 2.1 and later of the SalishSeaCmd package introduced changes in the run description file structure that are not backward compatible. Older run description files for NEMO-3.6 runs need to be updated before they can be successfully used with the salishsea run command. Please see SalishSeaCmd Changes That Break Backward Compatibility for details.
Example Run Description YAML File
Example (from SS-run-sets/SalishSea/nemo3.6/SalishSea.yaml
):
# salishsea command processor run description example for NEMO-3.6
# Name of the NEMO-3.6 configuration to use for the run;
# i.e. your NEMO-3.6-code/NEMOGCM/CONFIG/ configuration directory
config_name: SalishSea
# How is the domain to be distributed over the processors in the
# i and j grid dimensions?
# These values are used to set the nammpp namelist jpni & jpnj values
# and to set the number of processors and nodes in the PBS script
MPI decomposition: 8x18
# For runs on systems that use qsub/PBS/TORQUE or sbatch/SLURM
#
# The run_id value appears in the list of jobs display by the showq command
run_id: example
# Maximum run time requested/allowed for job
# Too low and you job will be terminated before it finishes
# Too high and you will have to wait longer on the queue for your job to start
# You have to experiment to find the "just right" value
walltime: 10:00:00
# Email address to send job begin, end, and abort notifications to
# If missing, email address is set to ${USER}@eos.ubc.ca
email: you@example.com
# For runs on systems that use sbatch/SLURM
#
# **Optional**
# Account name to charge resources used by the job to
# If missing, account defaults to rrg-allen on cedar and def-allen on
# other systems
account: def-allen
paths:
# Absolute path to NEMO-3.6 CONFIG/ directory.
# You can use ~, $USER, $HOME if you wish.
NEMO code config: $HOME/MEOPAR/NEMO-3.6-code/NEMOGCM/CONFIG
# If relative, paths below are taken from current directory
# You can use ~, $USER, $HOME if you wish.
XIOS: $HOME/MEOPAR/XIOS/
# Directory in which to create temporary run directories
# Typically NOT inside a version control repository
runs directory: $HOME/MEOPAR/SalishSea/
grid:
# If relative, paths are taken from the grid/ directory in the forcing
# path above
coordinates: $HOME/MEOPAR/grid/coordinates_seagrid_SalishSea201702.nc
bathymetry: $HOME/MEOPAR/grid/bathymetry_201702.nc
# Optional path/filename for land processor elimination MPI-LPE mapping
# file that matches bathymetry;
# If "land processor elimination:" key is absent or has the value "False",
# land processor elimination is disabled
land processor elimination: $HOME/MEOPAR/grid/bathymetry_201702.csv
forcing:
# The keys below are the names of symlinks that will be created.
# The targets of those symlinks will be the paths named by the associated
# "link to:" keys;
# e.g. a symlink named NEMO-atmos will be created to
# /results/forcing/atmospheric/GEM2.5/operational/
#
# The keys are directory names that you use as "cn_dir" values in your
# namelists.
#
# You only need to include keys that are used in the namelist(s) for
# your run.
#
# If relative, paths are taken from forcing path above
NEMO-atmos:
link to: /results/forcing/atmospheric/GEM2.5/operational/
check link:
type: atmospheric
namelist filename: namelist_cfg
rivers:
link to: $HOME/MEOPAR/rivers
tides:
link to: $HOME/MEOPAR/tides
tracers:
link to: $HOME/MEOPAR/tracers
restart:
# The keys below are the names of symlinks that will be created.
# The targets of those symlinks will be the paths/filenames associated
# with the keys;
# e.g. a symlink named restart.nc will be created to
# /results/SalishSea/nowcast/05may15/SalishSea_02056320_restart.nc
#
# You only need to include keys for the types of restart files
# that are used in your run.
#
# If relative, paths are taken from current directory
restart.nc: /results/SalishSea/nowcast/05may15/SalishSea_02056320_restart.nc
restart_trc.nc: /results/SalishSea/nowcast-green/06dec15/SalishSea_00004320_restart_trc.nc
namelists:
# The namelist section files in the lists below will be concatenated
# to create a namelist file whose name is the key under which the files
# are listed. The keys are the names of the namelist files that NEMO-3.6
# expects.
#
# The only required key is namelist_cfg.
#
# If relative, paths are taken from current directory
namelist_cfg:
- namelist.time
- namelist.domain
- namelist.surface
- namelist.lateral
- namelist.bottom
- namelist.tracer
- namelist.dynamics
- namelist.vertical
- namelist.compute
namelist_top_cfg:
- namelist_top_cfg
namelist_pisces_cfg:
- namelist_pisces_cfg
output:
# If relative, paths are taken from current directory
iodefs: iodef.xml
filedefs: file_def.xml
domaindefs: ../domain_def.xml
fielddefs: field_def.xml
separate XIOS server: True
XIOS servers: 1
vcs revisions:
git:
- $HOME/MEOPAR/grid
- $HOME/MEOPAR/NEMO-Cmd
- $HOME/MEOPAR/rivers-climatology
- $HOME/MEOPAR/SalishSeaCmd
- $HOME/MEOPAR/SS-run-sets
- $HOME/MEOPAR/tides
- $HOME/MEOPAR/tracers
- $HOME/MEOPAR/XIOS-ARCH
Basic Run Configuration
The following key-value pairs provide the basic configuration for the run:
- config_name
The name of the NEMO configuration to use for runs. It is the name of a directory in
NEMOGCM/CONFIG/
in the NEMO-3.6-code code directory given by the NEMO key in the paths Section.- MPI decomposition
Specify how the domain is to be distributed over the processors in the i (longitude) and j (latitude) directions; e.g. 8x18. Those values are used to set the namelist.compute nammpp namelist jpni and jpnj values, to set the number of processors and nodes in the
SalishSeaNEMO.sh
script that is generated by the salishsea run command, and to tell the REBUILD_NEMO tool how many files to process.- run_id
The job identifier that appears in the qstat listing.
- walltime
The wall-clock time requested for the run. It limits the time that the job will run for on all machines, and it also affects queue priority for runs on Westgrid machines. It is important to allow some buffer time when calculating your walltime limits to allow for indeterminacy of the NEMO run itself, as well as the time required to combine the per-processor results files into run results files at the end of the run.
The email address at which you want to receive notification of the beginning and end of execution of the run, as well as notification of abnormal abort messages. The email key is only required if the address is different than would be constructed by combining your user id on the machine that the job runs on with @eos.ubc.ca.
- account
Optional. The account name to include in the #SBATCH directives section of the
SalishSeaNEMO.sh
job script. This key-value pair is required on systems like cedar.computecanada.ca and graham.computecanada.ca that use the Slurm Workload Manager and that you submit runs to with the sbatch command.
paths Section
The paths section of the run description file is a collection of directory paths that salishsea uses to find files in other repos that it needs.
- NEMO code config
The path to the
CONFIG/
directory in the NEMO-3.6-code clone where the NEMO configuration directories are to be found; e.g.$HOME/MEOPAR/NEMO-3.6-code/NEMOGCM/CONFIG/
.An absolute path is required because the path is used in both the current directory and the temporary run directory created in the runs directory. The path may contain ~ or
$HOME
as alternative spellings of the user’s home directory, and$USER
as an alternative spelling of the user’s userid.This key may also be spelled NEMO-code-config for backward compatibility.
- XIOS
The path to the XIOS-1.0 (XML I/O Server) clone where the XIOS executable for the run is to be found.
This path may be either absolute or relative. The path may contain ~ or
$HOME
as alternative spellings of the user’s home directory, and$USER
as an alternative spelling of the user’s userid.- runs directory
The path to the directory where run directories will be created by the salishsea run (or salishsea prepare) sub-command.
This path may be relative or absolute, and may contain ~ or
$HOME
as alternative spellings of the user’s home directory, and$USER
as an alternative spelling of the user’s userid.- forcing
Optional path to the directory where the repository clones containing the netCDF files for the grid coordinates, bathymetry, initial conditions, open boundary conditions, etc. are to be found.
This path is used as the base for relative paths in the grid Section and forcing Section sections below.
This path may be relative or absolute, and may contain ~ or
$HOME
as alternative spellings of the user’s home directory, and$USER
as an alternative spelling of the user’s userid.
grid Section
The grid section of the run description file contains 3 key-value pairs that provide the paths/filenames of the grid files to use for the run:
coordinates
bathymetry
land processor elimination configuration
An example grid section:
grid:
coordinates: $HOME/MEOPAR/grid/coordinates_seagrid_SalishSea201702.nc
bathymetry: $HOME/MEOPAR/grid/bathymetry_201702.nc
land processor elimination: $HOME/MEOPAR/grid/bathymetry_201702.csv
Relative or absolute paths may contain ~ or $HOME
as alternative spellings of the user’s home directory,
and $USER
as an alternative spelling of the user’s userid.
If simple file names are provided,
those files are presumed to be in the grid/
sub-directory of the directory tree pointed to by the forcing key in the paths Section.
If relative paths are given,
they are appended to the grid/
directory of the forcing path.
- coordinates
The name of, or path to, the coordinates file to use for the run. It is symlinked in the run directory as
coordinates.nc
(the file name required by NEMO).- bathymetry
The name of, or path to, the bathymetry file to use for the run. It is symlinked in the run directory as
bathy_meter.nc
(the file name required by NEMO).- land processor elimination
The name of, or path to, the MPI-LPE mapping CSV file that corresponds to the bathymetry to use for the run. That file contains the information necessary to configure NEMO-3.6 to perform calculations only on the MPI subdomains that contain water. Please see Land Processor Elimination for a detailed explanation, and use the Preferred MPI LPE Decompositions table to guide your choice of value for MPI decomposition. The SalishSeaCast NEMO Command Processor takes care of setting the correct number of processors based on your chosen MPI decomposition.
A value of
False
disables Land Processor EliminationIf the land processor elimination key is missing, a warning message is printed to let you know that the run is proceeding on the assumption that you want to run without land processor elimination.
This key may also be spelled Land processor elimination for backward compatibility.
forcing Section
The forcing section of the run description file contains sub-sections that provide the names of directories and files that are to be symlinked in the run directory for NEMO to use to read initial conditions and forcing values from.
An example forcing section:
forcing:
NEMO-atmos:
link to: /results/forcing/atmospheric/GEM2.5/operational/
rivers:
link to: $HOME/MEOPAR/rivers
tides:
link to: $HOME/MEOPAR/tides
tracers:
link to: $HOME/MEOPAR/tracers
The sub-section keys (NEMO-atmos, rivers, tides, and tracers above) are the names of the symlinks that will be created in the run directory. Those names are expected to appear in the appropriate places in the namelists. The values associated with the link to keys are the targets of the symlinks that will be created in the run directory.
The paths may be relative or absolute,
and may contain ~ or $HOME
as alternative spellings of the user’s home directory,
and $USER
as an alternative spelling of the user’s userid.
If relative paths are given, they are appended to the forcing path given in the paths Section.
To provide links to directories containing initial fields for a run to use, include a sub-section like:
forcing:
...
initial_strat:
link to: initial_strat/
initial_green:
link to: initial_green/
You are free to use any keys that you wish with the understanding that the key will be the name of the symlink that will be created in the run directory, and that name will also need to appear as a directory name in the appropriate namelist.
The salishsea run and salishsea prepare commands and the salishsea_cmd.api.prepare()
API function confirm that the targets of the symlinks exist,
and exit with an error message if not.
Atmospheric Forcing File Checks
Additional checking can be performed on the files in the atmospheric forcing directory. That checking confirms the existence of all of the atmospheric forcing files for the date range of the run. Doing so ensures that a run won’t fail part way through due to a missing atmospheric forcing file. To enable the additional checking add a check link section at the same level as the link to key:
forcing:
NEMO-atmos:
link to: /results/forcing/atmospheric/GEM2.5/operational/
check link:
type: atmospheric
namelist filename: namelist_cfg
The type key provides the type of checking to perform on the link. The value associated with the namelist filename key is the name of the namelist file in which the atmospheric forcing link is used.
Link checking can be disabled by excluding the check link section,
or by setting the value associated with the type key to None
.
restart Section
The optional restart section of the run description file contains key-value pairs that provide paths and file names of restart files to be used to initialize the run.
The paths may be relative or absolute,
and may contain ~ or $HOME
as alternative spellings of the user’s home directory,
and $USER
as an alternative spelling of the user’s userid.
If relative paths are given, they are appended to the directory containing the run description file.
Here is an example restart section:
restart:
restart.nc: /results/SalishSea/nowcast/SalishSea_00475200_restart.nc
restart_trc.nc: /results/SalishSea/nowcast-green/06dec15/SalishSea_00004320_restart_trc.nc
NEMO requires that the name of the model restart file be restart.nc, so that is the key that you must use. For an (optional) tracers restart file the required file name (key) is restart_trc.nc.
The restart section is optional because it is not required for runs that are initialized with fields provided in a directory linked in the forcing Section.
The salishsea run and salishsea prepare commands and the salishsea_cmd.api.prepare()
API function confirm that the targets of the symlinks exist,
and exit with an error message if not.
namelists Section
The namelists section of the run description file contains one or more dicts of lists of NEMO namelist section files that will be concatenated to construct namelist*_cfg
files
(the file names required by NEMO)
file for the run.
The paths may be relative or absolute,
and may contain ~ or $HOME
as alternative spellings of the user’s home directory,
and $USER
as an alternative spelling of the user’s userid.
If relative paths are given, they are appended to the directory containing the run description file.
Here is an example namelist section:
namelists:
namelist_cfg:
- namelist.time
- namelist.domain
- namelist.surface
- namelist.lateral
- namelist.bottom
- namelist.tracer
- namelist.dynamics
- namelist.vertical
- namelist.compute
namelist_top_cfg:
- namelist_top_cfg
namelist_pisces_cfg:
- namelist_pisces_cfg
A namelist_cfg key must be present, other namelist*_cfg keys are optional. Each namelist*_cfg section must be a list containing at least 1 namelist section file.
Namelist sections that are specific to the run such as namelist.time
where the starting and ending timesteps and the restart configuration are defined are typically stored in the same directory as the run description file.
That mean that they are simply listed by name in the appropriate namelist*_cfg section:
namelists:
namelist_cfg:
- namelist.time
On the other hand,
when you want to use a namelist section that contains the group’s current consensus best values,
list it as a relative or absolute path from the location of your run description file to the “standard” nameslist section files in SS-run-sets/SalishSea/nemo3.6/
:
namelists:
namelist_cfg:
- ../../nemo3.6/namelist.bottom
For each namelist*_cfg key a NEMOGCM/CONFIG/config_name/EXP00/namelist*_ref
file is symlinked into the run directory to provide default values that will be used for any namelist variables not included in the namelist section files listed in the namelists section.
config_name is the value of the config name key in the run description file.
So,
NEMOGCM/CONFIG/config_name/EXP00/namelist_ref
will always be symlinked and,
if the namelist_top_cfg key is present,
the NEMOGCM/CONFIG/config_name/EXP00/namelist_top_ref
file will also be symlinked into the run directory.
You can override the use of *_ref
namelists from CONFIG/config_name/EXP00/
by including a *_ref
namelist key.
For example:
config name: SMELT
...
namelists:
namelist_ref:
- $HOME/MEOPAR/test-sponge/namelist_ref
will cause the namelist_ref
file in the $HOME/MEOPAR/test-sponge/namelist_ref
directory to be symlinked into the temporary run directory instead of CONFIG/SMELT/EXP00/namelist_ref
.
output Section
The output section of the run description file contains key-value pairs that provide the names of the files that define the output files, domains, and fields to be used by the XIOS server for the run.
The paths may be relative or absolute,
and may contain ~ or $HOME
as alternative spellings of the user’s home directory,
and $USER
as an alternative spelling of the user’s userid.
If relative paths are given, they are appended to the directory containing the run description file.
- iodefs
The path and name of the
iodef.xml
output files definitions file to use for the run. It is copied into the run directory asiodef.xml
(the file name required by XIOS). The value is typically either:a relative or absolute run-specific output files definitions file
a relative or absolute path to
SS-run-sets/SalishSea/nemo3.6/iodef.xml
This key may also be spelled files for backward compatibility.
- domaindefs
The path and name of the
domain_def.xml
output domains definitions file to use for the run. It is copied into the run directory asdomain_def.xml
(the file name required by XIOS). The value is typically either:a relative or absolute path to
SS-run-sets/SalishSea/nemo3.6/domain_def.xml
a relative or absolute run-specific output domains definitions file
This key may also be spelled domain for backward compatibility.
- fielddefs
The path and name of the
field_def.xml
output fields definitions file to use for the run. It is copied into the run directory asfield_def.xml
(the file name required by XIOS). The value is typically a relative or absolute path toNEMO-3.6-code/NEMOGCM/CONFIG/SHARED/field_def.xml
.This key may also be spelled fields for backward compatibility.
- filedefs (optional)
The path and name of the
file_def.xml
output domains definitions file to use for the run. This item is optional because it is only used by XIOS-2 (but it is required by XIOS-2). It is copied into the run directory asfile_def.xml
(the file name required by XIOS-2). The value is typically either:a relative or absolute run-specific output domains definitions file
a relative or absolute path to
SS-run-sets/SalishSea/nemo3.6/file_def.xml
The output section also contains key-value pairs that control how the XIOS server is run and, in the case where it is run as a separate server, the number of XIOS servers to run.
- separate XIOS server
Boolean flag indicating whether the XIOS server should be run on separate processors from NEMO (
True
), or in attached mode on every NEMO processor (False
). The salishsea prepare command sets the value of the using_server variable in the xios context in the copy of theiodef.xml
file in the temporary run directory to reflect the separate XIOS server value.- XIOS server
The number of XIOS servers to run when the value of separate XIOS server it
True
. The number of XIOS servers is added to the number of NEMO processors calculated from the MPI decomposition value to specify the total number of processors requested in the #PBS directives section of theSalishSeaNEMO.sh
script generated by the salishsea run command.
vcs revisions Section
The optional vcs revisions section of the run description file contains lists of version control system repositories for which the revision and status will be recorded in the temporary run and run results directories.
Note
Revision and status record files for the NEMO-3.6-code
,
and XIOS
Mercurial repositories listed in the paths Section section are always generated,
so those repository paths should not be included in the vcs revisions section.
Here is a (somewhat contrived) example vcs revisions section:
vcs revisions:
git:
- $HOME/MEOPAR/SS-run-sets
- $HOME/MEOPAR/grid
- $HOME/MEOPAR/rivers
- $HOME/MEOPAR/tides
- $HOME/MEOPAR/tracers
- $HOME/MEOPAR/NEMO-Cmd
- $HOME/MEOPAR/SalishSeaCmd
- $HOME/MEOPAR/XIOS-ARCH
hg:
- $HOME/MEOPAR/FVCOM-VHFR-config
The sub-section keys (hg and git above) are the names of the version control tools to use for the repositories listed below them. At present only Mercurial (hg) and Git (git) are supported.
The paths listed under the version control tool keys are the repositories of which the revision and status will be recorded.
The repository paths may be relative or absolute,
and may contain ~ or $HOME
as alternative spellings of the user’s home directory,
and $USER
as an alternative spelling of the user’s userid.
Absolute paths with environment variables are strongly recommended for portability and re-usability.
For each repository, a file will be created in the temporary run directory. The file names are the repository directory names with _rev.txt appended. So, from the example above, the files created will be:
SS-run-sets_rev.txt
grid_rev.txt
rivers_rev.txt
tides_rev.txt
tracers_rev.txt
NEMO-Cmd_rev.txt
SalishSeaCmd_rev.txt
XIOS-ARCH_rev.txt
FVCOM-VHFR-config_rev.txt
For Git repositories,
each _rev.txt
file will contain the output of the commands:
git branch --show-current
git log -1
git show --pretty="" --name-only
for the repository.
That is a record of the last committed revision of the repository that will be in effect for the run.
For example,
SS-run-sets_rev.txt
might contain:
branch: main
commit: b43b962a441971def65490620f59a168c2a44a3a
author: Doug Latornell
date: Tue Jan 31 11:14:50 2017 -0800
files: SalishSea/SalishSea.yaml SalishSea/nemo3.6/SalishSea.yaml SalishSea/nemo3.6/SalishSea_orcinus.yaml SalishSea/nemo3.6/nowcast-like/SalishSea_nowcast_green.yaml SalishSea/nemo3.6/nowcast-like/SalishSea_nowcast_green_orcinus.yaml
message:
Add 'NEMO code config' key to selected run description YAML files.
If any of the listed repositories contain uncommitted changes, the nemo prepare command that salishsea run uses will generate a warning message like:
nemo_cmd.prepare WARNING: There are uncommitted changes in $HOME/MEOPAR/SS-run-sets/
and the list of uncommitted changes and their status codes,
the output of the git diff --name-status command,
will be appended to the _rev.txt
file.
For Mercurial repositories,
each _rev.txt
file will contain the output of the hg parents -v command for the repository.
That is a record of the last committed revision of the repository that will be in effect for the run.
For example,
FVCOM-VHFR-config_rev.txt
might contain:
changset: 950:35fc362f3d77866df8c0a8b743aca81359295d59
tag: tip
user: Doug Latornell <djl@douglatornell.ca>
date: Fri Mar 08 15:08:15 2019 -08:00
files: namelists/namelist.numerics.template
description:
Change namelist.numerics to a template file.
If any of the listed repositories contain uncommitted changes, the nemo prepare command that salishsea run uses will generate a warning message like:
nemo_cmd.prepare WARNING: There are uncommitted changes in $HOME/MEOPAR/FVCOM-VHFR-config/
and the list of uncommitted changes and their status codes,
the output of the hg status -mardC command,
will be appended to the _rev.txt
file.