NEMO-3.6 Run Description File for AGRIF Runs

Warning

The SalishSeaCmd package provides support for running NEMO-3.6 with AGRIF sub-grids as of version 3.2. AGRIF is not supported for NEMO-3.4.

This section describes the additional sections that can be included in a NEMO-3.6 YAML run description file to enable SalishSeaCmd to handle runs that use AGRIF (Adaptive Grid Refinement in Fortran).

Example AGRIF Run Description YAML File

Example:

# salishsea command processor run description example for NEMO-3.6

config_name: SalishSeaAGRIF

# How is the domain to be distributed over the processors in the
# i and j grid dimensions?
# These values are used to set the nammpp namelist jpni & jpnj values
# and to set the number of processors and nodes in the PBS script
MPI decomposition: 8x18

# For runs on systems that use qsub/PBS/TORQUE or sbatch/SLURM
run_id: example
walltime: 10:00:00
# If missing, email address is set to ${USER}@eoas.ubc.ca
email: dlatornell@example.com

# For runs on systems that use sbatch/SLURM
#
# **Optional**
# Account name to charge resources used by the job to
# If missing, account defaults to rrg-allen on cedar and def-allen on
# other systems
account: def-allen


# Only required if your run uses AGRIF
AGRIF:
  fixed grids: $HOME/MEOPAR/NEMO-forcing-BS1/AGRIF_FixedGrids.in


paths:
  # Absolute path to NEMO-3.6 CONFIG/ directory.
  NEMO code config: $HOME/MEOPAR/NEMO-3.6-code/NEMOGCM/CONFIG
  # If relative, paths below are taken from current directory
  XIOS: $HOME/MEOPAR/XIOS-2
  runs directory: $HOME/MEOPAR/SalishSea/


grid:
  # If relative, paths are taken from the grid/ directory in the forcing
  # path above
  coordinates: $HOME/MEOPAR/grid/coordinates_seagrid_SalishSea.nc
  bathymetry: $HOME/MEOPAR/grid/bathy_meter_SalishSea2.nc
  land processor elimination: False
  AGRIF_1:
    # Symlinks will be prefixed with 1_
    coordinates: $HOME/MEOPAR/NEMO-forcing-BS1/1_coordinates_seagrid_SalishSea201702.nc
    bathymetry: $HOME/MEOPAR/NEMO-forcing-BS1/1_bathymetry_201702.nc


forcing:
  # The keys below are the names of symlinks that will be created.
  # The targets of those symlinks will be the paths named by the associated
  # "link to:" keys;
  # e.g. a symlink named NEMO-atmos will be created to
  # /results/forcing/atmospheric/GEM2.5/operational/
  #
  # You only need to include keys that are used in the namelist(s) for
  # your run.
  #
  # If relative, paths are taken from forcing path above
  NEMO-atmos:
    link to: /results/forcing/atmospheric/GEM2.5/operational/
    check link:
      type: atmospheric
      namelist filename: namelist_cfg
  open_boundaries:
    link to: $HOME/MEOPAR/NEMO-forcing-MD/open_boundaries/
  rivers:
    link to: $HOME/MEOPAR/NEMO-forcing-MD/rivers/
  bfr_coef.nc:
    link to: $HOME/MEOPAR/grid/jetty_mask_bathy201702.nc
  bio_climatology:
    link to: $HOME/MEOPAR/rivers/bio/
  bio_open_boundaries:
    link to: open_boundaries
  #
  # AGRIF inputs are specific to each subgrid and stored all in one directory
  #
  1_rivers:
    link to: $HOME/MEOPAR/NEMO-forcing-BS1/1_rivers/
  1_NEMO-atmos:
    link to: $HOME/MEOPAR/NEMO-forcing-BS1/1_atmospheric/
  1_bio_climatology:
    link to: $HOME/MEOPAR/NEMO-forcing-BS1/1_bio_climatology/


restart:
  # This section is only relevant if your namelist says that you are using
  # restart file(s) from a prior run to initialize your run,
  # if not, it is ignored
  #
  # The keys below are the names of symlinks that will be created.
  # The targets of those symlinks will be the paths provided as values
  # e.g. a symlink named restart_trc.nc will be created to
  # /results/SalishSea/nowcast-green/06dec15/SalishSea_00004320_restart_trc.nc
  #
  # You only need to include keys that are used in the namelist(s) for
  # your run.
  #
  # Paths must be absolute
  restart.nc: /results/SalishSea/nowcast/05may15/SalishSea_02056320_restart.nc
  restart_trc.nc: /results/SalishSea/nowcast-green/06dec15/SalishSea_00004320_restart_trc.nc
  AGRIF_1:
    # Symlinks will be prefixed with 1_
    restart.nc: /results/SalishSea/nowcast/05may15/1_SalishSea_02056320_restart.nc
    restart_trc.nc: /results/SalishSea/nowcast-green/06dec15/1_SalishSea_00004320_restart_trc.nc


namelists:
  # The namelist section files in the lists below will be concatenated
  # to create a namelist file whose name is the key under which the files
  # are listed. The keys are the names of the namelist files that NEMO-3.6
  # expects.
  #
  # The only required key is namelist_cfg.
  #
  # Corresponding namelist*_ref files will be copied from CONFIG/SHARED/
  # unless they are provided in this section.
  #
  # If relative, paths are taken from current directory
  #
  # Parent grid namelists
  namelist_cfg:
    - namelist.time.cont
    - namelist.domain.40
    - namelist.surface.green
    - namelist.lateral
    - namelist.bottom
    - namelist.tracer
    - namelist.dynamics
    - namelist.vertical
    - namelist.compute
  namelist_smelt_cfg:
    - namelist_smelt_cfg
  namelist_top_cfg:
    - namelist_top_cfg

  # Zoom domain namelists will be prefixed with 1_
  AGRIF_1:
    namelist_cfg:
      - 1_namelist.time.cont
      - 1_namelist.domain.20
      - 1_namelist.surface.green
      - 1_namelist.lateral
      - 1_namelist.bottom
      - 1_namelist.tracer
      - 1_namelist.dynamics
      - namelist.vertical
      - namelist.compute
      - 1_namelist.agrif
    namelist_smelt_cfg:
      - 1_namelist_smelt_cfg
    namelist_top_cfg:
      - namelist_top_cfg


output:
  separate XIOS server: True
  XIOS servers: 1
  # If relative, paths are taken from current directory
  iodefs: iodef.xml
  filedefs: ../file_def.xml
  domaindefs: ../domain_def.xml
  fielddefs: field_def.xml
  AGRIF_1:
    # xml file names will be prefixed with 1_
    domaindefs: xios2/1_domain_def.xml
    filedefs: xios2/1_file_def_green.xml


# Optional section
vcs revisions:
  git:
    # repos pointed to by paths: NEMO code config, paths: XIOS, and paths: forcing
    # will be automatically recorded
    #
    # List any additional repos to record revision & status of here
    - $HOME/MEOPAR/grid
    - $HOME/MEOPAR/NEMO-Cmd
    - $HOME/MEOPAR/rivers-climatolog
    - $HOME/MEOPAR/SalishSeaCmd
    - $HOME/MEOPAR/SS-run-sets
    - $HOME/MEOPAR/tides
    - $HOME/MEOPAR/tracers
    - $HOME/MEOPAR/XIOS-ARCH

Basic Run Configuration

There are no changes to the basic run configuration key-value pairs for AGRIF runs. Please see Basic Run Configuration.

`AGRIF` Section

The existence of an AGRIF section in the run description YAML file signals to salishsea run and salishsea prepare that the YAML file includes AGRIF sub-grid sections. It also contains the path to the AGRIF_FixedGrids.in file that NEMO-3.6 uses to configure itself for an AGRIF run.

An example AGRIF section:

AGRIF:
  fixed grids: $HOME/MEOPAR/NEMO-forcing-BS1/AGRIF_FixedGrids.in

The value associated with the fixed grids key must be an absolute path. The path may contain ~ or $HOME as alternative spellings of the user’s home directory, and $USER as an alternative spelling of the user’s userid.

`paths` Section

There are no changes to the paths section for AGRIF runs. Please see paths Section.

`grid` Section

The grid section must contain coordinates and bathymetry items for the main grid as described in grid Section.

Note

Land processor elimination is not supported for AGRIF runs.

An example grid section:

grid:
  coordinates: coordinates_seagrid_SalishSea201702.nc
  bathymetry: bathymetry_201702.nc
  land processor elimination: False
  AGRIF_1:
    coordinates: $HOME/MEOPAR/NEMO-forcing-BS1/1_coordinates_seagrid_SalishSea201702.nc
    bathymetry: $HOME/MEOPAR/NEMO-forcing-BS1/1_bathymetry_201702.nc

The AGRIF_1 key identifies a grid sub-section for the first AGRIF sub-grid. It must contain coordinates and bathymetry items for the sub-grid. Those items must follow the same rules as described in grid Section for the main grid coordinates and bathymetry. The AGRIF_1 coordinates file will be symlinked in the run directory as 1_coordinates.nc (the file name required by NEMO). Likewise, the AGRIF_1 bathymetry file will be symlinked as 1_bathy_meter.nc.

Additional AGRIF sub-grid sections are defined by adding additional AGRIF_n sub-sections. The n value from each AGRIF_n sub-section will be used as the prefix for the coordinates and bathymetry links for that sub-grid.

`forcing` Section

There are no changes to the forcing section for AGRIF runs. Please see forcing Section.

`restart` Section

The optional restart section of the run description file contains key-value pairs that provide paths and file names of restart files to be used to initialize the run.

Here is an example restart section:

restart:
  # Parent
  restart.nc: $HOME/MEOPAR/NEMO-forcing-MD/hindcast/28sep14/SalishSea_00025920_restart_expanded.nc
  AGRIF_1:
    restart.nc: $HOME/MEOPAR/NEMO-forcing-BS1/1_SalishSea_00025920_restart_expanded.nc

The AGRIF_1 key identifies a restart sub-section for the first AGRIF sub-grid. It contains key-value pairs that provide paths and file names of restart files to be used to initialize sub-grid for the run. Those items must follow the same rules as described in grid Section for the main grid restart files. The AGRIF_1 restart file will be symlinked in the run directory as 1_restart.nc

NEMO requires that the name of the model restart file be restart.nc, so that is the key that you must use. For an (optional) tracers restart file the required file name (key) is restart_trc.nc.

Additional AGRIF sub-grid sections are defined by adding additional AGRIF_n sub-sections. The n value from each AGRIF_n sub-section will be used as the prefix for the restart file links for that sub-grid.

`namelists` Section

The namelists section of the run description file must contain the dicts of lists of NEMO namelist section files for the main grid as described in namelists Section. It also contains AGRIF_n sub-section(s) that provide the namelists for the AGRIF sub-grid(s).

An example grid section:

namelists:
  # Parent grid namelists
  namelist_cfg:
    - namelist.time.cont
    - namelist.domain.40
    - namelist.surface.green
    - namelist.lateral
    - namelist.bottom
    - namelist.tracer
    - namelist.dynamics
    - namelist.vertical
    - namelist.compute
  namelist_smelt_cfg:
    - namelist_smelt_cfg
  namelist_top_cfg:
    - namelist_top_cfg

  # Zoom domain namelists
  AGRIF_1:
    namelist_cfg:
      - 1_namelist.time.cont
      - 1_namelist.domain.20
      - 1_namelist.surface.green
      - 1_namelist.lateral
      - 1_namelist.bottom
      - 1_namelist.tracer
      - 1_namelist.dynamics
      - namelist.vertical
      - namelist.compute
      - 1_namelist.agrif
    namelist_smelt_cfg:
      - 1_namelist_smelt_cfg
    namelist_top_cfg:
      - namelist_top_cfg

The AGRIF_1 key identifies a namelist sub-section for the first AGRIF sub-grid. Those items must follow the same rules as described in namelists Section for the main grid namelist files. The namelist section files associated with the namelist_cfg key in the AGRIF_1 sub-section will be concatenated to create the 1_namelist_cfg file in the run directory (the file name required by NEMO). Other namelists will be similarly prefixed.

Additional AGRIF sub-grid sections are defined by adding additional AGRIF_n sub-sections. The n value from each AGRIF_n sub-section will be used as the prefix for the namelist files for that sub-grid.

`output` Section

The output section of the run description file must contain key-value pairs that provide the names of the files that define the output files, domains, and fields to be used by the XIOS server for the run as described in output Section, as well as domain and files definitions for the AGRIF sub-grids.

Note

It is assumed here that AGRIF runs use XIOS-2.

Here is an example output section:

output:
  separate XIOS server: True
  XIOS servers: 1
  iodefs: xios2/iodef.xml
  domaindefs: xios2/domain_def.xml
  fielddefs: ../hindcast/field_def.xml
  filedefs: xios2/file_def.xml
  AGRIF_1:
    domaindefs: xios2/1_domain_def.xml
    filedefs: xios2/1_file_def.xml

The AGRIF_1 key identifies an output sub-section for the first AGRIF sub-grid. It must contain domaindefs and filedefs items for the sub-grid. Those items must follow the same rules as described in output Section for the main grid domain and files definitions. The AGRIF_1 domain definition file will be copied into the run directory as 1_domain_def.xml (the file name required by NEMO). Likewise, the AGRIF_1 files definition file will be copied as 1_file_def.xml.

Additional AGRIF sub-grid sections are defined by adding additional AGRIF_n sub-sections. The n value from each AGRIF_n sub-section will be used as the prefix for the domain and files definition files for that sub-grid.

NEMO-3.6 Run Description File for AGRIF Runs

Example AGRIF Run Description YAML File

Basic Run Configuration

AGRIF Section

paths Section

grid Section

forcing Section

restart Section

namelists Section

output Section