SalishSeaCmd Package Development
Continuous Integration |
|
Documentation |
|
Package |
|
Meta |
|
Python Versions
The SalishSeaCmd package is developed using Python 3.12.
The minimum supported Python version is 3.11.
The Continuous Integration workflow on GitHub ensures that the package
is tested for all versions of Python>=3.11.
An old version of the package running under Python 3.5 is depoloyed on the
Westgrid orcinus HPC platform.
That version is tagged in the repository as orcinus-python-3.5
.
Getting the Code
Clone the SalishSeaCmd code and documentation repository from GitHub with:
$ git clone git@github.com:SalishSeaCast/SalishSeaCmd.git
Development Environment
The SalishSeaCmd package depends on the NEMO-Cmd package, so you need to clone its repo, NEMO-Cmd, beside your clone of SalishSeaCmd.
Setting up an isolated development environment using Conda is recommended. Assuming that you have Anaconda Python Distribution or Miniconda3 installed, you can create and activate an environment called salishsea-cmd that will have all of the Python packages necessary for development, testing, and building the documentation with the commands:
$ conda env create -f SalishSeaCmd/envs/environment-dev.yaml
$ conda activate salishsea-cmd
(salishsea-cmd)$ pip install --editable NEMO-Cmd
(salishsea-cmd)$ pip install --editable SalishSeaCmd
The --editable option in the pip install commands above installs the NEMO-Cmd package and the SalishSeaCmd packages via symlinks so that salishsea in the salishsea-cmd environment will be automatically updated as the repos evolve.
To deactivate the environment use:
(salishsea-cmd)$ conda deactivate
Coding Style
The SalishSeaCmd
package uses Git pre-commit hooks managed by pre-commit to
maintain consistent code style and and other aspects of code,
docs,
and repo QA.
To install the pre-commit hooks in a newly cloned repo, activate the conda development environment, and run pre-commit install:
$ cd SalishSeaCmd
$ conda activate nemo-cmd
(nemo-cmd)$ pre-commit install
Note
You only need to install the hooks once immediately after you make a new clone of the SalishSeaCmd repository and build your Development Environment.
Building the Documentation
The documentation for the SalishSeaCmd package is written in reStructuredText and converted to HTML using Sphinx.
If you have write access to the repository on GitHub, whenever you push changes to GitHub the documentation is automatically re-built and rendered at https://salishseacmd.readthedocs.io/en/latest/.
Additions, improvements, and corrections to these docs are always welcome.
The quickest way to fix typos, etc. on existing pages is to use the Edit on GitHub link in the upper right corner of the page to get to the online editor for the page on GitHub.
For more substantial work, and to add new pages, follow the instructions in the Development Environment section above. In the development environment you can build the docs locally instead of having to push commits to GitHub to trigger a build on readthedocs.org and wait for it to complete. Below are instructions that explain how to:
build the docs with your changes, and preview them in Firefox
check the docs for broken links
Building and Previewing the Documentation
Building the documentation is driven by docs/Makefile
.
With your salishsea-cmd development environment activated,
use:
(salishsea-cmd)$ (cd docs && make clean html)
to do a clean build of the documentation. The output looks something like:
Removing everything under '_build'...
Running Sphinx v3.0.0
making output directory... done
loading intersphinx inventory from https://docs.python.org/3/objects.inv...
loading intersphinx inventory from http://salishsea-meopar-docs.readthedocs.io/en/latest/objects.inv...
loading intersphinx inventory from http://nemo-cmd.readthedocs.io/en/latest/objects.inv...
intersphinx inventory has moved: http://nemo-cmd.readthedocs.io/en/latest/objects.inv -> https://nemo-cmd.readthedocs.io/en/latest/objects.inv
intersphinx inventory has moved: http://salishsea-meopar-docs.readthedocs.io/en/latest/objects.inv -> https://salishsea-meopar-docs.readthedocs.io/en/latest/objects.inv
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 10 source files that are out of date
updating environment: [new config] 10 added, 0 changed, 0 removed
reading sources... [100%] subcommands
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
writing output... [100%] subcommands
generating indices... genindexdone
highlighting module code... [100%] salishsea_cmd.api
writing additional pages... searchdone
copying static files... ... done
copying extra files... done
dumping search index in English (code: en)... done
dumping object inventory... done
build succeeded.
Build finished. The HTML pages are in _build/html.
The HTML rendering of the docs ends up in docs/_build/html/
.
You can open the index.html
file in that directory tree in your browser to preview the results of the build before committing and pushing your changes to GitHub.
Whenever you push changes to the SalishSeaCmd on GitHub the documentation is automatically re-built and rendered at https://salishseacmd.readthedocs.io/en/latest/.
Link Checking the Documentation
Sphinx also provides a link checker utility which can be run to find broken or redirected links in the docs. With your salishsea-cmd environment activated, use:
(salishsea-cmd)$ cd SalishSeaCmd/docs/
(salishsea-cmd) docs$ make linkcheck
The output looks something like:
Removing everything under '_build'...
Running Sphinx v3.3.1
making output directory... done
loading intersphinx inventory from https://docs.python.org/3/objects.inv...
loading intersphinx inventory from https://salishsea-meopar-docs.readthedocs.io/en/latest/objects.inv...
loading intersphinx inventory from https://nemo-cmd.readthedocs.io/en/latest/objects.inv...
building [mo]: targets for 0 po files that are out of date
building [linkcheck]: targets for 10 source files that are out of date
updating environment: [new config] 10 added, 0 changed, 0 removed
reading sources... [100%] subcommands
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
writing output... [ 10%] api
(line 21) ok https://docs.python.org/3/library/pathlib.html#pathlib.Path
(line 21) ok https://docs.python.org/3/library/pathlib.html#pathlib.Path
(line 21) ok https://docs.python.org/3/library/functions.html#int
(line 21) ok https://docs.python.org/3/library/pathlib.html#pathlib.Path
(line 21) ok https://docs.python.org/3/library/stdtypes.html#str
(line 21) ok https://docs.python.org/3/library/stdtypes.html#str
(line 21) ok https://docs.python.org/3/library/stdtypes.html#str
(line 21) ok https://docs.python.org/3/library/stdtypes.html#str
(line 21) ok https://docs.python.org/3/library/stdtypes.html#str
(line 21) ok https://docs.python.org/3/library/constants.html#None
(line 21) ok https://docs.python.org/3/library/constants.html#None
(line 21) ok https://docs.python.org/3/library/constants.html#None
(line 21) ok https://docs.python.org/3/library/stdtypes.html#dict
(line 21) ok https://docs.python.org/3/library/stdtypes.html#dict
(line 21) ok https://docs.python.org/3/library/stdtypes.html#dict
writing output... [ 20%] breaking_changes
(line 97) ok https://docs.python.org/3/library/constants.html#False
(line 45) ok https://f90nml.readthedocs.io/en/latest/
(line 30) ok https://gitpython.readthedocs.io/en/stable/
(line 91) ok https://salishsea-meopar-docs.readthedocs.io/en/latest/code-notes/salishsea-nemo/land-processor-elimination/index.html#landprocessorelimination
(line 53) ok https://calver.org/
writing output... [ 30%] development
(line 21) ok https://docs.python.org/3.11/
(line 21) ok https://black.readthedocs.io/en/stable/
(line 21) ok https://salishseacmd.readthedocs.io/en/latest/
(line 21) ok https://codecov.io/gh/SalishSeaCast/SalishSeaCmd
(line 21) ok https://github.com/SalishSeaCast/NEMO-Cmd/issues
(line 58) ok https://www.python.org/
(line 58) ok https://www.python.org/
(line 21) ok https://www.apache.org/licenses/LICENSE-2.0
(line 58) ok https://nemo-cmd.readthedocs.io/en/latest/development.html#nemo-cmdcontinuousintegration
(line 80) ok https://salishsea-meopar-docs.readthedocs.io/en/latest/repos_organization.html#salishseacmd-repo
(line 21) ok https://github.com/SalishSeaCast/SalishSeaCmd
(line 94) ok https://docs.github.com/en/free-pro-team@latest/github/authenticating-to-github/connecting-to-github-with-ssh
(line 104) ok https://salishsea-meopar-docs.readthedocs.io/en/latest/repos_organization.html#nemo-cmd-repo
(line 74) ok https://github.com/SalishSeaCast/SalishSeaCmd
(line 109) ok https://salishsea-meopar-docs.readthedocs.io/en/latest/work_env/anaconda_python.html#anacondapythondistro
(line 80) ok https://github.com/SalishSeaCast/SalishSeaCmd
(line 143) ok https://www.python.org/dev/peps/pep-0008/
(line 179) ok https://www.sphinx-doc.org/en/master/usage/restructuredtext/index.html
(line 179) ok https://www.sphinx-doc.org/en/master/
(line 391) ok https://docs.pytest.org/en/latest/
(line 109) ok https://conda.io/en/latest/
(line 109) ok https://docs.conda.io/en/latest/miniconda.html
(line 21) ok https://github.com/SalishSeaCast/SalishSeaCmd/actions?query=workflow%3Apytest-with-coverage
(line 424) ok https://pytest-cov.readthedocs.io/en/latest/
(line 424) ok https://coverage.readthedocs.io/en/latest/
(line 469) ok https://docs.github.com/en/free-pro-team@latest/actions
(line 483) ok https://git-scm.com/
(line 449) ok https://github.com/SalishSeaCast/SalishSeaCmd/actions?query=workflow%3Apytest-with-coverage
(line 195) ok https://readthedocs.org/projects/salishseacmd/builds/
(line 458) ok https://github.com/SalishSeaCast/SalishSeaCmd/actions
(line 497) ok https://github.com/SalishSeaCast/SalishSeaCmd/issues
(line 491) ok https://github.com/SalishSeaCast/SalishSeaCmd/issues
(line 21) ok https://img.shields.io/badge/license-Apache%202-cb2533.svg
(line 21) ok https://img.shields.io/badge/Python-3.11%20%7C%203.12-blue?logo=python&label=Python&logoColor=gold
(line 21) ok https://img.shields.io/badge/version%20control-git-blue.svg?logo=github
(line 21) ok https://img.shields.io/badge/code%20style-black-000000.svg
(line 21) ok https://codecov.io/gh/SalishSeaCast/SalishSeaCmd/branch/main/graph/badge.svg
(line 509) ok https://github.com/SalishSeaCast/docs/blob/main/CONTRIBUTORS.rst
(line 21) ok https://github.com/SalishSeaCast/SalishSeaCmd/workflows/pytest-with-coverage/badge.svg
(line 458) ok https://github.com/SalishSeaCast/SalishSeaCmd/commits/main
(line 21) ok https://readthedocs.org/projects/salishseacmd/badge/?version=latest
(line 173) ok https://readthedocs.org/projects/salishseacmd/badge/?version=latest
(line 21) ok https://img.shields.io/github/issues/SalishSeaCast/SalishSeaCmd?logo=github
(line 491) ok https://img.shields.io/github/issues/SalishSeaCast/SalishSeaCmd?logo=github
writing output... [ 40%] index
(line 23) ok https://salishsea-meopar-docs.readthedocs.io/en/latest/code-notes/salishsea-nemo/index.html#salishseanemo
(line 30) ok https://salishsea-meopar-docs.readthedocs.io/en/latest/code-notes/salishsea-nemo/index.html#salishseanemo
(line 30) ok https://docs.openstack.org/cliff/latest/
(line 30) ok https://github.com/SalishSeaCast/NEMO-Cmd
(line 67) ok http://www.apache.org/licenses/LICENSE-2.0
writing output... [ 50%] installation
(line 63) ok https://en.wikipedia.org/wiki/Command-line_completion
writing output... [ 60%] run_description_file/3.6_agrif_yaml_file
(line 24) ok https://www-ljk.imag.fr/MOISE/AGRIF/index.html
(line 27) ok https://www-ljk.imag.fr/MOISE/AGRIF/index.html
writing output... [ 70%] run_description_file/3.6_yaml_file
(line 444) ok https://docs.python.org/3/library/constants.html#True
(line 89) ok https://salishsea-meopar-docs.readthedocs.io/en/latest/repos_organization.html#nemo-3-6-code-repo
(line 171) ok https://salishsea-meopar-docs.readthedocs.io/en/latest/code-notes/salishsea-nemo/land-processor-elimination/index.html#preferred-mpi-lpe-decompositions
(line 100) ok https://salishsea-meopar-docs.readthedocs.io/en/latest/repos_organization.html#xios-repo
(line 74) ok https://slurm.schedmd.com/
writing output... [ 80%] run_description_file/index
(line 23) ok https://pyyaml.org/wiki/PyYAMLDocumentation
(line 28) ok https://salishsea-meopar-docs.readthedocs.io/en/latest/repos_organization.html#ss-run-sets-repo
writing output... [ 90%] run_description_file/segmented_runs
writing output... [100%] subcommands
(line 374) ok https://nemo-cmd.readthedocs.io/en/latest/subcommands.html#nemo-combine
(line 285) ok https://en.wikipedia.org/wiki/Universally_unique_identifier
(line 218) ok https://nemo-cmd.readthedocs.io/en/latest/subcommands.html#nemo-deflate
(line 396) ok https://nemo-cmd.readthedocs.io/en/latest/subcommands.html#nemo-deflate
(line 416) ok https://nemo-cmd.readthedocs.io/en/latest/subcommands.html#nemo-gather
(line 388) ok https://github.com/SalishSeaCast/NEMO-Cmd/
(line 408) ok https://github.com/SalishSeaCast/NEMO-Cmd/
(line 366) ok https://github.com/SalishSeaCast/NEMO-Cmd/
(line 428) ok https://github.com/SalishSeaCast/SS-run-sets/blob/main/v201905/hindcast/file_def_dailysplit.xml
build succeeded.
Look for any errors in the above output or in _build/linkcheck/output.txt
make linkcheck is run monthly via a scheduled GitHub Actions workflow
Running the Unit Tests
The test suite for the SalishSeaCmd package is in SalishSeaCmd/tests/
.
The pytest tool is used for test fixtures and as the test runner for the suite.
With your salishsea-cmd development environment activated, use:
(salishsea-cmd)$ cd SalishSeaCmd/
(salishsea-cmd)$ pytest
to run the test suite. The output looks something like:
============================ test session starts =============================
platform linux -- Python 3.8.2, pytest-5.4.1, py-1.8.1, pluggy-0.13.1
Using --randomly-seed=1586216909
rootdir: /media/doug/warehouse/MEOPAR/SalishSeaCmd
plugins: randomly-3.2.1, cov-2.8.1
collected 279 items
tests/test_run.py ............................................................
..............................................................................
..............................................................................
............................. [ 87%]
tests/test_api.py ...... [ 89%]
tests/test_split_results.py ................ [ 95%]
tests/test_prepare.py ............ [100%]
============================ 279 passed in 1.96s =============================
You can monitor what lines of code the test suite exercises using the coverage.py and pytest-cov tools with the command:
(salishsea-cmd)$ cd SalishSeaCmd/
(salishsea-cmd)$ cpytest --cov=./
The test coverage report will be displayed below the test suite run output.
Alternatively, you can use
(salishsea-cmd)$ pytest --cov=./ --cov-report html
to produce an HTML report that you can view in your browser by opening
SalishSeaCmd/htmlcov/index.html
.
Continuous Integration
The SalishSeaCmd package unit test suite is run and a coverage report is generated whenever changes are pushed to GitHub. The results are visible on the repo actions page, from the green checkmarks beside commits on the repo commits page, or from the green checkmark to the left of the “Latest commit” message on the repo code overview page . The testing coverage report is uploaded to codecov.io
The GitHub Actions workflow configuration that defines the continuous integration
tasks is in the .github/workflows/pytest-with-coverage.yaml
file.
Version Control Repository
The SalishSeaCmd package code and documentation source files are available in the SalishSeaCmd Git repository at https://github.com/SalishSeaCast/SalishSeaCmd.
Issue Tracker
Development tasks, bug reports, and enhancement ideas are recorded and managed in the issue tracker at https://github.com/SalishSeaCast/SalishSeaCmd/issues.
License
The SalishSeaCast NEMO command processor and documentation are copyright 2013 – present by the SalishSeaCast Project Contributors and The University of British Columbia.
They are licensed under the Apache License, Version 2.0. https://www.apache.org/licenses/LICENSE-2.0 Please see the LICENSE file for details of the license.
Release Process
Releases are done at Doug’s discretion when significant pieces of development work have been completed.
The release process steps are:
Use hatch version release to bump the version from
.devn
to the next release version identifierConfirm that
docs/breaking_changes.rst
includes any relevant notes for the version being releasedCommit the version bump and breaking changes log update
Create an annotated tag for the release with Git -> New Tag… in PyCharm or git tag -e -a vyy.n
Push the version bump commit and tag to GitHub
Use the GitHub web interface to create a release, editing the auto-generated release notes as necessary
Use the GitHub Issues -> Milestones web interface to edit the release milestone:
Change the Due date to the release date
Delete the “when it’s ready” comment in the Description
Use the GitHub Issues -> Milestones web interface to create a milestone for the next release:
Set the Title to the next release version, prepended with a
v
; e.g.v23.1
Set the Due date to the end of the year of the next release
Set the Description to something like
v23.1 release - when it's ready :-)
Create the next release milestone
Review the open issues, especially any that are associated with the milestone for the just released version, and update their milestone.
Close the milestone for the just released version.
Use hatch version minor,dev to bump the version for the next development cycle, or use hatch version major,minor,dev for a year rollover version bump
Commit the version bump
Push the version bump commit to GitHub