auspice/DEV_DOCS.md

Web app for visualizing pathogen evolution

# Contributing to Auspice development
<!-- WARNING -->
<!-- Do not edit this file from within the docs.nextstrain.org repository. -->
<!-- It is fetched from another repository to be included in the docs.nextstrain.org build. -->
<!-- So, if you edit it after it is fetched into docs.nextstrain.org, your changes will be lost. -->
<!-- Instead, edit this file in its own repository and commit your changes there. -->
<!-- For more details on this (temporary) implementation, see https://github.com/nextstrain/docs.nextstrain.org#fetching-of-documents-from-other-repositories -->
<!-- This file is located at (fetched from): https://github.com/nextstrain/auspice/blob/master/DEV_DOCS.md -->
<!-- WARNING -->
<!-- WARNING -->
<!-- WARNING -->


Thank you for helping us to improve Nextstrain!

> **To get started, please see [the contributing guide](https://github.com/nextstrain/.github/blob/master/CONTRIBUTING.md) for useful information about how to pick an issue, submit your contributions, and so on.**

This project strictly adheres to the [Contributor Covenant Code of Conduct](https://github.com/nextstrain/.github/blob/master/CODE_OF_CONDUCT.md).

Please see the [project boards](https://github.com/orgs/nextstrain/projects) for currently available issues.

## Developer Installation

This is useful for debugging, modifying the source code, or using an unpublished feature branch.

The following steps assume you are using an environment with a suitable nodejs version such as a conda environment. Check the `engines` section of Auspice's [package.json](https://github.com/nextstrain/auspice/blob/master/package.json) for the currently supported nodejs versions.

```sh
git clone https://github.com/nextstrain/auspice.git
cd auspice

# install dependencies and build auspice
npm ci

# make `auspice` available globally
npm install --global .
```

When you update Auspice (e.g. via source code modifications or `git pull`) you may need to (i) update dependencies via `npm ci` and (ii) rebuild the client bundles via `auspice build` / `npm run build`.


## Contributing code

Please comment on an open issue if you are working on it.
For changes unrelated to an open issue, please make an issue outlining what you would like to change/add.

Where possible, **please rebase** your work onto master rather than merging changes from master into your PR.

From a fork: `git pull --rebase upstream master`


## Typescript

Auspice is in the process of moving to typescript, and currently supports both `.js(x)` and `.ts(x)` files.
New code should ideally be written in TS.
[This guide](https://www.typescriptlang.org/docs/handbook/migrating-from-javascript.html) is helpful to understand the bigger picture of migrating a JS project to TS. 
Typescript is supposed to help us, so there's no problem using `any` types as we move code from JS to TS, however new code would ideally be typed.

You can check the types via `npm type-check` or run `npm type-check:watch`; alternatively your editor should be able to show this.

Our CI (GitHub action) will type-check the project.

The various moving parts involved are:

* `typescript` (`tsc` command) is used by `npm run type-check` as well as other plugins/libraries
* `@typescript-eslint/eslint-plugin`, `@typescript-eslint/parser` allow ESLint to parse TypeScript syntax
* `@babel/preset-typescript` is used by babel-loader (via webpack) to parse `.ts(x)` files


## Tests

Auspice has various tests in place, although test coverage is sporadic.
When you submit a pull request to the auspice repository, certain tests will need to pass before it can be merged.
These tests can be run locally:

#### Unit tests

Unit tests use [`Jest`](https://github.com/facebook/jest) and can be run via `npm test`.
These tests rely on datasets which need to be fetched ahead of time via `npm run fetch-test-data`.

General advice for writing tests:

1. Run a single `describe()`, `it()`, or `test()` **within a file**, add `.only()`:

    E.g., `describe.only()`, `it.only()`, or `test.only()`

2. You can run a single test file like so:

    E.g., `npx jest test/dates.test.js`


#### Smoke tests

> Smoke tests are currently very limited but we may expand these in the future

1. Fetch datasets with `npm run get-data`.
2. Install the testing browser with `npx playwright install chromium`.
3. Run `npm run smoke-test`.


#### For integration tests

Auspice used to have integration testing using jest + puppeteer, however this was removed in [PR 1672](https://github.com/nextstrain/auspice/pull/1672).
Ideally this functionality will be brought back, and that PR would be a good place to start.
If embarking on this journey, consider using Playwright since it is already used for smoke tests.


## Linting & type-checking

Linting via eslint: `npm run lint`, typechecking via `npm type-check`


## Offline use

Auspice can register a [service worker](https://developer.mozilla.org/en-US/docs/Web/API/Service_Worker_API) for use without a network connection.
This is opt-in and primarily intended for [auspice.us](https://auspice.us); it is enabled by setting the environment variable `AUSPICE_ENABLE_SERVICE_WORKER=true` at build time.
It is **disabled by default** and is **always disabled in dev mode** (a caching service worker would interfere with hot-reloading).

When enabled, the service worker is registered at the site root with a scope of the whole origin.

What it intercepts:

* The build's static assets – `index.html` and referenced JS/CSS bundles.
* [Navigation requests](https://web.dev/articles/handling-navigation-requests) to the root path `/`.

What it does **not** intercept:

* **Charon API requests**
* **Map tiles**

These are runtime `fetch()` requests with no caching configured, so they always go to the network and will fail when offline.

### Removing a registered service worker

There are two options for removal:

1. Automatic – the easiest option for configurable origins (e.g. `localhost:4000`).

    Instructions: rebuild without `AUSPICE_ENABLE_SERVICE_WORKER`, serve the build **at the same origin**, and open a page. The service worker will be unregistered and Workbox caches removed. The service worker will be fully removed when all pages at the origin have been closed.

2. Manual – useful for ephemeral review apps (e.g. `*.herokuapp.com`).

    Instructions: find and remove the service worker and Workbox caches in browser developer tools. For Chromium-based browsers, this is typically under the "Application" tab > "Service workers" and "Cache storage".

## Heroku review apps


A Heroku pipeline for this repository is connected to GitHub under the nextstrain-bot user account. The Review Apps feature facilitates manual review of changes by automatically creating a test instance from the PR source branch and adding a link to it on the GitHub PR page. These apps are based on configuration in [app.json](./app.json).


#### Test on downstream repositories

Additionally, a GitHub Actions workflow has been set up to generate PRs in downstream repositories that reflect the new changes in Auspice. To use it, add the label [preview on auspice.us](https://github.com/nextstrain/auspice/labels/preview%20on%20auspice.us) and/or [preview on nextstrain.org](https://github.com/nextstrain/auspice/labels/preview%20on%20nextstrain.org).

## git-lfs

[Git Large File Storage](https://github.com/git-lfs/git-lfs) was removed in [PR 1672](https://github.com/nextstrain/auspice/pull/1672).
You may uninstall it via `git lfs uninstall` if needed.


## Contributing to Documentation

The main Nextstrain documentation is available at [docs.nextstrain.org](https://docs.nextstrain.org/en/latest/index.html). That documentation is built from the [docs.nextstrain.org GitHub repo](https://github.com/nextstrain/docs.nextstrain.org/) -- see the [contributing page for docs.nextstrain.org](https://docs.nextstrain.org/en/latest/guides/contribute/documentation.html) for more details on how to contribute to that documentation.

The Auspice technical reference guide is available at [docs.nextstrain.org/projects/auspice](https://docs.nextstrain.org/projects/auspice/en/stable/index.html). That documentation is built from the files in this repo in the `./docs` folder.

To preview the Auspice documentation locally,

1. Create and activate a Conda environment from `docs/conda.yml`. Example:

    ```bash
    conda env create --file docs/conda.yml
    conda activate auspice-docs
    ```

2. Build the docs from a clean slate.

    ```bash
    make -C docs clean livehtml
    ```

3. Open the preview link (http://127.0.0.1:8000) in a browser.

Changes to documentation source files (`.md` and `.rst` files under `docs/`) should automatically be reflected upon page refresh.

## Contributing to Internationalization and Localization (i18n/l18n)

If you can assist in efforts to translate the Auspice interface to more languages your assistance would be very much appreciated.
The currently available languages are displayed via a drop-down at the bottom of the sidebar.

## Adding a new language

  1) Add the language to the `getlanguageOptions` function in [this file](https://github.com/nextstrain/auspice/blob/master/src/components/controls/language.js#L24)
  2) If this is a new language, copy the folder (and the JSONs within it) `src/locales/en` and name it to match the language code for the new translation -- e.g. for Spanish this would be `src/locales/es`
  3) For each key-value in the JSONs, translate the english phrase to the new locale. (Do not modify the parts formatted as `{{...}}` or `<...>...</...>`.)

For example, a spanish translation would change the English:

```json
  "sampled between {{from}} and {{to}}": "sampled between {{from}} and {{to}}",
  "and comprising": "and comprising",
```

to

```json
  "sampled between {{from}} and {{to}}": "aislados entre {{from}} y {{to}}",
  "and comprising": "y compuesto de",
```

## Helper script to check what parts of a translation are out-of-date or missing

Run `npm run diff-lang -- X`, where `X` is the language you wish to check, for instance `es`.
This will display the strings which:

* need to be added to the translation
* are present but should be removed as they are no longer used
* are present but are simply a copy of the English version & need to be translated

> Running `npm run diff-lang` will check all available languages.

## Improving an existing translation

If a translation of a particular string is not yet available, then auspice will fall-back to the english version.

  1) Find the relevant key in the (EN) JSONs [in this directory](https://github.com/nextstrain/auspice/tree/master/src/locales/en)
  2) Add the key to the JSON with the same name, but in the directory corresponding to the language you are translating into (see above for an example).

## Releases & versioning

Releases are made by the [release GitHub Actions workflow](https://github.com/nextstrain/auspice/actions/workflows/release.yaml), which bumps the version, tags it, publishes to npm and creates a GitHub release.
The workflow will also trigger nextstrain/docker-base to update.
Other downstream targets (e.g. nextstrain.org, auspice.us, bioconda, conda-base) require additional steps; see below.

> The release workflow has a number of safeguards to prevent mistakes, see `test/compute-release-version.test.js` for full details.
  These include asserting that a normal release only comes from the `master` branch, and a prerelease only comes from a non-`master` branch. 



### Making a normal release

1. Compare the git history against the previous release (e.g. [view master vs release branch on GitHub](https://github.com/nextstrain/auspice/compare/release...master)) and ensure all changes are reflected in `CHANGELOG.md`.
   If new entries are needed, commit these and push them to the remote (GitHub).
   Put them at the top of the file, directly below the `# Changelog` heading; the workflow will add the version heading above them.
1. Go to the [release workflow](https://github.com/nextstrain/auspice/actions/workflows/release.yaml) and click "Run workflow". Then:
    * Leave **"Use workflow from"** set to `master`. This selects the branch being released, and Auspice's normal releases always come from the code present on `master`.
    * **bump** the version as necessary:
      * If the current version is a normal release, then use a `major` bump (e.g. 2.73.0 → 3.0.0), `feat` for a feature release (→ 2.74.0), or `minor` for a fix (→ 2.73.1).
      * If the current version is a prerelease (see below), then set bump to `promote` to ship the current prerelease as stable (e.g. `3.0.0-alpha.2` → `3.0.0`).
    * Leave **label** as `none` — a labelled release is a prerelease, which `master` doesn't make (see below).
    * Optionally tick **dry run** first: everything is computed and validated, and the npm tarball is built, but nothing is pushed or published.
1. The workflow then, in order:
    * validates the request via `scripts/compute-release-version.js`
    * bumps `package.json`, `package-lock.json` and `src/version.js`, prepends the `## version X.Y.Z - YYYY/MM/DD` heading to `CHANGELOG.md`, commits, and pushes to `master`;
    * pushes the annotated git tag `vX.Y.Z`, and fast-forwards the `release` branch to match `master`;
    * publishes to npm under the `latest` [dist-tag](https://docs.npmjs.com/adding-dist-tags-to-packages)
    * creates an entry on [github.com/nextstrain/auspice/releases](https://github.com/nextstrain/auspice/releases/) using the notes from the changelog section it just added;
    * triggers a rebuild of `nextstrain/docker-base`.
1. Check the results:
    * It takes some time for the new version to appear in the npm registry.
    To check if it has been released run `npm view auspice versions --json | tail`; you may wish to wait for the email to the slack channel `#nextstrain-admin` but it's unclear if there is still a lag after the email arrives before it's available to `npm` APIs.
    None of the "Updating downstream targets" steps will work until the new version is available on npm. 
    * Ensure the [docker-base CI workflow triggered by nextstrain-bot](https://github.com/nextstrain/docker-base/actions/workflows/ci.yml?query=branch%3Amaster+actor%3Anextstrain-bot) runs successfully.
    (This workflow in nextstrain/docker-base is triggered by the release workflow and installs Auspice from the release branch such that it's not dependant on availability in npm.)

Note that the version-bump commit does not itself get a CI run, because pushes made with the default `GITHUB_TOKEN` don't trigger other workflows.
The workflow instead requires CI to have passed on the commit being released, i.e. the one before the version bump.

### Making a prerelease

Prereleases come from any branch **other than** `master` — in practice a long-lived development branch such as `v3`.
They're published to npm under the `next` dist-tag, so `npm install auspice` continues to get the newest normal release and `npm install auspice@next` gets the prerelease.
They don't touch `CHANGELOG.md` or the `release` branch, and don't rebuild docker-base.

Dispatch the same workflow, but set **"Use workflow from"** to the branch you're releasing, and:

* To **start** a series, set **bump** to `major`/`feat`/`minor` and **label** to `alpha`/`beta`/`rc`.
  E.g. from 2.73.0, `major` + `alpha` gives `3.0.0-alpha.0`.
* To **continue** a series, set **bump** to `continue` and keep the same **label**. E.g.  `3.0.0-alpha.0` → `3.0.0-alpha.1` → …
* To **move to a later label**, set **bump** to `continue` and change **label**. E.g. `3.0.0-alpha.1` + `beta` → `3.0.0-beta.0`.

No entry is created on [GitHub's releases page](https://github.com/nextstrain/auspice/releases/), however the git tag `vX.Y.Z-label.n` is pushed as usual.

> Note: `release.yaml` is read from the branch being released, not from `master`.
> The workflow warns in its job summary when its own files differ from `master`'s.


### If a release fails partway through

The commit, tag and `release` branch are pushed *before* npm publish and the GitHub release.
So a failure in `publish-npm` or later leaves the repo already updated. Usually the right move is to fix the cause and **re-run the failed jobs** from the Actions UI.

To unwind instead:

```sh
git push --delete origin v$VERSION      # remove the git tag
git revert $BUMP_COMMIT && git push origin master
```

An npm version which was published can't be unpublished — release a new version instead.
If a bad version reached the `latest` dist-tag, move it back with `npm dist-tag add auspice@$GOOD_VERSION latest`.

### Updating downstream targets

These steps follow a normal release, and are still manual as there are so many downstream targets which are each to be updated with the new version.
None of them apply to prereleases, which deliberately don't move the `latest` dist-tag.

1. Updates to nextstrain sites which depend on Auspice:
    1. Nextstrain.org will check daily for a new version of Auspice and [automatically merge the resulting PR](https://github.com/nextstrain/nextstrain.org/blob/175171e0e1c1b331538729a1168598227d08698d/.github/workflows/dependabot-automation.yml), which results in [the canary site](https://next.nextstrain.org) being updated.
        * You can [trigger this check](https://github.com/nextstrain/nextstrain.org/blob/175171e0e1c1b331538729a1168598227d08698d/.github/dependabot.yml#L16-L17) as soon as the new version is available on npm if you wish.
        * To see the updated Auspice on [nextstrain.org](https://nextstrain.org) you'll need to promote the canary server.
    1. Auspice.us will similarly [check daily for a new version](https://github.com/nextstrain/auspice.us/blob/master/.github/dependabot.yml) however the resulting PR must be merged manually.
    1. Nextclade uses the same approach as auspice.us and is usually merged by Ivan.
1. Update the version on Bioconda:
    1. [Click to edit the Bioconda recipe](https://github.com/bioconda/bioconda-recipes/edit/master/recipes/auspice/meta.yaml). Or, consider using [the Nextstrain fork](https://github.com/nextstrain/bioconda-recipes) to create PRs that other team members can contribute to.
    2. Make the following changes:
        1. Update the version number in the first line.
        2. Change `source.sha256` to the `sha256` hash of the file at `source.url`. You can get the value using the following command:

            ```sh
            curl -s https://registry.npmjs.com/auspice/latest \
                | jq -r ".dist.tarball" \
                | xargs curl -s \
                | shasum -a 256 \
                | cut -f 1 -d " "
            ```
    3. Click through to create a Pull Request to the Bioconda GitHub repository.
1. When the new version of Auspice is available on Bioconda, manually run the [conda-base CI workflow](https://github.com/nextstrain/conda-base/actions/workflows/ci.yaml) on the `main` branch.