@openfin/automation-cli/README.md

CLI for running automation tests within the OpenFin ecosystem

# Automation CLI

This package contains a command line application that can be used to run automation tests within the OpenFin ecosystem.

The CLI is run by providing the location of the OpenFin application manifest and a set of test files.

During the execution of the CLI it will retrieve the required OpenFin runtimes as well as the correct version of the chromedriver to use with the runtime.

The CLI can also run in offline more where the OpenFin runtimes and chromedriver are already expected to be available locally. An `offline-versions.json` file is also expected to exist in the storage folder to define the real version numbers for labelled runtimes (see example below)

Currently Jasmine, Mocha and Jest are provided as in-built test runners.

When the test frameworks run they will automatically determine if your tests are written with JavaScript and TypeScript and behave accordingly.

The CLI process will exit with code 1 if any of the tests failed.

## Installation

Run `npm i @openfin/automation-cli`.

## Command Line Options

```shell
🚀 OpenFin Automation v1.2.0

Usage: of-automation [options] <manifestUrl <string>> <testGlobs <string>>

Run Automation Tests using Chrome Driver with an OpenFin UI

Arguments:
  manifestUrl <string>              The url of the manifest to load, can be http|https|file|none
  testGlobs <string>                Globs pointing to the tests to run, semi-colon separate for additional folders

Options:
  -V, --version                     output the version number
  --logLevel <level>                The log level for the webdriver (choices: "debug", "silent", default: "silent")
  --devToolsPort <number>           The port to run the dev tools on (default: 9090)
  --chromeDriverPort <number>       The port to run the chromedriver on (default: 4444)
  --storageFolder <path>            The path to store any downloaded or offline data (default: "./storage/")
  --offline                         In offline mode no resources are retrieved, they are expected to be in the storageFolder
  --testTimeout <number>            The timeout in seconds for running tests (default: 120)
  --testFailCount <number>          Will exit the tests after given number of failures, 0 will allow all failures (default: 0)
  --defaultRuntimeVersion <string>  The OpenFin runtime version to use if not specified in manifest (default: "stable")
  --framework <mocha>               The test framework to run the tests with (choices: "mocha", "jasmine", "jest", default: "mocha")
  --reporter <file.json>            Additional runner config to be passed to frameworks
  --driver <node>                   The webdriver used to access the runtime (choices: "node", "selenium", default: "node")
  --workspace <version>             The workspace version to use e.g. 9.2.5 for the DOS settings (default: "stable")
  --notifications <version>         The notifications version to use e.g. 1.20.2 for the DOS settings (default: "stable")
  --chromeDriverOverride <version>  Override the version of chromedriver e.g. 113
  --closeRuntime <mode>             When to close OpenFin runtimes in the workflow (choices: "never", "start", "end", "both", default: "both")
  --dos <file.json>                 Provide a custom DOS file to use when running tests
  --screenshotFolder <path>         The path to store any screen shots that have been taken (default: "./reports/screenshots/")
  --verbosity <mode>                What to display from the runner output (choices: "all", "none", "results", default: "all")
  --hooks <file>                    A file which contains global setup and teardown hooks (default: "./automation-hooks.{js,ts}")
  -h, --help                        display help for command
```

## Helper packages

Any tests that you write can make use of two helper packages [@openfin/automation-helpers](https://www.npmjs.com/package/@openfin/automation-helpers) and [@openfin/automation-native](https://www.npmjs.com/package/@openfin/automation-native) which provide features to aid in interacting with OpenFin.

## Example tests

There are JavaScript/TypeScript examples in the [Workspace Starter repo](https://github.com/built-on-openfin/workspace-starter/tree/main/how-to/automation-testing/README.md)

There are .NET examples in the [C# Starter repo](https://github.com/built-on-openfin/csharp-starter/tree/main/how-to/automation-testing/README.md)

## CI Environments

The CLI can be used to run tests in a headless CI environment such as GitHub Actions, it requires a `windows` platform to be executed on.

## Reporter config

To provide an additional reporter output for a test framework you can specify a reporter config file e.g. `--reporter mocha.reporter.json`. You must install the necessary npm package in the node_modules where you are running the tests from.

Some examples for each framework are.

### Mocha

```shell
npm install mochawesome
```

```json
{
    "reporter": "mochawesome",
    "options": {
        "reportDir": "./reports/mocha/",
        "reportFilename": "report.html",
        "html": true,
        "json": false
    }
}
```

### Jasmine

```shell
npm install jasmine-pretty-html-reporter
```

```json
{
    "reporter": "Reporter",
    "module": "jasmine-pretty-html-reporter",
    "options": {
        "path": "./reports/jasmine"
    }
}
```

### Jest

```shell
npm install jest-html-reporters
```

```json
{
    "reporter": "jest-html-reporters",
    "options": {
        "publicPath": "./reports/jest",
        "filename": "report.html"
    }
}
```

## Execution example

Assuming your manifest is already being served at `http://localhost:8080/manifest.fin.json` and your files are stored in `./tests`.

```shell
of-automation http://localhost:8080/manifest.fin.json ./tests/**/*.spec.js
```

You should see a summary similar to that shown below:

<details>
  <summary>CLI Output</summary>

```shell
🚀 OpenFin Automation v1.2.0

  ⚙️  Manifest Url http://localhost:8080/manifest.fin.json
  ⚙️  Test Glob Path ./tests/**/*.spec.js
  ⚙️  Log Level silent
  ⚙️  Dev Tools Port 9090
  ⚙️  Chrome Driver Port 4444
  ⚙️  Test Framework mocha
  ⚙️  Test Timeout 120
  ⚙️  Default Runtime Version stable
  ⚙️  Storage Folder ./storage/
  ⚙️  Offline false
_______________________________________________________

➡️  Loading manifest
   http://localhost:8080/manifest.fin.json

  ✅  Manifest loaded
_______________________________________________________

➡️  Resolving OpenFin runtime version
   Version 32.114.76.20

  ✅  Final Runtime version 32.114.76.20
_______________________________________________________

➡️  Get Chrome Driver
   Version 114

  ✅  Chrome Driver already exists .\storage\chromedriver\114\chromedriver.exe
_______________________________________________________

➡️  Creating temp profile directory
   .\tmp\\openfin-test-1651745687443

  ✅  Directory created
_______________________________________________________

➡️  Cleaning up OpenFin instances

  ✅  Cleanup instances complete
_______________________________________________________

➡️  Querying Desktop Owner Settings

  ✅  Writing Temporary Desktop Owner Settings .\tmp\\openfin-dos-1651745687531.json
  ✅  Setting Desktop Owner Setting file:\\\.\tmp\\openfin-dos-1651745687531.json
  ✅  Current Desktop Owner Setting http://localhost:5081/api/dos/openfin
_______________________________________________________

➡️  Running OpenFin
   C:\Users\marty\AppData\Local\OpenFin\OpenFinRVM.exe

  ✅  Args --config=http://localhost:8080/manifest.fin.json,--working-dir="C:\Users\user\AppData\Local\OpenFin",--runtime-arguments="--remote-debugging-port=9090"
  ✅  OpenFinRVM Process 22084
_______________________________________________________

➡️  Starting Chrome Driver
   .\storage\chromedriver\96\chromedriver.exe port 4444

  ✅  Chrome Driver Process 20880
  ✅  Waiting for Chrome Driver to be ready

Starting ChromeDriver 114.0.5615.49 (bd2a7bcb881c11e8cfe3078709382934e3916914-refs/branch-heads/5615@{#936}) on port 4444
Only local connections are allowed.
Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.
ChromeDriver was started successfully.
_______________________________________________________

➡️  Running Tests using Mocha
   Version 10.2.0

  Register with Home

    ✔ The title should be set
    ✔ The runtime version should be set
    ✔ Can open the home window (7263ms)
    ✔ Can search in the home window (1392ms)
    ✔ Can select entries in the home window by index (2077ms)
    ✔ Can select entries in the home window by id (2045ms)
    ✔ Can open the home window filters
    ✔ Can get the filter ids
    ✔ Set a filter by index
    ✔ Set a filter by id
    ✔ Can close the home window filters (3035ms)
    ✔ Can check selected entry content
    ✔ Can open an entry in the home window (1036ms)
    ✔ Can clear entries in the home window (1670ms)
    ✔ Can close the home window (640ms)
    ✔ Can perform operation in the interop window (2069ms)

  16 passing (21s)

_______________________________________________________

➡️  Cleaning Up

  ✅  Closing Chrome Driver
  ✅  Restoring DOS
  ✅  Setting Desktop Owner Setting http://localhost:5081/api/dos/openfin
  ✅  Removing temporary DOS Settings
  ✅  Cleaning up OpenFin instances
  ✅  Cleanup instances complete
  ✅  Removing temp data directory .\tmp\\openfin-test-1651745687443
_______________________________________________________

😀  Successfully ran the tests
```

</details>

### offline-versions.json

Example file for `offline-versions.json` required if running in offline mode.

```json
{
    "stable": "32.114.76.10",
    "release-candidate": "32.114.76.10",
    "beta": "32.114.76.10",
    "canary": "34.116.78.17",
    "stable-v32": "32.114.76.10",
    "stable-v31": "31.112.75.7",
    "stable-v30": "30.110.74.16",
    "stable-v29": "29.108.73.19",
    "stable-v28": "28.106.72.17",
    "stable-v27": "27.104.71.29",
    "stable-v26": "26.102.71.10",
    "stable-v25": "25.100.70.4",
    "stable-v24": "24.98.69.10",
    "stable-v23": "23.96.68.7",
    "stable-v22": "22.94.67.8",
    "stable-v21": "21.93.65.4",
    "stable-v20": "20.91.64.4",
    "stable-v19": "19.89.61.8",
    "stable-v18": "18.87.57.42",
    "stable-v17": "17.85.55.40",
    "stable-v16": "16.83.53.27",
    "stable-v15": "15.80.50.34",
    "stable-v14": "14.78.48.16",
    "stable-v13": "13.76.45.15",
    "stable-v12": "12.69.43.22",
    "stable-v11": "11.69.42.29",
    "stable-v10": "10.66.41.18",
    "stable-v9": "9.61.38.43",
    "stable-v8": "8.56.30.71",
    "stable-v7": "7.53.23.23",
    "stable-v6": "6.49.21.24"
}

```

## OpenFin Debugging

For more information on debugging in OpenFin see [https://developers.openfin.co/of-docs/docs/debugging](https://developers.openfin.co/of-docs/docs/debugging)

## Related Packages

- [@openfin/automation-helpers](https://www.npmjs.com/package/@openfin/automation-helpers)
- [@openfin/automation-native](https://www.npmjs.com/package/@openfin/automation-native)