siesta-lite/docs/guides/siesta_launchers/README.js

Stress-free JavaScript unit testing and functional testing tool, works in NodeJS and browsers

Ext.data.JsonP.siesta_launchers({"guide":"<h1 id='siesta_launchers-section-siesta-launchers%2C-reports.'>Siesta launchers, reports.</h1>\n<div class='toc'>\n<p><strong>Contents</strong></p>\n<ol>\n<li><a href='#!/guide/siesta_launchers-section-launchers'>Launchers</a></li>\n<li><a href='#!/guide/siesta_launchers-section-common-launcher-api'>Common launcher API</a></li>\n<li><a href='#!/guide/siesta_launchers-section-option-groups'>Option groups</a></li>\n<li><a href='#!/guide/siesta_launchers-section-selenium-webdriver-launcher'>Selenium WebDriver launcher</a></li>\n<li><a href='#!/guide/siesta_launchers-section-puppeteer-launcher'>Puppeteer launcher</a></li>\n<li><a href='#!/guide/siesta_launchers-section-remote-selenium-webdriver-launcher'>Remote Selenium WebDriver launcher</a></li>\n<li><a href='#!/guide/siesta_launchers-section-native-events'>Native events</a></li>\n<li><a href='#!/guide/siesta_launchers-section-parallelization'>Parallelization</a></li>\n<li><a href='#!/guide/siesta_launchers-section-reports-'>Reports </a></li>\n<li><a href='#!/guide/siesta_launchers-section-exit-codes'>Exit codes</a></li>\n<li><a href='#!/guide/siesta_launchers-section-buy-this-product'>Buy this product</a></li>\n<li><a href='#!/guide/siesta_launchers-section-support'>Support</a></li>\n<li><a href='#!/guide/siesta_launchers-section-see-also'>See also</a></li>\n<li><a href='#!/guide/siesta_launchers-section-copyright-and-license'>COPYRIGHT AND LICENSE</a></li>\n</ol>\n</div>\n\n<p>This guide describes how you can run the <a href=\"#!/guide/getting_started_browser\">Siesta browser project</a> from the command line.\nThis is a common requirement for the continuous integration process. It also describes which kind of reports can be generated for\nthe test suite results.</p>\n\n<p><em>Please note:</em> This functionality is only available in the Siesta Standard package.</p>\n\n<h2 id='siesta_launchers-section-launchers'>Launchers</h2>\n\n<p>To run the browser project from the command line Siesta uses \"launchers\" - executable files, sharing a common API. All of them are placed\nin the <code>bin</code> folder of the Siesta package.</p>\n\n<h2 id='siesta_launchers-section-common-launcher-api'>Common launcher API</h2>\n\n<p>All browser project launchers accept the URL of the Siesta project HTML wrapper file, as the only required positional argument.\nAll other options should start with double dash: <code>--option=value</code> and are optional.</p>\n\n<p>On MacOS and Linux it looks like:</p>\n\n<pre><code>&gt; __SIESTA__/bin/__LAUNCHER__ http://your/siesta/project/wrapper/siesta.html [OPTIONS]\n</code></pre>\n\n<p>On Windows:</p>\n\n<pre><code>&gt; __SIESTA__\\bin\\__LAUNCHER__ http://your/siesta/project/wrapper/siesta.html [OPTIONS]\n</code></pre>\n\n<p>Here, the <code>__SIESTA__</code> placeholder is the path to your Siesta package.</p>\n\n<p>Most important, commonly supported for all launchers options are:</p>\n\n<ul>\n<li><code>help</code>                - prints help message with all available options</li>\n<li><code>max-workers</code>         - maximum number of parallel testing \"threads\" that can be opened simultaneously.</li>\n<li><code>headless</code>            - some launchers supports headless mode (browser tests are not visible and not interfering with the working desktop).</li>\n<li><code>simulation</code>          - some launchers supports native events simulation mode (real OS cursor movement).</li>\n<li><code>filter pattern</code>      - a pattern to filter the tests tree, same as in the web interface. Pattern can be:</li>\n</ul>\n\n\n<pre><code>WORD - any word, will match with the test file name (no directory part)\nWORD | WORD - any of the alternative patterns\n\nGROUP &gt; TEST - first match GROUP with the group name, then TEST with tests inside that group\n\nGROUP1 | GROUP2 &gt; TEST1 | TEST2 - alternation support for groups too\n</code></pre>\n\n<p>For example, filtering pattern for \"mouse tests in the 'unit tests' group\" may look like: <code>unit&gt;mou</code>.\nThis option is processed before the following two ones.</p>\n\n<ul>\n<li><code>include regexp</code>      - a regexp to match against the full urls of the test files. Only matching tests will be launched.</li>\n<li><code>exclude regexp</code>      - a regexp to match against the full urls of the test files. Only non-matching tests will be launched.</li>\n<li><code>report-format</code>       - the format of the report, see the \"Reporting the results of a test suite execution in a structured format\" section below</li>\n<li><code>report-file</code>         - the file to save the report to</li>\n<li><code>project-url</code>         - URL of the project file (instead of the positional argument).</li>\n<li><code>verbose</code>             - will include the information about every individual assertions to the output. By default, only failed assertions will be shown</li>\n<li><code>debug</code>               - will enable various debugging messages</li>\n</ul>\n\n\n<p>Please also refer to the \"Parallelization\" section below.</p>\n\n<p>In case of any failures in the test suite the command will exit with non-zero exit code. See \"Exit codes\" section for details.</p>\n\n<h2 id='siesta_launchers-section-option-groups'>Option groups</h2>\n\n<p>Some command line options forms groups. Such \"grouped\" options starts with group name, then follows the dot, then option name. For example,\nthe <code>project.title</code> and <code>project.mouseMovePrecision</code> both belongs to the group <code>project</code>. Grouped options only supports assignments using\nthe <code>=</code> character: <code>project.title=\"Awesome test suite\"</code></p>\n\n<p>Currently there are 2 options groups:</p>\n\n<ul>\n<li><code>project.*</code> - Options in this group will override the <a href=\"#!/api/Siesta.Project-method-configure\" rel=\"Siesta.Project-method-configure\" class=\"docClass\">configuration options</a> of the Siesta project</li>\n<li><code>nyc.*</code> - Options in this group will be translated to the <a href=\"https://github.com/istanbuljs/nyc\">NYC</a> tool, which is used for\n<a href=\"#!/guide/code_coverage\">code coverage</a></li>\n</ul>\n\n\n<h2 id='siesta_launchers-section-selenium-webdriver-launcher'>Selenium WebDriver launcher</h2>\n\n<p>WebDriver launcher supports running your Siesta project in the following browsers: Chrome, Firefox, Safari, Edge, IE11. Older versions of IE will\nwork too, but support for them is limited to best-effort.</p>\n\n<p>This launcher requires the presence of the actual browsers in the operation system, installed in the default locations.</p>\n\n<p>Launcher executable is called <code>webdriver</code>, on MacOS and Linux it looks like:</p>\n\n<pre><code>&gt; __SIESTA__/bin/webdriver http://your/siesta/project/wrapper/siesta.html [--browser=chrome] [OPTIONS]\n</code></pre>\n\n<p>On Windows:</p>\n\n<pre><code>&gt; __SIESTA__\\bin\\webdriver http://your/siesta/project/wrapper/siesta.html [--browser=chrome] [OPTIONS]\n</code></pre>\n\n<p>The <code>--browser</code> option specifies the browser to open and should be one of: <code>chrome</code>, <code>firefox</code>, <code>safari</code>, <code>edge</code>, <code>ie</code>. Default value is <code>chrome</code></p>\n\n<p>The <code>--cap</code> option can specify various Selenium capabilities, for example <code>--cap browserName=firefox --cap platform=XP</code>,\nfor full list see <a href=\"https://github.com/SeleniumHQ/selenium/wiki/DesiredCapabilities\">https://github.com/SeleniumHQ/selenium/wiki/DesiredCapabilities</a></p>\n\n<p>The <code>--headless</code> option is supported for Chrome and Firefox. It prevents opening of browser window and interfering\nwith your normal work. For other browsers you need to make sure to not use real mouse / keyboard during tests execution.</p>\n\n<p>The <code>--simulation</code> option is supported, however the combination of <code>--simulation</code> and <code>--headless</code> option is only supported on Linux with Xvfb.</p>\n\n<p><strong>Important</strong>. Using the WebDriver requires some manual configuration steps for IE and Safari. Please refer to these\npages <a href=\"https://github.com/SeleniumHQ/selenium/wiki/InternetExplorerDriver\">https://github.com/SeleniumHQ/selenium/wiki/InternetExplorerDriver</a> and <a href=\"https://github.com/SeleniumHQ/selenium/wiki/SafariDriver\">https://github.com/SeleniumHQ/selenium/wiki/SafariDriver</a>.\nAlso, when running test suite in IE, make sure the IE window is focused (on top of other windows) and mouse cursor is outside of the IE window.</p>\n\n<h2 id='siesta_launchers-section-puppeteer-launcher'>Puppeteer launcher</h2>\n\n<p>Puppeteer is an automation library for the headless Chrome browser. Puppeteer uses real rendering engine of the Chromium browser -\nyou can trust the results from Puppeteer as if they were received from \"real\" Chrome.</p>\n\n<p>Puppeteer is the recommended launcher for integration with your IDE. Currently its the only one, that supports all combinations of the <code>--max-workers</code>,\n<code>--headless</code> and <code>--simulation</code> configuration options.</p>\n\n<p>On MacOS and Linux:</p>\n\n<pre><code>&gt; __SIESTA__/bin/puppeteer http://your/siesta/project/wrapper/siesta.html [OPTIONS]\n</code></pre>\n\n<p>On Windows:</p>\n\n<pre><code>&gt; __SIESTA__\\bin\\puppeteer http://your/siesta/project/wrapper/siesta.html [OPTIONS]\n</code></pre>\n\n<p>Puppeteer will download a required Chrome version on the 1st launch. Further launches will be instant.</p>\n\n<h2 id='siesta_launchers-section-remote-selenium-webdriver-launcher'>Remote Selenium WebDriver launcher</h2>\n\n<p>You can launch the test suite (and receive the results) on one machine, but physically open the browsers on some other machine using the RemoteWebDriver server.\nTo do that, first start the RemoteWebDriver server on target machine:</p>\n\n<p>On MacOS and Linux:</p>\n\n<pre><code>&gt; __SIESTA__/bin/webdriver-server [OPTIONS]\n</code></pre>\n\n<p>On Windows:</p>\n\n<pre><code>&gt; __SIESTA__\\bin\\webdriver-server [OPTIONS]\n</code></pre>\n\n<p>\"webdriver-server\" is a very thin wrapper around \"selenium-server-standalone-xx.jar\" which just specifies the location of binaries for various browsers. It bypass\nany command line options to that jar file. For example, to specify the port of server (default value is 4444), specify it as:</p>\n\n<pre><code>&gt; __SIESTA__/bin/webdriver-server -port 4444\n</code></pre>\n\n<p>For a list of available options for server, launch it with \"-help\" switch:</p>\n\n<pre><code>&gt; __SIESTA__/bin/webdriver-server -help\n</code></pre>\n\n<p>Also, please refer to: <a href=\"https://github.com/SeleniumHQ/selenium/wiki/RemoteWebDriverServer\">https://github.com/SeleniumHQ/selenium/wiki/RemoteWebDriverServer</a></p>\n\n<p>Then, when launching the test suite with <code>webdriver</code> launcher, specify an additional <code>--host</code>, <code>--port</code> options. Launcher will connect to the\nremote WebDriver server and opens a new browser window there:</p>\n\n<pre><code>&gt; __SIESTA__/bin/webdriver http://your/siesta/project/wrapper/siesta.html --host remote.webdriver.server.host --port 4444\n</code></pre>\n\n<p>The <code>--cap</code> option can be used to specify various Selenium capabilities, for example <code>--cap browserName=firefox --cap platform=XP</code>,\nfor full list see <a href=\"https://github.com/SeleniumHQ/selenium/wiki/DesiredCapabilities\">https://github.com/SeleniumHQ/selenium/wiki/DesiredCapabilities</a></p>\n\n<h2 id='siesta_launchers-section-native-events'>Native events</h2>\n\n<p>By default Siesta uses synthetic events simulation, which is very fast. However, in some cases, you may need to test some intrinsic browser\nbehavior which synthetic events can not reproduce. Most notable example will be testing the <code>:hover</code> CSS styles of some DOM elements.\nIn such case you can use native events simulation.</p>\n\n<h2 id='siesta_launchers-section-parallelization'>Parallelization</h2>\n\n<p>Siesta allows you to run the test suite in several parallel \"threads\" or \"workers\", speeding up the execution almost linearly by the\nnumber of \"workers\".</p>\n\n<p>This can be done using the <code>--max-workers</code> option, which specifies the maximum number of \"workers\".</p>\n\n<p>There are some caveats though, related to focus handling. Even if several browser windows are opened simultaneously,\nthe focus can remain only in one of them, and all browsers \"fights\" for it (focus jumps randomly from one window to\nanother). This naturally leads to tests instability, since code often relies on some element being focused.\nSo, in general, the <code>--max-workers</code> can be used safely only for the non-visual tests. However see below.</p>\n\n<ul>\n<li>XVFB</li>\n</ul>\n\n\n<p>Linux only. When using Linux with the \"Xvfb\" package installed, one can add special command-line switch <code>--xvfb</code>, which launch\nevery browser window inside the separate virtual X desktop (with own focus handling). This makes it safe to specify <code>--max-workers</code>\nmore than 1. For this option to work, the <code>Xvfb</code> is expected to be in PATH, same for the browser binaries,\n<code>google-chrome-stable</code> and <code>firefox</code>. This feature does the same thing as <code>--headless</code> browser mode, plus, it can be combined\nwith <code>--simulation=native</code> option.</p>\n\n<ul>\n<li>Headless</li>\n</ul>\n\n\n<p>The headless mode of the webdriver launcher provides an isolated environment for every browser window as well. It is safe\nto use several workers in this case. Note, however, that currently headless Chrome is known to be very slow on Windows 7.</p>\n\n<ul>\n<li>Cloud providers</li>\n</ul>\n\n\n<p>When using cloud-based infrastructure (SauceLabs or BrowserStack), each test page is running inside of the own VM,\nwhich guarantees the exclusive focus owning and makes it safe to use <code>--max-workers</code>.</p>\n\n<p>When value of <code>--max-workers</code> is more than 1, the order of tests execution is not defined. A test, that goes ealier\nin the <code>project.plan()</code> list, can be executed after the following tests. This is simply because all tests are divided in several\n\"threads\" and all threads are executed simultaneously. You should not rely on some test being run after another, instead,\nevery test should execute standalone (allocate exclusive resources, perform all necessary setup).</p>\n\n<h2 id='siesta_launchers-section-reports-'>Reports </h2>\n\n<p>You can export the results of a test suite execution in the structured format. To do that, provide the <code>--report-format</code> and <code>--report-file</code>\noptions to the launcher. Supported formats are <code>HTML</code>, <code>JSON</code>, <code>JSONS</code> and <code>JUnit</code> (case insensitive). These options can be repeated\nseveral times, to generate several reports.</p>\n\n<p>The <code>JSON</code> and <code>JSONS</code> formats are almost identical, the <code>JSON</code> format contains the test results as a flat list,\nthe <code>JSONS</code> contains results as a tree structure, based on the test groups information.</p>\n\n<p><code>HTML</code> report is basically a <code>JSONS</code> report, plus visualization files. It can be used to examine test suite execution results in the browser.\nWhen using <code>HTML</code> report, the <code>--report-file</code> option actually specifies the <em>directory</em> for files (as this report consists from several files).\nThe <code>JSONS</code> report will be created as <code>report-data.json</code> file in that directory.</p>\n\n<p>An example:</p>\n\n<pre><code>siesta/bin/webdriver http://your/siesta/project/wrapper/siesta.html --report-format JSON --report-file report.json\n</code></pre>\n\n<p>And this will generate a file named <code>report.json</code> containing the following JSON structure similar to (please refer to the actual output,\nas the sample below can be outdated):</p>\n\n<pre><code>{\n    \"testSuiteName\" : \"Siesta self-hosting test suite\",\n    \"startDate\"     : 1343114314723,\n    \"endDate\"       : 1343114315401,\n    \"passed\"        : true,\n    \"testCases\"     : [{\n        \"url\"           : \"010_sanity.t.js\",\n        \"startDate\"     : 1343114315390,\n        \"endDate\"       : 1343114315396,\n        \"passed\"        : true,\n        \"assertions\"    : [{\n            \"passed\"        : true,\n            \"description\"   : \"Siesta is here\",\n            \"type\"          : \"Siesta.Result.Assertion\"\n        }, {\n            \"passed\"        : true,\n            \"description\"   : \"<a href=\"#!/api/Siesta.Test\" rel=\"Siesta.Test\" class=\"docClass\">Siesta.Test</a> is here\",\n            \"type\"          : \"Siesta.Result.Assertion\"\n        }, {\n            \"passed\"        : true,\n            \"description\"   : \"<a href=\"#!/api/Siesta.Project\" rel=\"Siesta.Project\" class=\"docClass\">Siesta.Project</a> is here\",\n            \"type\"          : \"Siesta.Result.Assertion\"\n        }, {\n            \"passed\"        : false,\n            \"description\"   : \"Field 1 focused\",\n            \"annotation\"    : \"Failed assertion `ok` at line 27 of keyevents/050_tab_key_focus2.t.js\\nGot: false\\nNeed \\\"truthy\\\" value\",\n            \"sourceLine\"    : \"27\",\n            \"name\"          : \"ok\"\n            \"type\"          : \"Siesta.Result.Assertion\"\n        }]\n    }]\n}\n</code></pre>\n\n<p>Using the <code>JUnit</code> report option looks like this:</p>\n\n<pre><code>siesta/bin/webdriver http://localhost/YourApplication/tests --report-format junit --report-file report.xml\n</code></pre>\n\n<p>This will generate a file named <code>report.xml</code> containing the following JUnit XML structure:</p>\n\n<pre><code>&lt;testsuite errors=\"0\" failures=\"1\" hostname=\"localhost:8085\" name=\"Ext Scheduler Test Suite\" tests=\"2\" time=\"3.594\" timestamp=\"2012-06-06T08:55:21.520\"&gt;\n\n    &lt;testcase classname=\"Bryntum.Test\" name=\"lifecycle/040_schedulergrid.t.js\" time=\"1.238\"&gt;\n        &lt;failure message=\"Oops\" type=\"FAIL\"&gt;&lt;/failure&gt;\n    &lt;/testcase&gt;\n\n    &lt;testcase classname=\"Bryntum.Test\" name=\"lifecycle/042_schedulergrid_right_columns.t.js\" time=\"0.818\"&gt;&lt;/testcase&gt;\n&lt;/testsuite&gt;\n</code></pre>\n\n<h2 id='siesta_launchers-section-exit-codes'>Exit codes</h2>\n\n<ul>\n<li>0 - All tests passed successfully</li>\n<li>1 - Some tests failed</li>\n<li>3 - No supported browsers available on this machine</li>\n<li>4 - No tests to run (probably filter doesn't match any test url)</li>\n<li>5 - Can't open project page</li>\n<li>6 - Wrong command line arguments</li>\n<li>8 - Exit after showing the Siesta version (when <code>--version</code> is provided on the command line)</li>\n</ul>\n\n\n<h2 id='siesta_launchers-section-buy-this-product'>Buy this product</h2>\n\n<p>Visit our store: <a href=\"https://bryntum.com/store/siesta\">https://bryntum.com/store/siesta</a></p>\n\n<h2 id='siesta_launchers-section-support'>Support</h2>\n\n<p>Ask a question in our community forum: <a href=\"https://www.bryntum.com/forum/viewforum.php?f=20\">https://www.bryntum.com/forum/viewforum.php?f=20</a></p>\n\n<p>Subscribers can post expedited questions in Premium Forum: <a href=\"https://www.bryntum.com/forum/viewforum.php?f=21\">https://www.bryntum.com/forum/viewforum.php?f=21</a></p>\n\n<p>Please report any bugs through the web interface at <a href=\"https://www.assembla.com/spaces/bryntum/support/tickets\">https://www.assembla.com/spaces/bryntum/support/tickets</a></p>\n\n<h2 id='siesta_launchers-section-see-also'>See also</h2>\n\n<p>Siesta web-page: <a href=\"https://bryntum.com/products/siesta\">https://bryntum.com/products/siesta</a></p>\n\n<p>Other Bryntum products: <a href=\"https://bryntum.com/products\">https://bryntum.com/products</a></p>\n\n<h2 id='siesta_launchers-section-copyright-and-license'>COPYRIGHT AND LICENSE</h2>\n\n<p>Copyright (c) 2009-2022, Bryntum AB &amp; Nickolay Platonov</p>\n\n<p>All rights reserved.</p>\n","title":"Siesta launchers & reports"});