siesta-lite/docs/guides/code_coverage/README.js

Stress-free JavaScript unit testing and functional testing tool, works in NodeJS and browsers

Ext.data.JsonP.code_coverage({"guide":"<h1 id='code_coverage-section-code-coverage-with-siesta'>Code coverage with Siesta</h1>\n<div class='toc'>\n<p><strong>Contents</strong></p>\n<ol>\n<li><a href='#!/guide/code_coverage-section-code-instrumentation'>Code instrumentation</a></li>\n<li><a href='#!/guide/code_coverage-section-how-it-works'>How it works</a></li>\n<li><a href='#!/guide/code_coverage-section-collecting-code-coverage-'>Collecting code coverage </a></li>\n<li><a href='#!/guide/code_coverage-section-workaround-for-testing-from-the-%22localhost%22-'>Workaround for testing from the \"localhost\" </a></li>\n<li><a href='#!/guide/code_coverage-section-pre-instrumenting-the-source-code'>Pre-instrumenting the source code</a></li>\n<li><a href='#!/guide/code_coverage-section-additional-setup-for-ie-11'>Additional setup for IE 11</a></li>\n<li><a href='#!/guide/code_coverage-section-buy-this-product'>Buy this product</a></li>\n<li><a href='#!/guide/code_coverage-section-support'>Support</a></li>\n<li><a href='#!/guide/code_coverage-section-see-also'>See also</a></li>\n<li><a href='#!/guide/code_coverage-section-copyright-and-license'>COPYRIGHT AND LICENSE</a></li>\n</ol>\n</div>\n\n<p>Code coverage provides information about what parts of your codebase are executed while running your test suite.\nSuch code is called \"covered\" with tests and can be considered \"safer\" in comparison to \"uncovered\" code.</p>\n\n<p>However, the meaning of the code coverage information should not be overestimated. \"Covered\" code\nstill can contain algorithmic bugs and errors. Also \"covered\" code may not be exercised directly by the test logic, but\ninstead called as helper code (indirect coverage).</p>\n\n<p>The only reliable information code coverage report provides is what code is <em>not</em> covered with tests at all.</p>\n\n<p>Still, even this information is valuable. Definitely, all important parts of your codebase should not contain \"uncovered\" code.</p>\n\n<h2 id='code_coverage-section-code-instrumentation'>Code instrumentation</h2>\n\n<p>In order to collect the code coverage information, the code is <em>instrumented</em> by Siesta (transformed into different\ncode). Siesta uses the awesome <a href=\"https://istanbul.js.org/\">Istanbul</a> library for this purpose.</p>\n\n<p>For example if a \"sample_coverage.js\" file contains only one assignment:</p>\n\n<pre><code>var a = 1;\n</code></pre>\n\n<p>It will be instrumented to:</p>\n\n<pre><code class=\"javascript\">var cov_17swgm9fhy = function() {\n    var path = \"sample_coverage.js\",\n        hash = \"ddc9028ed765ed47d7780f65826df4a462e31fb2\",\n        Function = function() {}.constructor,\n        global = new Function('return this')(),\n        gcv = \"__coverage__\",\n        coverageData = {\n            path: \"examples/nodejs/tt.js\",\n            statementMap: { ... },\n            fnMap: {},\n            branchMap: {},\n            s: {\n                \"0\": 0\n            },\n            f: {},\n            b: {},\n            _coverageSchema: \"d34fc3e6b8297bcde183f5492bcb8fcb36775295\"\n        },\n        coverage = global[gcv] || (global[gcv] = {});\n    if (coverage[path] &amp;&amp; coverage[path].hash === hash) {\n        return coverage[path];\n    }\n    coverageData.hash = hash;\n    return coverage[path] = coverageData;\n}();\nvar a = (cov_17swgm9fhy.s[0]++, 1);\n</code></pre>\n\n<p>As you can see, the assignment to variable is converted to form, which increases the <code>cov_17swgm9fhy.s[0]</code> counter, when executed.\nSimilar counters are installed to other places in the code. Instrumentation does not change the semantic of the code.</p>\n\n<p>Instrumentation involves parsing and transformations of the original sources, so, running test suite with code coverage enabled\ntakes longer. Siesta utilizes Istanbul's caching, to not instrument the same file more then once.</p>\n\n<h2 id='code_coverage-section-how-it-works'>How it works</h2>\n\n<p>Siesta aims to provide just a thin integration level over the Istanbul facilities. This is to be able to freely upgrade Istanbul and stay up to date\nwith its new features and bug fixes. All command line options related to the code coverage are translated directly to <a href=\"https://github.com/istanbuljs/nyc\">Nyc</a>\n(Istanbul's command line tool). Such options forms a <code>--nyc.*</code> group (see more about command line option groups in <a href=\"#!/guide/siesta_launchers\">this guide</a>).</p>\n\n<p><strong>Important</strong>. Option groups supports assignments using the <code>=</code> character only:</p>\n\n<pre><code>--nyc.report-dir=\"Awesome test suite\"   # Ok\n--nyc.report-dir \"Awesome test suite\"   # Not recognized\n</code></pre>\n\n<p>For instrumentation of the browser code, Siesta implements an instrumentation proxy, which bypasses all traffic transparently,\nexcept the requests for JavaScript code. Such requests are determined by the MIME type - it should be one of: <code>text/javascript</code>, <code>application/javascript</code>,\n<code>application/x-javascript</code>. The URL of the request is transformed to a file name, for example <code>http://domain.com/app/file.js?query=string</code>\nbecomes <code>/http/domain.com/app/file.js</code>. Then the request is treated as a \"virtual\" file and passed to Istanbul for instrumentation. All regular\nNyc options, like <code>--nyc.exclude</code> and <code>--nyc.include</code> are applied.</p>\n\n<p>One caveat for instrumentation of the browser code is that, by default, <em>all</em> JavaScript code is instrumented. This may cause\na heavy CPU load (if your project uses a lot of 3rd party libraries). In such case, it is better to explicitly specify what code\nshould be instrumented with <code>--nyc.include</code> option.</p>\n\n<p>The proxying approach for browser code instrumentation, however, doesn't work, when testing <em>locally</em> with MS Edge or Safari,\nsimply because their WebDriver implementations don't support configuring the proxy (all other WebDrivers do).\nIf you really need to generate the coverage report from one of those browsers (using another browser is always an option),\nyou can either pre-instrument the codebase (see the section below), or use cloud testing providers, like <a href=\"#!/guide/saucelabs_integration\">SauceLabs</a>\nor <a href=\"#!/guide/browserstack_integration\">BrowserStack</a>. The latter will work, because traffic proxying will be managed by tunnel software.</p>\n\n<p><strong>Important</strong>. Currently, when collecting code coverage from the cloud testing providers, one can not use the manually launched tunnel, because it\nneeds to be configured to use the instrumentation proxy. The tunnel should be established by the launcher.</p>\n\n<h2 id='code_coverage-section-collecting-code-coverage-'>Collecting code coverage </h2>\n\n<p>To enable the code coverage, specify the value for the <code>--nyc.reporter</code> command line option. It will enable the\ncode instrumentation and generate a report, after the test suite has completed. Recognized values are the names of the Istanbul reporter packages,\nlisted <a href=\"https://github.com/istanbuljs/istanbuljs/tree/master/packages/istanbul-reports/lib\">here</a>.\nMost often used ones are: <code>html</code> and <code>text</code>. Note, that <code>text</code> report will just output the information to the <code>stdout</code> and not save it file.\nThis option can be repeated several times, resulting in several report generated.</p>\n\n<p>Here are some others, most important <code>--nyc.*</code> options. For a full list, please run <code>npx nyc --help</code> (or <code>node ./node_modules/.bin/nyc --help</code>)\nin the Siesta folder. See also the <a href=\"https://github.com/istanbuljs/nyc\">Nyc documentation</a></p>\n\n<ul>\n<li><p><code>--nyc.include</code> - The glob pattern for files to include for instrumentation, can be repeated several times.\nGlobs are resolved using <a href=\"https://www.npmjs.com/package/minimatch\">minimatch</a> package, the better documentation can be found in\n<a href=\"https://www.npmjs.com/package/glob\">glob</a> package. Note, that a glob pattern like <code>*.some.js</code> will only match the root-level <code>*.some.js</code> files,\nto match such files in all directories, specify it as <code>**/*.some.js</code>. By default all files are included.</p></li>\n<li><p><code>--nyc.exclude</code> - The glob pattern for files to exclude from instrumentation, can be repeated several times. Default value is:</p></li>\n</ul>\n\n\n<pre><code>    \"coverage/**\", \"packages/*/test/**\", \"test/**\",\"test{,-*}.js\",\n    \"**/*{.,-}test.js\", \"**/__tests__/**\", \"**/node_modules/**\" \n</code></pre>\n\n<p>Note, that specifying a value for this option will override the default, except the exclusion of <code>node_modules</code> folder. To force the inclusion\nof <code>node_modules</code> folder, specify the negated exclusion pattern: <code>!**/node_modules/</code></p>\n\n<ul>\n<li><p><code>--nyc.report-dir</code> - Specifies the output directory for the code coverage reports, default value is <code>./coverage/</code></p></li>\n<li><p><code>--nyc.extension</code> - Specifies the extension to handle for instrumentation, in addition to <code>.js</code>. Should include the leading dot, for example:\n<code>--nyc.extension=.mjs --nyc.extension=.jsx</code>.</p></li>\n<li><p><code>--nyc.instrument</code> - Whether the actual instrumentation should be performed, default value is <code>true</code>. Set this option to <code>false</code> if you\npre-instrument your source code.</p></li>\n<li><p><code>--nyc.temp-directory</code> - Specifies the directory for the \"raw\" code coverage information collected, default value is <code>./.nyc-output/</code>.</p></li>\n<li><p><code>--nyc.clean</code> - Default value is <code>true</code>. Setting this option to <code>false</code> will prevent nyc from clearing the <code>--nyc.temp-directory</code> folder\nbefore running the test suite. This option can be used to combine the code coverage information from several test suite launches into a single report.</p></li>\n</ul>\n\n\n<p><strong> Important note </strong>. On Windows, <a href=\"https://stackoverflow.com/questions/24173825/what-does-single-quote-do-in-windows-batch-files\">one need to use double quotes</a>,\nwhen providing value for the <code>--nyc.include</code> and <code>--nyc.exclude</code> options.</p>\n\n<p>For example, to collect the code coverage information from your Node.js test suite:</p>\n\n<pre><code>&gt; bin/nodejs --experimental-modules path/to/tests/folder --nyc.extension=mjs --nyc.include=my_tests\n</code></pre>\n\n<p>For example, to collect the code coverage information from the Firefox browser:</p>\n\n<pre><code>&gt; bin/webdriver http://url/of/siesta/project.html --nyc.include=my_tests --nyc.reporter=html --browser firefox\n</code></pre>\n\n<h2 id='code_coverage-section-workaround-for-testing-from-the-%22localhost%22-'>Workaround for testing from the \"localhost\" </h2>\n\n<p>Since Siesta's transparent code instrumentation relies on proxy connection, when running tests from the host <code>localhost</code> you may receive\nempty code coverage results (known issue with Chrome). This is because, it seems, ignoring proxy for requests to <code>localhost</code> is a hard-coded\nsetting in Chrome as a browser (there's an option for hosts to bypass in ChromeDriver, but setting it empty string does nothing,\nand providing it as a command line argument does not work either).</p>\n\n<p>The workaround is to use any other host name, like <code>local</code> or <code>lh</code>, which can be aliased to <code>127.0.0.1</code> in <code>/etc/hosts</code> or similar.</p>\n\n<p>If running from \"localhost\" is a must, you can pre-instrument the source code.</p>\n\n<h2 id='code_coverage-section-pre-instrumenting-the-source-code'>Pre-instrumenting the source code</h2>\n\n<p>In some scenarios, you may need to pre-instrument the codebase, before running the test suite. This may be needed for example, if you test\na browser code, concatenated into a single bundle. For pre-instrumented codebase, disable the on-the-fly instrumentation with the\n<code>--nyc.instrument=false</code> option.</p>\n\n<p>Currently there are 2 options for pre-instrumentation:</p>\n\n<ul>\n<li><p>Use the <a href=\"https://github.com/istanbuljs/babel-plugin-istanbul\">babel-plugin-istanbul</a>. Please find more information in the\n<a href=\"https://github.com/istanbuljs/nyc\">nyc documentation</a></p></li>\n<li><p>Use the <code>instrument</code> command of the <code>nyc</code> tool.  More information can be found by running <code>npx nyc instrument --help</code>\n(or <code>node node_modules/.bin/nyc instrument --help</code>)</p></li>\n</ul>\n\n\n<h2 id='code_coverage-section-additional-setup-for-ie-11'>Additional setup for IE 11</h2>\n\n<p>If, for some reason, you use IE 11 for your code coverage report generation, and test suite is using <code>https</code> protocol, you need to perform additional\nconfiguration step. This is because to intercept the https traffic, Siesta generates self-signed certificate and WebDriver implementation for IE 11\ndoes not support the <code>acceptSslCerts</code> capability.</p>\n\n<p>Please perform the following steps:</p>\n\n<ul>\n<li>Launch the test suite with code coverage enabled. It will fail with the error like \"Can't find Siesta on the project page\".</li>\n<li>In your user's home directory, find the <code>.node_easy_certs</code> folder and double click the <code>rootCA.crt</code> file in it.</li>\n<li>Click the \"Install certificate...\" button. In the opened wizard, select \"Place certificate in the following store\" and choose\nthe \"Trusted Root Certificate Authorities\"</li>\n</ul>\n\n\n<p>After these steps, the generated certificate will be trusted and code coverage can be collected from <code>https</code> sites.</p>\n\n<h2 id='code_coverage-section-buy-this-product'>Buy this product</h2>\n\n<p>Visit our store: <a href=\"https://bryntum.com/store/siesta\">https://bryntum.com/store/siesta</a></p>\n\n<h2 id='code_coverage-section-support'>Support</h2>\n\n<p>Ask a question in our community forum: <a href=\"https://www.bryntum.com/forum/viewforum.php?f=20\">https://www.bryntum.com/forum/viewforum.php?f=20</a></p>\n\n<p>Subscribers can post expedited questions in Premium Forum: <a href=\"https://www.bryntum.com/forum/viewforum.php?f=21\">https://www.bryntum.com/forum/viewforum.php?f=21</a></p>\n\n<p>Please report any bugs through the web interface at <a href=\"https://www.assembla.com/spaces/bryntum/support/tickets\">https://www.assembla.com/spaces/bryntum/support/tickets</a></p>\n\n<h2 id='code_coverage-section-see-also'>See also</h2>\n\n<p>Siesta web-page: <a href=\"https://bryntum.com/products/siesta\">https://bryntum.com/products/siesta</a></p>\n\n<p>Other Bryntum products: <a href=\"https://bryntum.com/products\">https://bryntum.com/products</a></p>\n\n<h2 id='code_coverage-section-copyright-and-license'>COPYRIGHT AND LICENSE</h2>\n\n<p>Copyright (c) 2009-2022, Bryntum &amp; Nickolay Platonov</p>\n\n<p>All rights reserved.</p>\n","title":"Code coverage with Siesta"});