suite-metrics
Version:
Easily keep track of metrics for many nested test suites
272 lines (187 loc) ā¢ 10.5 kB
Markdown
[](https://www.npmjs.com/package/suite-metrics)
[](https://www.npmjs.com/package/suite-metrics)
[](https://www.npmjs.com/package/suite-metrics)
[](https://buymeacoffee.com/reidmoffat)
Easily track and aggregate test timing metrics for many nested test suites
Features:
- **Precision Tracking**: Measure test execution time down to microseconds
- **Flexible Nesting**: Organize tests in any number of nested suites with any structure
- **Comprehensive Metrics**: Get aggregate test data, find outliers, and perform statistical tests
- **Easy Interface**: Simple methods calls provided with clear documentation
- **Concurrency Support**: Allows for tracking of multiple concurrent tests safely
```bash
npm i suite-metrics -D
pnpm i suite-metrics -D
yarn add suite-metrics -D
```
You can also use suite-metrics directly in the browser via CDN:
```html
<!-- Using unpkg -->
<script src="https://unpkg.com/suite-metrics"></script>
<!-- Using jsdelivr -->
<script src="https://cdn.jsdelivr.net/npm/suite-metrics"></script>
<!-- Specify version (recommended for production) -->
<script src="https://unpkg.com/suite-metrics@2.5.0"></script>
<script src="https://cdn.jsdelivr.net/npm/suite-metrics@2.5.0"></script>
```
When loaded via CDN, the library is available globally:
```html
<script>
// Same usage as below
const metrics = new SuiteMetrics();
metrics.startTest(["Suite Name", "Test Name"]);
// Test logic...
metrics.stopTest();
console.log(metrics.performance.getSlowestTest());
</script>
```
If you are not running tests concurrently, use SuiteMetrics:
```typescript
import SuiteMetrics from 'suite-metrics';
// Use as a lazy singleton for easy access across multiple files
const metricsSingleton = SuiteMetrics.getInstance();
// Alternatively, create a new instance for isolated metrics
const metrics = new SuiteMetrics();
```
For running multiple tests concurrently, ConcurrentSuiteMetrics is required:
```typescript
import { ConcurrentSuiteMetrics } from 'suite-metrics';
// Lazy singleton is also available, but is async to prevent multiple allocations
const concurrentMetricsSingleton = await ConcurrentSuiteMetrics.getInstance();
// or
const concurrentMetrics = new ConcurrentSuiteMetrics();
// Optionally set the start/stop test mutex timeout (default: 100ms)
const concurrentMetricsWithCustomMutexTimeout = new ConcurrentSuiteMetrics(10);
```
*Note: ConcurrentSuiteMetrics works fine for sequential tests; however, it is more complex (async calls, a parameter
for stopTest(), and some mutex lock overhead) so it is recommended to only use it when required*
### Tracking Tests
Standard SuiteMetrics is simple:
```typescript
// Start tracking a test (directly before test logic for best accuracy)
metrics.startTest(["Suite Name", "Sub-suite name", "Test Name"]);
// Execute your test logic here...
// Call directly after test logic completes to stop tracking
metrics.stopTest();
```
Concurrent metrics can run multiple at the same time:
```typescript
const promises = [
(async () => {
const testName = ["Suite Name", "Test Name 1"];
await concurrentMetrics.startTest(testName);
// Test logic...
await concurrentMetrics.stopTest(testName);
})(),
(async () => {
const testName = ["Suite Name", "Test Name 2"];
await concurrentMetrics.startTest(testName);
// Test logic...
await concurrentMetrics.stopTest(testName);
})(),
(async () => {
const testName = ["Suite Name", "Test Name 3"];
await concurrentMetrics.startTest(testName);
// Test logic...
await concurrentMetrics.stopTest(testName);
})()
];
await Promise.all(promises);
```
*Note: All internal returned data (Tests, Suites, arrays) are frozen to prevent accidental environment corruption*
Both `SuiteMetrics` and `ConcurrentSuiteMetrics` have extensive methods in composite classes:
- **BaseSuiteMetrics**: Base class with straightforward methods like `validatePath()` and `getAllData()`
- **queries**: Query for Suites and Tests, such as `getTest()` and `suiteExists()`
- **metrics**: Gets aggregate metrics for single or multiple suites
- **performance**: Gets the fastest or slowest test(s) in order
- **statistics**: Helpers for Z-scores and standard deviation
> ā ļø Important ā ļø: If you are using `ConcurrentSuiteMetrics`, these methods are NOT thread-safe. Do not call while
> concurrently running tests.
#### BaseSuiteMetrics
```typescript
metrics.validatePath(["Suite 1", "Test 1"], true); // -> true if that test exists
metrics.pathToString(['suite 1', 'sub-suite 2', 'test 3']); // -> "[suite 1, sub-suite 2, test 3]"
metrics.getTestsInOrder(); // -> all tests in the order they were completed
metrics.getAllData(); // -> all data in this metrics as-is
metrics.toJSONString(); // -> all data in this metrics, serialized to JSON
```
```typescript
// Check if a suite or test exists
if (metrics.queries.suiteExists(["Suite Name"])) {
// ...
}
if (metrics.queries.testExists(["Suite Name", "Test Name"])) {
// ...
}
// Gets a full Suite or Test object
metrics.queries.getSuite(["Suite 1", "Sub-suite"]); // -> suite's name, tests, sub-suites, and aggregate data
metrics.queries.getTest(["Suite 1", "Sub-suite", "Test 1"]); // -> test's name, timestamps, and metadata
// Get child suite/test names
metrics.queries.getSuiteNames(["Suite 1"]); // -> all suite names directly in this suite
metrics.queries.getTestNames(["Suite 2"]); // -> all test names directly in this suite
```
```typescript
metrics.metrics.getTotalTestCount(); // -> number of (completed) tests in this metrics instance
metrics.metrics.getAverageTestDuration(); // -> average test duration for all tests (microseconds)
metrics.metrics.getMedianTestDuration(); // -> median duration for all tests (microseconds)
metrics.metrics.getSuiteMetrics(["Suite Name"]); // -> suite's location and test metrics (direct and sub-suites)
console.log(metrics.metrics.printAllSuiteMetrics()); // -> human-readable summary of all tests
metrics.metrics.getStructureMetadata(); // -> high-level aggreagate summary (depth metrics, active time, test distribution, etc)
```
```typescript
metrics.performance.getSlowestTest(); // -> slowest test overall
metrics.performance.getKSlowestTests(5); // -> the 5 slowests tests overall, from slowest to fastest
metrics.performance.getAllTestsSlowestFirst(); // -> all tests, slowest first
metrics.performance.getFastestTest(); // -> fastest test overall
metrics.performance.getKFastestTests(10); // -> the 10 fastest tests overall, from fastest to slowest
metrics.performance.getAllTestsFastestFirst(); // -> all tests, fastest first
```
```typescript
metrics.statistics.getStandardDeviation(); // -> standard deviation for all test times combined
metrics.statistics.getTestZScore(/* <test object> */); // -> Z-score for the test (e.g. 0.7)
metrics.statistics.getAllTestsWithZScores(); // -> every test with their Z-score
metrics.statistics.interpretZScore(2); // -> human-readable z score interpretation (e.g. see below)
result = {
interpretation: 'Unusual performance',
severity: 'unusual',
description: 'Test is unusually slow'
}
```
This package uses **lazy loading** and **caching** to optimize performance, making most operations `O(1)` constant time.
| Operation | Complexity | Notes |
|-------------------------------------|------------|-----------------------------------------------------------------------------------------------------------------------|
| **ConcurrentSuiteMetrics methods** | `O(k)` | `k` = number of waiting operations. Very fast in practice (~few ms for 100 concurrent tests) |
| **Getting/Adding Suites/Tests** | `O(k)` | `k` = depth of Suite/Test in hierarchy. Minimal for typical use cases |
| **Returning multiple Suites/Tests** | `O(k)` | `k` = number of items returned, except getting all items after a cache rebuild has been performed |
| **Performance methods** | `O(n log n)` ā `O(k)` | Cache rebuild when tests added, then `O(k)` for subsequent calls (returning `k` Tests) |
| **Statistics methods** | `O(n)` ā `O(1)` | Cache rebuild (`O(m)` for `m` new tests) when tests added (except `interpretZScore()`), then `O(1)/O(k)` |
| **Data exporting** | `O(n)` | Methods `toJSONString()`, `printAllSuiteMetrics()`, `getStructureMetadata()` and `getAllData()` require a full traverse |
### ā” Performance Best Practices
- **Test execution**: Run all tests before gathering metrics to ensure cache rebuilds only run once
- **Suite depth**: Keep suite hierarchies reasonable (avoid 100s of deeply nested suites)
- **Bulk operations**: Minimize repeated calls to methods returning large datasets (10,000+ tests)
### Real-World Performance
In typical scenarios, performance overhead is **negligible** due to efficient caching. For large cases (~10,000+ tests),
following the recommended patterns above to reduce overhead.
## š Changelog
To view the release notes for each version, view the changelog:
* On GitHub: [Link](https://github.com/reid-moffat/suite-metrics/blob/main/CHANGELOG.md)
* On npm: [package page](https://www.npmjs.com/package/suite-metrics?activeTab=code) -> CHANGELOG.md
* In the repository: CHANGELOG.md
---
ā [Buy me a coffee](https://buymeacoffee.com/reidmoffat) if this package helped you!