calibre
Version:
Performance monitoring with Synthetic testing, Chrome UX Report, and Real User Metrics
613 lines (449 loc) • 24.5 kB
Markdown
# Calibre CLI v7.0.0 — Restructure & RUM/CrUX Implementation Plan
## Overview
Restructure the CLI to reflect Calibre's three monitoring pillars (Synthetic, CrUX, RUM), add 8 new commands, maintain backwards compatibility via hidden deprecated aliases, and update all documentation and examples.
Reference: `CHANGELOG-v7-DRAFT.md` in the repository root.
### Design Decisions (confirmed)
- **Naming:** Keep `synthetic` — matches Calibre's UI terminology.
- **Site flag:** Keep `--site` as a required flag (no env var or positional).
- **Pages collision:** `synthetic pages` and `rum pages` both use "pages" — context makes them distinct.
- **Grammar consistency:** Accept mixed patterns across namespaces (compound names in synthetic, clean verbs in deploy, nouns in crux/rum).
### Customer Experience Improvements (from review)
These items from the critical review are incorporated into the plan:
1. **Descriptive help text** — `crux` description must say "Chrome UX Report" in full so users unfamiliar with the abbreviation can discover it.
2. **Accessible grading output** — Always include text labels alongside colour (green `Good`, yellow `NI`, red `Poor`), not colour alone.
3. **Compact table output** — RUM summary uses abbreviated column headers (`Good%`, `NI%`, `Poor%`) to fit 80-column terminals.
4. **History truncation** — `crux history` and `rum history` default to showing the most recent 25 entries. Use `--limit` to adjust.
5. **Machine-readable deprecation warnings** — Prefix format: `[calibre:deprecated] "site pages" has moved to "synthetic pages"`. Support `CALIBRE_SUPPRESS_DEPRECATIONS=1` env var to silence.
6. **Empty data handling** — When CrUX/RUM returns no data, print a helpful message explaining why and link to docs, rather than an empty table or cryptic error.
7. **JSON output stability** — Every moved command gets an output stability test, not just a sample of three.
---
## Phase 1: Foundation
### 1.1 Create deprecation utility
**File:** `src/utils/deprecation.js`
```javascript
import chalk from 'chalk'
const deprecatedHandler = (oldCommand, newCommand, handler) => {
return async (args) => {
if (!process.env.CALIBRE_SUPPRESS_DEPRECATIONS) {
process.stderr.write(
chalk.yellow(
`[calibre:deprecated] "${oldCommand}" has moved to "${newCommand}"\n`
)
)
}
return handler(args)
}
}
export { deprecatedHandler }
```
Key design decisions:
- `process.stderr.write` — warnings go to stderr, never corrupting stdout JSON/CSV/markdown output
- `describe: false` — yargs hides the command from `--help` output
- The `commands` array exported from `site.js` only includes visible commands (for metadata generation), while the `builder` function registers both visible and hidden deprecated commands
- Prefix warnings with `[calibre:deprecated]` for machine parseability
- Support `CALIBRE_SUPPRESS_DEPRECATIONS=1` env var to silence warnings in CI pipelines
### 1.2 Create shared RUM filter options
**File:** `src/utils/rum-options.js`
Shared yargs option definitions for RUM filter flags used across `rum summary`, `rum history`, and `rum pages`:
- `--duration` (default: 7) — Number of days
- `--date-bin` (choices: day, month; default: day) — Time granularity
- `--country` (array) — Filter by country codes
- `--device` (choices: desktop, mobile, tablet) — Filter by device type
- `--connection` (array) — Filter by connection type
- `--path` (array) — Filter by URL paths
- `--page-grouping` (array) — Filter by page grouping UUIDs
### 1.3 Create shared CrUX options
**File:** `src/utils/crux-options.js`
- `--form-factor` (choices: desktop, phone, tablet) — Device type filter
- `--time-period` (choices: three-months, six-months, nine-months, twelve-months, eighteen-months, twenty-four-months; default: six-months) — History window
- `--metrics` (array) — Specific CrUX metric tags
---
## Phase 2: Move Synthetic Commands
Move 17 files from `src/cli/site/` to `src/cli/synthetic/`.
### Files to move (no rename needed):
| From | To |
|------|-----|
| `src/cli/site/pages.js` | `src/cli/synthetic/pages.js` |
| `src/cli/site/create-page.js` | `src/cli/synthetic/create-page.js` |
| `src/cli/site/update-page.js` | `src/cli/synthetic/update-page.js` |
| `src/cli/site/delete-page.js` | `src/cli/synthetic/delete-page.js` |
| `src/cli/site/snapshots.js` | `src/cli/synthetic/snapshots.js` |
| `src/cli/site/create-snapshot.js` | `src/cli/synthetic/create-snapshot.js` |
| `src/cli/site/delete-snapshot.js` | `src/cli/synthetic/delete-snapshot.js` |
| `src/cli/site/get-snapshot-metrics.js` | `src/cli/synthetic/get-snapshot-metrics.js` |
| `src/cli/site/metrics.js` | `src/cli/synthetic/metrics.js` |
| `src/cli/site/test-profiles.js` | `src/cli/synthetic/test-profiles.js` |
| `src/cli/site/create-test-profile.js` | `src/cli/synthetic/create-test-profile.js` |
| `src/cli/site/update-test-profile.js` | `src/cli/synthetic/update-test-profile.js` |
| `src/cli/site/delete-test-profile.js` | `src/cli/synthetic/delete-test-profile.js` |
| `src/cli/site/pull-request-reviews.js` | `src/cli/synthetic/pull-request-reviews.js` |
| `src/cli/site/create-pull-request-review.js` | `src/cli/synthetic/create-pull-request-review.js` |
| `src/cli/site/pull-request-review.js` | `src/cli/synthetic/pull-request-review.js` |
### File to move with rename:
| From | To | Command string change |
|------|-----|----------------------|
| `src/cli/site/download-snapshot-artifacts.js` | `src/cli/synthetic/download-artifacts.js` | `download-snapshot-artifacts [options]` → `download-artifacts [options]` |
### Import path updates
Each moved file has relative imports like `../../api/deploy.js` or `../../utils/cli.js`. After moving from `src/cli/site/` to `src/cli/synthetic/`, the relative paths remain identical (same depth). No import changes needed.
### Create `src/cli/synthetic.js` (group router)
```javascript
import * as Pages from './synthetic/pages.js'
import * as CreatePage from './synthetic/create-page.js'
// ... all 17 imports
const commands = [Pages, CreatePage, /* ... */]
const command = 'synthetic <command>'
const desc = 'Synthetic monitoring — manage scheduled Lighthouse tests, Pages, Test Profiles, Snapshots, and Pull Request Reviews.'
const builder = yargs => yargs.commands(commands)
const handler = () => {}
export { command, desc, builder, handler, commands }
```
---
## Phase 3: Move Deploy Commands
Move 3 files from `src/cli/site/` to `src/cli/deploy/` with command string renames.
| From | To | Command string change |
|------|-----|----------------------|
| `src/cli/site/deploys.js` | `src/cli/deploy/list.js` | `deploys [options]` → `list [options]` |
| `src/cli/site/create-deploy.js` | `src/cli/deploy/create.js` | `create-deploy [options]` → `create [options]` |
| `src/cli/site/delete-deploy.js` | `src/cli/deploy/delete.js` | `delete-deploy [options]` → `delete [options]` |
Additionally in `deploy/list.js`: update the pagination hint message from `calibre site deploys --site=calibre --cursor=...` to `calibre deploy list --site=... --cursor=...`.
### Create `src/cli/deploy.js` (group router)
```javascript
import * as DeployList from './deploy/list.js'
import * as DeployCreate from './deploy/create.js'
import * as DeployDelete from './deploy/delete.js'
const commands = [DeployList, DeployCreate, DeployDelete]
const command = 'deploy <command>'
const desc = 'Manage deployment markers — annotate your performance charts across Synthetic, CrUX, and RUM data.'
const builder = yargs => yargs.commands(commands)
const handler = () => {}
export { command, desc, builder, handler, commands }
```
---
## Phase 4: Wire Deprecations in `site.js`
Slim `site.js` to only export `create`, `list`, `delete` as visible commands. Register all 20 deprecated commands as hidden.
### Structure:
```javascript
import * as SiteCreate from './site/create.js'
import * as SiteList from './site/list.js'
import * as SiteDelete from './site/delete.js'
// Import from new locations for deprecated wrappers
import * as SyntheticPages from './synthetic/pages.js'
import * as SyntheticCreatePage from './synthetic/create-page.js'
// ... all moved modules
import * as DeployList from './deploy/list.js'
import * as DeployCreate from './deploy/create.js'
import * as DeployDelete from './deploy/delete.js'
import { deprecatedHandler } from '../utils/deprecation.js'
// Visible commands (shown in help, used for metadata/docs generation)
const commands = [SiteCreate, SiteList, SiteDelete]
// Hidden deprecated wrappers
const deprecatedCommands = [
{
command: 'pages [options]',
describe: false,
builder: SyntheticPages.builder,
handler: deprecatedHandler('site pages', 'synthetic pages', SyntheticPages.handler)
},
{
command: 'deploys [options]',
describe: false,
builder: DeployList.builder,
handler: deprecatedHandler('site deploys', 'deploy list', DeployList.handler)
},
// ... all 20 deprecated entries
]
const command = 'site <command>'
const desc = 'Manage your Sites.'
const builder = yargs => yargs.commands([...commands, ...deprecatedCommands])
const handler = () => {}
export { command, desc, builder, handler, commands }
```
Key: The exported `commands` array only has 3 entries. This is what `cli-metadata.js` reads for generating `CLI_COMMANDS.md`. The `builder` function registers all commands (visible + hidden) into yargs for runtime. Hidden commands work but don't appear anywhere in docs or help.
---
## Phase 5: Update `cli-commands.js`
```javascript
import * as ConnectionList from './cli/connection-list.js'
import * as DeviceList from './cli/device-list.js'
import * as LocationList from './cli/location-list.js'
import * as MetricList from './cli/metric-list.js'
import * as Request from './cli/request.js'
import * as Site from './cli/site.js'
import * as Synthetic from './cli/synthetic.js'
import * as Deploy from './cli/deploy.js'
import * as Crux from './cli/crux.js'
import * as Rum from './cli/rum.js'
import * as Team from './cli/team.js'
import * as Test from './cli/test.js'
import * as Token from './cli/token.js'
const commands = [
Site,
Synthetic,
Deploy,
Crux,
Rum,
Test,
Team,
ConnectionList,
DeviceList,
LocationList,
MetricList,
Token,
Request
]
export default commands
```
Order matters for `--help` display. Group by importance/frequency.
---
## Phase 6: Enhance `site list`
Update `src/api/site.js` `LIST_QUERY` to include:
```graphql
monitoringStatus {
synthetic
crux
rum
}
```
Update `src/cli/site/list.js` table output to show monitoring status:
```
3 sites
slug | name | monitoring | created
calibre | Calibre | synthetic crux rum | 2:30pm 5-May-2026
example | Example Corp | synthetic | 1:00pm 3-May-2026
```
Use chalk to colour active statuses green. JSON output includes the full `monitoringStatus` object unchanged.
---
## Phase 7: Enhance `metric-list`
Add `--type` flag to `src/cli/metric-list.js`:
```javascript
builder = {
json: options.json,
type: {
describe: 'Filter metrics by data source.',
choices: ['synthetic', 'crux', 'rum']
}
}
```
Update `src/api/metric.js` to support querying different metric sources:
- No `--type` → query `metrics` (existing behaviour, all synthetic metrics)
- `--type=synthetic` → query `metrics` (same)
- `--type=crux` → query `cruxMetrics`
- `--type=rum` → query `rumMetrics`
---
## Phase 8: CrUX Commands
### 8.1 API Module: `src/api/crux.js`
Four exported functions:
**`summary({ site, formFactor })`**
Query: `GetCruxData` — returns `cruxCvwAssessment`, `cruxFormFactorDensity`, `cruxAggregateMetrics`
**`history({ site, formFactor, timePeriod, metrics })`**
Query: `GetCruxData` — returns `cruxHistory` with per-metric collection period values
**`urls({ site, formFactor })`**
Query: `ListCruxUrls` — returns `cruxUrlsList` edges with aggregate metrics per URL
**`url({ site, uuid, formFactor, timePeriod, metrics })`**
Query: `GetCruxUrlData` — returns single URL detail with history
All functions query via `organisation { site(slug:) { ... } }`, consistent with existing commands. When the API returns no CrUX data (site lacks Chrome traffic), functions return a distinguishable empty result so the CLI can print a helpful message rather than an empty table.
### 8.2 CLI Commands
**`src/cli/crux/summary.js`**
- Description: `Display Chrome UX Report (CrUX) origin-level performance data and Core Web Vitals assessment.`
- Displays: CVW assessment (Good/Needs Improvement/Poor with text + colour), form factor density, aggregate metrics table
- Table format: `metric | p75 | grading` — compact, fits 80 columns
- When no data available: print `No CrUX data available for this site. CrUX requires sufficient Chrome user traffic. See: https://calibreapp.com/docs/crux`
**`src/cli/crux/history.js`**
- Description: `Display Chrome UX Report (CrUX) historical trends for a site.`
- Displays: per-metric time series with collection periods
- Table format: `period | metric | p75 | grading`
- Defaults to 25 most recent entries. Use `--limit` to adjust.
**`src/cli/crux/urls.js`**
- Description: `List Chrome UX Report (CrUX) monitored URLs with their metrics and Core Web Vitals assessment.`
- Displays: URL list with CVW assessment and key metrics
- Table format: `url | assessment | lcp | cls | inp | ttfb`
**`src/cli/crux/url.js`**
- Description: `Display Chrome UX Report (CrUX) data for a specific monitored URL.`
- Displays: single URL detail view — assessment, form factor density, metrics, history
- Combines summary + history into one detailed view
### 8.3 Views
**`src/views/crux-metrics-table.js`** — Reusable table for CrUX aggregate metrics with grading colours.
**`src/views/crux-history.js`** — Time series view for CrUX history data.
---
## Phase 9: RUM Commands
### 9.1 API Module: `src/api/rum.js`
Four exported functions:
**`summary({ site, metrics, filter })`**
Query: `GetSiteRumDashboard` — returns `liveVisitors`, `distinctCountriesCount`, `aggregate`, `uxRating`
**`history({ site, metrics, filter })`**
Query: `GetSiteRumDashboardHistory` — returns `history` array with per-date metrics
**`pages({ site, metrics, filter, limit, offset, sortBy })`**
Query: `GetSiteRumPages` — returns paginated page-level breakdown
**`config({ site })`**
Query: site query with `rumConfig` field — returns configuration details
The `filter` object maps CLI flags to the `RumFilterInput` GraphQL input:
- `--duration` → `filter.duration`
- `--date-bin` → `filter.dateBin` (mapped: `day` → `DAY`, `month` → `MONTH`)
- `--country` → `filter.countryCode`
- `--device` → mapped to `filter.isDesktopDevice` / `filter.isMobileDevice` / `filter.isTabletDevice`
- `--connection` → `filter.connectionType` (if available) or similar
- `--path` → `filter.path`
- `--page-grouping` → `filter.pageGroupingUuid`
### 9.2 CLI Commands
**`src/cli/rum/summary.js`**
- Description: `Display Real User Metrics (RUM) dashboard — live visitors, aggregate web vitals, and UX ratings.`
- Displays: live visitors, countries, aggregate metrics table with gradings, UX rating breakdown
- Format (compact, 80-column safe):
```
Live visitors: 342 | Countries: 28 | Sessions (7d): 12,450
Metric | p75 | Rating | Good% | NI% | Poor%
LCP | 2.1 sec | Good | 78% | 15% | 7%
CLS | 0.05 | Good | 85% | 10% | 5%
INP | 180 ms | Good | 72% | 20% | 8%
```
- Grading uses text label + colour: green `Good`, yellow `NI`, red `Poor`
- When no RUM data: print `No RUM data available. Check that RUM is enabled for this site with: calibre rum config --site=<slug>`
**`src/cli/rum/history.js`**
- Description: `Display Real User Metrics (RUM) historical trends.`
- Displays: per-date metrics table
- Table format: `date | lcp | cls | inp | ttfb | fcp | sessions`
- Defaults to 25 most recent entries. Use `--limit` to adjust.
**`src/cli/rum/pages.js`**
- Description: `Display Real User Metrics (RUM) page-level breakdown.`
- Displays: page-level metrics sorted by session count
- Table format: `path | sessions | lcp | cls | inp`
- Pagination hint (like deploy list): `To see more pages, run: calibre rum pages --site=x --offset=25`
**`src/cli/rum/config.js`**
- Description: `Display Real User Metrics (RUM) configuration for a site.`
- Simple key-value output: identifier, enabled, sample rate, allowed origins, retention
- When RUM not configured: print `RUM is not enabled for this site.`
### 9.3 Views
**`src/views/rum-summary.js`** — Dashboard view with UX ratings.
**`src/views/rum-pages-table.js`** — Page-level table with gradings.
**`src/views/grading.js`** — Shared utility for web vitals grading display. Always outputs text label + colour for accessibility: `Good` (green), `NI` (yellow), `Poor` (red). Works correctly when chalk colours are disabled (CI, piped output).
---
## Phase 10: Node.js API Exports
Update `index.js`:
```javascript
// Add new exports
export * as Crux from './src/api/crux.js'
export * as Rum from './src/api/rum.js'
```
---
## Phase 11: Testing
### 11.1 Update existing tests
Tests in `__tests__/cli/site/` that reference moved commands need updating:
- `__tests__/cli/site/snapshots.test.js` → `__tests__/cli/synthetic/snapshots.test.js`
- `__tests__/cli/site/pages.test.js` → `__tests__/cli/synthetic/pages.test.js`
- `__tests__/cli/site/get-snapshot-metrics.test.js` → `__tests__/cli/synthetic/get-snapshot-metrics.test.js`
Each test should invoke the NEW command path (e.g., `synthetic snapshots --site=test`) and update snapshot expectations.
### 11.2 Deprecation tests
**File:** `__tests__/cli/deprecation.test.js`
Verify for a sample of deprecated commands:
1. The command still works (stdout output matches expected)
2. stderr contains the deprecation warning
3. The warning names the correct replacement command
4. `--json` output on stdout is not affected
### 11.3 New command tests
For each new CrUX/RUM command:
- Fixture file with mock GraphQL response
- Integration test using mock server
- Snapshot assertion on formatted output
- JSON output test
**New fixture files:**
- `__tests__/fixtures/cruxSummary.json`
- `__tests__/fixtures/cruxHistory.json`
- `__tests__/fixtures/cruxUrls.json`
- `__tests__/fixtures/cruxUrl.json`
- `__tests__/fixtures/rumSummary.json`
- `__tests__/fixtures/rumHistory.json`
- `__tests__/fixtures/rumPages.json`
- `__tests__/fixtures/rumConfig.json`
### 11.4 Output stability tests
For **every** moved command, add an explicit test that verifies the JSON output shape is unchanged when invoked via the new path. This is critical because existing scripts consume this output. The test should call the command with `--json`, parse the output, and assert the expected top-level keys are present.
Commands requiring stability tests:
- `synthetic pages`, `synthetic create-page`, `synthetic update-page`, `synthetic delete-page`
- `synthetic snapshots`, `synthetic create-snapshot`, `synthetic delete-snapshot`
- `synthetic download-artifacts`, `synthetic get-snapshot-metrics`, `synthetic metrics`
- `synthetic test-profiles`, `synthetic create-test-profile`, `synthetic update-test-profile`, `synthetic delete-test-profile`
- `synthetic pull-request-reviews`, `synthetic create-pull-request-review`, `synthetic pull-request-review`
- `deploy list`, `deploy create`, `deploy delete`
---
## Phase 12: Documentation
### 12.1 `CLI_COMMANDS.md`
Regenerated via `npm run generate-cli-md`. This script calls `getCommandMetaData()` from `src/cli-metadata.js`, which reads the exported `commands` arrays from each command group.
How deprecated commands stay hidden:
- `site.js` exports a `commands` array with only 3 entries (create, list, delete)
- The deprecated wrappers are registered in the `builder` function (for yargs runtime) but NOT in the exported `commands` array (for metadata)
- `cli-metadata.js` reads `commands` for subcommand enumeration, so deprecated entries never appear
The `generate-cli-md.js` script iterates all commands from `cli-commands.js` and formats them using `formatHeader` and `formatOptions`. Since `cli-commands.js` now includes `Synthetic`, `Deploy`, `Crux`, and `Rum`, these all appear in the generated docs.
**Verification:** After all code changes, run `npm run generate-cli-md` and confirm:
1. No deprecated `site <command>` entries appear (no `site pages`, `site deploys`, etc.)
2. All new `synthetic`, `deploy`, `crux`, `rum` commands appear with correct flags
3. `site` section shows only `create`, `list`, `delete`
### 12.2 `README.md`
Update:
- Features section to mention Synthetic, CrUX, and RUM
- Usage examples to show new command namespaces
- Package exports to include `Crux` and `Rum`
- Remove "Google Lighthouse" from positioning — it's one component of Synthetic
- Update the "see all commands" link to reference the new structure
### 12.3 `examples/bash/`
Update existing:
- `create-test.sh` — unchanged (test commands didn't move)
- `get-agent-ip-address.sh` — unchanged
Add new:
- `crux-summary.sh` — fetch CrUX summary for a site
- `rum-pages.sh` — fetch RUM page breakdown sorted by LCP
- `deploy-create.sh` — create a deploy using new `deploy create` path
Update `examples/bash/README.md` to list new examples.
### 12.4 `examples/nodejs/`
Update existing:
- `examples/nodejs/README.md` — reflect new command structure and list new examples
- Existing deploy examples (`deploys/create.js`, etc.) — these use the `Deploy` Node.js API export which is unchanged, so the code works but we should update any CLI command references in comments
Add new:
- `crux/summary.js` — fetch CrUX summary
- `crux/history.js` — fetch CrUX history
- `crux/urls.js` — list CrUX URLs
- `rum/summary.js` — fetch RUM dashboard
- `rum/pages.js` — fetch RUM pages
- `rum/config.js` — fetch RUM configuration
### 12.5 `package.json`
- Version: `7.0.0`
- Description: Update from "Page speed performance testing with Google Lighthouse" to "Performance monitoring with Synthetic testing, Chrome UX Report, and Real User Metrics"
- Keywords: Update to include "rum", "crux", "web-vitals", "real-user-monitoring"
### 12.6 `CHANGELOG.md`
Prepend the content from `CHANGELOG-v7-DRAFT.md`, then delete the draft file.
---
## Phase 13: Verification
1. `npm run lint` — ESLint passes with zero warnings/errors
2. `npm test` — all tests pass (updated + new)
3. `npm run build` — esbuild bundles `dist/index.cjs` successfully
4. `npm run generate-cli-md` — produces correct `CLI_COMMANDS.md`:
- No deprecated `site <command>` entries visible
- All new `synthetic`, `deploy`, `crux`, `rum` commands present
- `site` section shows only `create`, `list`, `delete`
- All flags documented correctly
5. `calibre --help` — verify command list shows new structure, no deprecated commands
6. `calibre site --help` — verify only `create`, `list`, `delete` shown
7. `calibre synthetic --help` — verify all 17 subcommands listed
8. Manual smoke test: deprecated command (e.g., `calibre site pages --site=x --json`):
- stderr: deprecation warning
- stdout: valid JSON, identical to `calibre synthetic pages --site=x --json`
9. `CALIBRE_SUPPRESS_DEPRECATIONS=1 calibre site pages --site=x` — verify no warning
10. Manual smoke test of new commands against local API (`CALIBRE_HOST=http://localhost:3000`)
---
## Execution Order Summary
| Phase | Description | Depends on |
|-------|-------------|-----------|
| 1 | Foundation (deprecation util, shared options) | — |
| 2 | Move synthetic commands | Phase 1 |
| 3 | Move deploy commands | Phase 1 |
| 4 | Wire deprecations in site.js | Phases 2, 3 |
| 5 | Update cli-commands.js | Phases 2, 3, 4 |
| 6 | Enhance site list | — |
| 7 | Enhance metric-list | — |
| 8 | CrUX commands | Phase 1 |
| 9 | RUM commands | Phase 1 |
| 10 | Node.js exports | Phases 8, 9 |
| 11 | Testing | Phases 2–10 |
| 12 | Documentation | Phases 2–10 |
| 13 | Verification | All |
Phases 6, 7, 8, 9 can run in parallel once Phase 1 is complete.
---
## Critical Path
The minimum viable v7 release requires: Phases 1–5 (restructure), Phase 11 (tests), Phase 12 (docs), Phase 13 (verify). CrUX and RUM commands (Phases 8, 9) could ship in a 7.1 if needed, but the goal is to ship everything together.