paqad-ai/runtime/capabilities/coding/skills/design-test/SKILL.md

Spec-driven development framework — AI agents that think before they type

---
name: design-test
description: 'Audits the running UI against the project design-system contract by running the design review helpers in order and writing one report with a JSON sidecar, and retests an earlier report. Use when the workflow is design-test or design-retest.'
metadata:
  model_tier: reasoning
  cacheable: 'false'
  output_format: markdown
  paqad-meta: skill.meta.yaml
---

## What It Does

Audit the running UI against the project's own design-system contract. Mirror of the pentest workflow: same shape, same resumability, but graded against `docs/instructions/design-system/*` instead of OWASP WSTG. The contract is the standard; the framework holds no hard-coded color/spacing rules.

It runs the design review helpers in the order below and gathers their output into one report; each helper keeps its own procedure. A retest of an earlier report is the Retest mode of this skill.

## Use This When

Run this workflow whenever the user says anything equivalent to:

- "run design test", "design check", "design audit", "ui audit", "design-system audit", "check design system", "validate the ui", "validate design"

Do **not** improvise a design audit. Always follow the steps below in order. For a retest of an existing report, use the `design-retest` workflow instead.

Use the Retest mode below when the workflow is `design-retest`.

## Inputs

### Source-of-Truth Model

| Source                              | Role                                                                              |
| ----------------------------------- | --------------------------------------------------------------------------------- |
| Playwright walk of the running app  | **primary evidence**                                                              |
| UI source AST + CSS scan            | **primary evidence**                                                              |
| `docs/instructions/design-system/*` | **contract**: what reality is graded against                                      |
| `docs/modules/*/ui/**`              | **subject under audit**: never trusted as evidence; cross-checked against reality |
| Playwright test files               | **coverage evidence**: what's tested ≠ what exists; gap = finding                 |

Docs are help, not truth. Any disagreement between docs and reality produces a `documentation-drift` finding with bidirectional resolution ("either implement the state or remove the claim").

## Procedure

Progress is tracked in `.paqad/design-test/runs/<run_id>/progress.json` (same shape as pentest). The workflow is resumable: if a step is already marked `completed` with the same input hash, skip it.

### Step 0: readiness-gate

Grade `docs/instructions/design-system/` before any other step runs. The `design-system-coverage` skill produces the tier; it is always the first skill and gates everything else (analog of `stride-threat-model`).

| Tier         | Criteria                                                                    | Behavior                                                                                                                                                                                   |
| ------------ | --------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Missing**  | Directory absent, or all six contract files empty                           | **Stop.** Decision-pause packet via `AskUserQuestion`: "Run `create documentation` for the design system first?" Offers inline invocation of `documentation-update` workflow, then resume. |
| **Bare**     | Tokens partial; no component inventory; no a11y matrix                      | Warn + prompt. Run in exploratory mode, findings flagged `confidence: low`. Also emit `DT-DS-XXXX` findings listing gaps in the contract itself.                                           |
| **Adequate** | Tokens + components + a11y + at least one of {patterns, motion, responsive} | Proceed with normal scoring.                                                                                                                                                               |
| **Strong**   | All six contract files populated                                            | Proceed; if `design_test.strict: true` is set in `project-profile.yaml`, DT-blocker findings gate the dev workflow. Strict mode never auto-engages: explicit opt-in only.                  |

### Step 1: collect-context

- Load `docs/instructions/design-system/**` → the contract.
- Derive component inventory from **AST scan of `src/components/**`** + Storybook stories if present. **Do not derive from `docs/modules/\*/ui/**`**: instead audit those docs against the derived inventory.
- Load UI source: theme/style files, Tailwind config (`tailwind.config.*`), design-token sources (`src/design-tokens/`), CSS modules / CSS-in-JS.
- Load Playwright tests as coverage evidence.
- Emit `contract-summary.json`, `surface-summary.json`, `test-summary.json` to `.paqad/design-test/runs/<run_id>/artifacts/`.

Skills run **in this order** (`design-system-coverage` gates everything else, like STRIDE):

1. `design-system-coverage`: already ran in Step 0; readiness inventory reused as input to every downstream skill
2. `token-conformance-review`: scan UI source for hard-coded values that should reference tokens. **High severity by default**, hex literals, raw `px`/`rem`/`em`, font stacks not in `tokens.md`, color names not registered as tokens. The presence of a `primary` token while a component inlines `#1a73e8` is a finding, not a stylistic preference.
3. `component-conformance-review`: every used component must be in `components.md`; `!important`, inline `style=`, shadcn primitives wrapped without spec, undocumented Tailwind utility combinations → findings
4. `state-coverage-review`: every component in the AST-derived inventory must implement its declared states (default/hover/focus/disabled/loading/error/empty) AND be exercised by Playwright
5. `accessibility-review`: WCAG 2.2 A/AA: contrast, focus visibility, keyboard order, ARIA, target size, prefers-reduced-motion. Findings tagged `WCAG-2.2.x.x.x` (analog of WSTG)
6. `responsive-review`: declared breakpoints exercised; no horizontal scroll; touch targets ≥ spec
7. `motion-review`: animations match `motion.md`; `prefers-reduced-motion` respected
8. `copy-and-ia-review`: voice/tone/labeling consistency vs `patterns.md`

### Step 2: run-project-scripts

Five scripts, run in parallel, **shipped inside paqad-ai** (not seeded into the project: see "Footprint policy" below):

1. `runtime/scripts/design/scan-tokens.sh`: AST/grep for hard-coded design values
2. `runtime/scripts/design/scan-overrides.sh`: `!important`, inline `style=`, undocumented Tailwind class combos
3. `runtime/scripts/design/enumerate-surface.sh`: screen/route inventory (Next `app/` / `pages/` / framework router)
4. `runtime/scripts/design/axe-static.sh`: `axe-core` against component snapshots without booting the app
5. `runtime/scripts/design/coverage.sh`: Playwright tests cross-referenced against the AST-derived component inventory

Artefacts → `.paqad/design-test/runs/<run_id>/artifacts/`.

### Step 2.5: generate-local-design-playbook

For each suspected finding category, emit `docs/design-test/<timestamp>-local-playbook.md` with a Playwright snippet per category: same "break locally first" pattern pentest uses with curl. **Templates only, never auto-executed.** Categories: contrast probe, focus-ring probe, motion respect, touch-target probe, token-leak probe.

Example playbook entries:

```ts
// contrast probe: assert text/bg pair meets WCAG-2.2-1.4.3 (4.5:1)
import { test, expect } from '@playwright/test';
import AxeBuilder from '@axe-core/playwright';
test('contrast on /pricing', async ({ page }) => {
  await page.goto('/pricing');
  const results = await new AxeBuilder({ page }).withTags(['wcag2aa']).analyze();
  expect(results.violations.filter((v) => v.id === 'color-contrast')).toEqual([]);
});

// focus-ring probe: keyboard reach must produce a visible focus outline
test('focus ring visible on primary button', async ({ page }) => {
  await page.goto('/');
  await page.keyboard.press('Tab');
  const outline = await page
    .locator(':focus-visible')
    .evaluate((el) => getComputedStyle(el).outlineWidth);
  expect(outline).not.toBe('0px');
});

// token-leak probe: computed color must resolve to a declared token
test('button bg = color.primary.500', async ({ page }) => {
  await page.goto('/');
  const bg = await page
    .locator('button.primary')
    .evaluate((el) => getComputedStyle(el).backgroundColor);
  expect(bg).toBe('rgb(26, 115, 232)'); // declared in tokens.md as color.primary.500
});
```

### Step 3: live-validation (Playwright phase)

Equivalent of pentest's live-validation step. The runner brings up the app via `design_test.dev_command` (from the `design_test` block in `project-profile.yaml`) and walks it. It **reuses the project's `playwright.config.ts`** (browsers, projects, baseURL): overridable through the `design_test` config block.

- **Surface walk**: visit every route from the inventory; screenshot at declared breakpoints
- **Token diff**: inspect computed styles (color, font-family, font-size, line-height, radius, spacing, shadow) at each route; diff against `tokens.md`. One finding per mismatched property. This is the padding/margin/font/color/border check, mechanized.
- **State sweep**: for every component in the inventory, drive its declared states via Playwright
- **A11y dynamic**: `@axe-core/playwright` against each route in default + dark-mode + reduced-motion + RTL (if declared)
- **Visual regression**: first run creates a baseline in `.paqad/design-test/baselines/` (git-ignored); subsequent runs diff; `paqad design accept-baseline` action promotes approved diffs to a tracked path
- **Asset budget**: fonts loaded match `tokens.md`; no extra families
- **Performance budget**: LCP / CLS / INP vs declared budgets (only if `patterns.md` has a perf budget)

If the app can't be reached → record `blocked_checks` and proceed (identical to pentest behavior).

Skills consulted in this step: `accessibility-review`, `responsive-review`, `motion-review`, `state-coverage-review` (all keyed to the live walk's evidence).

### Step 4: synthesize-findings

Reuse `finding-normalizer`; the DT category vocabulary lives in its `assets/vocabulary.txt`. Stable `DT-XXXX` IDs. Each finding:

- `severity`: blocker / high / medium / low / nit
- `category`: token / component / state / a11y / responsive / motion / copy / performance / **documentation-drift**
- `surface`: screen + component
- `contract_ref`: clause in `docs/instructions/design-system/*.md` violated
- `evidence`: `file:line` for static; screenshot + selector for live
- `resolution`: concrete fix (e.g. "replace `#1a73e8` at `Button.tsx:42` with `color.primary.500` from `tokens.md`") so an LLM can resolve without re-deriving context
- `status`: open / accepted / waived (with reason) / fixed
- `playbook_ref`: link to playbook snippet that reproduces it

`token` findings default to **high** severity (raises visibility of hard-coded design values from day one). `documentation-drift` defaults to medium; rises to high if drift is in the contract itself (token declared in `tokens.md` never used, or used in code but not declared).

### Step 5: write-report

- `docs/design-test/<timestamp>.md`: human report
- `docs/design-test/<timestamp>.json`: machine sidecar (so `design-retest` preserves IDs)
- **DS-COV Coverage Matrix**: every contract clause × every route → covered/blocked (analog of WSTG matrix)
- Ordered findings (blocker → nit), remediation priorities, link to playbook

#### DS-COV Coverage Matrix Template

| Contract clause                      | Route      | Status            |
| ------------------------------------ | ---------- | ----------------- |
| tokens.md → color.primary.500        | /pricing   | covered / blocked |
| tokens.md → spacing.4                | /          | covered / blocked |
| components.md → Button > hover state | /signup    | covered / blocked |
| accessibility.md → WCAG-2.2-1.4.3    | (all)      | covered / blocked |
| motion.md → reduced-motion respected | (all)      | covered / blocked |
| responsive.md → breakpoint:sm        | (all)      | covered / blocked |
| patterns.md → override budget        | (codebase) | covered / blocked |

### Retest mode

Use this mode for the `design-retest` workflow.

Re-evaluate prior design-test findings against the current state of the UI so a retest report can classify each finding as `fixed`, `still-open`, or `needs-manual-verification` without re-deriving context. Mirror of `pentest-retest`.

Run this workflow whenever the user says anything equivalent to:

- "design retest", "design-retest", "retest design", "retest design findings"

#### Inputs

- The source `docs/design-test/<timestamp>.json` sidecar (the machine-readable companion of a prior design-test report).
- `docs/instructions/design-system/**` (contract: may have changed since the original run).
- Fresh evidence: AST scan + Playwright walk against the current working tree.

If the source sidecar path is not supplied in the request, default to the most recent file matching `docs/design-test/*.json`.

Progress is tracked in `.paqad/design-test/runs/<run_id>/progress.json` with `mode: retest` and `source_report: <path>`.

#### Step 1: load-source-findings

- Read the source sidecar. Validate against the design-test finding schema (same one `finding-normalizer` enforces).
- Build an in-memory index keyed by `DT-XXXX` id.
- Preserve every finding's original `severity`, `category`, `contract_ref`, and `playbook_ref`: retest does **not** re-derive these.

#### Step 2: gather-fresh-evidence

For each finding, gather only the evidence needed to re-classify it:

- `category: token` / `component` → re-run `scan-tokens.sh` and `scan-overrides.sh` over the file referenced in `evidence`; check whether the offending literal still exists at `file:line` (or has moved, in which case search the file).
- `category: state` / `a11y` / `responsive` / `motion` → re-run the relevant Playwright snippet from the original playbook (`playbook_ref`).
- `category: documentation-drift` → re-read the referenced `docs/modules/*/ui/**` file and the contract clause; check whether they now agree.
- `category: performance` → re-run the perf budget probe.

#### Step 3: classify-status

For each finding, pick a status from the `assets/status-vocabulary.txt` of the `retest-verification` skill:

- `fixed`: the offending evidence is gone AND the contract clause is now satisfied.
- `still-open`: the offending evidence remains.
- `needs-manual-verification`: automated evidence is inconclusive (e.g., a visual-regression diff that requires a human eye).

Findings whose `contract_ref` no longer exists in the contract get a synthetic `still-open` with a `contract-removed` note: the user removed the rule rather than fixing the code; the maintainer should confirm intent.

#### Step 4: write-retest-report

- `docs/design-test/<original-timestamp>-retest-<retest-timestamp>.md`: human report
- `docs/design-test/<original-timestamp>-retest-<retest-timestamp>.json`: machine sidecar with the same `DT-XXXX` ids and the new `status` field
- Summary block at the top: `fixed: N | still-open: N | needs-manual-verification: N | contract-removed: N`

Retest mode rules:

- Never invent new `DT-XXXX` ids during a retest. New findings discovered during retest are out of scope: run a fresh `design-test` for those.
- Never lower a finding's severity during retest. Severity is a property of the source report; retest only updates `status`.
- If the source sidecar is missing or malformed, stop and ask the user to point at a valid sidecar.
- Always write both the `.md` and `.json` retest outputs.

## Output Contract

### Footprint Policy

**Zero project footprint.** Diverges from pentest's "scripts live in the project" model: pentest scripts are meant to be tweakable by the project's security team; design-test scripts are infrastructure the project shouldn't touch. This divergence is intentional.

- All runners (scan-tokens, scan-overrides, enumerate-surface, axe-static, coverage, runtime-checks, retest) live inside paqad-ai under `runtime/scripts/design/`.
- `runtime-checks.ts` is a `.ts` file (Playwright is a Node API), run via `tsx` shipped as a paqad-ai dep.
- Stack-specific behavior is **framework code** inside the runners, they read the project's `package.json`, `playwright.config.ts`, `tailwind.config.*`, CSS-in-JS imports, and branch internally. One runner, many stacks.
- Outputs land in the project (`docs/design-test/*`, `.paqad/design-test/runs/*`): work products, not infrastructure. Same pattern as pentest.

### What Is and Isn't Project-Tunable

**Framework-owned, not project-editable:**

- This skill, including its Retest mode
- All 9 skills (the 8 review skills + `design-system-sync`)
- Runners under `runtime/scripts/design/`
- Routing rules
- Finding schema and severity vocabulary

**Project-tunable, via declarative inputs only:**

- The contract itself: `docs/instructions/design-system/*.md`
- A bounded `design_test` config block (schema-validated) in `project-profile.yaml`:
  - `design_test.strict: true|false` (default false: advisory unless opted in)
  - `design_test.app_url`
  - `design_test.dev_command`
  - `design_test.routes_include` / `routes_exclude`
  - `design_test.baselines_path`
  - `design_test.budgets` (schema framework-defined; values project-supplied)
  - `design_test.with: [visual-regression, dark-mode-parity, rtl-parity, i18n-width]` (optional checks)

**Explicitly off-limits:**

- Project-side plugins loaded by the runner
- Project-side scripts the workflow shells out to
- Project-overridable thresholds outside the declared schema
- Swapping a framework skill for a project-custom one
- Editing this skill in the project

## Escalate / Stop Conditions

- Never skip steps. If a script is unavailable, record it in `blocked_checks` and proceed.
- Do not run destructive operations (no writes to the live database, no mutations via the API). The Playwright walk is read-only.
- Always write both the `.md` report and the `.json` sidecar; the `design-retest` workflow depends on the sidecar to preserve `DT-XXXX` IDs.
- The local design playbook (Step 2.5) is **generated only**: never execute it automatically.
- `docs/modules/*/ui/**` is never evidence; any disagreement with the AST-derived inventory produces a `documentation-drift` finding.
- Hard-coded design values (hex, raw px/rem, ad-hoc font stacks) are never acceptable when a token exists. `token` findings default to high severity.

## Resources

- Helper skills, in the order the Procedure calls them: `design-system-coverage`, `token-conformance-review`, `component-conformance-review`, `state-coverage-review`, `accessibility-review`, `responsive-review`, `motion-review`, `copy-and-ia-review`, `finding-normalizer`.
- `references/helper-map.md`: read it when you need to see which helper a step calls.
- `agents/openai.yaml`