UNPKG

openclaw

Version:

Multi-channel AI gateway with extensible messaging integrations

github.com/openclaw/openclaw

openclaw/openclaw

60 lines (49 loc) • 2.98 kB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
--- doc-schema-version: 1 summary: "How taxonomy coverage IDs, scenario YAML, and profiles select what a QA run executes, plus the Multipass suite runner." read_when: - You are choosing which scenarios a run covers - You need the Multipass suite lane title: "Canonical scenario coverage" --- ## Canonical scenario coverage The root `taxonomy.yaml` defines semantic coverage IDs. Scenario YAML files under `qa/scenarios/` map each scenario to those IDs and own execution metadata; `execution.channel` or `execution.channels` declares channel requirements. Taxonomy profiles select coverage IDs or whole categories, and the catalog resolves their primary scenario owners. Transport runners apply channel and provider eligibility to that result instead of keeping scenario-ID allowlists. The channel driver is an interchangeable run-level implementation choice. For `qa suite`, `qa run --qa-profile`, and dedicated transport QA commands such as `qa telegram`, omit `--scenario` to use the default selection. When supplied, at least one non-empty scenario ID is required; surrounding whitespace and blank values alongside valid IDs are ignored. An all-blank selection is rejected before the transport runner is dispatched. Static `qa coverage` output reports the taxonomy-to-scenario mapping. Actual proof comes from `qa-evidence.json`, which records the executed scenario, coverage IDs, channel, driver actually used, and result. Channel and driver are report dimensions, not additional coverage-ID vocabularies or scenario eligibility axes. For a disposable Linux VM lane without bringing Docker into the QA path, run: ```bash pnpm openclaw qa suite --runner multipass --scenario channel-chat-baseline ``` This boots a fresh Multipass guest, installs dependencies, builds OpenClaw inside the guest, runs `qa suite`, then copies the normal QA report and summary back into `.artifacts/qa-e2e/...` on the host. It reuses the same scenario-selection behavior as `qa suite` on the host. Host and Multipass suite runs execute multiple selected scenarios in parallel with isolated gateway workers by default. `qa-channel` defaults to concurrency 4, capped by the selected scenario count. Use `--concurrency <count>` to tune the worker count, or `--concurrency 1` for serial execution. Use `qa run --qa-profile personal-agent --provider-mode mock-openai` for the personal assistant benchmark, or `--qa-profile observability` for the source checkout telemetry checks. CI uses the same profile resolver for `smoke-ci`; none of these selectors maintains a second scenario-ID list. The command exits non-zero when any scenario fails. Use `--allow-failures` when you want artifacts without a failing exit code. Live runs forward the supported QA auth inputs that are practical for the guest: env-based provider keys, the QA live provider config path, and `CODEX_HOME` when present. Keep `--output-dir` under the repo root so the guest can write back through the mounted workspace.