eve

--- title: "Assertions" description: "Scoped methods, value assertions, the matcher mini-language, and gate vs soft severity." --- Assertions are how an eval grades what its `test(t)` function produced. Each one records a result and returns a chainable handle. The runner reads every recorded result to compute the verdict, so a single run reports every failing assertion rather than dying on the first. There are two deterministic surfaces: scoped methods and `t.check` for grading a specific value. For model-graded assertions, see [Judge](./judge). ## Scoped assertions Scoped assertions take no explicit value and gate by default. Assertions on `t` inspect the whole run after `test` finishes. Session assertions snapshot that session when called, and turn assertions inspect one immutable response. A scope is **parked** when it paused on unanswered human-in-the-loop (HITL) input. | Assertion | Asserts | | ----------------------------------------------------- | ----------------------------------------------------------------------------- | | `t.succeeded()` | The run did not fail and did not park on unanswered HITL input | | `t.parked()` | The run cleanly parked on HITL input | | `t.messageIncludes(token)` | Joined assistant text contains `token` (string or RegExp) | | `turn.outputEquals(value)` / `.outputMatches(schema)` | Deep equality or Standard Schema validation of turn/session structured output | | `t.calledTool(name, opts?)` | A matching tool call completed (`input`, `output`, `status`, `count`) | | `t.loadedSkill(skill, opts?)` | Sugar for `t.calledTool("load_skill", { input: { skill }, ...opts })` | | `t.notCalledTool(name)` | No request for `name` in any lifecycle state | | `t.toolOrder([...names])` | Tool requests appear in order | | `t.usedNoTools()` | No tool calls at all | | `t.maxToolCalls(n)` | At most `n` tool calls | | `t.noFailedActions()` | No tool, subagent, or skill action reported a failure | | `t.calledSubagent(name, opts?)` | A subagent delegation happened (identity, remote, output, status, count) | | `t.event(type, opts?)` / `t.notEvent(type, opts?)` | Typed event presence, data, and count matching | | `t.eventOrder([...matchers])` | Matching event groups occur in order | | `t.eventsSatisfy(label, predicate)` | Escape hatch: any predicate over the typed event stream | `succeeded()` accepts both a closed session and a healthy session left open for the next user message; it rejects protocol failures and unanswered HITL. `parked()` requires a clean HITL park. Structured output assertions live on turns and independent sessions, where the output is unambiguous (see the [output schema guide](../guides/client/output-schema)). ```ts await t.send("What is the weather in Brooklyn?"); t.succeeded(); t.calledTool("get_weather"); ``` The same vocabulary narrows naturally in multi-turn and externally-created session evals: ```ts const first = await t.send("Call get_weather for Brooklyn"); first.calledTool("get_weather", { count: 1 }); const attached = await t.target.attachSession(sessionId); attached.succeeded(); attached.messageIncludes("Sunny"); ``` `t.calledTool` and `t.usedNoTools` are mutually exclusive; assert one or the other, never both in the same run. ## Value assertions with `t.check` `t.check(value, assertion)` grades an explicit value against a builder from `eve/evals/expect`. The value can be `t.reply`, a turn's `.message`, parsed JSON, or any local you computed: ```ts import { includes, equals, matches, satisfies, similarity } from "eve/evals/expect"; t.check(t.reply, includes(/sunny/i)); // substring or RegExp (gate) t.check(parsed, equals({ city: "Brooklyn" })); // deep structural equality (gate) t.check(parsed, matches(WeatherSchema)); // Standard Schema, e.g. Zod (gate) t.check(t.reply, similarity("Sunny, 72F")); // fuzzy 0–1 Levenshtein (soft) t.check( latencyMs, satisfies((value) => value < 1_000, "latency under one second"), ); ``` | Builder | Scores | Default | | ---------------------- | ------------------------------------------------------- | ------- | | `includes(value)` | coerced string contains a substring or matches a RegExp | gate | | `equals(value)` | deep structural equality | gate | | `matches(schema)` | validates against a Standard Schema | gate | | `similarity(expected)` | normalized Levenshtein similarity, 1 = identical | soft | | `satisfies(fn, label)` | custom boolean predicate | gate | Pick the cheapest builder that captures what "correct" means. When exact match is too strict but a judge model is overkill, `similarity` is the middle ground. For nuanced grading, reach for the [judge](./judge). ## The matcher mini-language `t.calledTool` and `t.calledSubagent` take matcher objects. Tools accept `{ input, output, status, count }`; subagents accept `{ callId, childSessionId, remoteUrl, output, status, count }`. Calls match `status: "completed"` by default; use `"pending"`, `"failed"`, or `"rejected"` explicitly for lifecycle checks. A numeric `count` requires an exact number of calls matching every supplied constraint. Use a predicate for ranges or other custom count requirements; it receives the observed number of matching calls. Matcher values accept a literal (objects partial-deep-match), a RegExp, or a predicate function that returns a boolean: ```ts t.calledTool("bash", { input: { command: /^pwd/ }, count: 1 }); t.calledTool("echo", { count: (count) => count >= 2 }); t.calledTool("echo", { output: (value) => String(value).includes(marker) }); parked.calledTool("guarded", { status: "pending", count: 1 }); t.calledTool("guarded", { output: /approved/, count: 1 }); t.calledSubagent("weather", { remoteUrl: (value) => value === process.env.WEATHER_AGENT_URL, output: /72F/, }); ``` `requireInputRequest` uses the same matcher language for `input`, `prompt`, and `display`. Its `optionIds` matcher receives option ids in request order; a literal array must match that complete ordered list exactly: ```ts const request = session.requireInputRequest({ toolName: "ask_question", optionIds: ["red", "blue"], }); ``` ## Run state and derived facts Beyond the raw `t.events` stream, the runner derives typed facts the assertions read: tool calls (name, input, output, lifecycle status), subagent calls, and HITL input requests. A turn that leaves the session open for a next message is the normal end state of a successful turn; parking on unanswered HITL input is tracked separately. Typed event matching covers presence, absence, numeric or predicate counts, partial event data, and ordering: ```ts turn.notEvent("result.completed"); turn.eventOrder([ { type: "subagent.called", data: { name: "researcher" }, count: 2 }, { type: "subagent.completed", data: { subagentName: "researcher" }, count: 2 }, ]); ``` When a protocol invariant needs cross-event correlation, `eventsSatisfy` remains the escape hatch: ```ts t.eventsSatisfy("assistant reply includes the marker", (events) => events.some((e) => e.type === "message.completed" && e.data.message?.includes(marker)), ); ``` ## Required preconditions Recorded assertions never throw and are not awaitable. When later control flow depends on a value, use `await t.require(value, assertion)`. It records a gate, returns the original value when it passes, and stops the test body without adding a duplicate execution error when it fails: ```ts await t.require( sessionIds, satisfies((ids) => ids.length > 0, "dispatch started a session"), ); await t.target.attachSession(sessionIds[0]!); ``` Use the matching `require*` lookups when dependent code needs protocol data: ```ts const call = turn.requireToolCall("search"); const request = session.requireInputRequest({ toolName: "guarded" }); ``` ## Severity Every assertion returns a chainable handle. Severity rides on the assertion, so there is no separate thresholds map to keep in sync. - `.gate(threshold?)` is hard. A miss marks the eval `failed` and `eve eval` exits non-zero. - `.soft(threshold?)` is tracked data. A below-threshold miss marks the eval `scored`, fatal only under `--strict`. With no threshold, it is tracked-only and never fails. - `.atLeast(threshold)` is soft with a bar (equivalent to `.soft(threshold)`). The defaults are chosen so you rarely set severity. Run-level methods and `includes`/`equals`/`matches` are gates; `similarity` and every `t.judge.*` assertion are soft. Annotate only when you deviate: ```ts t.calledTool("get_weather").soft(); // record the tool call as a metric, don't gate t.check(t.reply, similarity("Sunny")).atLeast(0.8); // gate the fuzzy match under --strict t.check(t.reply, includes("error")).soft(); // track without failing the build ``` ## What to read next - [Judge](./judge): LLM-graded assertions with thresholds - [Cases](./cases): where assertions attach - [Running evals](./running): how verdicts map to exit codes