eve/docs/evals/cases.mdx

Filesystem-first framework for durable backend AI agents that run anywhere.

---
title: "Cases"
description: "Author single-turn and multi-turn evals with test(t), and fan one file out over a dataset."
---

Each eval file is one graded case by default, and a single file can fan out over a dataset by default-exporting an array (covered below). The runner executes each `test(t)` function against the target, captures every event, and computes a verdict from the [assertions](./assertions) you recorded. Every eval shares one shape, whether single-turn, multi-turn, human-in-the-loop (HITL), or dataset-driven: one `async test(t)` function that drives the agent and asserts inline.

Before adding a case, create the required config at the root of `evals/`. An empty config is enough when you do not need shared judge, reporter, concurrency, or timeout settings:

```ts title="evals/evals.config.ts"
import { defineEvalConfig } from "eve/evals";

export default defineEvalConfig({});
```

## Single-turn evals

The common case sends one turn and asserts on the reply. `t.send(input)` resolves once the turn settles, and the returned turn’s `.message` is the assistant reply:

```ts title="evals/weather/brooklyn-forecast.eval.ts"
import { defineEval } from "eve/evals";
import { includes } from "eve/evals/expect";

export default defineEval({
  async test(t) {
    const turn = await t.send("What is the weather in Brooklyn?");
    t.succeeded();
    t.check(turn.message, includes("Sunny"));
  },
});
```

Some evals only care about behavior, not text. Assert on the run and skip the content check entirely:

```ts title="evals/weather/no-tools-for-greetings.eval.ts"
import { defineEval } from "eve/evals";

export default defineEval({
  async test(t) {
    await t.send("Hello!");
    t.succeeded();
    t.notCalledTool("get_weather");
  },
});
```

## Organizing with directories

Identity is the file path, so directories are the grouping mechanism. `evals/weather/brooklyn-forecast.eval.ts` gets the id `weather/brooklyn-forecast`, and `eve eval weather` runs everything under `evals/weather/`. Shared constants and helpers live in sibling non-eval files (any name that doesn't end in `.eval.ts`):

```text
evals/
├── evals.config.ts
├── weather/
│   ├── shared.ts                    # helpers, not an eval
│   ├── brooklyn-forecast.eval.ts
│   └── no-tools-for-greetings.eval.ts
└── smoke.eval.ts
```

## Multi-turn evals

Drive several turns in sequence for branching, HITL approvals, structured output, attachments, or multiple sessions. Because assertions live in the function, an intermediate value is a local variable. Keep each turn in a local variable and continue through its `.session`.

```ts title="evals/draft-then-send.eval.ts"
import { defineEval } from "eve/evals";
import { includes } from "eve/evals/expect";

export default defineEval({
  async test(t) {
    const draft = await t.send("Draft the follow-up email.");
    t.check(draft.message, includes("Best regards"));
    t.judge("professional tone", { on: draft.message }).atLeast(0.6);

    await draft.session.send("Now send it.");
    t.calledTool("send_email");
  },
});
```

Use scoped assertions for intermediate turns. When later control flow depends on a value-level check, `t.require` records a gate and stops the script if it fails:

```ts title="evals/session-continuity.eval.ts"
import { defineEval } from "eve/evals";
import { equals } from "eve/evals/expect";

export default defineEval({
  async test(t) {
    const first = await t.send("My favorite word is marigold.");

    const second = await first.session.send("What is my favorite word?");
    await t.require(second.sessionId, equals(first.sessionId));

    t.succeeded();
    second.messageIncludes("marigold");

    t.judge("The assistant remembers the user's favorite word across turns", {
      on: first.session.transcript,
    }).atLeast(0.8);
  },
});
```

## The drive API

Every `t.session()` or `t.send()` creates a new independent conversation. Continue an existing conversation through its session handle. Events from all sessions feed the same run-level assertions.

- `t.session(options?)` creates a session without starting a turn or consuming its stream. It resolves on `202 Accepted` with an `EveEvalSession` whose `.sessionId` and `.state` are available. Options accept per-request `headers` and `signal`; creation uses the eval timeout signal by default.
- `t.send(message, options?)` creates a new session with its first message in one request and waits for the turn to settle. The returned turn carries `.session`, `.message`, and `.expectOk()`.
- `session.send(message, options?)` sends a follow-up on that session and waits for it to settle.
- `session.start(message, options?)` starts a text turn and returns as soon as the server accepts it. The live turn exposes `.session`, `.sessionId`, `.waitForEvent(...)`, `.cancel()`, and `.result()`.
- `session.cancel()` requests cooperative cancellation of that session's active turn. Both `accepted` and `no_active_turn` are successful outcomes.
- `session.sendFile(text, path, mediaType?)` attaches a local file as a data URL.
- `session.requireInputRequest(filter?)` records a gate, requires exactly one pending request, and returns it. Filters match tool name, action input, prompt, display, and option ids.
- `session.respond(responses, options?)` answers specific pending input requests and sends them as the next turn. `session.startRespond(...)` returns a live turn for the same operation.
- `session.respondAll(optionId)` answers every pending input request with the same option and sends the responses as the next turn.
- `session.events` contains the events captured for that session. `session.transcript` formats its observed user and assistant messages in turn order.

The transcript uses `User:` and `Assistant:` labels separated by blank lines. It excludes reasoning, tool calls, tool results, and messages from other sessions. It updates after each turn settles, so read it after `await session.send(...)`, `await session.respond(...)`, or `await live.result()`.

Each `send` (and `respond`/`respondAll`) resolves to a turn with `.message`, `.data`, `.events`, `.inputRequests`, `.toolCalls`, `.session`, `.sessionId`, `.status`, and `.expectOk()`. Turn results retain their captured output and events; `.session` references the live conversation handle. `expectOk()` throws only when the turn ended failed; a session left open for a next message is the normal end state of a successful turn.

## Prewarming a session

Use `t.session()` to create a session before its first message:

```ts title="evals/prewarm.eval.ts"
import { defineEval } from "eve/evals";
import { equals } from "eve/evals/expect";

export default defineEval({
  async test(t) {
    const session = await t.session();
    const turn = await session.send("Greet Alice briefly.");
    await t.require(turn.sessionId, equals(session.sessionId));
    turn.event("session.started", { count: 1 });
    turn.event("turn.started", { count: 1, data: { turnId: "turn_0" } });
    t.succeeded();
  },
});
```

Acceptance is not an initialization barrier. The first message runs session initialization;
the TypeScript client handles bounded retries if the workflow inbox is still starting.
Creation failures reject `t.session()` without returning a handle. Each call creates an
independent session, including concurrent calls. Accepted sessions appear in eval results
and participate in timeout cleanup even when no message was sent.

## Migrating existing evals

Replace `t.newSession()` and `t.prewarm()` with `const session = await t.session()`.
Replace consecutive `t.send()` calls with `session.send()` calls, or continue the first
returned turn through `turn.session.send()`. Read identity, cursor, events, and transcript
from `session`; read replies from `turn.message`. Session operations such as `respond()`,
`requireInputRequest()`, and `cancel()` also move from `t` to `session`.

## In-flight turns

Use `start()` when the eval must observe or affect a turn before it settles. A live turn owns one stream consumer: `waitForEvent()` reads typed events from its buffer, and `result()` waits for the boundary and records the same buffered stream as an immutable turn. Event data matchers use the same partial-deep matcher language as `t.event(...)`.

```ts title="evals/cancel-running-tool.eval.ts"
import { defineEval } from "eve/evals";

export default defineEval({
  async test(t) {
    const session = await t.session();
    const live = await session.start("Run the long operation.");

    await live.waitForEvent("actions.requested", {
      data: {
        actions: (actions) =>
          actions.some(
            (action) => action.kind === "tool-call" && action.toolName === "long_operation",
          ),
      },
    });

    await live.cancel();
    const turn = await live.result();
    turn.eventOrder([{ type: "turn.cancelled" }, { type: "session.waiting" }]);
  },
});
```

`waitForEvent()` rejects if the stream fails or reaches its turn boundary before the requested event. Call `result()` after any coordination to settle the stream and make its events available to run-level assertions.

Use `expectOk()` only when the next operation depends on that intermediate turn succeeding. A final `t.succeeded()` already records a complete-run gate.

To intentionally omit an eval for the current target, call `t.skip(reason)` before creating sessions, sending messages, or recording assertions. Skipped evals are reported separately and do not affect the exit code.

Events from every session are captured in the result and artifacts. `t.log(message)` records debug lines into the eval artifact; `--verbose` also streams them to stdout as evals run. `t.signal` is an `AbortSignal` that fires on timeout.

For driving sessions created outside the eval, by a channel webhook or a schedule, see [Targets](./targets).

## Datasets: exporting an array

To fan one file out over a dataset, default-export an array of `defineEval(...)` values. Eval modules are ESM, so top-level `await` can load anything. Ids derive from the file name plus a zero-padded index in array order (`sql/0000`, `sql/0001`, and so on). The loaders (`loadJson`, `loadYaml` from `eve/evals/loaders`) parse fixture files relative to the app root:

```ts title="evals/sql.eval.ts"
import { defineEval } from "eve/evals";
import { loadYaml } from "eve/evals/loaders";
import { equals } from "eve/evals/expect";

const doc = await loadYaml("evals/data/cases.yaml");
const rows = doc.evals as readonly { task: string; prompt: string; sql: string }[];

export default rows.map((row) =>
  defineEval({
    description: row.task,
    async test(t) {
      const turn = await t.send(row.prompt);
      t.succeeded();
      t.check(turn.message, equals(row.sql));
    },
  }),
);
```

The loaders are meant for fixtures, not runtime agent code.

## What to read next

- [Assertions](./assertions): assert on what the eval did
- [Judge](./judge): grade quality with an LLM judge
- [TypeScript client](../guides/client/messages): the send/turn protocol eval sessions build on