eve/docs/evals/overview.mdx

Filesystem-first framework for durable backend AI agents that run anywhere.

---
title: "Overview"
description: "Define repeatable scored checks for an eve agent with defineEval and run them with eve eval."
---

An eval is a scored check that runs your agent against real sessions and grades the result, catching regressions when you change a prompt or a tool. Drive the agent through one or more turns, assert on what it did (the run completed, the right tool ran, the reply contains the right text), and optionally ship the results to Braintrust.

Evals exercise the same HTTP surface your users hit. The runner boots (or targets) a real agent server, drives sessions through the [TypeScript client](../guides/client/overview) protocol, and grades what comes back, so a passing eval means the agent booted, accepted a request, and produced the result you asserted.

## `defineEval`

eve discovers evals under the app-root `evals/` directory, in `.eval.ts` files. Each file is one eval by default. A file can also default-export an array to fan out over a dataset (see [Cases](./cases)). The file path is the eval's identity, so you don't author an `id` or `name`. Directories group related evals (`evals/weather/brooklyn-forecast.eval.ts` becomes id `weather/brooklyn-forecast`).

```text
my-agent/
├── agent/
├── evals/
│   ├── evals.config.ts
│   ├── smoke.eval.ts
│   └── weather/
│       ├── brooklyn-forecast.eval.ts
│       └── no-tools-for-greetings.eval.ts
└── package.json
```

An eval is a single `async test(t)` function. You drive the agent with `t` and assert on the run with the same `t`:

```ts title="evals/weather/brooklyn-forecast.eval.ts"
import { defineEval } from "eve/evals";
import { includes } from "eve/evals/expect";

export default defineEval({
  description: "Basic message and tool-usage coverage for the weather agent.",
  async test(t) {
    const turn = await t.send("What is the weather in Brooklyn?");
    t.succeeded();
    t.calledTool("get_weather");
    t.check(turn.message, includes("Sunny"));
  },
});
```

`test` is the only required field. The rest are optional: `description`, `judge`, `tags`, `metadata`, `timeoutMs`, and `reporters`. The init template adds `evals/**/*.ts` to `tsconfig.json`, so your eval code type-checks alongside the app.

## `evals.config.ts`

Every `evals/` directory needs exactly one `evals.config.ts` at its root. It declares the defaults every eval shares:

```ts title="evals/evals.config.ts"
import { defineEvalConfig } from "eve/evals";
import { Braintrust } from "eve/evals/reporters";

export default defineEvalConfig({
  judge: { model: "typesafe-ai/jev" },
  reporters: [Braintrust({ projectName: "my-agent" })],
});
```

Everything is optional. `judge` sets the default model for [LLM-as-judge](./judge) assertions (`t.judge(...)`); omit it to use the default evaluator, `typesafe-ai/jev`. Deterministic evals make no judge requests. `reporters`, `maxConcurrency`, and `timeoutMs` round out the defaults. Config `reporters` observe every eval in the run, so set one `Braintrust()` here instead of adding it to each eval. CLI flags (`--max-concurrency`, `--timeout`) and per-eval values take precedence over the config defaults.

### Run setup and teardown

Use `setup` to start shared resources or run migrations before eve starts the local agent server. Return shared context directly from `setup`, then release resources in `teardown`:

```ts title="evals/evals.config.ts"
import { defineEvalConfig } from "eve/evals";
import { startTestDatabase } from "./database.js";

export default defineEvalConfig({
  async setup() {
    const database = await startTestDatabase();
    return { database };
  },
  async teardown(context) {
    await context?.database.close();
  },
});
```

`context` is inferred from the setup return value. To use that type in an eval file, pass the config's type to `defineEval`:

```ts title="evals/database.eval.ts"
import { defineEval } from "eve/evals";
import { equals } from "eve/evals/expect";
import type config from "./evals.config.js";

export default defineEval<typeof config>({
  async test(t) {
    const result = await t.context.database.query("SELECT 1 AS value");
    await t.require(result.rows[0]?.value, equals(1));
  },
});
```

Setup, eval bodies, and teardown share the same context object in the runner process. Class instances and functions keep their identity; context is neither serialized nor sent to the agent server. Concurrent evals share these resources, so isolate any database records they modify.

`setup` can also return nothing.

- Setup runs once per `eve eval` invocation, after discovery and config loading, before target startup.
- Teardown runs after the local server and its sandbox handles shut down, including after setup, startup, or eval failures. Cleanup failures fail the command.

`teardown` receives `undefined` when setup returns no context or throws before returning. In that case, setup must clean up resources that only it can access. Teardown can also be used without setup. The per-eval `timeoutMs` does not apply to setup or teardown.

## Deterministic fixture models

Use `mockModel` when an eval fixture needs to exercise eve's runtime without calling a model provider. A static fixture can be one line:

```ts title="agent/agent.ts"
import { defineAgent } from "eve";
import { mockModel } from "eve/evals";

export default defineAgent({
  model: mockModel("A deterministic reply"),
});
```

Pass a callback when the reply depends on the conversation. The callback receives an eve-owned view of the prompt, including `lastUserMessage`, `userMessages`, `userMessageCount`, available `tools`, and prior `toolResults`:

```ts title="agent/agent.ts"
export default defineAgent({
  model: mockModel(
    ({ lastUserMessage, userMessageCount }) => `Turn ${userMessageCount}: ${lastUserMessage}`,
  ),
});
```

The callback may return `{ text, toolCalls, usage }` for deterministic tool loops or explicit token counts. Use the options form only when a fixture also needs a custom model identity:

```ts title="agent/agent.ts"
model: mockModel({
  modelId: "weather-script",
  provider: "my-fixtures",
  respond: ({ toolResults }) =>
    toolResults.length === 0
      ? { toolCalls: [{ name: "get_weather", input: { city: "Brooklyn" } }] }
      : `Weather: ${JSON.stringify(toolResults[0]?.output)}`,
});
```

`mockModel()` uses `"Mock response"` when no response is supplied. It handles both generated and streamed responses, derives deterministic response metadata, and estimates token usage. Because the model is part of the agent definition, use it for a dedicated fixture agent; it remains mocked whether that fixture runs locally or as a deployed eval target.

## The `t` context

`t` is both the driver and the assertion surface. There are no separate `input`, `run`, `checks`, or `scores` fields. You write ordinary control flow, sending turns and asserting inline.

- **Drive** the agent: `t.session()` creates an empty session; `t.send(...)` creates a session with its first message and returns a turn. Continue through `turn.session.send(...)`. Session handles also expose `start`, `cancel`, `respond`, `respondAll`, `sendFile`, and `requireInputRequest`. Read output from `turn.message`, and conversation state from `turn.session`. See [Cases](./cases).
- **Assert** with three surfaces, covered next.

## Three assertion surfaces

Each surface matches a genuinely different kind of judgment:

- **Scoped methods** read the final whole run on `t`, snapshot one independent session when invoked there, or inspect one immutable `EveEvalTurn`. See [Assertions](./assertions).
- **`t.check(value, assertion)`** grades an explicit value with a deterministic builder from `eve/evals/expect`, such as `t.check(turn.message, includes("sunny"))`. Grade `turn.message`, an intermediate draft, parsed JSON, or anything else. See [Assertions](./assertions).
- **`t.judge(...)`** is the LLM-as-judge surface, like `t.judge("cites a source")`. It grades the most recently settled turn’s assistant message by default; pass `{ on: turn.session.transcript }` to grade a multi-turn conversation. The judge uses the configured judge model, never the agent under test. See [Judge](./judge).

## Gate vs soft

Every assertion returns a chainable handle, so severity rides on the assertion itself. There is no separate thresholds map.

- **Gates** are hard. A failed gate marks the eval `failed` and `eve eval` exits non-zero. Run-level methods, `includes`, `equals`, and `matches` are gates by default.
- **Soft** assertions are tracked data. They land in reports and artifacts, and a below-threshold soft assertion marks the eval `scored` (visible but not fatal, unless you pass `--strict`). `similarity` and every `t.judge(...)` assertion are soft by default. A soft assertion with no threshold is tracked-only and never fails.

Override per assertion: `.gate(threshold?)` promotes to a hard gate, `.soft(threshold?)` demotes to tracked, and `.atLeast(threshold)` is a soft assertion with a bar.

```ts
t.succeeded(); // gate
t.calledTool("get_weather").soft(); // record as a metric, don't gate
t.judge("cites a source"); // soft, tracked (no threshold)
t.judge("The response is consistent with output.reference.", {
  on: { response: turn.message ?? "", reference },
}).atLeast(0.7); // soft, gated under --strict at 0.7
```

Use `await t.require(value, assertion)` for a gate that must pass before the script can safely continue. Use `t.skip(reason)` as the first operation for an intentionally unsupported target capability.

## Run evals with eve eval

```bash
eve eval                       # run all discovered evals against a local dev server
eve eval weather               # run one eval, or every eval under evals/weather/
eve eval --url https://<app>   # target an existing server or deployment
```

Exit code `0` means every eval passed its gates. See [Running evals](./running) for the full flag list, exit codes, and CI guidance.

## A good baseline

Most apps do fine with a few small smoke evals. Assert behavior with `t.succeeded()` plus one or two content checks, keep dataset fixtures in `evals/data/`, and reach for a judge or Braintrust only when you need fuzzy grading or shared result review. In CI, run `eve eval --strict` so soft threshold misses fail the build too.

## What to read next

The rest of this section covers each piece:

- [Cases](./cases): single-turn evals, scripted multi-turn evals, and dataset fan-out
- [Assertions](./assertions): run-level methods and `t.check` value assertions, with matchers and severity
- [Judge](./judge): LLM-as-judge grading and the judge model
- [Targets](./targets): local vs remote targets for the same eval files
- [Reporters](./reporters): Braintrust experiments and JUnit XML
- [Running evals](./running): the `eve eval` CLI, exit codes, and artifacts
- [Tools](../tools): the surface most evals assert on