eve/docs/evals/judge.mdx

Filesystem-first framework for durable backend AI agents that run anywhere.

---
title: "Judge"
description: "Grade evals with evaluation models using criteria, typed questions, or batches, and set thresholds on each assertion."
---

Use `t.judge(...)` when a deterministic [assertion](./assertions) cannot capture what a good response looks like. It calls [`evaluate` from `eve/ai`](../guides/evaluate#evaluate-inside-a-tool) and defaults to `typesafe-ai/jev`, TypeSafe AI's [Jev evaluation model](https://vercel.com/i/what-is-jev). The judge is separate from the agent under test and is used only for scoring.

```ts
import { defineEval } from "eve/evals";

export default defineEval({
  async test(t) {
    await t.send("Explain quantum tunneling to a 10-year-old.");
    t.succeeded();
    t.judge("The response uses no math beyond arithmetic.").atLeast(0.8);
  },
});
```

No judge configuration is required. Configure credentials for the evaluator as described under [Configuring the judge model](#configuring-the-judge-model).

## The graders

`t.judge` accepts a criteria string, one typed question, or a batch of named questions. A string becomes a boolean question asking whether the response meets those criteria. A single question returns an assertion handle; a batch returns one handle per question.

| Question                     | Assertion score                                                 |
| ---------------------------- | --------------------------------------------------------------- |
| Criteria string or `boolean` | The returned probability of true, from 0 to 1                   |
| `score`                      | The returned rubric position divided by the highest level index |
| `choice`                     | 1 when the selected option equals `expected`; otherwise 0       |

Boolean scores are model estimates, not guaranteed calibrated confidence. A rubric needs at least two levels, ordered from worst to best. The SDK can return a fractional position on that rubric; eve normalizes it to the 0–1 scale used by assertion thresholds.

```ts
t.judge({
  type: "score",
  instructions: "Grade the response's clarity.",
  criteria: ["Unclear", "Mostly clear", "Clear and concise"],
})
  .label("clarity")
  .atLeast(0.75);
```

Choice questions require an `expected` option key. eve uses that expectation locally for scoring; it does not send it to the evaluator. Instructions and rubric descriptions accept strings, JSON objects, or JSON arrays. Descriptions can also be `null`.

## Choose the value to grade

By default, the evaluator receives `{ input, output }`: the latest prompt and assistant message from the session whose turn most recently settled. Approval responses and stream reads preserve that prompt. For multipart messages, including `session.sendFile()`, the prompt contains the text parts.

Pass `on` to replace the output. JSON values stay structured rather than being converted to strings:

```ts
const draft = await t.send("Draft the welcome email.");
t.judge("The response has a professional tone.", { on: draft.message }).atLeast(0.6);
```

For a multi-turn eval, pass `session.transcript`:

```ts
const { session } = await t.send("My favorite word is marigold. Remember it.");
await session.send("What is my favorite word?");
t.judge("The assistant remembers the user's favorite word across turns.", {
  on: session.transcript,
}).atLeast(0.8);
```

The transcript contains user and assistant messages in turn order, excluding reasoning, tool calls, and tool results. See [Multi-turn evals](./cases#multi-turn-evals).

## Evaluate several questions together

A batch makes one evaluation request against one shared state. Pass `state` to replace the default `{ input, output }` completely. Batch calls use `state` instead of `on`.

```ts
const judgments = t.judge({
  state: { response: turn.message ?? "", reference },
  questions: {
    accurate: {
      type: "boolean",
      instructions: "Is the response consistent with the reference?",
    },
    outcome: {
      type: "choice",
      instructions: "Classify the response.",
      criteria: {
        answered: "Answers the request",
        declined: "Declines the request",
      },
      expected: "answered",
    },
  },
});
judgments.accurate.atLeast(0.9);
judgments.outcome.gate();
```

Keys identify questions and their assertion handles; instructions must explain what to judge. Every question sees the same state. Use separate calls for unrelated inputs. A batch must contain at least one question, and a provider or validation failure fails every assertion in the batch.

## Soft scoring and thresholds

Judges are soft by default:

- Without a threshold, the score is tracked in reports and artifacts without failing the eval.
- `.atLeast(threshold)` sets a soft bar. A lower score marks the eval `scored`, fatal only under `eve eval --strict`.
- `.gate(threshold)` makes a lower score fail the eval outright. An omitted gate threshold is 1.

```ts
t.judge("The response cites a source.");
t.judge("The response cites a source.").label("citation").atLeast(0.8);
t.judge("The response cites a source.").gate(0.9);
```

A boolean probability need not reach exactly 1 even for a good response, so choose an explicit gate threshold for boolean judgments. Choice scores are binary.

Calls capture their state and questions and start immediately. The runner waits for them during finalization; handles are not awaitable. Each standalone call makes its own request, while a batch shares one request. Eval timeouts cancel pending judgments and stop waiting even if a provider ignores cancellation.

## Configuring the judge model

Model settings resolve in this order:

1. Per-call: `t.judge("…", { model, modelOptions })`.
2. Per-eval: `defineEval({ judge: { model, modelOptions }, test })`.
3. Project default: `defineEvalConfig({ judge: { model, modelOptions } })`.
4. The default model used by `evaluate`, currently `typesafe-ai/jev`.

A per-eval `judge` replaces the project's judge configuration. Per-call fields override the resolved configuration. `judge` and its `model` are optional; deterministic evals make no evaluation requests.

```ts title="evals/evals.config.ts"
import { defineEvalConfig } from "eve/evals";

export default defineEvalConfig({
  judge: { model: "typesafe-ai/jev" },
});
```

String IDs resolve through the AI SDK default provider, or Vercel AI Gateway when none is configured. Gateway uses `AI_GATEWAY_API_KEY` or Vercel OIDC credentials; eve's existing local Gateway connection is also honored by `evaluate`. An explicit provider evaluation model uses that provider's credentials:

```ts
import { openai } from "@ai-sdk/openai";

t.judge("The response addresses the request.", {
  model: openai.evaluationModel("gpt-6-luna"),
  modelOptions: { providerOptions: { openai: { reasoningEffort: "high" } } },
});
```

Use the provider's `evaluationModel` factory, not a language model instance. AI SDK supplies evaluation adapters for supported language models. Provider support and the evaluation contract are experimental; see [AI SDK evaluation](https://ai-sdk.dev/docs/ai-sdk-core/evaluation).

String model IDs resolve through the default provider's evaluation API. Gateway accepts native evaluation models such as `typesafe-ai/jev`; passing a language model ID such as `openai/gpt-6-luna` does not automatically select the OpenAI adapter. Use an evaluation-model instance as shown above for language models.

Missing credentials, invalid answers, unsupported question types, and provider errors become failed gates even for tracked-only judgments. They do not silently skip the eval or select another model. Validation, authentication, and retries use the shared `evaluate` implementation.

## Diagnostics and migration

Reports use `judge.boolean`, `judge.score`, and `judge.choice`; batch assertion names also include their question key. Diagnostics include the captured state, question, and raw answer. Assertion metadata retains the normalized score, model identity, token usage, warnings, and available provider and response metadata. Usage for a batch is marked as shared across its assertions, not a separate charge for each question. Evaluation models do not promise a prose rationale.

The autoevals graders have been removed. Replace `t.judge.autoevals.closedQA(criteria, options)` with `t.judge(criteria, options)`. Scores now represent a probability rather than a binary yes/no answer, so review thresholds against your examples. For factuality, summaries, or SQL equivalence, write an explicit question and include the reference in shared state. No old grader buckets are preserved.

Change configured language model instances to evaluation model instances. `modelOptions.providerOptions`, assertion labels, and threshold methods remain available. [Braintrust reporting](./reporters) continues to accept judge scores independently of the grader implementation.

## What to read next

- [Assertions](./assertions): deterministic run-level and value assertions
- [Reporters](./reporters): publish scores and inspect results
- [Targets](./targets): local and remote eval targets