@mastra/core/dist/docs/references/docs-evals-multi-turn.md

> Discover all available pages from the documentation index: https://mastra.ai/llms.txt

# Multi-turn evals

Multi-turn evals test agent behavior across a conversation. Instead of a single `input`, you provide an `inputs` array. Each entry is sent sequentially to the agent on the same thread, and scorers see the accumulated output from all turns.

## When to use multi-turn evals

- The agent uses memory and must recall context from earlier turns
- The agent handles follow-up questions that depend on prior responses
- You need to verify tool-call sequences across a conversation
- The agent runs a multi-step workflow (search, confirm, execute)

## Quickstart

```typescript
import { runEvals } from '@mastra/core/evals'
import { checks } from '@mastra/evals/checks'
import { weatherAgent } from '../agents'

const result = await runEvals({
  data: [
    {
      inputs: [
        'What is the weather in Brooklyn?',
        'What about tomorrow?',
        'Compare the two forecasts.',
      ],
    },
  ],
  target: weatherAgent,
  scorers: [checks.calledTool('get_weather', { times: 2 }), checks.includes('Brooklyn')],
})
```

Each turn runs `agent.generate()` with the same thread ID, so the agent sees the full conversation history. Scorers receive the accumulated output messages from all turns.

## Memory is required for cross-turn recall

Multi-turn recall depends on the agent having a **memory store configured**. The shared thread ID is what lets each turn see the earlier ones, but a thread only persists history when the agent has memory. If the agent has no memory configured, the turns still run sequentially and their outputs still accumulate for scoring, but the agent won't recall earlier turns (each input runs in isolation). `runEvals` logs a warning when you use `inputs` on an agent without memory.

`runEvals` manages the conversation identity for you: it generates the shared `threadId` and injects a `resourceId` (Mastra memory scopes messages by resource + thread, so both are required for recall to work). By default the resource is derived from the generated thread so each conversation is isolated. To pin a specific resource, for example to reuse an existing user's memory, pass `targetOptions.memory.resource`; `runEvals` still owns the thread, so you don't provide one:

```typescript
await runEvals({
  target: weatherAgent,
  data: [{ inputs: ['What is the weather in Brooklyn?', 'What about tomorrow?'] }],
  scorers: [checks.similarity('weather forecast')],
  targetOptions: { memory: { resource: 'user-42' } },
})
```

See [Memory](https://mastra.ai/docs/memory/overview) for how to configure a memory store.

## How it works

When a data item has an `inputs` array, `runEvals`:

1. Creates a fresh thread (unique `threadId`) and resource for the conversation (a caller-provided `targetOptions.memory.resource` is preserved)
2. Sends each input sequentially via `agent.generate()` on that thread
3. Accumulates all output messages across turns
4. Passes the full accumulated output to scorers for evaluation

Scorers see the complete conversation output, including every turn.

## Scoring semantics

These details matter when writing scorers for multi-turn items:

- **`run.output` is the accumulated output from every turn.** Output-based scorers: `checks.includes`, `checks.calledTool`, `checks.similarity`, and similar: evaluate the whole conversation. For example, `checks.calledTool('get_weather', { times: 2 })` counts calls across all turns.
- **`run.input` is only the first turn's input.** Scorers that compare input against output (faithfulness, answer relevancy, and other input-relative LLM scorers) only see the first user message, not the full conversation. Prefer output-based checks for multi-turn, or build scorers that read the accumulated `run.output` directly.

## Per-turn assertions with `turns`

The `inputs` form scores the **accumulated** output as a whole, a single score over every turn's output. That can hide per-turn failures: an output-based check like `checks.includes('Brooklyn')` passes if _any_ turn mentions Brooklyn, even when the follow-up turn is broken.

When you need to assert that a **specific turn** behaved correctly, use `turns` instead. Each turn is an object with its own `input` and optional `gates`/`scorers` that evaluate **only that turn's** input and output:

```typescript
import { runEvals } from '@mastra/core/evals'
import { checks } from '@mastra/evals/checks'
import { weatherAgent } from '../agents'

const result = await runEvals({
  data: [
    {
      turns: [
        {
          input: 'What is the weather in Brooklyn?',
          gates: [checks.calledTool('get_weather')],
        },
        {
          // The follow-up must call the tool again — it can't be satisfied
          // by the first turn's tool call.
          input: 'What about tomorrow?',
          gates: [checks.calledTool('get_weather')],
          scorers: [{ scorer: checks.similarity('tomorrow forecast'), threshold: 0.5 }],
        },
      ],
    },
  ],
  target: weatherAgent,
})

result.verdict // 'passed' | 'scored' | 'failed'
result.turnResults // per-turn gate/threshold/scorer outcomes
```

Semantics:

- A per-turn gate or scorer sees **only that turn's** `run.input` and `run.output`: never the accumulated conversation. This fixes both blind spots of `inputs`: the wrong turn can't satisfy a check, and `run.input` is correct for each turn.
- Per-turn outcomes fold into the [verdict](https://mastra.ai/docs/evals/gates-and-verdicts): a failing turn gate makes the verdict `failed`. A missed turn threshold (with gates passing) makes it `scored`.
- `result.turnResults[i]` reports each turn's `gateResults`, `thresholdResults`, and `scores`, so a failure points at the exact turn. Across multiple conversations, turn results are averaged by turn index.
- A turn with no `gates` or `scorers` advances the conversation.
- Top-level `scorers`/`gates` still run as a whole over the accumulated output, so you can combine "this turn must call the tool" with "the answer mentions Brooklyn."
- When the agent has storage configured, each per-turn scorer/gate result is persisted like top-level scores, so per-turn outcomes appear in your scores store. Each stored per-turn score is labeled with its turn index (`metadata.turnIndex`), shares the conversation's `threadId`, and links to that turn's own trace span.

Use `inputs` when a single whole-conversation score over the whole conversation is enough. Use `turns` when correctness depends on individual turns. `turns` can't be combined with `input` or `inputs` in the same data item.

## Combining with gates and thresholds

Multi-turn data items work with [gates and verdicts](https://mastra.ai/docs/evals/gates-and-verdicts). Use output-based scorers so gating reflects the full conversation:

```typescript
import { runEvals } from '@mastra/core/evals'
import { checks } from '@mastra/evals/checks'

const result = await runEvals({
  data: [
    {
      inputs: ['My favorite city is Brooklyn.', 'What is the weather in my favorite city?'],
    },
  ],
  target: weatherAgent,
  gates: [checks.calledTool('get_weather')],
  scorers: [{ scorer: checks.similarity('Brooklyn weather forecast'), threshold: 0.5 }],
})

result.verdict // 'passed' | 'scored' | 'failed'
```

## Mixing single-turn and multi-turn

A single `runEvals` call can include both single-turn and multi-turn data items:

```typescript
const result = await runEvals({
  data: [
    { input: 'What is the weather in Brooklyn?' },
    {
      inputs: ['My favorite city is Brooklyn.', 'What is the weather in my favorite city?'],
    },
  ],
  target: weatherAgent,
  scorers: [checks.includes('Brooklyn')],
})
```

Single-turn items use `input` as usual. Multi-turn items use `inputs`, `input` can be omitted entirely.

## Validation

`runEvals` throws a `MastraError` if `inputs` is present but empty:

```typescript
// Throws: 'inputs' must be a non-empty array
await runEvals({
  data: [{ inputs: [] }],
  target: myAgent,
  scorers: [myScorer],
})
```

## Related

- [`runEvals()` reference](https://mastra.ai/reference/evals/run-evals): Full API for `runEvals` parameters and returns
- [Gates and verdicts](https://mastra.ai/docs/evals/gates-and-verdicts): Enforce hard requirements and quality thresholds
- [Quick Checks](https://mastra.ai/docs/evals/quick-checks): Zero-LLM composable micro-scorers