@mastra/core/dist/docs/references/docs-datasets-running-experiments.md

> Discover all available pages from the documentation index: https://mastra.ai/llms.txt

# Running experiments

**Added in:** `@mastra/core@1.4.0`

An experiment runs every item in a dataset through a target (an agent, a workflow, or a scorer) and then optionally scores the outputs. Use a scorer as the target when you want to evaluate an LLM judge itself. By default, results are persisted to storage so you can compare runs across different prompts, models, or code changes.

**For AI agents:** Run `npx mastra api experiment run dataset_123 '{"name":"translation-baseline"}'` to start an experiment directly instead of opening Studio or writing a temporary script. Use a dataset ID returned by `npx mastra api dataset list` in place of the sample ID. The command requires a running Mastra server with dataset storage and registered experiment targets; start the local server with `npx mastra dev`, or pass the reachable server's base URL with `--url`. Run `npx mastra api experiment run --schema` before constructing different input, and get user approval before starting an experiment because it can make model calls. Install Mastra's skill with `npx skills add mastra-ai/skills --skill mastra` for complete API CLI discovery, targeting, schema, authentication, and error-handling guidance.

## Basic experiment

Call [`startExperiment()`](https://mastra.ai/reference/datasets/startExperiment) with a target and scorers:

```typescript
import { mastra } from '../index'

const dataset = await mastra.datasets.get({ id: 'translation-dataset-id' })

const summary = await dataset.startExperiment({
  name: 'gpt-5.1-baseline',
  targetType: 'agent',
  targetId: 'translation-agent',
  scorers: ['accuracy', 'fluency'],
})

console.log(summary.status) // 'completed' | 'failed'
console.log(summary.succeededCount) // number of items that ran successfully
console.log(summary.failedCount) // number of items that failed
```

`startExperiment()` blocks until all items finish. For fire-and-forget execution, see [async experiments](#async-experiments).

## Studio

You can also run experiments in [Studio](https://mastra.ai/docs/studio/overview). After you've added a dataset item, open it and select **Run Experiment** and configure the target, scorers, and options.

After running an experiment, the **Experiments** tab shows all runs for that dataset (with status, counts, and timestamps). Select an experiment to see per-item results, scores, and execution traces.

In the **Experiments** tab, select **Compare** and choose two or more experiments to compare their scores and results side by side.

## Experiment targets

You can point an experiment at a registered agent, workflow, or scorer.

### Registered agent

Point to an agent registered on your Mastra instance:

```typescript
const summary = await dataset.startExperiment({
  name: 'agent-v2-eval',
  targetType: 'agent',
  targetId: 'translation-agent',
  scorers: ['accuracy'],
})
```

Each item's `input` is passed directly to `agent.generate()`, so it must be a `string`, `string[]`, or `CoreMessage[]`.

### Registered workflow

Point to a workflow registered on your Mastra instance:

```typescript
const summary = await dataset.startExperiment({
  name: 'workflow-eval',
  targetType: 'workflow',
  targetId: 'translation-workflow',
  scorers: ['accuracy'],
})
```

The workflow receives each item's `input` as its trigger data.

### Registered scorer

Point to a scorer to evaluate an LLM judge against ground truth:

```typescript
const summary = await dataset.startExperiment({
  name: 'judge-accuracy-eval',
  targetType: 'scorer',
  targetId: 'accuracy',
})
```

The scorer receives each item's `input` and `groundTruth`. LLM-based judges can drift over time as underlying models change, so it's important to periodically realign them against known-good labels. A dataset gives you a stable benchmark to detect that drift.

## Scoring results

Scorers automatically run after each item's target execution. Pass scorer instances or registered scorer IDs:

**Scorer IDs**:

```typescript
// Reference scorers registered on the Mastra instance
const summary = await dataset.startExperiment({
  name: 'with-registered-scorers',
  targetType: 'agent',
  targetId: 'translation-agent',
  scorers: ['accuracy', 'fluency'],
})
```

**Scorer instances**:

```typescript
import { createAnswerRelevancyScorer } from '@mastra/evals/scorers/prebuilt'

const relevancy = createAnswerRelevancyScorer({ model: 'openai/gpt-5-mini' })

const summary = await dataset.startExperiment({
  name: 'with-scorer-instances',
  targetType: 'agent',
  targetId: 'translation-agent',
  scorers: [relevancy],
})
```

Each item's results include per-scorer scores:

```typescript
for (const item of summary.results) {
  console.log(item.itemId, item.output)
  for (const score of item.scores) {
    console.log(`  ${score.scorerName}: ${score.score} — ${score.reason}`)
  }
}
```

Visit the [Scorers overview](https://mastra.ai/docs/evals/overview) for details on available and custom scorers.

## Control persistence per run

Use `persistence` to skip storage writes for a specific run. Experiment records and score records can be disabled independently:

```typescript
const summary = await dataset.startExperiment({
  targetType: 'agent',
  targetId: 'translation-agent',
  scorers: ['accuracy'],
  persistence: {
    experiments: 'none',
    scores: 'none',
  },
})
```

The target and scorers still run, and `startExperiment()` still returns the item results and scores in `summary`. The settings are independent. For example, set only `scores: 'none'` to persist the experiment and its item results without creating score records.

Omitted settings default to `'default'`, which preserves the standard storage behavior. This policy only controls experiment and score records created by the run. It doesn't disable storage used by the target, such as agent memory, vectors, observability, or custom tool storage.

When `startExperimentAsync()` runs with `experiments: 'none'`, it doesn't persist an experiment record, progress updates, or item results. Score persistence remains controlled separately by `persistence.scores`. Without an experiment event observer, the run is fire-and-forget, and the experiment API can't report whether it completed or failed.

Use synchronous `startExperiment()` when the caller needs the returned summary. An experiment event observer can receive lifecycle events and the terminal summary.

## Observe experiment events

Use `onEvent` to receive versioned, JSON-safe lifecycle events while an experiment runs. This works with `startExperiment()`, `startExperimentAsync()`, and `runExperiment()`.

```typescript
import type { ExperimentEvent } from '@mastra/core/datasets'

const events: ExperimentEvent[] = []

await dataset.startExperimentAsync({
  task: async ({ input }) => processItem(input),
  persistence: { experiments: 'none' },
  onEvent: async event => {
    events.push(event)
    await publishEvent(event)
  },
})
```

The observer receives these event types:

- `experiment.run.started`: Identifies the run, target, resolved dataset version, and item count.
- `experiment.item.completed`: Reports a committed item result after scoring, including scores, errors, retry count, tool mock details, and stable item identity.
- `experiment.run.finished`: Reports the terminal outcome and summary counters.

Mastra awaits each observer call before delivering the next event. This serialized delivery applies backpressure and ensures event `sequence` values match delivery order, while item execution can remain concurrent.

If the observer throws or rejects, Mastra aborts the remaining run and rejects `runExperiment()` with a `MastraError` whose `id` is `EXPERIMENT_EVENT_OBSERVER_FAILED`. It doesn't send a terminal event through the failed observer. For `startExperimentAsync()`, the method has already returned when a detached observer fails, so handle delivery failures inside the observer when the caller needs direct error reporting.

The `experiment.run.finished` event is awaited before Mastra persists the final experiment status. Treat the event as the authoritative terminal signal when experiment persistence is disabled, but don't use it as a read-after-write signal for storage.

The exported event types are `ExperimentEvent`, `ExperimentRunStartedEvent`, `ExperimentItemCompletedEvent`, and `ExperimentRunFinishedEvent`. Use the discriminated `type` field to narrow an event before reading event-specific properties.

## Tool mocks

When an experiment runs an agent that calls side-effecting tools, attach static tool mocks to individual dataset items to make the run deterministic. During the experiment, a mocked tool returns its declared output instead of executing. Tools without a mock on the item run live by default.

Mocks live on the dataset item, so they version with the row and travel with the test case. Each mock declares a tool name, the arguments it expects, and the output to return:

```typescript
await dataset.addItem({
  input: 'What is the weather in Seattle?',
  toolMocks: [
    {
      toolName: 'getWeather',
      args: { city: 'Seattle' },
      output: { temperature: 60, conditions: 'rainy' },
    },
  ],
})
```

Tool mocks are supported for `agent` targets only.

### Block undeclared tools

Set `unmockedToolPolicy: 'deny'` on an experiment to block every tool call that doesn't have a mock. This is useful when a live call could cause side effects:

```typescript
const summary = await dataset.startExperiment({
  targetType: 'agent',
  targetId: 'weather-agent',
  unmockedToolPolicy: 'deny',
})
```

The default policy is `'allow'`. You can override the experiment policy on an individual stored or inline item:

```typescript
await dataset.addItem({
  input: 'What is the weather in Seattle?',
  unmockedToolPolicy: 'allow',
})
```

The item value takes precedence over the experiment value. A denied call fails with `TOOL_MOCK_NOT_DECLARED` before the tool executes. The failure isn't retried or added to `liveCalls`.

### Matching and consumption

Arguments are matched strictly: object key order is ignored and array order is substantial, plus there is no type coercion. A mock is served only when the agent calls the tool with arguments that deep-equal the mock's `args`.

When an item declares several mocks for the same tool and arguments, they're consumed in order, the first call gets the first mock, the next call gets the second, and so on. Ordering is tracked per `(toolName, args)` group and is independent across different arguments.

### Matching mode

By default each mock matches strictly on its `args`. Set `matchArgs: 'ignore'` to match on the tool name only, the mock's `args` aren't compared and the next unconsumed mock for that tool is served regardless of how the agent called it:

```typescript
const subAgentMock = {
  toolName: 'agent-balanceAgent',
  args: { prompt: 'look up the balance for YJ' },
  output: { text: "YJ's balance is $100." },
  matchArgs: 'ignore',
}
```

This is useful when a tool's arguments are noisy or generated by the model. The most common case is mocking a **sub-agent's response**: a delegated sub-agent is exposed to the parent as an `agent-<name>` tool, and its arguments include an LLM-authored `prompt` plus runtime-injected fields. Mocking `agent-<name>` returns the canned response in place of running the sub-agent and its inner tools. When you create a mock from a trace, sub-agent delegation calls are derived with `matchArgs: 'ignore'` automatically. You can change it to `'strict'` to pin the exact arguments.

### Failures

A tool call fails the item when it violates the mock configuration:

- `TOOL_MOCK_MISMATCH`: the tool was called with arguments that no mock matches.
- `TOOL_MOCK_EXHAUSTED`: every matching mock has already been consumed.
- `TOOL_MOCK_NOT_DECLARED`: the tool has no mock and the effective `unmockedToolPolicy` is `'deny'`.

On any of these failures, the agent run is aborted immediately, so the model can't go on to call any further tools, including unmocked, side-effecting tools that would otherwise run live. These failures are deterministic, so they're not retried. Mocks that are declared but never used don't fail the item, they're reported as unconsumed.

While mock interception is active, the agent's tools execute sequentially so repeated `(toolName, args)` mocks are consumed in the provider's call order. Interception is active when the item declares mocks or its effective `unmockedToolPolicy` is `'deny'`.

### Diagnostics

Each item result carries a `toolMockReport` describing what the run did with the item's mocks:

```typescript
for (const item of summary.results) {
  const report = item.toolMockReport
  if (!report) continue

  console.log(report.served) // mocks matched and returned
  console.log(report.unconsumed) // mocks declared but never used
  console.log(report.liveCalls) // undeclared tools allowed to run live
  console.log(report.failure) // the first deterministic mock failure, if any
}
```

In [Studio](https://mastra.ai/docs/studio/overview), edit a dataset item to author tool mocks as a JSON array, and open an experiment result to see the same report.

### Limitations

- **No tool span for mocked calls.** A mocked call returns its output before the tool executes, so it doesn't create a tool span. Trajectory scorers backed by stored traces may therefore not see mocked tool calls. Trajectory extraction that falls back to the agent's message output still sees them, so trajectory scoring can differ depending on your observability configuration.
- **Storage support.** Tool mocks and tool mock reports are persisted by the LibSQL, PostgreSQL, MongoDB, and Spanner adapters. The MySQL adapter doesn't support them and rejects writes that carry tool mocks or a tool mock report. All dataset storage adapters persist `unmockedToolPolicy`.

## Async experiments

`startExperiment()` blocks until every item completes. For long-running datasets, use [`startExperimentAsync()`](https://mastra.ai/reference/datasets/startExperimentAsync) to start the experiment in the background:

```typescript
const { experimentId, status } = await dataset.startExperimentAsync({
  name: 'large-dataset-run',
  targetType: 'agent',
  targetId: 'translation-agent',
  scorers: ['accuracy'],
})

console.log(experimentId) // UUID
console.log(status) // 'pending'
```

Poll for completion using [`getExperiment()`](https://mastra.ai/reference/datasets/getExperiment):

```typescript
let experiment = await dataset.getExperiment({ experimentId })

while (experiment.status === 'pending' || experiment.status === 'running') {
  await new Promise(resolve => setTimeout(resolve, 5000))
  experiment = await dataset.getExperiment({ experimentId })
}

console.log(experiment.status) // 'completed' | 'failed'
```

## Configuration options

### Concurrency

Control how many items run in parallel (default: 5):

```typescript
const summary = await dataset.startExperiment({
  targetType: 'agent',
  targetId: 'translation-agent',
  maxConcurrency: 10,
})
```

### Timeouts and retries

Set a per-item timeout (in milliseconds) and retry count:

```typescript
const summary = await dataset.startExperiment({
  targetType: 'agent',
  targetId: 'translation-agent',
  itemTimeout: 30_000, // 30 seconds per item
  maxRetries: 2, // retry failed items up to 2 times
})
```

Retries use exponential backoff. Abort errors are never retried.

### Aborting an experiment

Pass an `AbortSignal` to cancel a running experiment:

```typescript
const controller = new AbortController()

// Cancel after 60 seconds
setTimeout(() => controller.abort(), 60_000)

const summary = await dataset.startExperiment({
  targetType: 'agent',
  targetId: 'translation-agent',
  signal: controller.signal,
})
```

Remaining items are marked as skipped in the summary.

### Pinning a dataset version

Run against a specific snapshot of the dataset:

```typescript
const summary = await dataset.startExperiment({
  targetType: 'agent',
  targetId: 'translation-agent',
  version: 3, // use items from dataset version 3
})
```

## Viewing results

### Listing experiments

```typescript
const { experiments, pagination } = await dataset.listExperiments({
  page: 0,
  perPage: 10,
})

for (const exp of experiments) {
  console.log(`${exp.name} — ${exp.status} (${exp.succeededCount}/${exp.totalItems})`)
}
```

### Experiment details

```typescript
const experiment = await dataset.getExperiment({
  experimentId: 'exp-abc-123',
})

console.log(experiment.status)
console.log(experiment.startedAt)
console.log(experiment.completedAt)
```

> **📹 Watch:** Watch [Mastra datasets and experiments workflow](https://www.youtube.com/watch?v=R6pjAdGhxhQ) to see how datasets and experiments help improve reliability.

### Item-level results

```typescript
const { results, pagination } = await dataset.listExperimentResults({
  experimentId: 'exp-abc-123',
  page: 0,
  perPage: 50,
})

for (const result of results) {
  console.log(result.itemId, result.output, result.error)
}
```

## Understanding the summary

`startExperiment()` returns an `ExperimentSummary` with counts and per-item results:

- `completedWithErrors` is `true` when the experiment finished but some items failed.
- Items cancelled via `signal` appear in `skippedCount`.

Visit the [`startExperiment` reference](https://mastra.ai/reference/datasets/startExperiment) for the full parameter and return type documentation.

## Related

- [Datasets overview](https://mastra.ai/docs/datasets/overview)
- [Scorers overview](https://mastra.ai/docs/evals/overview)
- [`startExperiment` reference](https://mastra.ai/reference/datasets/startExperiment)
- [`listExperimentResults` reference](https://mastra.ai/reference/datasets/listExperimentResults)