@mastra/core/dist/docs/references/reference-evals-mastra-scorer.md

> Discover all available pages from the documentation index: https://mastra.ai/llms.txt

# MastraScorer

The `MastraScorer` class is the base class for all scorers in Mastra. It provides a standard `.run()` method for evaluating input/output pairs and supports multi-step scoring workflows with preprocess → analyze → generateScore → generateReason execution flow.

Most users should use [`createScorer`](https://mastra.ai/reference/evals/create-scorer) to create scorer instances. Direct instantiation of `MastraScorer` isn't recommended.

## How to get a `MastraScorer` instance

Use the `createScorer` factory function, which returns a `MastraScorer` instance:

```typescript
import { createScorer } from '@mastra/core/evals'

const scorer = createScorer({
  name: 'My Custom Scorer',
  description: 'Evaluates responses based on custom criteria',
}).generateScore(({ run, results }) => {
  // scoring logic
  return 0.85
})

// scorer is now a MastraScorer instance
```

## `.run()` method

The `.run()` method is the primary way to execute your scorer and evaluate input/output pairs. It processes the data through your defined steps (preprocess → analyze → generateScore → generateReason) and returns a detailed result object with the score, reasoning, and intermediate results.

```typescript
const result = await scorer.run({
  input: 'What is machine learning?',
  output: 'Machine learning is a subset of artificial intelligence...',
  runId: 'optional-run-id',
  requestContext: {/* optional context */},
})
```

## `.run()` input

**input** (`any`): Input data to be evaluated. Can be any type depending on your scorer's requirements.

**output** (`any`): Output data to be evaluated. Can be any type depending on your scorer's requirements.

**runId** (`string`): Optional unique identifier for this scoring run.

**requestContext** (`any`): Optional request context from the agent or workflow step being evaluated.

**groundTruth** (`any`): Optional expected or reference output for comparison during scoring. Automatically passed when using runEvals.

## `.run()` returns

**runId** (`string`): The unique identifier for this scoring run.

**score** (`number`): Numerical score computed by the generateScore step.

**reason** (`string`): Explanation for the score, if generateReason step was defined (optional).

**preprocessStepResult** (`any`): Result of the preprocess step, if defined (optional).

**analyzeStepResult** (`any`): Result of the analyze step, if defined (optional).

**preprocessPrompt** (`string`): Preprocess prompt, if defined (optional).

**analyzePrompt** (`string`): Analyze prompt, if defined (optional).

**generateScorePrompt** (`string`): Generate score prompt, if defined (optional).

**generateReasonPrompt** (`string`): Generate reason prompt, if defined (optional).

**judge** (`ScorerJudgeResults`): Execution details for prompt-based scorer steps, if any (optional).

### Judge results

The optional `judge` record contains details about the judge model calls made by prompt-based scorer steps. Its known keys are `preprocess`, `analyze`, `generateScore`, and `generateReason`. Each key contains an ordered `executions` array.

```typescript
interface ScorerJudgeExecutionBase {
  prompt: string
  judgeModelId: string
  judgeProvider?: string
  attemptCount: number
  modelCallCount: number
  durationMs: number
}

interface ScorerJudgeExecutionSuccess extends ScorerJudgeExecutionBase {
  status: 'success'
  output: JSONValue
  usage: ScorerJudgeUsage
  cost?: {
    amount: number
    unit: string
    source: string
  }
}

interface ScorerJudgeExecutionFailure extends ScorerJudgeExecutionBase {
  status: 'failed'
  output?: JSONValue
  rawOutput?: string
  usage?: ScorerJudgeUsage
  finishReason?: string
  error: {
    name: string
    message: string
    code?: string
  }
}

type ScorerJudgeExecution = ScorerJudgeExecutionSuccess | ScorerJudgeExecutionFailure

interface ScorerJudgeUsage {
  inputTokens?: number
  outputTokens?: number
  totalTokens?: number
  reasoningTokens?: number
  cachedInputTokens?: number
  cacheCreationInputTokens?: number
}

type ScorerJudgeResults = Partial<
  Record<
    'preprocess' | 'analyze' | 'generateScore' | 'generateReason',
    { executions: ScorerJudgeExecution[] }
  >
>
```

Use the step key to access its judge execution details:

```typescript
const execution = result.judge?.generateScore?.executions[0]

console.log(execution?.status)
console.log(execution?.judgeModelId)
console.log(execution?.usage?.totalTokens)
console.log(execution?.durationMs)
```

The `status` value describes the outcome of the logical prompt-step execution, not the quality of the evaluated response. A structured-output fallback that eventually succeeds creates one `success` execution with an `attemptCount` greater than one. Exhausted attempts create one `failed` execution.

Successful executions require validated `output` and normalized `usage`. Failed executions require an `error` summary and include only the evidence the runtime received. A failed execution includes `output` only when the output was validated before a later callback or orchestration failure. Mastra doesn't parse `rawOutput` to create `output`.

`attemptCount` counts judge invocations, including a structured-output fallback. `modelCallCount` counts the completed model steps across those attempts. `durationMs` covers the full prompt-step execution.

Function steps don't create `judge` entries. Usage in this record belongs to the scorer's judge model, not the agent or workflow being evaluated. Filter by `status` when aggregating successful executions. Include both statuses when aggregating all completed provider usage. The optional `cost` field is present only on successful executions that directly report an authoritative cost, source, and unit.

Use Mastra metrics to query aggregate usage, latency, and estimated cost across scorer runs. The `judge` record describes one scorer run and doesn't query metrics or traces.

### Failed runs

A failed scorer stage still rejects the `.run()` promise. Catch `ScorerRunError` to inspect completed stages and any results they produced:

```typescript
import { ScorerRunError } from '@mastra/core/evals'

try {
  const result = await scorer.run({ input, output })
  console.log(result.score)
} catch (error) {
  if (error instanceof ScorerRunError) {
    console.log(error.failedStep)
    console.log(error.completedSteps)
    console.log(error.result?.score)

    const failedExecution = error.result?.judge?.[error.failedStep]?.executions.find(
      execution => execution.status === 'failed',
    )
    console.log(failedExecution?.error)
  }

  throw error
}
```

`ScorerRunError` exposes these properties:

**failedStep** (`ScorerStepName`): The scorer stage that failed.

**completedSteps** (`ScorerStepName[]`): The scorer stages that completed before the failure, in execution order.

**result** (`ScorerRunResultSnapshot | undefined`): Outputs from completed scorer stages and judge execution evidence from attempted prompt stages. This property is omitted when neither is available.

The `result` snapshot contains completed stage outputs and judge execution evidence. For example, if `generateReason` fails after `generateScore` returns `0`, `error.result.score` is `0`, the `generateScore` execution has `status: 'success'`, and the `generateReason` execution has `status: 'failed'`. The run remains failed.

A prompt failure can create `error.result` with only run identity, input, and a failed `judge` entry. A function stage that fails before producing a scorer field doesn't create a result.

`JSON.stringify(error)` uses the standard `MastraError` serialization and omits `result`, including successful and failed judge evidence. Read `result` explicitly when you need scorer artifacts or raw failed output.

An in-memory experiment result can keep a completed score or reason from a failed scorer along with `error`, `failedStep`, and `completedSteps`. The scorer is still treated as failed, and a recovered score isn't written to legacy successful-score storage.

## Step execution flow

When you call `.run()`, the MastraScorer executes the defined steps in this order:

1. **preprocess** (optional): Extracts or transforms data
2. **analyze** (optional): Processes the input/output and preprocessed data
3. **generateScore** (required): Computes the numerical score
4. **generateReason** (optional): Provides explanation for the score

Each step receives the results from previous steps, allowing you to build complex evaluation pipelines.

## Usage example

```typescript
const scorer = createScorer({
  name: 'Quality Scorer',
  description: 'Evaluates response quality',
})
  .preprocess(({ run }) => {
    // Extract key information
    return { wordCount: run.output.split(' ').length }
  })
  .analyze(({ run, results }) => {
    // Analyze the response
    const hasSubstance = results.preprocessStepResult.wordCount > 10
    return { hasSubstance }
  })
  .generateScore(({ results }) => {
    // Calculate score
    return results.analyzeStepResult.hasSubstance ? 1.0 : 0.0
  })
  .generateReason(({ score, results }) => {
    // Explain the score
    const wordCount = results.preprocessStepResult.wordCount
    return `Score: ${score}. Response has ${wordCount} words.`
  })

// Use the scorer
const result = await scorer.run({
  input: 'What is machine learning?',
  output: 'Machine learning is a subset of artificial intelligence...',
})

console.log(result.score) // 1.0
console.log(result.reason) // "Score: 1.0. Response has 12 words."
```

## Integration

MastraScorer instances can be used for agents and workflow steps

See the [createScorer reference](https://mastra.ai/reference/evals/create-scorer) for detailed information on defining custom scoring logic.