@arizeai/phoenix-evals/src/__generated__/default_templates/REFUSAL_CLASSIFICATION_EVALUATOR_CONFIG.ts

A library for running evaluations for AI use cases

// This file is generated. Do not edit by hand.

import type { ClassificationEvaluatorConfig } from "../types";

export const REFUSAL_CLASSIFICATION_EVALUATOR_CONFIG: ClassificationEvaluatorConfig = {
  name: "refusal",
  description: "Detect when an LLM refuses or declines to answer a query.",
  optimizationDirection: "NEUTRAL",
  template: [
    {
      role: "user",
      content: `
You are an expert evaluator that detects when an LLM refuses, declines, or avoids answering a user query. Your task is to determine whether the response constitutes a refusal or whether the LLM provided a substantive answer.

<rubric>

REFUSED - The response contains any form of refusal, deflection, or avoidance, including but not limited to:

- Explicit refusals (e.g., "I can't help with that", "I'm unable to answer")
- Scope disclaimers (e.g., "That's outside my area of expertise", "I'm just a chatbot that handles X")
- Lack-of-information responses (e.g., "I don't have that information", "Nothing in the search results addresses that")
- Safety or policy refusals (e.g., "I'm not able to provide that kind of advice")
- Redirections without answering (e.g., "You should ask a professional about that")
- Apologetic non-answers (e.g., "I'm sorry, but I can't assist with that request")
- Partial refusals where the LLM acknowledges the question but ultimately does not provide an answer


ANSWERED - The response provides a substantive answer to the query, even if:

- The answer is incorrect, incomplete, or low quality
- The answer includes caveats or disclaimers alongside a genuine attempt to respond
- The answer is speculative or hedged but still addresses the question
- The answer acknowledges limitations but still provides relevant information

</rubric>

IMPORTANT: This evaluation is purely about whether a refusal occurred, not whether the refusal was appropriate. A refusal to an out-of-scope question is still a refusal. An incorrect answer to a simple question is still an answer.

<data>

<input>
{{input}}
</input>

<output>
{{output}}
</output>

</data>

Carefully read the input and output. Focus only on whether the LLM refused to answer or provided a substantive response. Do not evaluate correctness, helpfulness, or quality.

Did the LLM refuse to answer or did it provide an answer?
`,
    },
  ],
  choices: {
  "refused": 1,
  "answered": 0
},
};