@twilio-alpha/assistants-eval/README.md

promptfoo extension for writing AI evaluations for Twilio AI Assistants

# Twilio AI Assistants Evaluation

This repository contains the extentions to the [promptfoo.dev](https://www.promptfoo.dev/) framework for test your [Twilio AI Assistant](https://www.twilio.com/docs/alpha/ai-assistants).

## Installation

1. Install and setup `promptfoo` as described in the [promptfoo.dev](https://www.promptfoo.dev/docs/installation/) documentation.
2. Install this extension using `npm install @twilio-alpha/assistants-eval`.
3. Create a target in your promptfoo config yaml:

```
targets:
  - id: package:@twilio-alpha/assistants-eval:TwilioAgentProvider
    label: TwilioProvider
    config:
      assistantId: aia_asst_abc123
      accountSid: ACXXX
      authTokenKey: TWILIO_AUTH_TOKEN
```

## Usage

Consult [promptfoo.dev](https://www.promptfoo.dev/docs/intro/) for general usage instructions on how to use promptfoo.

### Configuration Options

- `assistantId: string` The ID of the Twilio AI Assistant you want to test.
- `accountSid: string` The Twilio Account SID.
- `authTokenKey: string` The environment variable key that contains the Twilio Auth Token.
- `sessionId?: string` The session ID to use for the conversation. If no sessionId is provided a random UUID will be generated.
- `identity?: string` The identity of the user. Defaults to `test-evaluator`.