@botpress/adk-cli
Version:
Command-line interface for the Botpress Agent Development Kit (ADK)
368 lines (274 loc) • 8.57 kB
Markdown
# Eval File Format
Evals are TypeScript files in the `evals/` directory. Each file exports one or more eval definitions using `new Eval`.
## File Location
Evals live in `evals/` at the **project root** (not inside `src/`). Create it if it doesn't exist:
```
my-agent/
├── agent.config.ts
├── src/
│ ├── actions/
│ └── workflows/
└── evals/ ← create this
├── greeting.eval.ts
└── billing.eval.ts
```
- **Naming**: `*.eval.ts` convention (recommended)
- **Auto-discovery**: All files in `evals/` are picked up by `adk evals`
## Full Structure
```typescript
import { Eval } from '@botpress/evals'
export default new Eval({
name: 'my-eval', // unique identifier (required)
description: 'What this tests', // optional
type: 'regression', // 'capability' or 'regression' — optional, for filtering
tags: ['tools', 'multi-turn'], // optional, for filtering
setup: {
// Seed state or trigger a workflow before the conversation (optional)
},
conversation: [
{
user: 'message from user', // or: event, expectSilence
assert: {
/* per-turn assertions */
},
},
],
outcome: {
/* post-conversation assertions (optional) */
},
options: {
/* per-eval overrides (optional) */
},
})
```
## Types
| Type | Purpose |
| ------------ | -------------------------------------- |
| `capability` | Verify the bot can do something new |
| `regression` | Verify the bot still does it correctly |
## Multiple Evals Per File
```typescript
export const greeting = new Eval({ name: 'greeting', ... })
export const farewell = new Eval({ name: 'farewell', ... })
```
## Conversation Turns
Each entry in `conversation` is one turn. A turn must have either `user` or `event`.
### User Message
```typescript
{
user: 'What is my account balance?',
assert: { /* assertions on the bot's response */ },
}
```
### Event Trigger
Push a custom event instead of a user message. The turn carries only a `payload`; the bot receives it as a `chat:custom` event and reads the data at `event.payload.payload`.
```typescript
{
event: {
payload: { orderId: 'ORD-001', total: 49.99 },
},
assert: {
workflow: [{ name: 'orderConfirmation', entered: true }],
},
}
```
### Expect Silence
Assert the bot does **not** respond. Add `expectSilence: true` to any turn.
```typescript
// Silence after a user message
{ user: 'Please ignore this.', expectSilence: true }
// Silence after an event
{ event: { payload: { kind: 'ping' } }, expectSilence: true }
```
> **Note:** `expectSilence` is mutually exclusive with `assert.response`. Every turn must have `user` or `event` — `expectSilence` is a flag on top of that, not a standalone turn type.
## Assertion Categories
### Response
What the bot said back.
```typescript
assert: {
response: [
{ contains: 'ticket' }, // substring present
{ not_contains: 'error' }, // substring absent
{ matches: 'TKT-\\d{3}' }, // regex match
{ llm_judge: 'Response confirms the ticket was created' }, // AI judge, scores 1–5
],
}
```
### Tools
Which tools the bot called and with what parameters.
```typescript
assert: {
tools: [
{ called: 'createTicket' }, // tool was invoked
{ called: 'createTicket', params: { // with specific params
priority: { equals: 'high' },
department: { contains: 'Engineering' },
}},
{ not_called: 'deleteTicket' }, // tool was NOT invoked
{ call_order: ['lookupUser', 'createTicket'] }, // ordered calls
],
}
```
### State
Bot, user, or conversation state values after the turn.
```typescript
assert: {
state: [
{ path: 'conversation.topic', equals: 'support' }, // exact value
{ path: 'conversation.topic', changed: true }, // value changed from before
{ path: 'bot.ticketCount', equals: 3 },
],
}
```
### Workflow
Workflow execution (verified via trace spans).
```typescript
assert: {
workflow: [
{ name: 'onboarding', entered: true }, // workflow was started
{ name: 'onboarding', completed: true }, // workflow finished
],
}
```
### Timing
How long the bot took to respond (milliseconds).
```typescript
assert: {
timing: [
{ response_time: { lte: 5000 } }, // must respond within 5s
{ response_time: { gte: 100 } }, // sanity-check: not suspiciously fast
],
}
```
## Match Operators
Used in tool params, state values, and workflow params:
| Operator | Example | Description |
| -------------- | ---------------------------- | ------------------ |
| `equals` | `{ equals: 'urgent' }` | Exact match |
| `contains` | `{ contains: 'HR' }` | Substring |
| `not_contains` | `{ not_contains: 'test' }` | Excludes substring |
| `matches` | `{ matches: '\\d+' }` | Regex |
| `in` | `{ in: ['high', 'urgent'] }` | One of |
| `exists` | `{ exists: true }` | Property exists |
**Numeric-only operators** (for `timing` and numeric state values):
| Operator | Example | Description |
| -------- | --------------- | --------------------- |
| `gte` | `{ gte: 100 }` | Greater than or equal |
| `lte` | `{ lte: 5000 }` | Less than or equal |
## Eval Setup
Use `setup` to put the bot in a known state before the conversation starts.
### Seed State
Pre-populate bot, user, or conversation state.
```typescript
setup: {
state: {
bot: { welcomeMessageSent: true },
user: { plan: 'pro' },
conversation: { topic: 'billing' },
},
}
```
The seeded state becomes the baseline for `changed` assertions — `{ changed: false }` passes if the value matches the seeded value at the end.
### Trigger a Workflow
Start a workflow before the conversation begins.
```typescript
setup: {
workflow: {
trigger: 'onboarding',
input: { userId: 'test-user-1' },
},
}
```
Both can be combined:
```typescript
setup: {
state: {
bot: { sleepDurationMs: 3000 }, // 3 seconds instead of default 10 minutes
},
workflow: {
trigger: 'reminderFlow',
input: { userId: 'test-user-1' },
},
}
```
> **Testing `step.sleep()`:** Server-side scheduling can't be fast-forwarded. Workaround: make the sleep duration configurable by reading it from bot state in the workflow, then seed a short value in ms (e.g. `3000`) in `setup.state.bot`.
## Outcome Assertions
Run once after all conversation turns complete. Supports `state` and `workflow` (not `response` or `tools` — those are per-turn only).
```typescript
outcome: {
state: [
{ path: 'conversation.resolved', equals: true },
],
workflow: [
{ name: 'ticketFlow', completed: true },
],
}
```
## Options
Override defaults for a specific eval. Cascades: **eval options → agent config → default**.
```typescript
options: {
idleTimeout: 60000, // ms to wait for bot response (default: 30000)
judgePassThreshold: 4, // llm_judge score required to pass, 1–5 (default: 3)
}
```
Agent-level defaults in `agent.config.ts`:
```typescript
export default defineConfig({
evals: {
idleTimeout: 20000,
judgePassThreshold: 3,
judgeModel: 'fast', // 'fast', 'best', or a model ref like 'openai:gpt-4o'
},
})
```
## Common Mistakes
❌ **Turn with neither `user` nor `event`**
```typescript
// WRONG — every turn needs a trigger
{
expectSilence: true
}
```
✅ **Correct**
```typescript
{ user: 'hello', expectSilence: true }
```
❌ **`expectSilence` with `assert.response`**
```typescript
// WRONG — mutually exclusive
{ user: 'hello', expectSilence: true, assert: { response: [{ contains: 'hi' }] } }
```
✅ **Correct — pick one**
```typescript
{ user: 'hello', expectSilence: true }
// or
{ user: 'hello', assert: { response: [{ contains: 'hi' }] } }
```
❌ **Both `user` and `event` on the same turn**
```typescript
// WRONG — mutually exclusive
{ user: 'hello', event: { payload: { amount: 50 } } }
```
✅ **Correct — use separate turns**
```typescript
{
event: {
payload: {
amount: 50
}
}
}
```
## See Also
- [testing-workflow.md](./testing-workflow.md) — Running evals, interpreting output, the write → test → iterate loop
- [test-patterns.md](./test-patterns.md) — Per-primitive testing patterns