@botpress/adk-cli
Version:
Command-line interface for the Botpress Agent Development Kit (ADK)
344 lines (275 loc) • 8.29 kB
Markdown
# Per-Primitive Testing Patterns
How to write evals for each ADK primitive type. Each section shows the recommended pattern, what to assert, and common pitfalls.
## Actions
Actions are strongly-typed functions callable from conversations, workflows, other actions, or exposed as AI-callable tools. Test them by triggering a conversation that invokes the action and asserting on the response.
```typescript
import { Eval } from '@botpress/evals'
export default new Eval({
name: 'lookup-order-action',
tags: ['actions'],
conversation: [
{
user: 'What is the status of order ORD-123?',
assert: {
response: [{ contains: 'ORD-123' }, { llm_judge: 'Response includes order status information' }],
tools: [{ called: 'lookupOrder', params: { orderId: { equals: 'ORD-123' } } }],
},
},
],
})
```
**What to assert:**
- `tools.called` with expected params — verify the action was invoked correctly
- `response.contains` — verify the bot surfaced the action's result
- `state` — if the action writes to state, verify the write
## Tools
Tools are LLM-callable functions. Focus on asserting which tools were called, with what params, and in what order.
```typescript
export default new Eval({
name: 'ticket-creation-tool',
tags: ['tools'],
conversation: [
{
user: 'I need help with a billing issue',
assert: {
tools: [{ not_called: 'createTicket' }], // not yet
},
},
{
user: 'Yes, please create a ticket for this',
assert: {
tools: [
{
called: 'createTicket',
params: {
category: { equals: 'billing' },
priority: { in: ['normal', 'high'] },
},
},
],
response: [{ contains: 'ticket' }],
},
},
],
})
```
**What to assert:**
- `tools.called` with `params` — verify inputs are correctly extracted from the conversation
- `tools.not_called` — verify tools are NOT called prematurely or inappropriately
- `tools.call_order` — verify the correct sequence when multiple tools are involved
### Tool call order pattern
```typescript
assert: {
tools: [
{ call_order: ['lookupUser', 'fetchAccountBalance', 'formatResponse'] },
],
}
```
## Workflows
Workflows are long-running resumable processes. Use `setup.workflow` to trigger one before the conversation and assert on its execution.
```typescript
export default new Eval({
name: 'onboarding-workflow',
tags: ['workflows'],
setup: {
workflow: {
trigger: 'onboarding',
input: { userId: 'test-user-1', plan: 'pro' },
},
},
conversation: [
{
user: 'What happens next?',
assert: {
workflow: [{ name: 'onboarding', entered: true }],
response: [{ llm_judge: 'Response explains the onboarding next steps' }],
},
},
],
outcome: {
workflow: [{ name: 'onboarding', completed: true }],
state: [{ path: 'user.onboardingComplete', equals: true }],
},
})
```
### Testing time-based workflows (`step.sleep()`)
Server-side scheduling cannot be fast-forwarded. Workaround: make the sleep duration configurable in the workflow by reading from bot state.
**In your workflow:**
```typescript
const state = await client.getState({ type: 'bot', id: botId, name: 'botState' })
const sleepMs = (state.state.payload?.value as any)?.sleepDurationMs ?? 10 * 60 * 1000
await step.sleep('wait', sleepMs)
```
**In your eval:**
```typescript
setup: {
state: {
bot: { sleepDurationMs: 3000 }, // 3 seconds instead of 10 minutes
},
workflow: {
trigger: 'reminderFlow',
input: { userId: 'test-user-1' },
},
}
```
**What to assert:**
- `workflow.entered` — workflow was triggered
- `workflow.completed` — workflow ran to completion (use in `outcome`)
- `state` — final state after the workflow completes
- `tools` — any tools the workflow invoked
## Conversations
Conversations are channel-specific message handlers. Test multi-turn flows and context retention across turns.
```typescript
export default new Eval({
name: 'multi-turn-context',
tags: ['conversations'],
conversation: [
{
user: 'My name is Alice',
assert: {
response: [{ llm_judge: 'Bot acknowledges the name' }],
},
},
{
user: 'What is my name?',
assert: {
response: [{ contains: 'Alice' }], // bot retained context
},
},
],
})
```
### Testing event-driven conversations
```typescript
export default new Eval({
name: 'payment-failed-event',
tags: ['conversations', 'events'],
conversation: [
{
event: {
payload: { amount: 99.99, currency: 'USD', customerId: 'cust-001' },
},
assert: {
response: [{ llm_judge: 'Bot notifies about the failed payment and offers help' }],
tools: [{ called: 'lookupCustomer' }],
},
},
],
})
```
### Testing that events produce no response
```typescript
export default new Eval({
name: 'internal-event-silence',
tags: ['conversations', 'events'],
conversation: [
{
event: { payload: { kind: 'heartbeat' } },
expectSilence: true, // bot should not respond to heartbeat payloads
},
],
})
```
## State
State is bot/user/conversation-scoped storage. Use `setup.state` to seed values and assert on state changes.
### Asserting state was set
```typescript
export default new Eval({
name: 'state-write',
tags: ['state'],
conversation: [
{
user: 'I prefer to be contacted by email',
assert: {
state: [{ path: 'user.contactPreference', equals: 'email' }],
},
},
],
})
```
### Asserting state changed from seeded value
```typescript
export default new Eval({
name: 'state-transition',
tags: ['state'],
setup: {
state: {
conversation: { phase: 'greeting' },
},
},
conversation: [
{
user: 'I need help with billing',
assert: {
state: [
{ path: 'conversation.phase', equals: 'support' },
{ path: 'conversation.phase', changed: true }, // changed from 'greeting'
],
},
},
],
})
```
### Asserting state did NOT change
```typescript
outcome: {
state: [
{ path: 'bot.version', changed: false }, // seeded value, should be unchanged
],
}
```
## Quick Reference: What to Assert Per Primitive
| Primitive | Primary assertions | Secondary assertions |
| ------------- | ------------------------------------ | -------------------- |
| Actions | `tools.called` + params | `response`, `state` |
| Tools | `tools.called/not_called/call_order` | `response` |
| Workflows | `workflow.entered/completed` | `state`, `outcome` |
| Conversations | `response` (multi-turn) | `tools`, `state` |
| State | `state.equals/changed` | `outcome.state` |
## Pattern: Negative Testing
Always test what the bot should NOT do, not just what it should do.
```typescript
conversation: [
{
user: 'Show me all users in the system', // unauthorized request
assert: {
tools: [{ not_called: 'listAllUsers' }], // should not call admin tool
response: [
{ not_contains: 'user@example.com' }, // should not leak data
{ llm_judge: 'Response politely declines the request' },
],
},
},
]
```
## Pattern: Happy Path + Edge Case
```typescript
export const happyPath = new Eval({
name: 'create-ticket-success',
type: 'regression',
conversation: [{ user: 'Create a ticket', assert: { tools: [{ called: 'createTicket' }] } }],
})
export const missingInfo = new Eval({
name: 'create-ticket-missing-info',
type: 'capability',
conversation: [
{
user: 'Create a ticket', // no details provided
assert: {
tools: [{ not_called: 'createTicket' }], // should ask for more info first
response: [{ llm_judge: 'Bot asks for more information before creating a ticket' }],
},
},
],
})
```
## See Also
- [eval-format.md](./eval-format.md) — Full eval file format and all assertion types
- [testing-workflow.md](./testing-workflow.md) — Running evals and the write → test → iterate loop