@cloudkinetix/bmad-enhanced

# {{Project Name}} LLM Agent Evaluation Suite [[LLM: Initial Setup 1. Replace {{Project Name}} with actual project name 2. Define comprehensive test scenarios covering all agent capabilities 3. Include edge cases, adversarial inputs, and performance benchmarks 4. Configure for automated execution with PromptFoo or similar tools Output file location: `docs/ai-agents/evaluation-suite.md`]] ## Evaluation Overview - **Project**: {{project-name}} - **Agent(s) Under Test**: {{agent-names}} - **Evaluation Framework**: PromptFoo {{version}} - **Last Updated**: {{date}} - **Test Coverage**: {{percentage}}% ## Test Categories ### Functional Tests [[LLM: Define tests that verify core functionality works as expected.]] <<REPEAT: functional_test>> #### {{Test Name}} - **Test ID**: `{{test-id}}` - **Category**: {{functional-category}} - **Priority**: {{High/Medium/Low}} - **Automated**: {{Yes/No}} **Test Scenario**: ````yaml description: '{{test-description}}' vars: input: '{{test-input}}' context: '{{optional-context}}' expected: - output: '{{expected-output}}' - contains: - '{{keyword1}}' - '{{keyword2}}' - not_contains: - '{{forbidden1}}' - '{{forbidden2}}' - format: '{{json/markdown/plain}}' assertions: - type: equals value: '{{exact-match}}' - type: contains value: '{{partial-match}}' - type: javascript value: | output.length > {{min-length}} && output.length < {{max-length}} ```text **Success Criteria**: - {{criterion-1}} - {{criterion-2}} <</REPEAT>> ### Edge Case Tests [[LLM: Test boundary conditions and unusual inputs.]] <<REPEAT: edge_case>> #### {{Edge Case Name}} - **Test ID**: `{{test-id}}` - **Risk Level**: {{High/Medium/Low}} - **Failure Impact**: {{impact-description}} **Scenarios**: 1. **Empty Input**: {{empty-behavior}} 2. **Oversized Input**: {{large-input-behavior}} 3. **Special Characters**: {{special-char-handling}} 4. **Language Mix**: {{multilingual-handling}} 5. **Concurrent Requests**: {{concurrency-behavior}} **Test Configuration**: ```yaml edge_cases: - name: '{{case-name}}' input: '{{edge-input}}' expected_behavior: '{{graceful-handling}}' error_type: '{{expected-error}}' ```` <</REPEAT>> ### Safety and Alignment Tests [[LLM: Verify safety measures and alignment constraints.]] #### Content Safety ````yaml safety_tests: harmful_content: - prompt: '{{harmful-prompt}}' expected: refusal message_contains: cannot assist bias_detection: - prompt: '{{biased-prompt}}' expected: neutral bias_score: < 0.3 privacy_protection: - prompt: '{{pii-prompt}}' expected: redacted pii_detected: false ```text #### Jailbreak Attempts <<REPEAT: jailbreak>> - **Technique**: {{jailbreak-type}} - **Test Prompt**: `{{jailbreak-prompt}}` - **Expected Response**: {{safe-response}} - **Detection Method**: {{how-detected}} <</REPEAT>> ### Performance Benchmarks [[LLM: Define performance tests with specific metrics.]] #### Latency Tests ```yaml performance: latency: - scenario: simple_query max_latency_ms: '[object Object]': null percentiles: p50: '[object Object]': null p95: '[object Object]': null p99: '[object Object]': null - scenario: complex_analysis max_latency_ms: '[object Object]': null timeout_ms: '[object Object]': null ```` #### Throughput Tests ````yaml throughput: - concurrent_users: {{number}} duration_seconds: {{duration}} target_rps: {{requests-per-second}} error_rate_threshold: {{percentage}}% ```text #### Resource Usage ```yaml resources: - metric: token_usage limit_per_request: '[object Object]': null average_target: '[object Object]': null - metric: memory_usage max_mb: '[object Object]': null - metric: cost_per_request max_usd: '[object Object]': null ```` ### Quality Evaluation [[LLM: Define quality metrics and evaluation criteria.]] #### Accuracy Tests <<REPEAT: accuracy_test>> ##### {{Domain}} Accuracy - **Test Set**: {{dataset-name}} - **Sample Size**: {{n-samples}} - **Target Accuracy**: {{percentage}}% **Evaluation Method**: ````python def evaluate_{{domain}}_accuracy(response, ground_truth): # {{evaluation-logic}} return accuracy_score ```text **Results**: | Model | Accuracy | Precision | Recall | F1 Score | |-------|----------|-----------|--------|----------| | {{model}} | {{acc}}% | {{prec}}% | {{rec}}% | {{f1}}% | <</REPEAT>> #### Consistency Tests ```yaml consistency: - test: "deterministic_responses" runs: {{number-of-runs}} temperature: 0 expected_variance: < {{threshold}} - test: "semantic_consistency" paraphrased_queries: {{number}} similarity_threshold: > {{threshold}} ```` ### Integration Tests [[LLM: Test agent integration with other systems.]] <<REPEAT: integration>> #### {{System}} Integration - **Integration Type**: {{API/Database/Service}} - **Test Coverage**: {{percentage}}% **Test Scenarios**: 1. **Connection Test**: {{connectivity-check}} 2. **Data Flow Test**: {{data-validation}} 3. **Error Handling**: {{failure-recovery}} 4. **Performance Impact**: {{latency-addition}} **Mock Configuration**: ````yaml mocks: '[object Object]': response_time_ms: '[object Object]': null success_rate: '[object Object]': null data_format: '[object Object]': null ```text <</REPEAT>> ### Regression Tests [[LLM: Ensure new changes don't break existing functionality.]] #### Test Suite Version Control ```yaml regression_suite: version: {{suite-version}} baseline_date: {{date}} test_count: {{number}} coverage_delta: {{+/-percentage}}% ```` #### Critical Path Tests <<REPEAT: critical_path>> - **Path**: {{user-journey}} - **Steps**: {{number-of-steps}} - **Must Pass**: {{Yes/No}} - **Last Verified**: {{date}} <</REPEAT>> ## Evaluation Automation ### PromptFoo Configuration ````yaml providers: - id: '[object Object]': null config: apiKey: ${API_KEY} model: '[object Object]': null temperature: '[object Object]': null max_tokens: '[object Object]': null prompts: - file://prompts/{{prompt-file}}.txt tests: - file://tests/functional/*.yml - file://tests/safety/*.yml - file://tests/performance/*.yml - file://tests/quality/*.yml outputPath: ./results/{{timestamp}}/ assertions: - type: python file: ./evaluators/custom_evaluator.py ```text ### Continuous Evaluation #### Scheduled Runs ```yaml schedule: daily: - time: 02:00 UTC suite: full notify: - '{{email}}' hourly: - time: '*/1 * * * *' suite: smoke alert_on_failure: true on_deploy: - trigger: deployment suite: critical_path block_on_failure: true ```` #### Results Tracking ````yaml metrics_storage: backend: '[object Object]': null retention_days: '[object Object]': null dashboards: - name: LLM Agent Health url: '{{dashboard-url}}' ```text ## Evaluation Reports ### Report Generation ```python # Generate evaluation report def generate_report(test_results): report = { "summary": calculate_summary(test_results), "passed": filter_passed_tests(test_results), "failed": filter_failed_tests(test_results), "performance": calculate_performance_metrics(test_results), "recommendations": generate_recommendations(test_results) } return format_report(report) ```` ### Key Metrics Dashboard | Metric | Target | Current | Trend | Status | | -------------------- | ------------ | ------------ | --------- | ------- | | Functional Pass Rate | >{{target}}% | {{current}}% | {{↑/↓/→}} | {{✓/✗}} | | Safety Score | >{{target}} | {{current}} | {{↑/↓/→}} | {{✓/✗}} | | Avg Latency (ms) | <{{target}} | {{current}} | {{↑/↓/→}} | {{✓/✗}} | | Error Rate | <{{target}}% | {{current}}% | {{↑/↓/→}} | {{✓/✗}} | | Cost per 1K requests | <${{target}} | ${{current}} | {{↑/↓/→}} | {{✓/✗}} | ## Test Data Management ### Test Dataset - **Location**: {{dataset-path}} - **Format**: {{json/csv/parquet}} - **Size**: {{number-of-examples}} - **Categories**: {{list-categories}} - **Update Frequency**: {{how-often}} ### Synthetic Data Generation ```python # Generate test data def generate_test_data(category, count): generator = TestDataGenerator(category) return generator.create_examples(count) ``` ## Failure Analysis ### Common Failure Patterns <<REPEAT: failure_pattern>> #### {{Pattern Name}} - **Frequency**: {{occurrences-per-day}} - **Root Cause**: {{cause-analysis}} - **Impact**: {{user-impact}} - **Mitigation**: {{fix-strategy}} <</REPEAT>> ### Debugging Guide 1. **Identify Failure Category**: {{classification-method}} 2. **Collect Context**: {{what-to-gather}} 3. **Reproduce Issue**: {{reproduction-steps}} 4. **Analyze Root Cause**: {{analysis-tools}} 5. **Implement Fix**: {{fix-process}} 6. **Verify Resolution**: {{verification-method}} ## Continuous Improvement ### Evaluation Roadmap - **Q{{quarter}} {{year}}**: {{planned-improvements}} - **New Test Categories**: {{upcoming-tests}} - **Tool Upgrades**: {{tool-updates}} - **Coverage Goals**: {{target-coverage}}% ### Feedback Loop - **User Reports**: {{feedback-channel}} - **Automated Detection**: {{monitoring-integration}} - **Review Cycle**: {{frequency}} - **Action Items**: {{improvement-tracking}} ## Resources and References - **Test Repository**: {{git-repo}} - **Documentation**: {{docs-link}} - **Issue Tracker**: {{issue-system}} - **Team Contact**: {{contact-info}}