claude-flow-novice
Version:
Claude Flow Novice - Advanced orchestration platform for multi-agent AI workflows with CFN Loop architecture Includes Local RuVector Accelerator and all CFN skills for complete functionality.
289 lines (219 loc) • 6.24 kB
Markdown
# CFN Test Runner Skill
**Version:** 1.0.0
**Status:** Production
**Purpose:** Unified test execution with benchmarking and regression detection
## Overview
Provides comprehensive test execution across all CFN test suites:
- **Hello World Tests** (Layer 0-7)
- **CFN Loop E2E Tests** (9 integration tests)
- **Benchmark Tracking** (SQLite storage)
- **Regression Detection** (automated alerts)
**Key Features:**
- Single command test execution
- Historical benchmark comparison
- Performance regression alerts
- Git-aware baseline tracking
- Parallel test execution
- Comprehensive reporting
## Architecture
### Skill Components
```
.claude/skills/cfn-test-runner/
├── SKILL.md # This file
├── run-all-tests.sh # Main test runner
├── store-benchmarks.sh # SQLite benchmark storage
├── detect-regressions.sh # Regression analysis
├── init-benchmark-db.sh # Database initialization
└── generate-report.sh # Test report generation
```
## Usage
### Run All Tests
```bash
./.claude/skills/cfn-test-runner/run-all-tests.sh \
--suite all \
--benchmark \
--detect-regressions
```
### Run Specific Suite
```bash
# Hello World only
./.claude/skills/cfn-test-runner/run-all-tests.sh --suite hello-world
# CFN E2E only
./.claude/skills/cfn-test-runner/run-all-tests.sh --suite cfn-e2e
# Both
./.claude/skills/cfn-test-runner/run-all-tests.sh --suite all
```
### With Regression Detection
```bash
./.claude/skills/cfn-test-runner/run-all-tests.sh \
--suite all \
--benchmark \
--detect-regressions \
--threshold 0.10
```
## Parameters
### run-all-tests.sh
| Parameter | Required | Description | Default |
|-----------|----------|-------------|---------|
| `--suite` | No | Test suite: `all`, `hello-world`, `cfn-e2e` | `all` |
| `--benchmark` | No | Store results in SQLite | `false` |
| `--detect-regressions` | No | Run regression analysis | `false` |
| `--threshold` | No | Regression threshold (%) | `0.10` |
| `--parallel` | No | Run tests in parallel | `false` |
| `--output` | No | Output format: `text`, `json`, `html` | `text` |
## Test Suites
### Hello World (Layer 0-7)
**Purpose:** Validate agent spawning patterns and coordination
| Layer | Test | Duration | Critical |
|-------|------|----------|----------|
| 0 | Tool Validation | 60s | ✅ |
| 1-4 | [Future] | - | - |
| 5 | Coordinator Spawning | 120s | ✅ |
| 6 | Review Handoff | 180s | ✅ |
| 7 | Error Retry | 150s | ✅ |
**Location:** `tests/hello-world/`
### CFN Loop E2E (9 Tests)
**Purpose:** Validate full CFN Loop workflow
| Test | Component | Duration | Critical |
|------|-----------|----------|----------|
| 1 | Coordinator → Orchestrator | 30s | ✅ |
| 2 | Loop 3 → Gate Check | 60s | ✅ |
| 3 | Gate Pass → Loop 2 | 30s | ⚠️ |
| 4 | Loop 2 → Product Owner | 90s | ✅ |
| 5 | Product Owner Decision | 60s | ✅ |
| 6 | Iteration Cycle | 120s | ✅ |
| 7 | Redis Key Structure | 10s | ✅ |
| 8 | Error Recovery | 60s | ⚠️ |
| 9 | Cleanup | 10s | ⚠️ |
**Location:** `tests/cfn-v3/test-e2e-cfn-loop.sh`
## Benchmark Storage
### SQLite Schema
```sql
test_suites -- Suite definitions
test_runs -- Execution records
test_cases -- Individual tests
test_results -- Per-test outcomes
performance_metrics -- Performance data
regression_alerts -- Detected regressions
```
**Database:** `.artifacts/test-benchmarks.db`
### Stored Metrics
**Test-Level:**
- Duration (ms)
- Status (passed/failed/skipped)
- Assertion count
- Error messages
**Suite-Level:**
- Total duration (seconds)
- Success rate (%)
- Pass/fail/skip counts
- Git commit/branch
**Performance:**
- Agent spawn time
- Redis operation latency
- File I/O duration
- Memory usage
## Regression Detection
### Automatic Alerts
```bash
# Critical: Test started failing
alert_type: test_failure
severity: critical
message: "TEST 5: Product Owner Decision FAILED (was passing)"
# Warning: Performance degraded
alert_type: performance_regression
severity: warning
message: "Loop 3 spawn time increased 25% (baseline: 5s, current: 6.25s)"
# Info: Success rate dropped
alert_type: success_rate_drop
severity: info
message: "Suite success rate: 88% → 77% (threshold: 10%)"
```
### Threshold Configuration
```bash
# Default: 10% regression threshold
--threshold 0.10
# Strict: 5% threshold
--threshold 0.05
# Relaxed: 20% threshold
--threshold 0.20
```
## Output Formats
### Text (Console)
```
==========================================
CFN Test Suite Results
==========================================
Suite: Hello World
Layer 0: ✅ PASSED (58.3s)
Layer 5: ✅ PASSED (115.7s)
Layer 6: ✅ PASSED (172.4s)
Layer 7: ✅ PASSED (148.2s)
Suite: CFN E2E
TEST 1: ✅ PASSED (28.1s)
TEST 2: ✅ PASSED (54.6s)
TEST 3: ⚠️ SKIPPED
TEST 4: ✅ PASSED (87.3s)
TEST 5: ✅ PASSED (59.2s)
TEST 6: ✅ PASSED (118.9s)
TEST 7: ✅ PASSED (9.4s)
Total: 11 tests, 9 passed, 0 failed, 2 skipped
Duration: 851.1s
Success Rate: 81.8%
Regressions Detected: 0
==========================================
```
### JSON (API/CI)
```json
{
"timestamp": "2025-11-04T02:00:00Z",
"git_commit": "abc123",
"git_branch": "main",
"suites": {
"hello-world": {
"total": 4,
"passed": 4,
"failed": 0,
"skipped": 0,
"duration": 494.6,
"success_rate": 1.0
},
"cfn-e2e": {
"total": 9,
"passed": 7,
"failed": 0,
"skipped": 2,
"duration": 356.5,
"success_rate": 0.778
}
},
"regressions": [],
"baseline_comparison": {
"duration_change": "+2.3%",
"success_rate_change": "+0.0%"
}
}
```
### HTML (Report)
Generates interactive HTML report with:
- Trend graphs (Chart.js)
- Regression highlights
- Test history
- Performance metrics
## Baseline Management
### Establish Baseline
```bash
# First run establishes baseline
./.claude/skills/cfn-test-runner/run-all-tests.sh --suite all --benchmark
# Mark as baseline
sqlite3 .artifacts/test-benchmarks.db << EOF
UPDATE test_runs SET environment = 'baseline' WHERE id = (SELECT MAX(id) FROM test_runs);