claude-flow-novice
Version:
Claude Flow Novice - Advanced orchestration platform for multi-agent AI workflows with CFN Loop architecture Includes CodeSearch (hybrid SQLite + pgvector), mem0/memgraph specialists, and all CFN skills.
405 lines (311 loc) • 11.4 kB
Markdown
# Edge Case Tracking System
## Overview
The Edge Case Tracking System captures skill execution failures, detects recurring patterns, and automatically generates skill update proposals to improve system resilience.
## Architecture
```
Skill Execution
├─ Success (exit 0) ──> Continue
└─ Failure (exit ≠0) ──> track-edge-case.sh
├─ Record edge case
├─ Calculate occurrence count
└─ Check threshold (≥3)
├─ Below threshold ──> Store only
└─ Above threshold ──> generate-skill-update.sh
├─ Create test case
├─ Propose logic changes
├─ Update documentation
└─ Increment version
```
## Database Schema
### edge_cases Table
```sql
CREATE TABLE edge_cases (
id INTEGER PRIMARY KEY AUTOINCREMENT,
skill_name TEXT NOT NULL,
skill_version TEXT NOT NULL,
exit_code INTEGER NOT NULL,
input_params TEXT NOT NULL,
expected_output TEXT,
actual_output TEXT,
error_message TEXT,
timestamp TEXT DEFAULT (datetime('now')),
occurrence_count INTEGER DEFAULT 1,
edge_case_hash TEXT NOT NULL UNIQUE,
status TEXT DEFAULT 'new' CHECK(status IN ('new', 'analyzing', 'proposal_generated', 'resolved')),
metadata TEXT
);
```
### Indexes
- `idx_edge_cases_skill`: (skill_name, skill_version)
- `idx_edge_cases_status`: (status)
- `idx_edge_cases_hash`: (edge_case_hash)
- `idx_edge_cases_timestamp`: (timestamp)
## Usage
### 1. Record Edge Case
```bash
track-edge-case.sh --action record \
--skill-name "cfn-coordination" \
--skill-version "1.0.0" \
--exit-code 1 \
--input-params "task-id=123 timeout=30" \
--expected-output "success" \
--actual-output "" \
--error-message "Connection timeout after 30s"
```
**Output:**
```
Recorded new edge case: a3f8b2c1d4e5f6...
```
### 2. Query Recurring Edge Cases
```bash
track-edge-case.sh --action query --skill-name "cfn-coordination"
```
**Output:**
```
skill_name skill_version exit_code occurrence_count status timestamp
----------------- ------------- --------- ---------------- -------------------- -------------------
cfn-coordination 1.0.0 1 5 proposal_generated 2025-11-15 10:30:00
cfn-agent-spawn 2.1.0 127 3 new 2025-11-15 09:15:00
```
### 3. Get Edge Case Details
```bash
track-edge-case.sh --action details --edge-case-hash "a3f8b2c1d4e5f6..."
```
**Output (JSON):**
```json
[
{
"id": 1,
"skill_name": "cfn-coordination",
"skill_version": "1.0.0",
"exit_code": 1,
"input_params": "task-id=123 timeout=30",
"expected_output": "success",
"actual_output": "",
"error_message": "Connection timeout after 30s",
"timestamp": "2025-11-15 10:30:00",
"occurrence_count": 5,
"edge_case_hash": "a3f8b2c1d4e5f6...",
"status": "proposal_generated",
"metadata": "{}"
}
]
```
### 4. Update Edge Case Status
```bash
track-edge-case.sh --action update-status \
--edge-case-hash "a3f8b2c1d4e5f6..." \
--status "resolved"
```
## Edge Case Hash Calculation
The system generates a unique hash for each edge case pattern:
```bash
hash = SHA256(skill_name:exit_code:input_params)
```
**Example:**
- skill_name: `cfn-coordination`
- exit_code: `1`
- input_params: `task-id=123 timeout=30`
- hash: `a3f8b2c1d4e5f6a7b8c9d0e1f2...`
This ensures identical failures are tracked together, incrementing the occurrence count.
## Recurrence Detection
**Threshold:** ≥3 occurrences
When an edge case reaches the recurrence threshold, the system:
1. Logs warning message
2. Automatically triggers `generate-skill-update.sh`
3. Updates status to `analyzing`
4. Creates proposal artifacts
## Skill Update Proposal
When threshold is reached, `generate-skill-update.sh` creates:
```
proposals/
└── cfn-coordination_a3f8b2c1_v1.0.1/
├── PROPOSAL_SUMMARY.md # Overview and next steps
├── test_case.md # Regression test
├── logic_proposal.md # Proposed code changes
└── documentation_update.md # Changelog and migration guide
```
### Proposal Contents
#### PROPOSAL_SUMMARY.md
- Edge case overview
- Pattern analysis
- Approval checklist
- Execution commands
#### test_case.md
- Input parameters
- Expected vs actual output
- Executable test script
- Validation criteria
#### logic_proposal.md
- Input validation enhancements
- Error handling improvements
- Edge case handling logic
- Implementation priority
#### documentation_update.md
- Changelog entry (semantic versioning)
- Migration guide
- Updated examples
- Testing recommendations
## Integration Patterns
### Pattern 1: Skill Wrapper
```bash
#!/usr/bin/env bash
set -euo pipefail
SKILL_NAME="cfn-coordination"
SKILL_VERSION="1.0.0"
INPUT_PARAMS="$*"
# Execute skill
START_TIME=$(date +%s%3N)
output=$(./cfn-coordination.sh "$@" 2>&1) || exit_code=$?
END_TIME=$(date +%s%3N)
if [[ ${exit_code:-0} -ne 0 ]]; then
# Track edge case
/path/to/track-edge-case.sh --action record \
--skill-name "$SKILL_NAME" \
--skill-version "$SKILL_VERSION" \
--exit-code "$exit_code" \
--input-params "$INPUT_PARAMS" \
--actual-output "$output" \
--error-message "$output"
fi
exit ${exit_code:-0}
```
### Pattern 2: Post-Execution Hook
```bash
# .claude/hooks/cfn-skill-execution.sh
#!/usr/bin/env bash
set -euo pipefail
SKILL_NAME="$1"
SKILL_VERSION="$2"
EXIT_CODE="$3"
INPUT_PARAMS="$4"
OUTPUT="$5"
if [[ $EXIT_CODE -ne 0 ]]; then
/path/to/track-edge-case.sh --action record \
--skill-name "$SKILL_NAME" \
--skill-version "$SKILL_VERSION" \
--exit-code "$EXIT_CODE" \
--input-params "$INPUT_PARAMS" \
--actual-output "$OUTPUT"
fi
```
### Pattern 3: Agent Integration
```bash
# Within agent execution
execute_skill() {
local skill_name="$1"
local skill_version="$2"
shift 2
local params="$*"
local output exit_code
output=$("${skill_name}.sh" "$@" 2>&1) || exit_code=$?
if [[ ${exit_code:-0} -ne 0 ]]; then
# Track edge case
track-edge-case.sh --action record \
--skill-name "$skill_name" \
--skill-version "$skill_version" \
--exit-code "$exit_code" \
--input-params "$params" \
--actual-output "$output" \
--metadata "{\"agent_id\": \"$AGENT_ID\"}"
fi
return ${exit_code:-0}
}
```
## Pattern Analysis
The system performs automatic pattern analysis to categorize failures:
### Timeout Patterns
- **Detection:** Error message contains "timeout"
- **Proposal:** Increase timeout threshold, implement retry logic
### Connection Patterns
- **Detection:** Error message contains "connection"
- **Proposal:** Implement connection retry with exponential backoff
### Not Found Patterns
- **Detection:** Error message contains "not found"
- **Proposal:** Add existence check before operation
### Permission Patterns
- **Detection:** Error message contains "permission"
- **Proposal:** Validate permissions before execution
### Command Not Found
- **Detection:** Exit code 127
- **Proposal:** Add dependency validation at script initialization
## Lifecycle Management
### Edge Case States
1. **new** - Newly recorded, below threshold
2. **analyzing** - Threshold reached, analysis in progress
3. **proposal_generated** - Proposal created, awaiting review
4. **resolved** - Fix implemented and validated
### State Transitions
```
new ─────────────> analyzing ─────────> proposal_generated ─────────> resolved
(threshold=3) (auto) (manual review) (deployment)
```
## Metrics and Reporting
### Query All Edge Cases
```bash
sqlite3 workflow-codification.db "SELECT skill_name, COUNT(*), SUM(occurrence_count) FROM edge_cases GROUP BY skill_name ORDER BY SUM(occurrence_count) DESC;"
```
### Query by Status
```bash
sqlite3 workflow-codification.db "SELECT * FROM edge_cases WHERE status = 'proposal_generated';"
```
### Monthly Trend
```bash
sqlite3 workflow-codification.db "
SELECT
date(timestamp) as date,
COUNT(*) as new_edge_cases,
SUM(occurrence_count) as total_occurrences
FROM edge_cases
WHERE timestamp >= date('now', '-30 days')
GROUP BY date(timestamp)
ORDER BY date;
"
```
## Best Practices
### 1. Always Record Failures
Track all skill failures, even if they seem transient. Pattern detection requires data.
### 2. Include Context
Provide `expected_output` when possible to help pattern analysis.
### 3. Review Proposals Promptly
Edge cases impact production. Review and implement proposals within 48 hours.
### 4. Update Status
Mark edge cases as `resolved` after fix deployment to track effectiveness.
### 5. Monitor Occurrence Trends
Weekly review of high-occurrence edge cases to prioritize fixes.
### 6. Use Metadata
Store additional context (agent_id, task_id, environment) for debugging.
## Troubleshooting
### Issue: Duplicate Edge Cases
**Cause:** Hash collision or parameter variations
**Solution:** Review `edge_case_hash` calculation, ensure parameters are normalized
### Issue: Threshold Not Triggering
**Cause:** Each variation creates new hash
**Solution:** Normalize input parameters before hashing
### Issue: Proposal Generation Fails
**Cause:** Missing database record or invalid hash
**Solution:** Verify edge case exists: `track-edge-case.sh --action details --edge-case-hash <hash>`
## Security Considerations
### Sensitive Data in Parameters
- **Risk:** Input parameters may contain credentials
- **Mitigation:** Sanitize parameters before storage, use `metadata` for sensitive context
### Database Access
- **Risk:** Unauthorized access to failure patterns
- **Mitigation:** Set restrictive file permissions (600) on database file
### Proposal Review
- **Risk:** Automatic proposals may introduce vulnerabilities
- **Mitigation:** Always manually review before implementation
## Environment Variables
- `DB_PATH` - Database location (default: `../../../../data/workflow-codification.db`)
- `RECURRENCE_THRESHOLD` - Threshold for proposal generation (default: 3)
- `PROPOSALS_DIR` - Proposal output directory (default: `./proposals`)
## Dependencies
- `bash` ≥4.0
- `sqlite3` ≥3.0
- `coreutils` (sha256sum, date, bc)
## Version History
- **v1.0.0** (2025-11-15) - Initial implementation
- Edge case recording
- Recurrence detection
- Automatic proposal generation
- Pattern analysis