claude-flow-novice
Version:
Claude Flow Novice - Advanced orchestration platform for multi-agent AI workflows with CFN Loop architecture Includes CodeSearch (hybrid SQLite + pgvector), mem0/memgraph specialists, and all CFN skills.
565 lines (434 loc) • 16.7 kB
Markdown
# Confidence Score Aggregation Module
**Phase 4 · P1 Priority · Loop 3 Score Aggregation**
## Overview
The Confidence Score Aggregator is a comprehensive TypeScript module for collecting, analyzing, and reporting on confidence scores from CFN Loop 3 (implementer) agents. It provides advanced statistical analysis, outlier detection, and weighted averaging capabilities.
**Module Location:** `src/helpers/confidence-aggregator.ts`
**Test Suite:** `tests/confidence-aggregator.test.ts`
**Implementation:** 230+ LOC
**Test Coverage:** 53 comprehensive tests
## Key Features
### 1. Score Validation
- Validates confidence scores are within valid range [0.0, 1.0]
- Filters invalid scores while preserving valid ones
- Provides detailed validation error reporting
### 2. Statistical Analysis
- **Min/Max:** Identifies extreme values
- **Average:** Calculates mean confidence
- **Median:** Finds middle value (robust to outliers)
- **Standard Deviation:** Measures consistency
- **Variance:** Calculates squared deviation
- **Range:** Shows span of scores
### 3. Outlier Detection
- **IQR Method:** Interquartile Range analysis
- **Customizable Threshold:** Adjust sensitivity (default 1.5)
- **Outlier Identification:** Lists anomalous scores
- **Percentage Calculation:** Shows outlier prevalence
### 4. Aggregation Metrics
- **Aggregate Score:** Confidence average
- **Confidence Rating:** Composite metric based on consistency and agent count
- **Validity Status:** Boolean indicating data quality
### 5. Weighted Averaging
- **Default Weighting:** Equal distribution
- **Custom Weights:** Agent-specific contribution
- **Weight Normalization:** Automatic scaling to sum = 1.0
- **Contribution Map:** Shows each agent's impact
### 6. Analysis by Agent Type
- **Grouping:** Organize scores by agent type
- **Type Statistics:** Separate analysis per type
- **Performance Comparison:** Identify high/low performing types
- **Low Performer Detection:** Find problematic agent types
### 7. Reporting
- **Summary Generation:** Human-readable reports
- **Structured Output:** Complete aggregation result object
- **Error Reporting:** Detailed validation messages
- **Timestamp Tracking:** When aggregation occurred
## Core Interfaces
```typescript
/**
* Individual confidence score from an agent
*/
export interface ConfidenceScore {
agentId: string; // Unique agent identifier
agentType: string; // Type of agent (e.g., 'backend-dev')
score: number; // Confidence value [0.0, 1.0]
timestamp: number; // Unix timestamp of score
metadata?: Record<string, any>; // Optional additional data
}
/**
* Statistical measures for aggregated scores
*/
export interface ScoreStatistics {
min: number; // Minimum score
max: number; // Maximum score
average: number; // Mean score
median: number; // Median score
stddev: number; // Standard deviation
count: number; // Number of scores
range: number; // Max - min
variance: number; // Squared standard deviation
}
/**
* Aggregated confidence with full analysis
*/
export interface AggregatedConfidence {
scores: ConfidenceScore[]; // Valid scores only
statistics: ScoreStatistics; // Statistical analysis
outliers: ConfidenceScore[]; // Detected outliers
aggregateScore: number; // Average of scores
confidence: number; // Composite confidence metric
isValid: boolean; // Data quality indicator
validationErrors: string[]; // List of validation issues
timestamp: number; // When aggregation occurred
}
/**
* Weighted confidence score result
*/
export interface WeightedAggregation {
weights: Map<string, number>; // Original weights
weightedScore: number; // Calculated weighted average
normalizedWeights: Map<string, number>; // Normalized to sum = 1.0
contributionMap: Map<string, number>; // Per-agent contribution
}
```
## API Reference
### validateScoreRange(score: number): boolean
Validates that a score is within valid range [0.0, 1.0].
```typescript
validateScoreRange(0.85); // → true
validateScoreRange(1.5); // → false
validateScoreRange(-0.1); // → false
```
### aggregateScores(scores: ConfidenceScore[]): AggregatedConfidence
Core function for aggregating confidence scores with complete statistical analysis.
```typescript
const scores: ConfidenceScore[] = [
{ agentId: 'a1', agentType: 'backend', score: 0.92, timestamp: Date.now() },
{ agentId: 'a2', agentType: 'backend', score: 0.88, timestamp: Date.now() },
{ agentId: 'a3', agentType: 'frontend', score: 0.95, timestamp: Date.now() }
];
const result = aggregateScores(scores);
console.log(result.statistics.average); // 0.917
console.log(result.confidence); // ~0.85
console.log(result.isValid); // true
```
**Returns:**
- Statistical analysis of all valid scores
- Detected outliers (IQR method)
- Composite confidence metric
- Validation error list
### detectOutliers(scores: ConfidenceScore[], threshold?: number): OutlierDetectionResult
Identifies anomalous scores using Interquartile Range (IQR) method.
```typescript
const result = detectOutliers(scores, 1.5); // Default threshold = 1.5
console.log(result.outlierCount); // Number of outliers
console.log(result.outlierPercentage); // % of data
console.log(result.outliers); // List of outlier scores
console.log(result.isOutlier(score)); // Predicate function
```
**Parameters:**
- `scores`: Array of confidence scores
- `threshold`: IQR multiplier for sensitivity (default: 1.5)
- Lower threshold = stricter outlier detection
- Higher threshold = lenient outlier detection
**Returns:**
- `outliers`: Identified outlier scores
- `isOutlier()`: Function to test if a score is outlier
- `outlierCount`: Number of outliers
- `outlierPercentage`: Percentage of scores that are outliers
### calculateWeightedAverage(scores: ConfidenceScore[], weightMap?: Map): WeightedAggregation
Calculates weighted average with custom or equal weights.
```typescript
// Equal weights (default)
const result1 = calculateWeightedAverage(scores);
console.log(result1.weightedScore); // 0.917
// Custom weights - prioritize senior developers
const weights = new Map([
['senior-1', 3],
['mid-1', 2],
['junior-1', 1]
]);
const result2 = calculateWeightedAverage(scores, weights);
console.log(result2.weightedScore);
console.log(result2.normalizedWeights); // Normalized to sum = 1.0
console.log(result2.contributionMap); // Per-agent contribution
```
### groupByAgentType(scores: ConfidenceScore[]): Map<string, ConfidenceScore[]>
Groups scores by agent type for type-specific analysis.
```typescript
const grouped = groupByAgentType(scores);
// Map {
// 'backend-dev': [ ... ],
// 'frontend-dev': [ ... ],
// 'devops-engineer': [ ... ]
// }
```
### analyzeByAgentType(scores: ConfidenceScore[]): Map<string, ScoreStatistics>
Calculates statistics for each agent type separately.
```typescript
const analysis = analyzeByAgentType(scores);
const backendStats = analysis.get('backend-dev');
console.log(backendStats?.average); // Backend avg confidence
console.log(backendStats?.stddev); // Backend consistency
console.log(backendStats?.count); // Number of backend agents
```
### identifyLowPerformers(scores: ConfidenceScore[], threshold?: number): ConfidenceScore[]
Identifies agents from types with below-threshold average scores.
```typescript
// Find agents from poorly performing agent types
const lowPerformers = identifyLowPerformers(scores, 0.75);
console.log(lowPerformers.length); // Number of low performers
// Returns all agents from agent types averaging < 0.75
```
### generateSummary(aggregated: AggregatedConfidence): string
Generates human-readable summary report.
```typescript
const aggregated = aggregateScores(scores);
const summary = generateSummary(aggregated);
console.log(summary);
// Output:
// === Confidence Score Aggregation Summary ===
// Total Agents: 5
// Aggregate Score: 0.904
// Confidence: 0.856
//
// Statistics:
// Min: 0.870
// Max: 0.950
// Average: 0.904
// Median: 0.900
// StdDev: 0.028
// Range: 0.080
```
## Usage Examples
### Example 1: Basic Aggregation
```typescript
import {
ConfidenceScore,
aggregateScores
} from '@orchestrator/helpers/confidence-aggregator';
const scores: ConfidenceScore[] = [
{ agentId: 'impl-1', agentType: 'backend-dev', score: 0.92, timestamp: Date.now() },
{ agentId: 'impl-2', agentType: 'backend-dev', score: 0.88, timestamp: Date.now() },
{ agentId: 'impl-3', agentType: 'frontend-dev', score: 0.95, timestamp: Date.now() }
];
const result = aggregateScores(scores);
if (result.isValid) {
console.log(`Average confidence: ${result.aggregateScore.toFixed(3)}`);
console.log(`Overall confidence: ${result.confidence.toFixed(3)}`);
} else {
console.error('Validation errors:', result.validationErrors);
}
```
### Example 2: Outlier Detection
```typescript
import { aggregateScores, detectOutliers } from '@orchestrator/helpers/confidence-aggregator';
const aggregated = aggregateScores(scores);
if (aggregated.outliers.length > 0) {
console.warn('Outliers detected:');
for (const outlier of aggregated.outliers) {
console.warn(` ${outlier.agentId}: ${outlier.score.toFixed(3)}`);
}
}
```
### Example 3: Performance Analysis by Type
```typescript
import { analyzeByAgentType, identifyLowPerformers } from '@orchestrator/helpers/confidence-aggregator';
const analysis = analyzeByAgentType(scores);
// Show statistics per type
for (const [type, stats] of analysis.entries()) {
console.log(`${type}:`);
console.log(` Avg: ${stats.average.toFixed(3)}`);
console.log(` StdDev: ${stats.stddev.toFixed(3)}`);
console.log(` Count: ${stats.count}`);
}
// Identify problematic types
const lowPerformers = identifyLowPerformers(scores, 0.80);
if (lowPerformers.length > 0) {
console.warn('Low performing agent types detected');
}
```
### Example 4: Weighted Aggregation
```typescript
import { calculateWeightedAverage } from '@orchestrator/helpers/confidence-aggregator';
// Prioritize scores from more experienced agents
const weights = new Map([
['senior-backend-1', 3],
['mid-backend-1', 2],
['junior-backend-1', 1]
]);
const weighted = calculateWeightedAverage(scores, weights);
console.log(`Standard average: ${weighted.weightedScore.toFixed(3)}`);
console.log('Agent contributions:');
for (const [agentId, contribution] of weighted.contributionMap.entries()) {
console.log(` ${agentId}: ${(contribution * 100).toFixed(1)}%`);
}
```
### Example 5: Loop 3 Integration
```typescript
import { aggregateScores, generateSummary } from '@orchestrator/helpers/confidence-aggregator';
// Collect scores from all Loop 3 implementers
async function aggregateLoop3Scores(taskId: string, iteration: number) {
const scores = await collectScoresFromRedis(taskId, iteration);
const result = aggregateScores(scores);
// Generate report
const summary = generateSummary(result);
console.log(summary);
// Use aggregate for gate check
const passGate = result.aggregateScore >= 0.85 && result.isValid;
return { result, passGate };
}
```
## Statistical Details
### Confidence Calculation
The confidence metric combines three factors:
```
confidence = (aggregateScore + consistencyBonus + agentCountBonus) / 3
where:
aggregateScore = average of all valid scores
consistencyBonus = max(0, 1 - (stddev / 0.5))
agentCountBonus = min(1, agentCount / 5)
```
This favors:
- Higher average scores
- More consistent scores (lower stddev)
- More agents reporting confidence
### Outlier Detection (IQR Method)
```
Q1 = value at 25th percentile
Q3 = value at 75th percentile
IQR = Q3 - Q1
Lower Bound = Q1 - (threshold × IQR)
Upper Bound = Q3 + (threshold × IQR)
Outlier = score < Lower Bound OR score > Upper Bound
```
Default threshold 1.5 is standard statistical practice.
## Error Handling
### Validation Errors
The module gracefully handles invalid data:
```typescript
const mixed = [
{ agentId: 'a1', agentType: 'dev', score: 0.90, timestamp: Date.now() },
{ agentId: 'a2', agentType: 'dev', score: 1.5, timestamp: Date.now() }, // Invalid
{ agentId: 'a3', agentType: 'dev', score: -0.1, timestamp: Date.now() } // Invalid
];
const result = aggregateScores(mixed);
console.log(result.isValid); // false
console.log(result.statistics.count); // 1 (only valid score counted)
console.log(result.validationErrors); // Array of error messages
```
### Edge Cases
The module handles:
- Empty score arrays
- Null/undefined input
- Single score (stddev = 0)
- All identical scores (stddev = 0)
- Insufficient data for outlier detection (< 4 scores)
## Testing
### Test Coverage: 53 Tests
**Test Categories:**
1. **Score Validation (7 tests)**
- Valid ranges
- Invalid scores
- Non-numeric values
2. **Outlier Detection (6 tests)**
- Uniform scores
- Extreme values
- Insufficient data
- Custom thresholds
3. **Aggregation (17 tests)**
- Statistics calculation
- Validity checking
- Confidence scoring
- Invalid score filtering
- Metadata preservation
4. **Weighted Averaging (7 tests)**
- Equal weights
- Custom weights
- Weight normalization
- Contribution mapping
5. **Grouping & Analysis (7 tests)**
- Type-based grouping
- Per-type statistics
- Low performer detection
6. **Reporting (5 tests)**
- Summary generation
- Error reporting
- Statistics inclusion
7. **Integration (4 tests)**
- End-to-end pipeline
- Real-world scenarios
- Loop 2 consensus validation
**Running Tests:**
```bash
# All confidence aggregator tests
npm test -- confidence-aggregator.test.ts
# Watch mode
npm test -- confidence-aggregator.test.ts --watch
# With coverage
npm test -- confidence-aggregator.test.ts --coverage
```
## Performance Characteristics
- **Time Complexity:**
- Aggregation: O(n log n) due to sorting for median/quartiles
- Weighted averaging: O(n)
- Outlier detection: O(n log n)
- **Space Complexity:** O(n) for storing results
- **Typical Performance:**
- 1,000 scores: < 1ms
- 10,000 scores: < 5ms
- 100,000 scores: < 50ms
## Integration with CFN Loop v3
### Loop 3 Integration
```typescript
// In orchestrator after Loop 3 completion
const aggregated = aggregateScores(loop3Scores);
// Use for gate check
if (aggregated.aggregateScore >= gateThreshold && aggregated.isValid) {
// Proceed to Loop 2
} else {
// Iterate Loop 3
}
```
### Loop 2 Consensus
```typescript
// Alias confidence scores as consensus for Loop 2
const consensusScores: ConfidenceScore[] = validatorScores.map(v => ({
agentId: v.validatorId,
agentType: 'validator',
score: v.consensusScore,
timestamp: v.reportedAt
}));
const consensus = aggregateScores(consensusScores);
```
## Dependencies
- **No external dependencies** - Pure TypeScript
- **Target Environment:** Node.js 18+, ES2022+
- **Type Safety:** Full strict mode compliance
## Export Path
```typescript
import {
aggregateScores,
detectOutliers,
calculateWeightedAverage,
analyzeByAgentType,
identifyLowPerformers,
generateSummary,
type ConfidenceScore,
type AggregatedConfidence,
type ScoreStatistics,
type WeightedAggregation
} from '@cfn/loop-orchestration';
```
## Future Enhancements
Potential future improvements:
1. **Temporal Analysis** - Track score trends over iterations
2. **Prediction** - Estimate future confidence trajectories
3. **Regression Detection** - Identify quality decline patterns
4. **Custom Metrics** - Pluggable statistical algorithms
5. **Persistence** - Store aggregation history
6. **Visualization** - Generate charts/graphs
## References
- **Test Suite:** `tests/confidence-aggregator.test.ts` (53 tests)
- **Type Definitions:** Core interfaces in module header
- **CFN Loop Documentation:** `.claude/skills/cfn-loop-orchestration/`
- **Gate Checking:** Uses aggregateScore for Loop 3 gate validation