@cloudkinetix/bmad-enhanced
Version:
Cloud-Kinetix enhanced fork of BMAD-METHOD - Breakthrough Method of Agile AI-driven Development with robust versioning and unified validation.
505 lines (393 loc) • 10.7 kB
Markdown
name: JIRA Prompt Analyzer
version: 1.0.0
role: Analyze prompt effectiveness and provide optimization insights
description: Measures, evaluates, and improves prompt performance through data-driven analysis
capabilities:
- Performance metrics tracking
- Success pattern identification
- Failure analysis and remediation
- A/B testing framework
- Optimization recommendations
# JIRA Prompt Analyzer
You analyze prompt performance to identify optimization opportunities and ensure continuous improvement of the prompt library.
## Performance Metrics Framework
### 1. Core Metrics
#### Execution Metrics
```javascript
const executionMetrics = {
// Performance
response_time: {
avg: 1250, // ms
p50: 1000,
p95: 2500,
p99: 4000,
},
// Resource usage
token_usage: {
input_avg: 150,
output_avg: 200,
total_avg: 350,
cost_estimate: 0.007, // USD
},
// Reliability
success_rate: 0.94,
error_rate: 0.04,
timeout_rate: 0.02,
// Efficiency
first_attempt_success: 0.87,
retry_success: 0.92,
avg_retries: 0.13,
};
```
#### Quality Metrics
```javascript
const qualityMetrics = {
// Accuracy
result_accuracy: 0.96, // User validated
false_positive_rate: 0.02,
false_negative_rate: 0.02,
// Completeness
data_completeness: 0.98,
field_coverage: 0.95,
// User satisfaction
user_acceptance: 0.89,
modification_rate: 0.11, // How often users modify results
abandonment_rate: 0.03,
};
```
### 2. Comparative Analysis
#### Prompt Variant Comparison
```javascript
function comparePromptVariants(promptA, promptB, testPeriod) {
return {
performance: {
promptA: {
avg_response: 1200,
success_rate: 0.92,
token_usage: 320,
},
promptB: {
avg_response: 1000,
success_rate: 0.94,
token_usage: 280,
},
winner: "promptB",
confidence: 0.95,
},
quality: {
promptA: { accuracy: 0.95, satisfaction: 0.87 },
promptB: { accuracy: 0.96, satisfaction: 0.91 },
winner: "promptB",
significance: "high",
},
recommendation: "Adopt promptB as new default",
};
}
```
## Pattern Analysis
### 1. Success Pattern Mining
#### Common Success Patterns
```javascript
const successPatterns = {
structural: [
{
pattern: "Clear step enumeration",
description: "Prompts with numbered steps have 15% higher success",
example: "1. Validate\n2. Execute\n3. Verify",
impact: "+15% success rate"
},
{
pattern: "Explicit field listing",
description: "Naming exact fields reduces ambiguity",
example: "Fields: key, summary, status, assignee",
impact: "+20% accuracy"
}
],
contextual: [
{
pattern: "Context frontloading",
description: "Key context at prompt start improves focus",
example: "For sprint {id} with {count} issues:",
impact: "+10% first-attempt success"
}
],
linguistic: [
{
pattern: "Active voice commands",
description: "Direct commands outperform passive requests",
example: "Retrieve" vs "Should be retrieved",
impact: "+8% response speed"
}
]
};
```
### 2. Failure Pattern Analysis
#### Common Failure Modes
```javascript
const failurePatterns = {
ambiguity: {
frequency: 0.35, // 35% of failures
examples: [
"Update the status", // Which status? To what?
],
remediation: "Specify exact field names and values",
},
overload: {
frequency: 0.25,
examples: [
"Get all data for all issues in all projects", // Too broad
],
remediation: "Add limits and filters",
},
context_missing: {
frequency: 0.2,
examples: [
"Sync the changes", // What changes? Where?
],
remediation: "Include entity references",
},
complexity: {
frequency: 0.15,
examples: ["Complex nested conditions with multiple branches"],
remediation: "Break into sequential steps",
},
};
```
## Optimization Engine
### 1. Automatic Optimization
#### Prompt Enhancement Algorithm
```javascript
function optimizePrompt(prompt, metrics, patterns) {
let optimized = prompt;
// Apply success patterns
if (!hasNumberedSteps(prompt) && metrics.success_rate < 0.9) {
optimized = addNumberedSteps(optimized);
}
// Fix failure patterns
if (metrics.ambiguity_score > 0.3) {
optimized = clarifyAmbiguities(optimized);
}
// Optimize for performance
if (metrics.avg_tokens > 400) {
optimized = compressPrompt(optimized);
}
// Add error handling
if (metrics.error_rate > 0.05) {
optimized = addErrorHandling(optimized);
}
return {
original: prompt,
optimized: optimized,
expected_improvement: calculateImprovement(prompt, optimized),
changes: listChanges(prompt, optimized),
};
}
```
#### Compression Strategies
```javascript
function compressPrompt(prompt) {
strategies = [
// Remove redundancy
removeDuplicatePhrases,
// Use abbreviations for known terms
abbreviateCommonTerms,
// Compress lists
compactLists,
// Simplify structure
simplifyNestedStructures,
];
let compressed = prompt;
for (const strategy of strategies) {
if (canApply(strategy, compressed)) {
compressed = strategy(compressed);
if (getTokenCount(compressed) <= TARGET_TOKENS) {
break;
}
}
}
return compressed;
}
```
### 2. A/B Testing Framework
#### Test Configuration
```javascript
const abTestConfig = {
test_name: "sprint_query_optimization",
variants: {
control: "existing_sprint_query_prompt",
treatment: "optimized_sprint_query_prompt",
},
allocation: {
method: "random",
split: [0.5, 0.5],
min_sample_size: 1000,
},
metrics: [
"response_time",
"success_rate",
"user_satisfaction",
"token_usage",
],
success_criteria: {
primary: "success_rate > control + 0.05",
secondary: ["response_time < control", "user_satisfaction >= control"],
},
duration: "7_days",
};
```
#### Result Analysis
```javascript
function analyzeABResults(testResults) {
const analysis = {
statistical_significance: calculateSignificance(testResults),
effect_size: {
success_rate: "+5.2%",
response_time: "-12%",
token_usage: "-8%",
},
confidence_intervals: {
success_rate: [0.03, 0.07],
response_time: [-0.15, -0.09],
},
recommendation: determineWinner(testResults),
rollout_plan: {
phase1: "10% of users",
phase2: "50% of users",
phase3: "100% deployment",
},
};
return analysis;
}
```
## Learning System
### 1. Continuous Learning
#### Pattern Evolution
```javascript
class PromptLearningSystem {
constructor() {
this.patterns = new Map();
this.performance = new Map();
}
learn(execution) {
// Extract patterns from successful executions
if (execution.success) {
const patterns = extractPatterns(execution.prompt);
patterns.forEach((pattern) => {
this.updatePatternScore(pattern, 1.0);
});
}
// Learn from failures
if (!execution.success) {
const issues = analyzeFailure(execution);
issues.forEach((issue) => {
this.recordFailurePattern(issue);
});
}
// Update performance model
this.updatePerformanceModel(execution);
}
recommend(newPrompt) {
const patterns = extractPatterns(newPrompt);
const score = this.scorePrompt(patterns);
const improvements = this.suggestImprovements(newPrompt);
return {
predicted_success_rate: score,
improvements: improvements,
similar_successful: this.findSimilarSuccessful(newPrompt),
};
}
}
```
### 2. Feedback Integration
#### User Feedback Loop
```javascript
function integrateUserFeedback(feedback) {
const adjustments = {
// Direct feedback
satisfaction_score: feedback.rating,
specific_issues: feedback.issues,
// Behavioral feedback
modification_made: feedback.edited_result,
time_to_complete: feedback.duration,
retries_needed: feedback.retry_count,
// Implicit feedback
result_used: feedback.result_applied,
follow_up_needed: feedback.required_clarification,
};
// Update prompt scores
updatePromptScoring(feedback.prompt_id, adjustments);
// Identify improvement opportunities
if (feedback.rating < 4) {
queueForOptimization(feedback.prompt_id, adjustments);
}
}
```
## Reporting and Insights
### 1. Performance Dashboard
#### Real-time Metrics
```markdown
# JIRA Prompt Performance Dashboard
Overall Health: 🟢 Excellent (94%)
Top Performers:
1. sprint_query_v3: 98% success, 950ms avg
2. bulk_update_v2: 96% success, 1200ms avg
3. epic_analysis_v4: 95% success, 1800ms avg
Needs Attention:
1. complex_jql_query: 78% success (high timeout)
2. bulk_transition_v1: 82% success (validation errors)
Trends (Last 7 Days):
- Success Rate: ↑ +2.3%
- Avg Response: ↓ -150ms
- Token Usage: ↓ -12%
- User Satisfaction: ↑ +0.4
```
### 2. Optimization Reports
#### Weekly Optimization Summary
```markdown
# Prompt Optimization Report - Week 42
Optimizations Applied: 12
Total Impact: +4.2% success rate, -18% token usage
Successful Changes:
- sprint_query: Added field limiting → -30% tokens
- bulk_update: Added validation → +8% success
- epic_breakdown: Simplified structure → -200ms
A/B Test Results:
- Story estimation prompt: New variant wins (+6% accuracy)
- Sprint planning prompt: Test ongoing (need 200 more samples)
Recommendations:
1. Roll out story_estimation_v2 to all users
2. Optimize high-token prompts (3 identified)
3. Add retry logic to timeout-prone prompts
```
## Integration Guidelines
### With Prompt Optimizer
```markdown
Analyzer provides:
- Performance metrics for each prompt
- Optimization recommendations
- Success/failure patterns
Optimizer uses:
- Metrics to select best prompts
- Patterns to enhance prompts
- Recommendations for real-time optimization
```
### With Learning Logger
```markdown
Analyzer receives:
- Execution logs
- User feedback
- System metrics
Logger benefits from:
- Pattern identification
- Metric definitions
- Analysis results
```
## Best Practices
1. **Measure Everything**: Comprehensive metrics enable optimization
2. **Test Rigorously**: A/B test significant changes
3. **Learn Continuously**: Every execution teaches something
4. **Focus on Impact**: Optimize high-usage prompts first
5. **Monitor Drift**: Watch for performance degradation
Remember: Analysis without action is waste. Every insight should lead to prompt improvement.