aiwg
Version:
Deployment tool and support utility for AI context. Copies agents, skills, commands, rules, and behaviors into the paths each AI platform reads (Claude Code, Codex, Copilot, Cursor, Warp, OpenClaw, and 6 more) so one source of truth works across 10 platfo
386 lines (298 loc) • 13.2 kB
Markdown
# Gap Analysis Report
## Metadata
```yaml
report_type: gap-analysis
report_date: YYYY-MM-DD
corpus_size: 0 # Number of sources analyzed
coverage_score: 0 # 0-100 overall coverage
identified_gaps: 0 # Total gaps found
high_priority_gaps: 0
medium_priority_gaps: 0
low_priority_gaps: 0
analyst: gap-analysis-agent
version: 1.0.0
status: draft # draft | final | archived
```
## Executive Summary
**Corpus Coverage:** X% (Target: 100%)
**Gaps Identified:** X total (Y high priority, Z medium, W low)
**Recommendation:** [Fill high-priority gaps | Coverage sufficient | Expand corpus]
**Key Findings:**
- [Most critical gap]
- [Second critical gap]
- [Notable insight about corpus composition]
## Corpus Overview
### Size and Composition
| Metric | Count | Percentage |
|--------|-------|------------|
| **Total Sources** | 0 | 100% |
| Academic Papers | 0 | 0% |
| Preprints | 0 | 0% |
| Blog Posts | 0 | 0% |
| Documentation | 0 | 0% |
| Books/Chapters | 0 | 0% |
### Date Range Distribution
| Year Range | Count | Percentage |
|------------|-------|------------|
| 2024-2026 | 0 | 0% |
| 2022-2023 | 0 | 0% |
| 2020-2021 | 0 | 0% |
| 2019 or earlier | 0 | 0% |
### Quality Distribution
| GRADE Rating | Count | Percentage |
|--------------|-------|------------|
| High | 0 | 0% |
| Moderate | 0 | 0% |
| Low | 0 | 0% |
| Very Low | 0 | 0% |
| Not Assessed | 0 | 0% |
## Coverage Analysis
### Topic Coverage
#### Expected Topics (Based on Taxonomy)
| Topic | Expected Coverage | Actual Coverage | Gap Size | Priority |
|-------|------------------|-----------------|----------|----------|
| Multi-agent frameworks | 30% (7-8 sources) | 48% (12 sources) | ✅ Exceeded | - |
| Prompt engineering | 25% (6-7 sources) | 32% (8 sources) | ✅ Met | - |
| Agent memory systems | 20% (5 sources) | 12% (3 sources) | ⚠️ Under-represented | Medium |
| Agent tool use | 15% (3-4 sources) | 8% (2 sources) | ⚠️ Under-represented | Medium |
| **Agent safety** | **10% (2-3 sources)** | **0% (0 sources)** | **❌ Missing** | **High** |
| **Evaluation methods** | **10% (2-3 sources)** | **0% (0 sources)** | **❌ Missing** | **High** |
**Overall Topic Coverage:** 66% (4/6 expected topics covered)
#### Topic Coverage Gaps Identified
##### GAP-T-001: Agent Safety (Missing Topic)
- **Expected Coverage:** 2-3 sources (10%)
- **Actual Coverage:** 0 sources (0%)
- **Gap Size:** 100% missing
- **Priority:** High
- **Impact:** Critical for production AIWG deployment
- **Recommendation:** Acquire 3-5 sources on AI safety, agent alignment, constitutional AI
- **Search Query:** `"AI safety" OR "agent alignment" OR "constitutional AI"`
- **Expected Sources:** REF-025 (Constitutional AI), safety frameworks, guardrails research
##### GAP-T-002: Evaluation Methods (Missing Topic)
- **Expected Coverage:** 2-3 sources (10%)
- **Actual Coverage:** 0 sources (0%)
- **Gap Size:** 100% missing
- **Priority:** High
- **Impact:** Cannot validate AIWG agent performance without evaluation framework
- **Recommendation:** Acquire 2-3 sources on agent benchmarking, task performance metrics
- **Search Query:** `"agent evaluation" OR "benchmarking" OR "task performance metrics"`
##### GAP-T-003: Agent Tool Use (Under-Represented)
- **Expected Coverage:** 3-4 sources (15%)
- **Actual Coverage:** 2 sources (8%)
- **Gap Size:** 47% under-represented
- **Priority:** Medium
- **Impact:** Limited evidence for tool integration patterns
- **Recommendation:** Acquire 1-2 additional sources on function calling, API usage
- **Search Query:** `"agent tool use" OR "function calling" OR "API integration"`
### Citation Completeness
#### Citation Network Analysis
| Metric | Count | Status |
|--------|-------|--------|
| **Total Citations** | 245 | - |
| **In Corpus** | 233 | ✅ |
| **Orphaned Citations** | 12 | ⚠️ |
| **Citation Completeness** | 95% | ✅ Good |
#### Orphaned Citations (Frequently Cited, Not in Corpus)
##### GAP-C-001: Constitutional AI Paper (5 citations, not in corpus)
- **Title:** "Constitutional AI: Harmlessness from AI Feedback"
- **Authors:** Bai, Y., et al. (Anthropic)
- **Year:** 2022
- **Cited By:** REF-012, REF-016, REF-021, REF-023, REF-024
- **Priority:** High
- **Rationale:** Cited by 20% of corpus (5/25 papers), foundational for AI safety
- **Action:** Acquire as REF-025
##### GAP-C-002: Chain-of-Thought Prompting (4 citations, not in corpus)
- **Title:** "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models"
- **Authors:** Wei, J., et al. (Google)
- **Year:** 2022
- **Cited By:** REF-016, REF-017, REF-021, REF-023
- **Priority:** High
- **Rationale:** Foundational for prompt engineering approaches
- **Action:** Acquire as REF-026
##### GAP-C-003: ReAct Framework (3 citations, not in corpus)
- **Title:** "ReAct: Synergizing Reasoning and Acting in Language Models"
- **Authors:** Yao, S., et al.
- **Year:** 2022
- **Cited By:** REF-018, REF-023, REF-024
- **Priority:** Medium
- **Rationale:** Influential for reasoning + action agents
- **Action:** Acquire as REF-027
[... list all 12 orphaned citations with priority ...]
### Source Diversity
#### Publication Type Diversity
| Type | Count | Percentage | Target | Gap |
|------|-------|------------|--------|-----|
| Academic Papers | 18 | 72% | 60-70% | ✅ Met |
| Blog Posts | 5 | 20% | 10-20% | ✅ Met |
| Documentation | 2 | 8% | 5-10% | ✅ Met |
| **Books/Chapters** | **0** | **0%** | **5-10%** | **❌ Missing** |
##### GAP-D-001: No Books or Book Chapters
- **Expected:** 1-2 sources (5-10%)
- **Actual:** 0 sources (0%)
- **Priority:** Low
- **Impact:** Missing foundational/comprehensive treatments
- **Recommendation:** Acquire 1-2 authoritative books on multi-agent systems or LLM applications
- **Search Query:** `"multi-agent systems" book OR "LLM applications" book`
#### Date Range Analysis
| Range | Count | Percentage | Assessment |
|-------|-------|------------|------------|
| 2024-2026 | 15 | 60% | ✅ Good (recent research) |
| 2022-2023 | 8 | 32% | ✅ Good (recent history) |
| 2020-2021 | 2 | 8% | ⚠️ Limited |
| **2019 or earlier** | **0** | **0%** | **❌ Missing foundational work** |
##### GAP-D-002: No Foundational Papers (2019 or Earlier)
- **Expected:** 1-2 sources (5%)
- **Actual:** 0 sources (0%)
- **Priority:** Medium
- **Impact:** Lacks historical context, may miss fundamental concepts
- **Recommendation:** Acquire 2-3 foundational papers on multi-agent systems (pre-LLM era)
- **Search Query:** `"multi-agent systems" AND year:2015-2019`
#### Author Diversity
| Metric | Count | Assessment |
|--------|-------|------------|
| **Unique Authors** | 45 | Good diversity |
| **Top Author Concentration** | 10 sources (40%) | ⚠️ High concentration |
| **Unique Organizations** | 15 | Good diversity |
##### GAP-D-003: Author Concentration (Top 3 Authors in 40% of Sources)
- **Top Authors:** [Author A: 5 sources, Author B: 3 sources, Author C: 2 sources]
- **Priority:** Low
- **Impact:** May indicate limited perspective diversity
- **Recommendation:** Seek sources from different research groups
- **Action:** Review author affiliation distribution in next gap analysis
## Coverage Score Calculation
### Dimension Scores
| Dimension | Weight | Score | Weighted Score |
|-----------|--------|-------|----------------|
| **Topic Coverage** | 50% | 66/100 | 33.0 |
| **Citation Completeness** | 30% | 95/100 | 28.5 |
| **Source Diversity** | 20% | 75/100 | 15.0 |
**Overall Coverage Score:** 76.5 / 100
**Interpretation:**
- **70-79:** Good coverage with room for improvement
- **Target:** 85+ (comprehensive coverage)
## Recommendations
### High Priority (Address Before Construction Phase)
#### Recommendation 1: Fill "Agent Safety" Topic Gap
- **Action:** Acquire 3-5 sources on AI safety, agent alignment, constitutional AI
- **Search Query:** `"AI safety" OR "agent alignment" OR "constitutional AI"`
- **Target Sources:**
- REF-025: Constitutional AI (Anthropic)
- Safety frameworks for LLM applications
- Guardrails and monitoring approaches
- **Estimated Time:** 2 hours (discovery + acquisition + documentation)
- **Impact:** +8% topic coverage
- **Owner:** Discovery Agent → Acquisition Agent → Documentation Agent
#### Recommendation 2: Fill "Evaluation Methods" Topic Gap
- **Action:** Acquire 2-3 sources on agent evaluation, benchmarking
- **Search Query:** `"agent evaluation" OR "benchmarking" OR "task performance metrics"`
- **Target Sources:**
- Agent benchmarking papers (HELM, BIG-Bench)
- Human evaluation frameworks
- Automatic evaluation metrics
- **Estimated Time:** 1.5 hours
- **Impact:** +6% topic coverage
- **Owner:** Discovery Agent
#### Recommendation 3: Acquire Orphaned Citations (Top 3)
- **Action:** Add Constitutional AI, Chain-of-Thought, ReAct papers
- **Search Query:** Exact titles (known works)
- **Estimated Time:** 1 hour (known sources, direct acquisition)
- **Impact:** +3% citation completeness (12 → 9 orphaned)
- **Owner:** Acquisition Agent
### Medium Priority (Address During Construction Phase)
#### Recommendation 4: Add Foundational Papers (2019 or Earlier)
- **Action:** Acquire 2-3 foundational multi-agent systems papers
- **Search Query:** `"multi-agent systems" AND year:2015-2019`
- **Rationale:** Establish historical context
- **Estimated Time:** 2 hours
- **Impact:** +8% date diversity
- **Owner:** Discovery Agent
#### Recommendation 5: Address Under-Represented Topics
- **Topics:** Agent tool use (needs 1-2 more sources), Agent memory (needs 2 more)
- **Estimated Time:** 1.5 hours total
- **Impact:** +10% topic coverage
- **Owner:** Discovery Agent
### Low Priority (Post-v1.0 Enhancement)
#### Recommendation 6: Add Book Sources
- **Action:** Acquire 1-2 authoritative books on multi-agent systems
- **Rationale:** Comprehensive treatments for reference
- **Estimated Time:** 3 hours (books require careful review)
- **Impact:** +8% publication diversity
- **Owner:** Manual review + Acquisition
#### Recommendation 7: Complete Citation Metadata
- **Action:** Document reference lists for 5 sources missing citation metadata
- **Estimated Time:** 30 minutes
- **Impact:** Citation network completeness
- **Owner:** Documentation Agent
## Gap-Filling Action Plan
### Immediate Actions (Next 7 Days)
| Action | Priority | Owner | Time | Impact |
|--------|----------|-------|------|--------|
| Acquire Agent Safety sources | High | Discovery Agent | 2h | +8% topic |
| Acquire Evaluation sources | High | Discovery Agent | 1.5h | +6% topic |
| Acquire orphaned citations (top 3) | High | Acquisition Agent | 1h | +3% citation |
**Total Time Investment:** 4.5 hours
**Expected Coverage Improvement:** 76.5% → 85%+ (Target achieved)
### Follow-Up Actions (Next 30 Days)
| Action | Priority | Owner | Time | Impact |
|--------|----------|-------|------|--------|
| Acquire foundational papers | Medium | Discovery Agent | 2h | +8% date diversity |
| Fill under-represented topics | Medium | Discovery Agent | 1.5h | +10% topic |
### Future Enhancements (Post-v1.0)
| Action | Priority | Owner | Time | Impact |
|--------|----------|-------|------|--------|
| Add book sources | Low | Manual | 3h | +8% pub diversity |
| Complete citation metadata | Low | Documentation Agent | 30min | Citation network |
## Gap Analysis Changelog
### 2026-01-25 (Initial Analysis)
- **Corpus Size:** 25 sources
- **Coverage Score:** 76.5%
- **Gaps Identified:** 8 total (3 high, 3 medium, 2 low)
- **Action:** Begin high-priority gap-filling
### [Future Entry Template]
- **Date:** YYYY-MM-DD
- **Corpus Size:** [Updated size]
- **Coverage Score:** [Updated score]
- **Changes:** [What was added/removed]
- **Gaps Resolved:** [Which gaps were filled]
- **New Gaps:** [Any new gaps identified]
## Validation Rules
### Required Metrics
- `corpus_size`: Integer >0
- `coverage_score`: Integer 0-100
- `identified_gaps`: Integer ≥0
- `report_date`: ISO 8601 date
### Coverage Thresholds
- **Excellent:** 85-100% (comprehensive)
- **Good:** 70-84% (sufficient for most purposes)
- **Fair:** 50-69% (needs improvement)
- **Poor:** <50% (significant gaps)
### Gap Priority Criteria
- **High:** Missing expected topics (0% coverage), >10 orphaned citations
- **Medium:** Under-represented topics (<50% expected coverage), 5-10 orphaned citations
- **Low:** Minor diversity gaps, metadata incompleteness
## Agent Responsibilities
**Produced by:** Gap Analysis Agent (UC-RF-009)
**Updated by:** Gap Analysis Agent (periodic re-analysis)
**Used by:** Discovery Agent (gap-filling search), User (strategic planning), Workflow Agent (phase gate decisions)
## References
- @$AIWG_ROOT/agentic/code/frameworks/research-complete/elaboration/use-cases/UC-RF-009-perform-gap-analysis.md - Gap analysis use case
- @.aiwg/research/sources/metadata/ - Source metadata for analysis
- @.aiwg/research/knowledge/extractions/ - Citation network data
- @.aiwg/research/config/topic-taxonomy.yaml - Expected topic coverage (if defined)
**Template Version:** 1.0.0
**Last Updated:** 2026-01-25
**Owner:** Gap Analysis Agent