aiwg
Version:
Deployment tool and support utility for AI context. Copies agents, skills, commands, rules, and behaviors into the paths each AI platform reads (Claude Code, Codex, Copilot, Cursor, Warp, OpenClaw, and 6 more) so one source of truth works across 10 platfo
398 lines (306 loc) • 15.4 kB
Markdown
# ADR-RF-003: Quality Assessment Model
## Metadata
- **ID**: ADR-RF-003
- **Title**: Quality Assessment Model for Research Sources
- **Status**: Accepted
- **Created**: 2026-01-25
- **Updated**: 2026-01-25
- **Decision Makers**: Research Framework Architecture Team
- **Related ADRs**: ADR-RF-002 (Provenance Storage), ADR-RF-004 (Artifact Storage)
## Context
The Research Framework requires a systematic approach to assessing source quality to ensure that research decisions are informed by credible, reliable evidence. Quality assessment serves multiple purposes:
1. **Source Selection**: Filtering high-quality sources from large discovery results
2. **Evidence Grading**: Informing how much weight to give claims from different sources
3. **Gap Identification**: Revealing areas lacking high-quality evidence
4. **Reproducibility**: Providing transparent quality rationale for external verification
### Decision Drivers
1. **Academic Standards**: GRADE is the gold standard for evidence quality in systematic reviews
2. **Multi-Dimensional**: Quality is not single-score; different dimensions matter for different uses
3. **Automation**: Must support automated scoring while allowing human override (NFR-QA-03: 95%+ agreement)
4. **Accessibility**: Developers need usable quality signals without PhD-level training
5. **FAIR Compliance**: Quality metadata contributes to FAIR R1.2 (community standards)
6. **Performance**: NFR-QA-01 requires <30 seconds per source assessment
### GRADE Framework Overview
GRADE (Grading of Recommendations Assessment, Development and Evaluation) provides:
| Level | Confidence | Meaning |
|-------|------------|---------|
| High | Strong | Further research unlikely to change confidence |
| Moderate | Moderate | Further research may change estimate |
| Low | Limited | Further research likely to change estimate |
| Very Low | Very limited | Estimate very uncertain |
GRADE factors: Study design, risk of bias, inconsistency, indirectness, imprecision, publication bias.
## Decision
**Adopt a Multi-Dimensional Quality Model combining GRADE principles with weighted dimension scoring.**
### Model Architecture
```
┌────────────────────────────────────────┐
│ Quality Assessment │
│ Model │
└────────────────────┬───────────────────┘
│
┌──────────────┬───────────────┼───────────────┬──────────────┐
│ │ │ │ │
▼ ▼ ▼ ▼ ▼
┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐
│ Authority│ │ Currency │ │ Accuracy │ │ Coverage │ │Objectivity│
│ 30% │ │ 20% │ │ 25% │ │ 15% │ │ 10% │
└────┬─────┘ └────┬─────┘ └────┬─────┘ └────┬─────┘ └────┬─────┘
│ │ │ │ │
└─────────────┴──────────────┴──────────────┴──────────────┘
│
▼
┌─────────────────┐
│ Overall Score │
│ (0-100) │
└────────┬────────┘
│
┌─────────────────┼─────────────────┐
▼ ▼ ▼
┌───────────┐ ┌───────────┐ ┌───────────┐
│ GRADE │ │ FAIR │ │ Source │
│ Rating │ │Compliance │ │ Type │
│(H/M/L/VL) │ │ (0-4) │ │ Context │
└───────────┘ └───────────┘ └───────────┘
```
### Dimension Definitions
#### 1. Authority (30% weight)
Measures the credibility of the source creator and publisher.
| Score | Criteria |
|-------|----------|
| 90-100 | Peer-reviewed journal, established author, high citation count |
| 70-89 | Reputable conference, known organization, moderate citations |
| 50-69 | Technical blog by verified expert, company documentation |
| 30-49 | Self-published, unknown author, limited verification |
| 0-29 | Anonymous, unverifiable, known unreliable source |
**Automated Signals**: Citation count (Semantic Scholar), h-index, venue impact factor, author affiliation.
#### 2. Currency (20% weight)
Measures how recent and up-to-date the source is relative to topic dynamics.
| Score | Criteria |
|-------|----------|
| 90-100 | Published within 2 years, actively maintained |
| 70-89 | Published 2-5 years ago, topic evolves slowly |
| 50-69 | Published 5-10 years ago, foundational work |
| 30-49 | Older than 10 years, may be outdated |
| 0-29 | Significantly outdated, superseded by newer work |
**Automated Signals**: Publication date, last update date, topic velocity (via citation trends).
#### 3. Accuracy (25% weight)
Measures methodological rigor and verifiability.
| Score | Criteria |
|-------|----------|
| 90-100 | RCT/systematic review, methodology documented, data available |
| 70-89 | Peer-reviewed, methodology clear, reproducible |
| 50-69 | Internal review, methodology partially documented |
| 30-49 | No peer review, methodology unclear |
| 0-29 | Unverifiable claims, no methodology |
**Automated Signals**: Study design (extracted from abstract), data availability statement, conflict of interest disclosure.
#### 4. Coverage (15% weight)
Measures depth and comprehensiveness of the source.
| Score | Criteria |
|-------|----------|
| 90-100 | Comprehensive treatment, multiple perspectives, synthesis |
| 70-89 | Detailed coverage of specific aspect |
| 50-69 | Moderate depth, focused scope |
| 30-49 | Surface-level treatment |
| 0-29 | Incomplete, fragmentary coverage |
**Automated Signals**: Document length, section count, citation breadth, abstract completeness.
#### 5. Objectivity (10% weight)
Measures bias and balance.
| Score | Criteria |
|-------|----------|
| 90-100 | Balanced perspectives, conflicts disclosed, neutral tone |
| 70-89 | Mostly objective, minor bias indicators |
| 50-69 | Some bias evident, single perspective dominant |
| 30-49 | Clearly biased, commercial interest |
| 0-29 | Propaganda, undisclosed conflicts |
**Automated Signals**: Sentiment analysis, conflict of interest statements, funding sources.
### Overall Score Calculation
```
Overall Score = (Authority × 0.30) + (Currency × 0.20) + (Accuracy × 0.25) +
(Coverage × 0.15) + (Objectivity × 0.10)
```
### Quality Thresholds
Per BR-QA-001:
| Range | Classification | Recommendation |
|-------|---------------|----------------|
| 70-100 | High Quality | Approved for integration |
| 50-69 | Moderate Quality | Requires review, conditional approval |
| 0-49 | Low Quality | Not recommended, seek alternatives |
### GRADE Rating Mapping
The overall score maps to GRADE-style ratings for academic contexts:
| Overall Score | GRADE Rating | Confidence |
|---------------|--------------|------------|
| 85-100 | High | Strong confidence in evidence |
| 70-84 | Moderate | Moderate confidence, likely suitable |
| 50-69 | Low | Limited confidence, use with caution |
| 0-49 | Very Low | Very limited confidence, seek better sources |
### Source Type Context
Different source types have adjusted expectations:
| Source Type | Authority Expectation | Accuracy Expectation |
|-------------|----------------------|---------------------|
| Peer-reviewed paper | 80+ | 85+ |
| Conference paper | 70+ | 75+ |
| Technical blog | 50+ | 60+ |
| Documentation | 60+ | 70+ |
| Opinion/perspective | 70+ (author) | 40+ (acceptable) |
## Consequences
### Positive
1. **Transparency**: Multi-dimensional scoring explains why sources are rated
2. **Flexibility**: Different use cases can weight dimensions differently
3. **Academic Alignment**: GRADE mapping provides academic credibility
4. **Automation**: Automated signals enable batch processing (NFR-QA-02)
5. **Human Override**: Dimension scores can be manually adjusted with rationale
6. **Progressive Disclosure**: Simple overall score for quick use, detailed dimensions for deep analysis
### Negative
1. **Complexity**: Five dimensions plus GRADE plus FAIR is conceptually heavy
2. **Subjectivity**: Some dimensions (objectivity, coverage) difficult to automate accurately
3. **Domain Variation**: Weights may need adjustment for different research domains
4. **Calibration**: Automated scoring requires ongoing calibration against human judgment
### Risks
| Risk | Likelihood | Impact | Mitigation |
|------|-----------|--------|------------|
| Automated scores disagree with experts | Medium | Medium | NFR-QA-03 target 95% agreement, calibration dataset |
| Over-reliance on metrics | Medium | Low | Display qualitative rationale alongside scores |
| Domain-inappropriate weights | Medium | Medium | Domain-specific weight profiles, user customization |
| Gaming/manipulation | Low | Low | Multiple signals per dimension, human review for critical sources |
## Alternatives Considered
### Option A: GRADE-Only Assessment
**Description**: Use pure GRADE methodology with its four levels and specific criteria.
**Pros**:
- Academic gold standard
- Well-documented methodology
- Recognized credibility
**Cons**:
- Designed for clinical evidence, not software research
- Binary/categorical (not continuous scores)
- Requires expert judgment, hard to automate
- Steep learning curve for developers
**Decision**: Rejected as sole approach. GRADE informs the model but doesn't provide actionable scores for automated filtering.
### Option B: Single Numeric Score
**Description**: Simple 1-5 or 1-100 quality score without dimensions.
**Pros**:
- Maximum simplicity
- Easy to sort/filter
- Quick assessment
**Cons**:
- No explanation of why sources score as they do
- Different users value different quality aspects
- Hard to improve scores (what needs fixing?)
- Opaque decision-making
**Decision**: Rejected. Single scores obscure important quality nuances.
### Option C: Machine Learning Quality Prediction
**Description**: Train ML model to predict quality scores from metadata patterns.
**Pros**:
- Potentially higher accuracy
- Learns patterns humans miss
- Scales infinitely
**Cons**:
- Requires training data (labeled quality assessments)
- Black box explanations
- Model drift over time
- Infrastructure complexity
**Decision**: Deferred to v2.0 (TODO-002 in UC-RF-006). Current approach provides explainable baseline.
### Option D: Crowdsourced Quality Ratings
**Description**: Aggregate quality ratings from multiple users.
**Pros**:
- Diverse perspectives
- Scales with user base
- Self-correcting over time
**Cons**:
- Requires user community
- Cold start problem
- Manipulation risk
- Solo developer context lacks crowd
**Decision**: Rejected. Framework targets solo developers initially; crowdsourcing may be added later.
## Implementation Notes
### Quality Report Schema
```json
{
"$schema": "https://aiwg.io/research/schemas/quality-report.json",
"source_id": "REF-025",
"assessed_at": "2026-01-25T16:00:00Z",
"assessed_by": "quality-agent-v1.0.0",
"dimensions": {
"authority": {
"score": 85,
"signals": {
"citation_count": 1250,
"h_index": 45,
"venue_impact": "high",
"author_affiliation": "Stanford University"
},
"rationale": "High-impact venue, well-cited author with strong affiliation"
},
"currency": {
"score": 90,
"signals": {
"publication_date": "2024-03-15",
"topic_velocity": "high"
},
"rationale": "Recent publication in fast-moving field"
},
"accuracy": {
"score": 95,
"signals": {
"study_design": "systematic_review",
"peer_reviewed": true,
"data_available": true
},
"rationale": "Systematic review with transparent methodology and available data"
},
"coverage": {
"score": 80,
"signals": {
"page_count": 45,
"section_count": 8,
"references_count": 127
},
"rationale": "Comprehensive treatment with extensive references"
},
"objectivity": {
"score": 85,
"signals": {
"conflicts_disclosed": true,
"funding_disclosed": true,
"tone": "neutral"
},
"rationale": "No conflicts, balanced perspective"
}
},
"overall_score": 87,
"grade_rating": "High",
"fair_compliance": 4,
"source_type": "systematic_review",
"recommendation": "Approved for integration",
"limitations": [],
"human_override": null
}
```
### Configurable Weights
Users can customize dimension weights via `.aiwg/research/config/quality-criteria.yaml`:
```yaml
# Custom weight profile for software engineering research
weights:
authority: 0.25 # Less emphasis on traditional credentials
currency: 0.30 # More emphasis on recency (fast-moving field)
accuracy: 0.20 # Methodology still important but flexible
coverage: 0.15 # Unchanged
objectivity: 0.10 # Unchanged
thresholds:
high_quality: 70
moderate_quality: 50
```
### Batch Assessment Performance
To meet NFR-QA-02 (100 sources in <15 minutes):
- Parallel API calls for metadata (Semantic Scholar, CrossRef)
- Cached venue/author metrics
- Progressive scoring (quick estimate first, detailed on demand)
## References
- @$AIWG_ROOT/agentic/code/frameworks/research-complete/elaboration/use-cases/UC-RF-006-assess-source-quality.md - Quality assessment use case
- @$AIWG_ROOT/agentic/code/frameworks/research-complete/elaboration/nfr/NFR-RF-specifications.md - NFR-QA-01 through NFR-QA-06
- @$AIWG_ROOT/agentic/code/frameworks/research-complete/inception/solution-profile.md - Section 3.2 (GRADE-inspired scoring)
- [GRADE Handbook](https://gdt.gradepro.org/app/handbook/handbook.html) - GRADE methodology
- [FAIR Principles](https://www.go-fair.org/fair-principles/) - R1.2 community standards
**Document Status**: Accepted
**Review Date**: 2026-01-25
**Next Review**: End of Construction Phase