aiwg
Version:
Deployment tool and support utility for AI context. Copies agents, skills, commands, rules, and behaviors into the paths each AI platform reads (Claude Code, Codex, Copilot, Cursor, Warp, OpenClaw, and 6 more) so one source of truth works across 10 platfo
401 lines (328 loc) • 15.3 kB
Markdown
# ADR-RF-004: Artifact Storage Structure
## Metadata
- **ID**: ADR-RF-004
- **Title**: Artifact Storage Structure for Research Framework
- **Status**: Accepted
- **Created**: 2026-01-25
- **Updated**: 2026-01-25
- **Decision Makers**: Research Framework Architecture Team
- **Related ADRs**: ADR-RF-002 (Provenance Storage), ADR-RF-005 (API Integration)
## Context
The Research Framework generates and manages numerous artifacts across the research lifecycle: search strategies, PDFs, metadata files, summaries, quality reports, notes, knowledge graphs, and provenance logs. The storage structure must:
1. **Support the 5-Stage Lifecycle**: Discovery, Acquisition, Documentation, Integration, Archival
2. **Enable FAIR Compliance**: Findable, Accessible, Interoperable, Reusable
3. **Scale to 1,000+ Papers**: Per NFR-RF-S-001
4. **Integrate with AIWG**: Follow `.aiwg/` directory conventions
5. **Support Git Versioning**: Work well with Git for provenance
6. **Enable Cross-Project Sharing**: research-papers repo pattern
### Decision Drivers
1. **AIWG Consistency**: Must follow `.aiwg/` directory structure patterns from SDLC framework
2. **Discoverability**: Artifacts must be easily locatable without framework tools
3. **Scalability**: Structure must handle 1,000+ papers without performance degradation
4. **Interoperability**: Export/import with Zotero, Obsidian, BibTeX
5. **Versioning**: All artifacts version-controlled via Git
6. **Portability**: Copy/move research corpus between projects
7. **Offline Access**: NFR-RF-X-001 requires local-first architecture
### Current AIWG Patterns
SDLC framework uses:
```
.aiwg/
├── intake/ # Project initiation
├── requirements/ # Use cases, user stories
├── architecture/ # SAD, ADRs
├── planning/ # Phase plans
├── testing/ # Test strategy
└── working/ # Temporary files
```
## Decision
**Adopt a lifecycle-aligned directory structure within `.aiwg/research/` with source-centric artifact organization.**
### Directory Structure
```
.aiwg/research/
├── discovery/ # Stage 1: Discovery artifacts
│ ├── search-strategies/ # PRISMA-style reproducible queries
│ │ ├── SS-001-oauth-security.yaml
│ │ └── SS-002-agentic-frameworks.yaml
│ ├── screening-results/ # Screening decisions with rationale
│ │ └── SR-001-oauth-screening.md
│ ├── gap-analysis/ # Automated gap detection reports
│ │ └── GA-001-coverage-report.md
│ └── preregistration/ # Research plans (open science)
│ └── PR-001-systematic-review-plan.md
│
├── sources/ # Stage 2-3: Source artifacts (per-source organization)
│ ├── pdfs/ # Original source files
│ │ ├── REF-001-constitutional-ai.pdf
│ │ └── REF-025-oauth-security-patterns.pdf
│ ├── REF-001/ # Source-specific directory
│ │ ├── metadata.json # FAIR-compliant structured metadata
│ │ ├── summary.md # AI-generated summary
│ │ ├── extraction.json # Structured data extraction
│ │ ├── quality-report.json # Quality assessment
│ │ └── notes/ # Source-specific notes
│ │ ├── literature-note.md
│ │ └── annotations.md
│ ├── REF-025/
│ │ └── ...
│ ├── index.json # Searchable source index (F4 compliance)
│ └── checksums.json # Integrity verification
│
├── knowledge/ # Stage 3-4: Knowledge artifacts
│ ├── literature-notes/ # Processed insights (Zettelkasten)
│ │ ├── LN-001-token-rotation-patterns.md
│ │ └── LN-002-agent-orchestration.md
│ ├── permanent-notes/ # Refined ideas
│ │ ├── PN-001-security-tradeoffs.md
│ │ └── PN-002-multi-agent-design.md
│ ├── maps-of-content/ # Topic MoCs
│ │ ├── MOC-security.md
│ │ └── MOC-agentic-ai.md
│ └── claims-index.md # Claims needing/having citations
│
├── networks/ # Stage 4: Integration artifacts
│ ├── citation-network.json # Citation relationships
│ ├── concept-graph.json # Knowledge graph
│ ├── author-network.json # Collaboration networks
│ └── visualizations/ # Generated graph images
│ ├── citation-network.svg
│ └── concept-graph.svg
│
├── analysis/ # Cross-cutting analysis
│ ├── quality-summary.md # Corpus quality overview
│ ├── fair-compliance.md # FAIR compliance report
│ ├── reproducibility-status.md # Replication readiness
│ └── gap-summary.md # Research gap synthesis
│
├── provenance/ # W3C PROV tracking (see ADR-RF-002)
│ ├── prov-2026-01-25.json # Daily provenance logs
│ ├── index.yaml # Human-readable summary
│ ├── lineage/ # Data lineage graphs
│ └── failed-logs/ # Recovery for failed logging
│
├── workflows/ # Stage 5: Workflow artifacts
│ ├── WF-001-literature-review/ # Per-workflow directory
│ │ ├── status.json # Workflow state
│ │ ├── summary.md # Completion summary
│ │ └── checkpoints/ # Resume points
│ ├── pipelines/ # Reusable workflow definitions
│ │ └── systematic-review.yaml
│ └── reproducibility-packages/ # Containerized environments
│ └── RP-001-oauth-review.zip
│
├── outputs/ # Generated deliverables
│ ├── literature-review-oauth.md # Final outputs
│ ├── bibliography.bib # BibTeX export
│ └── obsidian-export/ # Obsidian vault export
│
├── archive/ # Stage 5: Archival
│ ├── AIP-001/ # Archival Information Package
│ │ ├── manifest.json
│ │ ├── preservation-metadata.json
│ │ └── content/
│ └── tombstones/ # Deleted source metadata (A2 compliance)
│
└── config/ # Framework configuration
├── search-config.yaml # Search preferences
├── quality-criteria.yaml # Quality weights/thresholds
├── fair-requirements.yaml # FAIR compliance rules
├── lifecycle-rules.yaml # Automation rules
└── metadata-schema.json # JSON schema for metadata
```
### Naming Conventions
| Artifact Type | Pattern | Example |
|---------------|---------|---------|
| Search Strategy | `SS-NNN-{topic}.yaml` | `SS-001-oauth-security.yaml` |
| Screening Results | `SR-NNN-{topic}.md` | `SR-001-oauth-screening.md` |
| Gap Analysis | `GA-NNN-{topic}.md` | `GA-001-coverage-report.md` |
| Source Reference | `REF-NNN` or `REF-NNN-{slug}` | `REF-025-oauth-patterns` |
| Literature Note | `LN-NNN-{topic}.md` | `LN-001-token-rotation.md` |
| Permanent Note | `PN-NNN-{topic}.md` | `PN-001-security-tradeoffs.md` |
| Map of Content | `MOC-{topic}.md` | `MOC-security.md` |
| Workflow | `WF-{date}-NNN` | `WF-2026-01-25-001` |
| Archival Package | `AIP-NNN` | `AIP-001` |
### Source-Centric Organization
Each source (REF-XXX) has its own directory containing all related artifacts:
```
sources/REF-025/
├── metadata.json # Core bibliographic metadata
├── summary.md # AI-generated summary
├── extraction.json # Structured data from paper
├── quality-report.json # Quality assessment
└── notes/ # Source-specific notes
├── literature-note.md # Processed insights
└── annotations.md # Reading annotations
```
**Rationale**: Source-centric organization keeps related artifacts together, simplifies backup/export of specific sources, and enables easy deletion (just remove REF-XXX directory).
## Consequences
### Positive
1. **AIWG Alignment**: Follows `.aiwg/` conventions, consistent with SDLC framework
2. **Lifecycle Clarity**: Directory structure mirrors 5-stage research lifecycle
3. **Discoverability**: Intuitive navigation without framework tools
4. **Scalability**: Source-centric organization handles 1,000+ sources
5. **Git-Friendly**: Text-based artifacts, reasonable directory depth
6. **Export-Ready**: Clear boundaries for Obsidian, BibTeX, OAIS exports
7. **FAIR Compliance**: Structure supports all FAIR principles
### Negative
1. **Deep Nesting**: Some paths are long (`.aiwg/research/sources/REF-025/notes/literature-note.md`)
2. **Index Maintenance**: `index.json` must be updated on every source change
3. **Cross-Cutting Queries**: Finding all summaries requires traversing source directories
4. **Duplication Risk**: Notes in both `sources/REF-XXX/notes/` and `knowledge/literature-notes/`
### Risks
| Risk | Likelihood | Impact | Mitigation |
|------|-----------|--------|------------|
| Directory count exceeds filesystem limits | Very Low | High | Tested to 10,000 sources; FAT32 limit (65,536) far above use case |
| Index desync with actual files | Medium | Medium | Regenerate index on startup, validate on operations |
| Git performance with many files | Low | Medium | `.gitignore` for PDFs in large repos, shallow clones |
| Path length issues on Windows | Low | Low | Recommend short project paths, test Windows compatibility |
## Alternatives Considered
### Option A: Flat File Structure
**Description**: All artifacts in a single directory with naming conventions to distinguish types.
**Example**:
```
.aiwg/research/
├── REF-001-metadata.json
├── REF-001-summary.md
├── REF-001-quality.json
├── REF-002-metadata.json
├── search-001-oauth.yaml
├── note-001-token-rotation.md
...
```
**Pros**:
- Simple, no nesting
- Easy to list all artifacts
- No directory creation overhead
**Cons**:
- Unmanageable at scale (1,000 sources = 5,000+ files in one directory)
- No clear lifecycle organization
- Hard to export specific sources
- Difficult to find related artifacts
**Decision**: Rejected. Does not scale to target corpus size.
### Option B: Type-Centric Organization
**Description**: Organize by artifact type rather than lifecycle stage or source.
**Example**:
```
.aiwg/research/
├── metadata/
│ ├── REF-001.json
│ └── REF-002.json
├── summaries/
│ ├── REF-001.md
│ └── REF-002.md
├── quality-reports/
│ ├── REF-001.json
│ └── REF-002.json
├── pdfs/
│ └── ...
```
**Pros**:
- Easy to find all artifacts of a type
- Simple queries ("list all summaries")
- Clear separation of concerns
**Cons**:
- Scattered source information (REF-001 in 5+ directories)
- Hard to export/delete specific sources
- Doesn't reflect lifecycle stages
- Requires cross-directory navigation for source review
**Decision**: Rejected. Source-centric organization better supports common workflows (review a paper, export a source, delete a source).
### Option C: Database Storage
**Description**: Store metadata and relationships in SQLite/PostgreSQL, files on disk.
**Example**:
```
.aiwg/research/
├── research.db # SQLite database
├── pdfs/ # Binary files
└── exports/ # Generated files
```
**Pros**:
- Powerful querying (SQL)
- Referential integrity
- Compact storage
- Efficient relationships (citation networks)
**Cons**:
- Binary database not Git-friendly (no diff, no merge)
- Requires SQLite/ORM dependency
- Harder to inspect without tools
- Violates offline/local-first principle
- Conflicts with AIWG file-based patterns
**Decision**: Rejected. Database storage conflicts with AIWG's file-based philosophy and Git-based provenance.
### Option D: Obsidian Vault Structure
**Description**: Match Obsidian vault conventions for native integration.
**Example**:
```
.aiwg/research/
├── 00 - Inbox/
├── 01 - Sources/
├── 02 - Notes/
├── 03 - Projects/
└── Attachments/
```
**Pros**:
- Native Obsidian compatibility
- Familiar to PKM users
- Graph view works out of box
**Cons**:
- Obsidian-specific conventions
- Not aligned with AIWG patterns
- Numbering clutters programmatic access
- Lifecycle stages not represented
**Decision**: Rejected. Support Obsidian via export, not native structure. Framework should be tool-agnostic.
## Implementation Notes
### Index File Schema
`sources/index.json` enables F4 compliance (searchable resource):
```json
{
"$schema": "https://aiwg.io/research/schemas/source-index.json",
"updated_at": "2026-01-25T16:00:00Z",
"source_count": 25,
"sources": [
{
"id": "REF-025",
"doi": "10.1234/example",
"title": "OAuth2 Security Patterns for Modern Applications",
"authors": ["Smith, J.", "Doe, A."],
"year": 2024,
"type": "academic-paper",
"quality_score": 87,
"grade_rating": "High",
"keywords": ["oauth", "security", "authentication"],
"path": "sources/REF-025/"
}
]
}
```
### Cross-Reference Pattern
Notes reference sources and other notes via AIWG @-mention syntax:
```markdown
# Token Rotation Patterns
Key insight from @sources/REF-025/summary.md: Token rotation reduces CSRF risk by 80%.
Related: @knowledge/permanent-notes/PN-001-security-tradeoffs.md
```
### Git Integration
Recommended `.gitignore` for large corpora:
```gitignore
# Ignore PDFs to reduce repo size (optional)
.aiwg/research/sources/pdfs/*.pdf
# Keep metadata and summaries versioned
!.aiwg/research/sources/pdfs/.gitkeep
# Ignore working files
.aiwg/research/workflows/*/checkpoints/
.aiwg/research/provenance/failed-logs/
```
### Migration Path
For existing research-papers repos:
1. Create `.aiwg/research/` structure
2. Move PDFs to `sources/pdfs/`
3. Generate REF-XXX directories from existing metadata
4. Build index.json from migration
5. Validate with `aiwg research validate`
## References
- @$AIWG_ROOT/agentic/code/frameworks/research-complete/inception/solution-profile.md - Section 4.2 (Artifact Structure)
- @$AIWG_ROOT/agentic/code/frameworks/research-complete/elaboration/nfr/NFR-RF-specifications.md - NFR-RF-S-001 (Scalability), NFR-RF-CMP-001 through CMP-004 (FAIR)
- @$AIWG_ROOT/agentic/code/frameworks/research-complete/elaboration/use-cases/UC-RF-007-archive-research-artifacts.md - Archival structure
- @CLAUDE.md - `.aiwg/` directory conventions
- @$AIWG_ROOT/agentic/code/frameworks/sdlc-complete/README.md - SDLC artifact patterns
**Document Status**: Accepted
**Review Date**: 2026-01-25
**Next Review**: End of Construction Phase