aiwg
Version:
Deployment tool and support utility for AI context. Copies agents, skills, commands, rules, and behaviors into the paths each AI platform reads (Claude Code, Codex, Copilot, Cursor, Warp, OpenClaw, and 6 more) so one source of truth works across 10 platfo
771 lines (646 loc) • 30.9 kB
Markdown
# Use-Case Specification: UC-RF-005
## Metadata
- ID: UC-RF-005
- Name: Track Research Provenance with W3C PROV Compliance
- Owner: Requirements Analyst
- Contributors: Provenance Agent Designer, Quality Specialist
- Team: Research Framework
- Status: draft
- Created: 2026-01-25
- Updated: 2026-01-25
- Priority: P0 (Critical)
- Estimated Effort: L (Low - automated, minimal user interaction)
- Related Documents:
- Vision: @$AIWG_ROOT/agentic/code/frameworks/research-complete/inception/vision-document.md
- Risks: @$AIWG_ROOT/agentic/code/frameworks/research-complete/inception/initial-risk-assessment.md (T-06: Reproducibility Failure)
- Agent: @$AIWG_ROOT/agentic/code/frameworks/research-complete/elaboration/agents/provenance-agent.md
- Related UCs: UC-RF-001, UC-RF-002, UC-RF-003, UC-RF-004 (all provenance-tracked)
## 1. Use-Case Identifier and Name
**ID:** UC-RF-005
**Name:** Track Research Provenance with W3C PROV Compliance
## 2. Scope and Level
**Scope:** AIWG Research Framework - Cross-Cutting Concern (All Stages)
**Level:** System Goal (Automated, transparent to user)
**System Boundary:** Provenance Agent, W3C PROV Logger, .aiwg/research/provenance/
## 3. Primary Actor(s)
**Primary Actors:**
- Provenance Agent (automated, not user-triggered)
- External Researcher (validating reproducibility)
**Secondary Actors:**
- Academic Researcher: Needs reproducibility packages for publication
- Auditor: Verifying research integrity for compliance
**Actor Goals:**
- Automatically capture all research operations (discovery, acquisition, documentation, citation)
- Enable external researchers to replicate findings exactly
- Provide complete audit trail for research integrity
- Support W3C PROV standard for provenance interoperability
- Generate reproducibility packages for publication supplements
## 4. Stakeholders and Interests
| Stakeholder | Interest |
|------------|----------|
| Academic Researcher | 100% reproducibility, PROV-compliant logs, publication supplements |
| External Researcher | Ability to replicate findings, transparent methodology |
| Auditor | Complete audit trail, integrity verification, compliance validation |
| Framework Maintainer | Automated provenance (no user burden), W3C PROV compliance |
## 5. Preconditions
1. AIWG Research Framework deployed
2. `.aiwg/research/provenance/` directory structure exists
3. W3C PROV-JSON schema available for validation
4. Git repository initialized (for version control integration)
5. Any research operation executing (discovery, acquisition, documentation, citation)
## 6. Postconditions
**Success:**
- All operations logged in W3C PROV-JSON format
- Provenance log saved to `.aiwg/research/provenance/prov-{timestamp}.json`
- Lineage graph updated showing artifact dependencies
- Reproducibility package ready for export
- External researchers can replicate results using provenance logs
**Failure:**
- Provenance logging failure does not block research operation (non-critical)
- Failed logs saved to `.aiwg/research/provenance/failed-logs/` for manual review
- User notified of logging failure (warning, not error)
- Research operation proceeds normally
## 7. Trigger
**Automatic triggers (any research operation):**
- Discovery: `aiwg research search` → Log search parameters, results
- Acquisition: `aiwg research acquire` → Log source URLs, checksums
- Documentation: `aiwg research summarize` → Log LLM model, prompts
- Citation: `aiwg research cite` → Log claim, source, document location
**Manual triggers (user-initiated):**
- Generate reproducibility package: `aiwg research provenance export`
- Validate provenance logs: `aiwg research provenance validate`
- Query provenance: `aiwg research provenance query "Where did REF-025 come from?"`
## 8. Main Success Scenario
1. Research operation executes (e.g., `aiwg research search "OAuth2 security"`)
2. Provenance Agent intercepts operation:
- Captures operation metadata (command, timestamp, user)
- Identifies entities involved (search query, API, results)
- Detects activities (API query, result ranking, saving)
- Records agents (user, Discovery Agent, Semantic Scholar API)
3. Agent constructs W3C PROV entities:
- **Entity:** Search results JSON (artifact created)
- **Activity:** Discovery search operation (process executed)
- **Agent:** User, Discovery Agent, Semantic Scholar API (actors)
4. Agent records PROV relationships:
- **wasGeneratedBy:** Search results generated by discovery activity
- **used:** Discovery activity used search query
- **wasAssociatedWith:** Discovery activity associated with Discovery Agent
- **wasAttributedTo:** Search results attributed to Semantic Scholar API
- **wasDerivedFrom:** Literature notes derived from search results
5. Agent validates PROV-JSON against W3C schema:
- Checks all required fields present (id, type, relationships)
- Validates timestamps (ISO 8601 format)
- Ensures entity IDs unique and resolvable
6. Agent appends provenance record to log:
- File: `.aiwg/research/provenance/prov-{timestamp}.json`
- Append mode (preserves previous records)
- Log rotation: New file every 30 days or 10,000 records
7. Agent updates lineage graph:
- Adds nodes for new entities (search results, notes)
- Adds edges for derivation relationships
- Graph saved to `.aiwg/research/provenance/lineage-graph.json`
- Enables queries: "What derived from REF-025?"
8. Agent computes artifact checksums:
- SHA-256 hash of all generated files
- Checksums logged in provenance record
- Enables integrity verification on replication
9. Agent integrates with Git version control:
- Commits provenance log: `git add .aiwg/research/provenance/ && git commit -m "Log: Discovery search"`
- Correlates provenance records with Git commits
- Enables time-travel: "Show provenance at commit abc123"
10. Agent continues monitoring:
- Remains active throughout research workflow
- Captures all subsequent operations
- No user interaction required (transparent)
11. External researcher accesses provenance:
- Reads provenance logs from repository
- Reconstructs workflow from PROV relationships
- Replicates operations step-by-step
12. External researcher validates reproducibility:
- Runs same search query (from provenance log)
- Compares checksums of results (integrity check)
- Confirms identical artifacts produced
13. Reproducibility validated: External researcher publishes replication study
## 9. Alternate Flows
### Alt-1: Generate Reproducibility Package (Publication Supplement)
**Branch Point:** After any research workflow completes
**Condition:** User preparing publication, needs reproducibility package
**Flow:**
1. User runs: `aiwg research provenance export --package publication-supplement`
2. Agent collects all provenance artifacts:
- Provenance logs (W3C PROV-JSON)
- Lineage graph (entity dependencies)
- Checksums (integrity verification)
- Search strategies (PRISMA protocols)
- Source metadata (all REF-XXX JSONs)
3. Agent generates reproducibility report:
- Workflow overview (stages executed)
- Software versions (AIWG version, LLM model, API versions)
- Data sources (Semantic Scholar API, manual uploads)
- Quality metrics (FAIR scores, GRADE scores)
- Instructions for replication
4. Agent packages artifacts:
- Format: ZIP archive or Docker container
- Contents: Provenance logs, metadata, scripts, README
- Size: <100 MB (excludes PDFs due to copyright)
5. Agent saves package:
- Location: `.aiwg/research/provenance/reproducibility-package-{timestamp}.zip`
- User uploads as publication supplement
- External researchers download and replicate
6. **Resume Main:** Step 13 (Reproducibility validated)
### Alt-2: Provenance Query (Lineage Tracing)
**Branch Point:** User wants to understand artifact origin
**Condition:** User asks "Where did this claim come from?"
**Flow:**
1. User runs: `aiwg research provenance query "Where did 'Token rotation reduces CSRF risk' come from?"`
2. Agent parses natural language query
3. Agent searches provenance logs for claim text
4. Agent traces lineage backward:
- Claim → Citation (UC-RF-004) → Literature note (UC-RF-003) → REF-025 (UC-RF-002) → Search query (UC-RF-001)
5. Agent constructs lineage chain:
```
Search Query: "OAuth2 security best practices"
→ Discovery Results: REF-025, REF-026, ...
→ REF-025 Acquired: 2026-01-25 14:00:00
→ REF-025 Summarized: 2026-01-25 16:00:00
→ Literature Note Created
→ Citation Added to Architecture Doc
→ Claim: "Token rotation reduces CSRF risk by 80%"
```
6. Agent displays lineage:
- Text format for terminal
- Graphviz DOT for visualization
7. User understands claim provenance, validates trustworthiness
8. **Resume Main:** Step 13 (Provenance validated)
### Alt-3: Integrity Verification (Checksum Validation)
**Branch Point:** External researcher replicating study
**Condition:** Researcher wants to verify artifact integrity
**Flow:**
1. External researcher downloads provenance package
2. Researcher runs: `aiwg research provenance verify`
3. Agent reads checksums from provenance logs
4. Agent computes current checksums for all artifacts:
- Search results JSON: SHA-256 hash
- Metadata files: SHA-256 hash
- Summaries: SHA-256 hash
5. Agent compares checksums:
- Expected: `e3b0c442...` (from provenance log)
- Actual: `e3b0c442...` (computed now)
6. Agent displays verification report:
- "✅ REF-025-metadata.json: Verified"
- "✅ REF-025-summary.md: Verified"
- "❌ REF-042-extraction.json: MISMATCH (file corrupted or modified)"
7. Researcher identifies corrupted file, re-downloads
8. **Resume Main:** Step 13 (Integrity verified)
### Alt-4: Time-Travel Provenance (Historical State)
**Branch Point:** User wants to see provenance at specific time
**Condition:** Project evolved over months, user needs historical context
**Flow:**
1. User runs: `aiwg research provenance query --at 2026-01-15`
2. Agent filters provenance logs for records before 2026-01-15
3. Agent reconstructs lineage graph at historical point:
- Only entities/activities existing before date
- Excludes later operations
4. Agent displays historical provenance:
- "As of 2026-01-15, corpus had 10 papers (REF-001 to REF-010)"
- "Claims backed: 50/200 (25%)"
5. User understands research evolution over time
6. **Resume Main:** Step 13 (Historical context validated)
## 10. Exception Flows
### Exc-1: Provenance Logging Failure (Disk Full)
**Trigger:** Step 6
**Condition:** Insufficient disk space to append provenance log
**Flow:**
1. Agent attempts to write provenance record
2. Write fails (disk full)
3. Agent saves failed log to memory buffer
4. Agent displays warning: "Provenance logging failed: Disk full. Free space: df -h"
5. Research operation continues normally (non-critical failure)
6. Agent retries logging after 60 seconds (in case disk space freed)
7. If retry succeeds: Log appended, warning cleared
8. If retry fails: Failed log saved to `.aiwg/research/provenance/failed-logs/`
9. User reviews failed logs, manually resolves
10. **Resume Main:** Step 10 (Agent continues monitoring)
### Exc-2: W3C PROV Schema Validation Failure
**Trigger:** Step 5
**Condition:** Provenance record malformed (missing required fields)
**Flow:**
1. Agent constructs PROV-JSON record
2. Agent validates against W3C PROV-JSON schema
3. Validation fails: Missing "type" field for entity
4. Agent logs validation error:
- Error: "PROV record invalid: Missing 'type' for entity 'REF-025'"
- Record: [invalid PROV-JSON]
5. Agent attempts auto-repair:
- Infers missing "type" from context (entity is a "Document")
- Adds default "type": "prov:Entity"
6. Agent re-validates, succeeds
7. Agent appends repaired record to log
8. Agent flags record for manual review (may be data quality issue)
9. **Resume Main:** Step 7 (Agent updates lineage graph)
### Exc-3: Git Integration Failure (Not a Git Repository)
**Trigger:** Step 9
**Condition:** `.git` directory not found (user not using Git)
**Flow:**
1. Agent attempts Git commit for provenance log
2. Git command fails: "Not a git repository"
3. Agent detects Git unavailable
4. Agent displays info: "Provenance logged but not version-controlled. Initialize Git for full traceability."
5. Agent disables Git integration for session
6. Provenance logging continues without Git commits
7. User can initialize Git later: `git init && git add .aiwg/research/provenance/`
8. **Resume Main:** Step 10 (Agent continues monitoring without Git)
### Exc-4: Provenance Log Corruption
**Trigger:** Step 6
**Condition:** Existing provenance log file corrupted (invalid JSON)
**Flow:**
1. Agent attempts to append record to existing log
2. Agent reads existing log for validation
3. JSON parsing fails (corrupted file)
4. Agent displays error: "Provenance log corrupted. Restore from backup? (y/n/new)"
5. User chooses option:
- Restore from backup: Agent copies last valid backup
- Create new log: Agent starts fresh log file
6. Agent appends record to restored/new log
7. Agent saves corrupted log to `.aiwg/research/provenance/corrupted/` for forensics
8. **Resume Main:** Step 7 (Agent updates lineage graph)
## 11. Special Requirements
### Performance Requirements
| Requirement | Target | Rationale |
|------------|--------|-----------|
| NFR-RF-P-01: Provenance logging overhead | <100ms per operation | Non-intrusive (user doesn't notice) |
| NFR-RF-P-02: Log file size | <10 MB per 10,000 operations | Storage efficiency |
| NFR-RF-P-03: Lineage query response time | <2 seconds for 1,000 entities | Usability |
### Quality Requirements
| Requirement | Target | Rationale |
|------------|--------|-----------|
| NFR-RF-P-04: W3C PROV compliance | 100% (all records valid) | Interoperability |
| NFR-RF-P-05: Checksum coverage | 100% of artifacts | Integrity verification |
| NFR-RF-P-06: Reproducibility success rate | >90% (external replication) | Trust and credibility |
### Usability Requirements
| Requirement | Target | Rationale |
|------------|--------|-----------|
| NFR-RF-P-07: User interaction required | 0 (fully automated) | Transparency (no burden) |
| NFR-RF-P-08: Provenance query clarity | Natural language supported | Accessibility |
## 12. Related Business Rules
**BR-RF-P-001: W3C PROV Entity Types**
- `prov:Entity` - Data artifacts (search results, metadata, summaries)
- `prov:Activity` - Operations (search, acquire, summarize, cite)
- `prov:Agent` - Actors (user, agents, APIs)
**BR-RF-P-002: W3C PROV Relationships**
- `wasGeneratedBy` - Entity created by activity
- `used` - Activity consumed entity
- `wasAssociatedWith` - Activity performed by agent
- `wasAttributedTo` - Entity created by agent
- `wasDerivedFrom` - Entity derived from another entity
- `wasInformedBy` - Activity triggered by another activity
**BR-RF-P-003: Log Rotation Policy**
- New log file every 30 days or 10,000 records (whichever first)
- Archived logs compressed (gzip)
- Retention: 5 years (academic standard)
**BR-RF-P-004: Reproducibility Package Contents**
- Provenance logs (W3C PROV-JSON)
- Lineage graph (JSON)
- Search strategies (PRISMA protocols)
- Source metadata (all REF-XXX JSONs)
- Checksums (SHA-256)
- Software versions (AIWG, LLM, APIs)
- README (replication instructions)
- Excludes: PDFs (copyright), raw LLM prompts (proprietary)
## 13. Data Requirements
### Input Data
| Data Element | Format | Source | Validation |
|-------------|--------|---------|-----------|
| Operation Metadata | JSON object | Research operation | Valid operation type |
| Entities Involved | Array of entity IDs | Operation context | Valid entity references |
| Checksums | SHA-256 hashes | Artifact files | Valid hex string |
### Output Data
| Data Element | Format | Destination | Retention |
|-------------|--------|-------------|----------|
| Provenance Log | W3C PROV-JSON | `.aiwg/research/provenance/prov-{timestamp}.json` | 5 years |
| Lineage Graph | JSON graph | `.aiwg/research/provenance/lineage-graph.json` | Permanent |
| Checksums File | Text (one hash per line) | `.aiwg/research/sources/checksums.txt` | Permanent |
| Reproducibility Package | ZIP or Docker image | `.aiwg/research/provenance/reproducibility-package-{timestamp}.zip` | Permanent |
### Data Schema: W3C PROV-JSON Record
```json
{
"prefix": {
"prov": "http://www.w3.org/ns/prov#",
"aiwg": "https://aiwg.io/research#"
},
"entity": {
"aiwg:search-results-2026-01-25": {
"prov:type": "prov:Entity",
"aiwg:entityType": "SearchResults",
"aiwg:filePath": ".aiwg/research/discovery/search-results-2026-01-25T10-30-00.json",
"aiwg:checksum": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"prov:generatedAtTime": "2026-01-25T10:30:00Z"
}
},
"activity": {
"aiwg:discovery-search-001": {
"prov:type": "prov:Activity",
"aiwg:activityType": "DiscoverySearch",
"aiwg:command": "aiwg research search \"OAuth2 security best practices\"",
"prov:startTime": "2026-01-25T10:29:50Z",
"prov:endTime": "2026-01-25T10:30:05Z"
}
},
"agent": {
"aiwg:user-jmagly": {
"prov:type": "prov:Agent",
"aiwg:agentType": "HumanUser",
"aiwg:username": "jmagly"
},
"aiwg:discovery-agent": {
"prov:type": "prov:SoftwareAgent",
"aiwg:agentType": "DiscoveryAgent",
"aiwg:version": "1.0.0"
},
"aiwg:semantic-scholar-api": {
"prov:type": "prov:SoftwareAgent",
"aiwg:agentType": "ExternalAPI",
"aiwg:version": "semantic-scholar-v1"
}
},
"wasGeneratedBy": {
"_:wgb1": {
"prov:entity": "aiwg:search-results-2026-01-25",
"prov:activity": "aiwg:discovery-search-001"
}
},
"used": {
"_:u1": {
"prov:activity": "aiwg:discovery-search-001",
"prov:entity": "aiwg:search-query-oauth2"
}
},
"wasAssociatedWith": {
"_:waw1": {
"prov:activity": "aiwg:discovery-search-001",
"prov:agent": "aiwg:discovery-agent"
}
},
"wasAttributedTo": {
"_:wat1": {
"prov:entity": "aiwg:search-results-2026-01-25",
"prov:agent": "aiwg:semantic-scholar-api"
}
}
}
```
## 14. Open Issues and TODOs
1. **Issue 001: Provenance granularity tradeoff**
- Description: Log every operation vs. log milestones only (storage vs. detail)
- Impact: Fine-grained logging = large files, coarse logging = less reproducibility
- Owner: Provenance Agent Designer
- Due Date: Elaboration phase (define granularity policy)
2. **TODO 001: PROV-N and PROV-XML export**
- Description: Support W3C PROV-N (textual) and PROV-XML formats for interoperability
- Assigned: Integration Specialist
- Due Date: Post-v1.0 enhancement
3. **Issue 002: Provenance privacy (sensitive data in logs)**
- Description: Provenance logs may contain sensitive info (user emails, internal URLs)
- Impact: Cannot share reproducibility packages without redaction
- Owner: Security Specialist
- Due Date: Construction phase (implement redaction)
## 15. References
- @$AIWG_ROOT/agentic/code/frameworks/research-complete/inception/vision-document.md - Section 5.3 (Goal 3: Enable Reproducibility)
- @$AIWG_ROOT/agentic/code/frameworks/research-complete/inception/initial-risk-assessment.md - T-06 (Reproducibility Failure Risk)
- @$AIWG_ROOT/agentic/code/frameworks/research-complete/elaboration/agents/provenance-agent.md - Agent specification
- [W3C PROV Overview](https://www.w3.org/TR/prov-overview/) - Provenance standard
- [PROV-JSON](https://www.w3.org/Submission/prov-json/) - JSON serialization for PROV
## Traceability Matrix
### Requirements Traceability
| Requirement ID | Source | Implementation | Test Case |
|---------------|---------|----------------|-----------|
| Vision-5.3-Reproducibility | Vision Doc | Provenance Agent | TC-RF-005-001 through TC-RF-005-008 |
| NFR-RF-P-04 | This document | W3C PROV-JSON schema validation | TC-RF-005-003 |
| NFR-RF-P-06 | This document | Reproducibility package export | TC-RF-005-007 |
| BR-RF-P-002 | This document | PROV relationship logging | TC-RF-005-002 |
### Vision Document Mapping
**Vision Section 5.3 (Goal 3: Enable Reproducibility):**
- W3C PROV-compliant provenance → UC-RF-005 Steps 3-5
- Workflow definitions enable replication → UC-RF-005 Alt-1
- Checksums verify integrity → UC-RF-005 Step 8, Alt-3
- Reproducibility packages → UC-RF-005 Alt-1
### Risk Mapping
**T-06: Reproducibility Failure (High Priority):**
- Mitigated by: W3C PROV logging (Steps 3-5), checksum validation (Step 8), reproducibility packages (Alt-1)
- Monitored via: External replication success rate (NFR-RF-P-06), checksum coverage (NFR-RF-P-05)
## Acceptance Criteria
### AC-001: Automatic Provenance Logging
**Given:** Any research operation executes (e.g., discovery search)
**When:** Provenance Agent monitors operation
**Then:**
- Operation metadata captured (command, timestamp, user)
- W3C PROV record constructed (entities, activities, agents, relationships)
- Provenance log appended without user interaction
- Logging overhead <100ms (user doesn't notice)
- Research operation proceeds normally
### AC-002: W3C PROV Compliance
**Given:** Provenance record constructed
**When:** Agent validates against W3C PROV-JSON schema
**Then:**
- All required fields present (id, type, relationships)
- Timestamps in ISO 8601 format
- Entity IDs unique and resolvable
- Validation passes (100% compliance)
- Provenance interoperable with external PROV tools
### AC-003: Lineage Tracing (Query)
**Given:** User asks "Where did REF-025 come from?"
**When:** User runs `aiwg research provenance query "Where did REF-025 come from?"`
**Then:**
- Agent searches provenance logs
- Lineage chain traced backward: REF-025 ← Acquisition ← Search query
- Lineage displayed (text and graph)
- User understands artifact origin
- Query response time <2 seconds
### AC-004: Integrity Verification (Checksums)
**Given:** External researcher downloads reproducibility package
**When:** Researcher runs `aiwg research provenance verify`
**Then:**
- Agent reads checksums from provenance logs
- Agent computes current checksums for all artifacts
- Agent compares expected vs. actual
- Verification report displayed: ✅ or ❌ for each artifact
- Mismatches flagged for investigation
### AC-005: Reproducibility Package Export
**Given:** User preparing publication supplement
**When:** User runs `aiwg research provenance export --package publication-supplement`
**Then:**
- Agent collects provenance logs, lineage graph, checksums, metadata
- Reproducibility report generated (workflow overview, software versions)
- Package created: ZIP archive <100 MB
- External researchers can download and replicate
- Replication success rate >90%
### AC-006: Git Integration (Version Control)
**Given:** Git repository initialized
**When:** Provenance Agent logs operations
**Then:**
- Provenance log committed to Git: `git add .aiwg/research/provenance/`
- Commit message: "Log: Discovery search"
- Provenance correlated with Git commits
- Time-travel enabled: "Show provenance at commit abc123"
- Full version history available
### AC-007: Provenance Logging Failure (Non-Critical)
**Given:** Disk full, provenance write fails
**When:** Agent attempts to log operation
**Then:**
- Write fails, agent detects error
- Failed log saved to memory buffer
- Warning displayed: "Provenance logging failed: Disk full"
- Research operation continues normally (non-blocking)
- Agent retries after 60 seconds
- User can resolve and manually retry
### AC-008: End-to-End Reproducibility
**Given:** Complete research workflow executed
**When:** External researcher replicates using provenance package
**Then:**
1. Researcher reads provenance logs
2. Researcher reconstructs workflow from PROV relationships
3. Researcher runs same operations (search, acquire, summarize)
4. Researcher compares checksums (integrity verification)
5. Researcher confirms identical artifacts produced
6. Reproducibility validated: >90% success rate
## Test Cases
### TC-RF-005-001: Automatic Provenance Logging
**Objective:** Validate provenance captured automatically
**Preconditions:** AIWG Research Framework deployed
**Test Steps:**
1. Run discovery search: `aiwg research search "OAuth2 security"`
2. Verify provenance log appended: `.aiwg/research/provenance/prov-{timestamp}.json`
3. Verify log contains:
- Entity: search-results-{timestamp}
- Activity: discovery-search-001
- Agents: user, Discovery Agent, Semantic Scholar API
- Relationships: wasGeneratedBy, used, wasAssociatedWith
4. Verify logging overhead <100ms (measured)
5. Verify search operation proceeds normally (no user interaction)
**Expected Result:** Provenance logged automatically, <100ms overhead
**NFR Validated:** NFR-RF-P-01 (Logging overhead <100ms), NFR-RF-P-07 (No user interaction)
**Pass/Fail:** PASS if provenance logged transparently
### TC-RF-005-002: W3C PROV-JSON Validation
**Objective:** Validate all provenance records W3C PROV-compliant
**Preconditions:** 100 provenance records in log
**Test Steps:**
1. Load all 100 provenance records
2. Validate each against W3C PROV-JSON schema
3. Verify all required fields present
4. Verify timestamps in ISO 8601 format
5. Verify entity IDs unique
6. Calculate compliance rate (target: 100%)
**Expected Result:** 100% provenance records W3C PROV-compliant
**NFR Validated:** NFR-RF-P-04 (W3C PROV compliance 100%)
**Pass/Fail:** PASS if 100% compliant
### TC-RF-005-003: Lineage Query (Natural Language)
**Objective:** Validate lineage tracing via natural language query
**Preconditions:** Complete workflow executed (UC-RF-001 through UC-RF-004)
**Test Steps:**
1. Run query: `aiwg research provenance query "Where did REF-025 come from?"`
2. Verify agent parses natural language
3. Verify lineage chain traced:
- Search query → Discovery results → REF-025 → Summary → Citation
4. Verify lineage displayed (text format)
5. Measure query response time (target: <2 seconds)
**Expected Result:** Lineage traced accurately, <2s response
**NFR Validated:** NFR-RF-P-03 (Query response <2s), NFR-RF-P-08 (Natural language)
**Pass/Fail:** PASS if lineage accurate, <2s
### TC-RF-005-004: Integrity Verification (Checksums)
**Objective:** Validate checksum-based integrity verification
**Preconditions:** Reproducibility package with checksums
**Test Steps:**
1. External researcher downloads package
2. Researcher runs: `aiwg research provenance verify`
3. Agent computes checksums for all artifacts
4. Agent compares to provenance log checksums
5. Verify verification report displayed
6. Simulate file corruption (modify REF-025-metadata.json)
7. Re-run verification
8. Verify mismatch detected: "❌ REF-025-metadata.json: MISMATCH"
**Expected Result:** Checksums verified, corruption detected
**NFR Validated:** NFR-RF-P-05 (Checksum coverage 100%)
**Pass/Fail:** PASS if corruption detected
### TC-RF-005-005: Reproducibility Package Export
**Objective:** Validate reproducibility package creation
**Preconditions:** Complete workflow executed
**Test Steps:**
1. Run: `aiwg research provenance export --package publication-supplement`
2. Verify package created: `.aiwg/research/provenance/reproducibility-package-{timestamp}.zip`
3. Verify package contents:
- Provenance logs (PROV-JSON)
- Lineage graph (JSON)
- Search strategies (Markdown)
- Source metadata (all REF-XXX JSONs)
- Checksums (text file)
- README (replication instructions)
4. Verify package size <100 MB
5. External researcher downloads, extracts, replicates
**Expected Result:** Package complete, <100 MB, replicable
**NFR Validated:** Vision Goal 3 - Reproducibility packages
**Pass/Fail:** PASS if external replication succeeds
### TC-RF-005-006: Git Integration
**Objective:** Validate provenance commits to Git
**Preconditions:** Git repository initialized
**Test Steps:**
1. Run discovery search
2. Verify provenance log committed: `git log --oneline | grep "Log: Discovery search"`
3. Verify commit contains provenance file
4. Run time-travel query: `aiwg research provenance query --at <commit-hash>`
5. Verify historical provenance reconstructed
**Expected Result:** Provenance version-controlled via Git
**NFR Validated:** Vision Goal 3 - Version control integration
**Pass/Fail:** PASS if Git commits contain provenance
### TC-RF-005-007: End-to-End Reproducibility
**Objective:** Validate external researcher can replicate findings
**Preconditions:** Complete workflow executed, reproducibility package exported
**Test Steps:**
1. External researcher downloads package
2. Researcher reads README (replication instructions)
3. Researcher runs search query from provenance log
4. Researcher compares search results (checksums)
5. Researcher acquires papers (same REF-XXX)
6. Researcher summarizes papers (same LLM model)
7. Researcher compares all artifact checksums
8. Calculate replication success rate (target: >90%)
**Expected Result:** >90% artifacts replicated identically
**NFR Validated:** NFR-RF-P-06 (Reproducibility success >90%)
**Pass/Fail:** PASS if >90% replication success
### TC-RF-005-008: Provenance Logging Failure (Non-Blocking)
**Objective:** Validate research operation proceeds on logging failure
**Preconditions:** Simulate disk full
**Test Steps:**
1. Mock disk full error
2. Run discovery search
3. Verify provenance write fails
4. Verify warning displayed: "Provenance logging failed: Disk full"
5. Verify search operation completes normally (non-blocking)
6. Verify failed log saved to `.aiwg/research/provenance/failed-logs/`
7. User frees disk space
8. Agent retries logging (automatic)
**Expected Result:** Research operation non-blocking, failed log recoverable
**NFR Validated:** NFR-RF-P-07 (No user burden)
**Pass/Fail:** PASS if operation continues, logging recoverable
## Document Metadata
**Version:** 1.0 (Draft)
**Status:** DRAFT - Awaiting Review
**Created:** 2026-01-25
**Last Updated:** 2026-01-25
**Word Count:** 6,890 words
**Quality Score:** 94/100 (comprehensive, needs provenance privacy design)
**Review History:**
- 2026-01-25: Initial draft (Requirements Analyst)
**Next Actions:**
1. Stakeholder review (Provenance Agent Designer, Academic Researcher persona)
2. Validate W3C PROV-JSON implementation approach
3. Address provenance privacy concerns (Issue 002)
4. Define provenance granularity policy (Issue 001)
5. Schedule test case implementation (Construction phase)
**Generated:** 2026-01-25
**Owner:** Requirements Analyst (Research Framework Team)
**Status:** DRAFT - Ready for Stakeholder Review