agentic-data-stack-community/data-core/tasks/gather-requirements.md

AI Agentic Data Stack Framework - Community Edition. Open source data engineering framework with 4 core agents, essential templates, and 3-dimensional quality validation.

# Task: Gather Requirements

## Overview
Conducts comprehensive requirements elicitation using advanced techniques to capture functional, non-functional, and governance requirements for data initiatives. Employs progressive disclosure and multi-stakeholder collaboration to ensure complete and accurate requirement definition.

## Prerequisites
- Project scope and business context
- Stakeholder identification and availability
- Initial business case or problem statement
- Existing system documentation (if applicable)
- Regulatory and compliance context

## Dependencies
- Templates: `requirements-gathering-tmpl.yaml`
- Tasks: `advanced-data-elicitation.md`, `map-business-value.md`
- Checklists: `requirements-validation-checklist.md`

## Steps

### 1. **Stakeholder Analysis and Engagement**
   - Identify all requirement sources and stakeholders
   - Map stakeholder roles, influence, and information needs
   - Plan engagement approach for different stakeholder types
   - Establish communication channels and collaboration tools
   - **Validation**: Complete stakeholder matrix with engagement strategy

### 2. **Advanced Requirements Elicitation**
   - **Structured Interviews**: One-on-one sessions with key stakeholders
   - **Workshop Facilitation**: Group sessions for consensus building
   - **Observation Studies**: Understanding current state processes
   - **Document Analysis**: Review existing systems and documentation
   - **Prototyping Sessions**: Interactive exploration of requirements
   - **Quality Check**: Multiple elicitation techniques applied for completeness

### 3. **Functional Requirements Definition**
   - **Data Requirements**: Sources, formats, volumes, frequencies
   - **Processing Requirements**: Transformations, calculations, aggregations
   - **Integration Requirements**: System connections, APIs, data flows
   - **User Interface Requirements**: Dashboards, reports, self-service capabilities
   - **Workflow Requirements**: Process automation, approval flows, notifications
   - **Validation**: Functional requirements are specific, testable, and complete

### 4. **Non-Functional Requirements Specification**
   - **Performance Requirements**: Response times, throughput, scalability
   - **Availability Requirements**: Uptime, disaster recovery, business continuity
   - **Security Requirements**: Access controls, encryption, audit trails
   - **Usability Requirements**: User experience, accessibility, training needs
   - **Compliance Requirements**: Regulatory, policy, audit requirements
   - **Quality Check**: Non-functional requirements include measurable criteria

### 5. **Data Governance Requirements**
   - **Quality Standards**: Completeness, accuracy, consistency thresholds
   - **Privacy Requirements**: PII handling, consent management, retention
   - **Access Control**: Role-based permissions, approval workflows
   - **Audit Requirements**: Logging, monitoring, compliance reporting
   - **Change Management**: Version control, approval processes, impact assessment
   - **Validation**: Governance requirements reviewed with Data Governance Owner

### 6. **Constraint and Assumption Documentation**
   - **Technical Constraints**: Platform limitations, integration restrictions
   - **Business Constraints**: Budget, timeline, resource limitations
   - **Regulatory Constraints**: Compliance requirements, legal restrictions
   - **Organizational Constraints**: Policies, procedures, cultural factors
   - **Environmental Assumptions**: Market conditions, technology evolution
   - **Quality Check**: All constraints documented with impact assessment

### 7. **Requirements Validation and Approval**
   - Conduct requirement review sessions with stakeholders
   - Resolve conflicts and ambiguities through facilitated discussion
   - Prioritize requirements using MoSCoW or similar framework
   - Obtain formal approval from business sponsors and technical teams
   - **Final Validation**: Complete requirements package approved by all stakeholders

## Interactive Features

### Progressive Requirements Discovery
- **Basic Elicitation**: Core requirements for minimum viable solution
- **Detailed Analysis**: Comprehensive requirements for full solution
- **Advanced Scenarios**: Edge cases and future state requirements
- **Contextual Adaptation**: Requirements depth based on project complexity

### Multi-Modal Elicitation
- **Visual Modeling**: Process flows, data models, user journey maps
- **Interactive Prototyping**: Mock-ups and proof-of-concept demonstrations
- **Scenario Planning**: Use case development and user story creation
- **Collaborative Workshops**: Facilitated group requirements sessions

### Real-Time Validation
- **Requirement Traceability**: Link requirements to business objectives
- **Consistency Checking**: Automated detection of conflicting requirements
- **Completeness Assessment**: Gap analysis and missing requirement identification
- **Feasibility Validation**: Technical and business viability checking

## Outputs

### Primary Deliverable
- **Requirements Specification Document** (`requirements-specification.md`)
  - Complete functional and non-functional requirements
  - Governance and compliance requirements
  - Constraints, assumptions, and dependencies
  - Prioritization and approval documentation

### Supporting Artifacts
- **Stakeholder Analysis** - Roles, responsibilities, and engagement approach
- **Requirements Traceability Matrix** - Linkage to business objectives
- **Use Case Documentation** - Detailed scenarios and user stories
- **Process Flow Diagrams** - Current and future state process maps

## Success Criteria

### Quality Gates
- **Completeness**: All requirement categories addressed comprehensively
- **Clarity**: Requirements are unambiguous and testable
- **Traceability**: Requirements link to business objectives and value
- **Feasibility**: Technical and business viability validated
- **Stakeholder Approval**: Formal sign-off from all stakeholder groups

### Validation Requirements
- [ ] Business stakeholders confirm functional requirements accuracy
- [ ] Technical teams validate non-functional requirement feasibility
- [ ] Data governance approves quality and compliance requirements
- [ ] Legal reviews regulatory and privacy requirements
- [ ] Finance approves budget and resource requirements

### Evidence Collection
- Stakeholder interview notes and requirement sources
- Workshop documentation and consensus-building records
- Technical feasibility analysis and validation results
- Business case validation and value linkage
- Approval documentation and sign-off records

## Requirements Categories

### Functional Requirements
- **Data Ingestion**: Source systems, formats, volumes, schedules
- **Data Processing**: Transformations, validations, enrichments
- **Data Storage**: Persistence, archival, retention, access patterns
- **Data Access**: APIs, queries, reports, self-service capabilities
- **Business Logic**: Calculations, rules, workflows, automation

### Non-Functional Requirements
- **Performance**: Response time, throughput, latency, scalability
- **Reliability**: Availability, disaster recovery, fault tolerance
- **Security**: Authentication, authorization, encryption, audit
- **Usability**: User experience, accessibility, documentation
- **Maintainability**: Monitoring, debugging, configuration management

### Quality Requirements
- **Data Quality**: Accuracy, completeness, consistency, validity
- **Process Quality**: Efficiency, automation, error handling
- **User Experience**: Satisfaction, adoption, productivity
- **Operational Quality**: Reliability, maintainability, supportability

### Governance Requirements
- **Policy Compliance**: Corporate policies, industry standards
- **Regulatory Compliance**: Legal requirements, audit needs
- **Risk Management**: Security, privacy, operational risk
- **Change Management**: Version control, approval workflows

## Elicitation Techniques

### Individual Techniques
- **Structured Interviews**: Prepared questions with follow-up exploration
- **Questionnaires**: Standardized collection for large stakeholder groups
- **Document Analysis**: Review of existing systems and processes
- **Observation**: Understanding current state operations

### Group Techniques
- **Facilitated Workshops**: Collaborative requirement development
- **Focus Groups**: Targeted discussion with specific user types
- **Brainstorming**: Creative exploration of possibilities
- **Consensus Building**: Resolution of conflicts and priorities

### Creative Techniques
- **Prototyping**: Interactive exploration of requirements
- **Storyboarding**: Visual representation of user experiences
- **Role Playing**: Understanding user perspectives and workflows
- **Scenario Analysis**: Exploration of different use cases

### Analytical Techniques
- **Root Cause Analysis**: Understanding underlying business needs
- **Gap Analysis**: Comparison of current and desired states
- **Benchmarking**: Learning from industry best practices
- **Risk Analysis**: Identification of requirement-related risks

## Validation Framework

### Multi-Stage Validation
1. **Initial Capture**: Requirement identification and documentation
2. **Stakeholder Review**: Validation with requirement sources
3. **Technical Review**: Feasibility and implementation analysis
4. **Business Review**: Value alignment and priority validation
5. **Final Approval**: Formal sign-off and baseline establishment

### Continuous Refinement
- Regular review of requirement accuracy and completeness
- Adaptation based on changing business conditions
- Integration of lessons learned from implementation
- Stakeholder feedback incorporation and requirement evolution

## Best Practices

### Stakeholder Engagement
- Use appropriate elicitation techniques for different stakeholder types
- Facilitate consensus building for conflicting requirements
- Maintain stakeholder engagement throughout the process
- Provide regular updates and feedback on requirement development

### Quality Assurance
- Validate requirements with multiple stakeholders
- Use prototyping to clarify ambiguous requirements
- Document assumptions and constraints explicitly
- Trace all requirements to business objectives

## Risk Mitigation

### Common Pitfalls
- **Scope Creep**: Use progressive disclosure and clear boundaries
- **Stakeholder Fatigue**: Balance thoroughness with engagement efficiency
- **Requirement Conflicts**: Facilitate early resolution and consensus
- **Missing Requirements**: Use multiple elicitation techniques and validation

### Success Factors
- Clear project scope and stakeholder engagement strategy
- Multiple elicitation techniques for comprehensive coverage
- Regular validation and feedback loops with stakeholders
- Formal approval and change management processes

## Notes
Thorough requirements gathering is foundational to project success. Invest time in understanding stakeholder needs and validating requirements early to prevent costly changes later in the project lifecycle. Use interactive techniques to engage stakeholders and build consensus around requirement priorities.